GitHub - MARCCHERGGI/reflex: Demonstrate a GUI workflow once - compile it to a visual state machine - replay with zero LLM calls against moved windows, shuffled rows, changed themes. 100/100 + 29/29 unattended benchmark runs. · GitHub
/" data-turbo-transient="true" />
Skip to content
Type / to search
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
MARCCHERGGI
reflex
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>10 Commits<br>10 Commits
.github/workflows
.github/workflows
benchmarks
benchmarks
docs
docs
journal
journal
kernel
kernel
reflex
reflex
.gitignore
.gitignore
LICENSE
LICENSE
README.md
README.md
View all files
Repository files navigation
REFLEX — a visual skill compiler for screen automation
Demonstrate a GUI workflow once. REFLEX compiles the recording into a visual state<br>machine and replays it faster than the demonstration — with zero LLM calls per action —<br>against a window that moved, rows that shuffled, and a theme that changed color.
Computer-use agents re-pay a model call for every click, which makes them slow, expensive,<br>and nondeterministic on workflows a human could teach in fifteen seconds. RPA tools solved<br>replay decades ago but break on anything that moves. REFLEX sits between them:
Record : a screen ring buffer (~12fps, mss) + global input hooks (pynput) capture<br>a human demonstration as frames + actions.
Compile : each click becomes a state — target crop, tight band, context crop, and a<br>label strip, all cut from the frame the human aimed at. No coordinates are stored as<br>truth; crops are the truth.
Replay : each state matches its crops on the live screen (OpenCV template matching<br>with a pyramid search), clicks at the matched position, and verifies the screen<br>actually responded before advancing. Timeouts pause the run as BLOCKED — the runtime<br>never guesses.
Repair (prototype): a BLOCKED run builds a compact repair package and asks a model<br>for ONE plan; a successful repair is written back into the skill as a new version, so<br>the same exception never costs a second model call.
Benchmark
The benchmark app (reflex/mock_app.py, tkinter) randomizes window position, row order,<br>accent color, and spinner duration per launch and writes its own ground-truth log of what<br>was actually clicked and typed — replays are scored against reality, not against the<br>runtime's opinion of itself.
Each run: fresh demonstration → compile → relaunch randomized → replay from crops only.
Mock app, 100 unattended runs
outcome<br>count
PERFECT (all fields exact, correct shuffled row, DONE)<br>100
MISROUTE (text in a wrong field)
BLOCKED (honest refusal, nothing wrong typed)
CRASH/other
Median replay: 6.5s (demonstration: 14.4s). LLM calls per replay: 0.
Real Chrome tab (throwaway profile), 30 reps
Window position AND form field order randomized per replay; ground truth is the page itself confessing which fields received which text (document.title).
outcome<br>count
DEMO-INVALID
PERFECT<br>29
Valid reps (contention/demo-invalid excluded): 29.
The matching rules were earned, not designed
Every rule in matcher.py exists because a measured failure demanded it:
Rule<br>The failure that earned it
Change detection counts moved pixels, not mean diff<br>a menu opening changed 0.2% of the screen and mean-diff called it "no change"
Featureless crops are refused, not matched<br>a blank patch of one app matched a blank patch of a different app at confidence 1.0 and 33 characters went into the wrong window
Tight band beats square crop in lists<br>reordered rows dragged old neighbors into the crop → 0.81 confidence on the right row
Label strip through the cursor row, matched globally<br>three pixel-identical empty form fields are distinguishable only by the labels beside them; every cursor-local crop excluded the label and text landed in the wrong field
Twin margin: the best row must beat the runner-up by ≥0.02<br>a strip win inside noise of a sibling row is a coin flip, not a localization
Edge-domain rescue (raw float Sobel magnitude)<br>a randomized accent color dropped grayscale correlation to 0.72–0.85 on the same button; gradient structure survives color
Focus clicks verify via the next type-state's visible text<br>a caret is 1px — invisible to change detection; the honest evidence that focus landed is that typing appears
reflex/matcher.py journals every decision (context confidence, strip margin, fallbacks<br>taken, final box) to diag_find.jsonl — when a run misbehaves, the evidence chain is one<br>read away.
Run it
Windows, Python 3.13+, pip install mss pynput opencv-python...