GitHub - ShawnPana/phone-harness: let your agent control your phone · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
ShawnPana
phone-harness
Public
Notifications<br>You must be signed in to change notification settings
Fork<br>180
Star<br>1.9k
main
BranchesTags
Go to file
CodeOpen more actions menu
Latest commit
History<br>66 Commits<br>66 Commits
Folders and files<br>NameNameLast commit message<br>Last commit date<br>.github/workflows
.github/workflows
agent-workspace
agent-workspace
src/phone_harness
src/phone_harness
.gitignore
.gitignore
LICENSE
LICENSE
README.md
README.md
SKILL.md
SKILL.md
install.md
install.md
onboarding.md
onboarding.md
phone-harness
phone-harness
pyproject.toml
pyproject.toml
View all files
Repository files navigation
Phone Harness 📱
phone-harness · let your agent control your phone.
Connect an AI agent — Claude Code, Codex, or any LLM — directly to your real<br>phone with a thin, editable harness. iPhone through the Mac's iPhone<br>Mirroring window, or Android over adb (USB or Wi‑Fi). No jailbreak, no<br>Xcode, no WebDriverAgent, no app on the phone.
The Mac is the whole transport. iPhone: screencapture + Vision-framework OCR<br>for eyes, HID-level CGEvents for hands. Android: screencap for eyes plus the<br>phone's own accessibility tree — exact text, exact boxes — and input for<br>hands. Nothing between the agent and the phone. The agent writes what's<br>missing during execution in agent-workspace/agent_helpers.py.
● agent: wants to open Weather<br>● ocr() → "Weather" at (400, 468)<br>● tap(400, 468) → wait_stable() → ocr() confirms the forecast<br>✓ done
Your phone, driven by an agent.
Setup prompt
Paste into Claude Code or Codex:
Set up phone-harness for me. Clone https://github.com/ShawnPana/phone-harness into ~/.phone-harness (its canonical home), read `install.md` first, install it so `phone-harness` is a command on my PATH, and register it as an agent skill named phone-harness using `phone-harness skill` as the body, so you reach for it automatically. Then read `SKILL.md` for normal usage, and always read `src/phone_harness/helpers.py` because that is where the functions are. Then read `onboarding.md` and walk me through it.
The agent then follows onboarding.md: it asks one thing —<br>which phone is your default, iPhone or Android — and walks you through only<br>what needs your hands. iPhone: pairing iPhone Mirroring once, and granting<br>the terminal Accessibility and Screen Recording in System Settings →<br>Privacy & Security (Screen Recording takes effect after the terminal restarts).<br>Android: turning on Developer options, then either plugging in and tapping<br>Allow, or Wireless debugging + one 6‑digit pairing code. Then<br>phone-harness --doctor verifies the chain, and phone-harness config set platform ios|android sets the default (both can be set up).
A fresh machine may prompt for more permissions the first time an action runs<br>— if --doctor passes but taps or capture silently do nothing, watch for a<br>macOS prompt. See install.md for details.
Why this works
iPhone. iPhone Mirroring (macOS Sequoia+) renders the phone as a Mac window<br>and forwards real mouse and keyboard input as touches. That gives an agent<br>everything it needs for real-device iOS automation:
See — capture just the mirroring window, OCR it with Apple's Vision<br>framework: every visible string with a tap-ready coordinate. The poor man's<br>DOM.
Act — CGEvents posted at the HID tap: taps, long-presses, drags/flicks,<br>scroll gestures, unicode typing, and the app's own shortcuts (Cmd+1 Home,<br>Cmd+2 App Switcher, Cmd+3 Spotlight). For icon-only controls found in a<br>screenshot, use tap_image_point(x, y, image_size=...): screenshot coordinates<br>are image pixels, while tap(x, y) expects global Mac screen points.<br>image_point() performs the conversion when a raw tap() is needed.
Verify — screenshot again. No DOM means the capture is the ground truth.
Things that do NOT work, learned the hard way: AppleScript click at (silently<br>ignored — the window is a video stream with no accessibility tree), unicode key<br>payloads (mirroring forwards raw HID keycodes, so typing must use keycodes), a<br>slow touch-drag (barely moves an iOS list — use wheel scroll for lists, a fast<br>flick for pages), and input while the window isn't frontmost (swallowed).
Android. adb reaches the phone directly, over USB or Wi‑Fi — no window, no<br>focus, nothing on the Mac has to be in front. screencap is the capture; the<br>phone's own accessibility tree (uiautomator) is the text source, so ocr()<br>returns exact strings and boxes and tap_ui("url_bar") finds elements OCR<br>never could; input tap/swipe/text are the hands; Back exists. Coordinates<br>are device pixels, so a...