ScreenHand

Let AI control your desktop — click buttons, fill forms, automate workflows in ~50ms with zero extra AI calls.

An open-sourceMCP serverfor macOS and Windows. Works with Claude, Cursor, Codex CLI, and any MCP-compatible client.

Quick Start|What It Does|Example|All 111 Tools|Architecture|Website

claude mcp add screenhand -- npx -y screenhand

{ "mcpServers": { "screenhand": { "command": "npx", "args": ["-y", "screenhand"] } } }

{ "mcpServers": { "screenhand": { "command": "npx", "args": ["-y", "screenhand"] } } }

[mcp.screenhand] command = "npx" args = ["-y", "screenhand"] transport = "stdio"

ScreenHand is a standard MCP server over stdio. Run withnpx -y screenhand.

macOS: System Settings > Privacy & Security > Accessibility > enable your terminal app.

Windows: No special permissions needed.

Launch Chrome with remote debugging to enable browser tools:

open -a "Google Chrome" --args --remote-debugging-port=9222

That's it. Your AI client now has 111 tools for desktop automation — and ships with prebuilt knowledge for 36 apps so you don't start from zero.

git clone https://github.com/manushi4/screenhand.git cd screenhand && npm install && npm run build:native

On Windows, usenpm run build:native:windowsinstead.

Every install ships with battle-tested knowledge so AI starts from EXPERT level on day one — no re-exploration needed:

These load automatically when the matching app or website is detected. No setup required.

ScreenHand gives AI agents eight capabilities:

Click buttons, type text, read UI trees, navigate menus, drag, scroll — all via native Accessibility APIs in ~50ms. Works with any app: Finder, Notes, VS Code, Xcode, System Settings, etc.

Full Chrome control via DevTools Protocol. Navigate, click, type, run JavaScript, fill forms — all in the background at ~10ms. Built-in anti-detection (browser_stealth,browser_human_click) for sites with bot protection.

click_with_fallback,type_with_fallback, etc. automatically try Accessibility → CDP → OCR → coordinates. You don't have to pick the right method — ScreenHand figures it out.

Gets smarter every session. Logs tool calls, saves winning strategies, tracks error patterns with fixes. Zero config, zero latency overhead (in-memory cache, async disk writes). Ships with 12 seed strategies for common macOS workflows. 6 learning policies: locator stability, sensor effectiveness, recovery ranking, pattern recognition, adaptive timing, and topology (navigation edge reliability).

App Mastery Map — automatic per-app spatial understanding

Builds a persistent reverse-engineered blueprint of every app from normal tool usage. 8 features record automatically: page zones, navigation graph (BFS pathfinding), hierarchy, I/O contracts, state machine, element visibility, timing profiles, and ready signals. Mastery levels (beginner → pro → expert → grandmaster) honestly reflect how well ScreenHand knows each app. Maps stored at~/.screenhand/app-maps/.

Website Feature Discovery — real features, not generic ladders

discover_featuresfetches an app's official website and extracts real product features (headings, feature cards, definition lists). Assigns difficulty tiers automatically and generates value-add features only ScreenHand can provide: bulk operations, cross-app export, content summarization, auto-organize, and change monitoring. No LLM calls needed — pure rule-based extraction. Features merge into the reference file and enrich the mastery ladder.

Queue multi-step jobs, run them via background worker daemon, coordinate multiple AI agents with session leases, detect stalls, auto-recover. Survives client restarts.

Continuous screen awareness (3-rate perception loop at 100ms/300ms/1000ms), real-time world model with entity tracking, goal-oriented planning with auto-decomposition, recovery engine with self-healing. The system always knows what's on screen and feeds observations into the App Mastery Map.

Full reference: See all111 tools with descriptions.

You: Search for "screenhand" on Instagram → browser_tabs() # ~10ms [34DF5DE1] Instagram — https://www.instagram.com/ → browser_js({ code: "/ click Search icon /" }) # ~10ms → browser_fill_form({ selector: "input", text: "screenhand" }) # ~50ms (human-like) → browser_js({ code: "/ extract results /" }) # ~10ms Found @screenhand_ as the top result.

Desktop— native app control without screenshots:

→ apps() # List running apps ~10ms → focus("com.apple.Notes") # Bring Notes to front ~10ms → ui_tree() # Read full UI element tree ~50ms → ui_press("New Note") # Click "New Note" button ~50ms → type_text("Hello world") # Type text ~30ms

Cross-app— chain actions across your whole desktop:

→ browser_js(...) # Extract data from Chrome → focus("com.apple.Notes") # Switch to Notes → type_text(extractedData) # Paste it in → key("cmd+s") # Save

If you use Claude Code, ScreenHand includes a plugin with13 skills and 5 agentsthat wrap all 111 tools into intent-oriented workflows.

./install-plugin.sh # after npm install && npm run build:native

5 specialized agents:marketing,design,QA,scraper,orchestrator.

AI Client (Claude, Cursor, Codex CLI) ↓ MCP protocol (stdio) ScreenHand MCP Server (TypeScript) ↓ JSON-RPC (stdio) Native Bridge (Swift on macOS / C# on Windows) ↓ OS APIs Accessibility, CoreGraphics, Vision, UI Automation, SendInput

ScreenHand reads the UI tree and DOM directly — no screenshots needed for most operations. When screenshots are needed (canvas apps, visual verification), OCR runs in ~600ms via the native Vision framework.

Computer Use is cloud-based and screenshot-driven. ScreenHand is local-first, uses native OS APIs (50ms vs 3-5s per action), costs zero API calls for clicks/typing, and runs entirely on your machine.

Any app with Accessibility support (most macOS/Windows apps). Chrome and Electron apps get full DOM access via CDP. Canvas-heavy apps (games, Photoshop viewport) use OCR as fallback.

Ships with EXPERT-level prebuilt knowledge for: Terminal, Mail, Finder, Calendar, Reminders, Keynote, Pages, Notes, Photos, Apple Music, WhatsApp, Figma, Discord, DaVinci Resolve, Canva, Instagram, X/Twitter, LinkedIn, YouTube, Reddit, Notion, n8n, and more. Any other app gets explored and learned automatically on first use.

Runs locally, never sends screen data externally. PII is redacted from all persisted data (memory, playbooks, strategies). Dangerous protocols (javascript:,data:) are blocked. AppleScript and browser JS execution are audit-logged.

Yes. Session leases with heartbeat prevent conflicts. The supervisor daemon detects stalls and recovers. Each agent claims its own app window.

Accessibility: ~50ms. Chrome CDP: ~10ms (background, no focus needed). OCR: ~600ms. Memory lookups: ~0ms (in-memory cache). All disk writes are async and non-blocking.

git clone https://github.com/manushi4/screenhand.git cd screenhand && npm install && npm run build:native npm test # 1331 tests, 54 files

- Email:khushi@clazro.com
- Issues:github.com/manushi4/screenhand/issues
- Website:screenhand.com

screenhand.com|khushi@clazro.com| A product ofClazro Technology Private Limited

A faster, parallel browser for AI agents: it drives your own Chrome from Claude Desktop or any MCP client, on fewer round trips and tokens than Playwright MCP.