PyMCPAutoGUI 🖱️⌨️🖼️ - GUI Automation via MCP
About
PyMCPAutoGUI bridges AI agents (e.g., in Cursor or other MCP-compatible environments) with the computer’s graphical user interface. It lets an agent see the screen, control the mouse and keyboard, and interact with windows — just like a human user.
Details
- Author
- kitfactory
- Downloads
- 388
- Categories
- Productivity
Jump to
- Empowers AI agents to control any desktop application.
- Simple plug‑and‑play integration with MCP‑compatible clients.
- Comprehensive mouse, keyboard, and window automation functions.
- Screen perception: take screenshots and locate images on screen.
- Window management: move, resize, minimize, maximize, restore, close.
- User interaction: display alert
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
PyMCPAutoGUI 🖱️⌨️🖼️ - GUI Automation via MCPCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Install via pip install pymcpautogui in a Python 3.11+ virtual environment. Run python -m pymcpautogui.server to start the MCP server (defaults to port 6789). Connect from Cursor by adding the provided JSON configuration to mcp.json; then use @PyMCPAutoGUI in chats to invoke tools like move_to, write, screenshot, or activate_window.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
"PyMCPAutoGUI": {
"command": "python",
"args": [
"-m",
"venv",
".venv"
]
}
}
}
}McpServers
{
"PyMCPAutoGUI": {
"command": "python",
"args": [
"-m",
"venv",
".venv"
]
}
}PyMCPAutoGUI 🖱️⌨️🖼️ - GUI Automation via MCP
Supercharge your AI Agent's capabilities! ✨ PyMCPAutoGUI provides a bridge between your AI agents (like those in Cursor or other MCP-compatible environments) and your computer's graphical user interface (GUI). It allows your agent to see the screen 👁️, control the mouse 🖱️ and keyboard ⌨️, and interact with windows 🪟, just like a human user!
Stop tedious manual GUI tasks and let your AI do the heavy lifting 💪. Perfect for automating repetitive actions, testing GUIs, or building powerful AI assistants 🤖.
🤔 Why Choose PyMCPAutoGUI?
🤖 Empower Your Agents: Give your AI agents the power to interact directly with desktop applications.
✅ Simple Integration: Works seamlessly with MCP-compatible clients like the Cursor editor. It's plug and play!
🚀 Easy to Use: Get started with a simple server command. Seriously, it's that easy.
🖱️⌨️ Comprehensive Control: Offers a wide range of GUI automation functions from the battle-tested PyAutoGUI and PyGetWindow.
🖼️ Screen Perception: Includes tools for taking screenshots and locating images on the screen – let your agent see!
🪟 Window Management: Control window position, size, state (minimize, maximize), and more. Tidy up that desktop!
💬 User Interaction: Display alert, confirmation, and prompt boxes to communicate with the user.
🛠️ Supported Environments
Operating Systems: Windows, macOS, Linux (Requires appropriate dependencies for pyautogui on each OS)
Python: 3.11+ 🐍
MCP Clients: Cursor Editor, any client supporting the Model Context Protocol (MCP)
🚀 Getting Started - It's Super Easy!
1. Installation (Recommended: Use a Virtual Environment!)
Using a virtual environment keeps your project dependencies tidy.
```bash
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
