PyMCPAutoGUI 🖱️⌨️🖼️ - GUI Automation via MCP

by kitfactory

388 downloads Not rated yet MIT license

GitHub

Productivity

About

PyMCPAutoGUI bridges AI agents (e.g., in Cursor or other MCP-compatible environments) with the computer’s graphical user interface. It lets an agent see the screen, control the mouse and keyboard, and interact with windows — just like a human user.

Details

License: MIT license

Explore

- Empowers AI agents to control any desktop application.
- Simple plug‑and‑play integration with MCP‑compatible clients.
- Comprehensive mouse, keyboard, and window automation functions.
- Screen perception: take screenshots and locate images on screen.
- Window management: move, resize, minimize, maximize, restore, close.
- User interaction: display alert

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below

Plugin Name PyMCPAutoGUI 🖱️⌨️🖼️ - GUI Automation via MCP

Command (node, npx, python, etc.)

Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

From the repository

Operating Systems: Windows, macOS, Linux (Requires appropriate dependencies for pyautogui on each OS)
Python: 3.11+ 🐍
MCP Clients: Cursor Editor, any client supporting the Model Context Protocol (MCP)

Using a virtual environment keeps your project dependencies tidy.

python -m venv .venv
pip install pymcpautogui # Or pip install . if installing from local source

(Note: pyautogui might have system dependencies like scrot on Linux for screenshots. Please check the pyautogui documentation for OS-specific installation requirements.)*

Once installed, simply run the server from your terminal:


python -m pymcpautogui.server

The server will start and listen for connections (defaulting to port 6789). Look for this output:

INFO:     Started server process [XXXXX]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:6789 (Press CTRL+C to quit)

Keep this terminal running while you need the GUI automation magic! ✨

PyMCPAutoGUI exposes most functions from pyautogui and pygetwindow. Examples include:

Mouse 🖱️: move_to, click, move_rel, drag_to, drag_rel, scroll, mouse_down, mouse_up, get_position
Keyboard ⌨️: write, press, key_down, key_up, hotkey
Screenshots 🖼️: screenshot, locate_on_screen, locate_center_on_screen
Windows 🪟: get_all_titles, get_windows_with_title, get_active_window, activate_window, minimize_window, maximize_window, restore_window, move_window, resize_window, close_window
Dialogs 💬: alert, confirm, prompt, password
Config ⚙️: set_pause, set_failsafe

For the full list and details, check the pymcpautogui/server.py file or use @PyMCPAutoGUI list_tools in your MCP client.

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "pymcpautogui \ud83d\uddb1\ufe0f\u2328\ufe0f\ud83d\uddbc\ufe0f - gui automation via mcp": {
            "PyMCPAutoGUI": {
                "command": "python",
                "args": [
                    "-m",
                    "venv",
                    ".venv"
                ]
            }
        }
    }
}

McpServers

{
    "PyMCPAutoGUI": {
        "command": "python",
        "args": [
            "-m",
            "venv",
            ".venv"
        ]
    }
}

Supercharge your AI Agent's capabilities! ✨ PyMCPAutoGUI provides a bridge between your AI agents (like those in Cursor or other MCP-compatible environments) and your computer's graphical user interface (GUI). It allows your agent to see the screen 👁️, control the mouse 🖱️ and keyboard ⌨️, and interact with windows 🪟, just like a human user!

Stop tedious manual GUI tasks and let your AI do the heavy lifting 💪. Perfect for automating repetitive actions, testing GUIs, or building powerful AI assistants 🤖.

🤔 Why Choose PyMCPAutoGUI?

🤖 Empower Your Agents: Give your AI agents the power to interact directly with desktop applications.
✅ Simple Integration: Works seamlessly with MCP-compatible clients like the Cursor editor. It's plug and play!
🚀 Easy to Use: Get started with a simple server command. Seriously, it's that easy.
🖱️⌨️ Comprehensive Control: Offers a wide range of GUI automation functions from the battle-tested PyAutoGUI and PyGetWindow.
🖼️ Screen Perception: Includes tools for taking screenshots and locating images on the screen – let your agent see!
🪟 Window Management: Control window position, size, state (minimize, maximize), and more. Tidy up that desktop!
💬 User Interaction: Display alert, confirmation, and prompt boxes to communicate with the user.

🛠️ Supported Environments

🚀 Getting Started - It's Super Easy!

1. Installation (Recommended: Use a Virtual Environment!)

Using a virtual environment keeps your project dependencies tidy.

```bash

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

About

Details

Explore

Setting up with Highlight

From the repository

Claude Desktop / Cursor

McpServers

🤔 Why Choose PyMCPAutoGUI?

🛠️ Supported Environments

🚀 Getting Started - It's Super Easy!

1. Installation (Recommended: Use a Virtual Environment!)

More like this

Time

Github SSE

Npx Y convex@latest mcp start

Inbox Zero