ScrapeGraph MCP Server

<p align="center">

</p>

A production-ready Model Context Protocol (MCP) server that provides seamless integration with the ScrapeGraph AI API. This server enables language models to leverage advanced AI-powered web scraping capabilities with enterprise-grade reliability.

Table of Contents

- Key Features
- Quick Start
- Available Tools
- Setup Instructions
- Remote Server Usage
- Local Usage
- Google ADK Integration
- Example Use Cases
- Error Handling
- Common Issues
- Development
- Contributing
- Documentation
- Technology Stack
- License

API v2

This MCP server targets ScrapeGraph API v2 (https://v2-api.scrapegraphai.com/api), aligned 1:1 with
scrapegraph-py PR #84. Auth uses the
SGAI-APIKEY header. Environment variables mirror the Python SDK:

- SGAI_API_URL — override the base URL (default https://v2-api.scrapegraphai.com/api)
- SGAI_TIMEOUT — request timeout in seconds (default 120)
- SGAI_API_KEY — API key (can also be passed via MCP scrapegraphApiKey or X-API-Key header)

> Legacy aliases (still honored): SCRAPEGRAPH_API_BASE_URL for SGAI_API_URL, SGAI_TIMEOUT_S for SGAI_TIMEOUT.

Key Features

- Scrape & extract: scrape (POST /scrape, multi-format), extract (POST /extract, URL + prompt)
- Search: search (POST /search; num_results clamped 3–20)
- Crawl: Async multi-page crawl with crawl_start / crawl_get_status / crawl_stop / crawl_resume
- Schema: schema (POST /schema) — generate or augment a JSON Schema from a prompt
- Monitors: Scheduled jobs via monitor_create, monitor_list, monitor_get, pause/resume/delete, monitor_activity (paginated tick history)
- Account: credits, history
- Easy integration: Claude Desktop, Cursor, Smithery, HTTP transport
- Developer docs: .agent/ folder

Migration: v2 → v3

v3 renames every MCP tool that diverged from the v2 API docs. Hard rename, no aliases.

| v2 (old) | v3 (new) |
|---|---|
| smartscraper | extract |
| searchscraper | search |
| smartcrawler_initiate | crawl_start |
| smartcrawler_fetch_results | crawl_get_status |
| sgai_history | history |
| generate_schema | schema |
| markdownify | removed — use scrape with output_format="markdown" |

Quick Start

1. Get Your API Key

Sign up and get your API key from the ScrapeGraph Dashboard

2. Install with Smithery (Recommended)

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

3. Start Using

Ask Claude or Cursor:
- "Convert https://scrapegraphai.com to markdown"
- "Extract all product prices from this e-commerce page"
- "Research the latest AI developments and summarize findings"

That's it! The server is now available to your AI assistant.

Available Tools

| Tool | Role |
|------|------|
| scrape | POST /scrape (output_format: markdown, html, screenshot, branding, links, images, summary) |
| extract | POST /extract (requires website_url + user_prompt; optional output_schema) |
| search | POST /search (num_results 1–20; supports country_search, time_range, output_schema) |
| crawl_start | POST /crawl — extraction_mode markdown / html / links / images / summary / branding / screenshot |
| crawl_get_status | GET /crawl/:id (poll until status: completed) |
| crawl_stop, crawl_resume | POST /crawl/:id/stop \| resume |
| schema | POST /schema (generate or augment a JSON Schema from a prompt) |
| credits | GET /credits |
| history | GET /history (paginated, service filter) |
| monitor_create, monitor_list, monitor_get, monitor_pause, monitor_resume, monitor_delete | /monitor API |
| monitor_activity | GET /monitor/:id/activity (paginated tick history: id, createdAt, status, changed, elapsedMs, diffs) |

Removed: sitemap, agentic_scrapper, async-status polling, and (in v3) markdownify — use scrape with output_format="markdown".

Setup Instructions

To utilize this server, you'll need a ScrapeGraph API key. Follow these steps to obtain one:

1. Navigate to the ScrapeGraph Dashboard
2. Create an account and generate your API key

Automated Installation via Smithery

For automated installation of the ScrapeGraph API Integration Server using Smithery:

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

Claude Desktop Configuration

Update your Claude Desktop configuration file with the following settings (located on the top rigth of the Cursor page):

(remember to add your API key inside the config)

{
    "mcpServers": {
        "@ScrapeGraphAI-scrapegraph-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@smithery/cli@latest",
                "run",
                "@ScrapeGraphAI/scrapegraph-mcp",
                "--config",
                "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
            ]
        }
    }
}

The configuration file is located at:
- Windows: %APPDATA%/Claude/claude_desktop_config.json
- macOS: ~/Library/Application\ Support/Claude/claude_desktop_config.json

Cursor Integration

Add the ScrapeGraphAI MCP server on the settings:

Remote Server Usage

Connect to our hosted MCP server - no local installation required!

Claude Desktop Configuration (Remote)

Add this to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "scrapegraph-mcp": {
      "command": "npx",
      "args": [
        "[email protected]",
        "https://mcp.scrapegraphai.com/mcp",
        "--header",
        "X-API-Key:YOUR_API_KEY"
      ]
    }
  }
}

Cursor Configuration (Remote)

Cursor supports native HTTP MCP connections. Add to your Cursor MCP settings (~/.cursor/mcp.json):

{
  "mcpServers": {
    "scrapegraph-mcp": {
      "url": "https://mcp.scrapegraphai.com/mcp",
      "headers": {
        "X-API-Key": "YOUR_API_KEY"
      }
    }
  }
}

Benefits of Remote Server

- No local setup - Just configure and start using
- Always up-to-date - Automatically receives latest updates
- Cross-platform - Works on any OS with Node.js

Local Usage

To run the MCP server locally for development or testing, follow these steps:

Prerequisites

- Python 3.13 or higher
- pip or uv package manager
- ScrapeGraph API key

Installation

1. Clone the repository (if you haven't already):

git clone https://github.com/ScrapeGraphAI/scrapegraph-mcp
cd scrapegraph-mcp

2. Install the package:

```bash