Fastcrw

SSE

by us

286 2.1k downloads Not rated yet AGPL-3.0

GitHub Website SSE

About

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary.

Details

Transport: SSE
License: AGPL-3.0

Explore

- Single static binary (~8 MB), no Redis, Node.js, or Python required.
- ~50 MB RAM idle, runs on a $5 VPS.
- Native /v1/* API and Firecrawl v2 compatibility layer.
- Change tracking with markdown git-diff and optional LLM judge.
- AGPL-3.0 open core with managed commercial option.
- Built-in MCP server and SDKs (npm, PyPI, Homebrew, Cargo).

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below

Plugin Name Fastcrw

Command (node, npx, python, etc.)

Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

From the repository

Hit the managed API at api.fastcrw.com, or self-host the same binary.

fastCRW ships a built-in MCP server so any MCP-compatible agent (Claude Code, Cursor, Windsurf, Cline, Continue.dev, Codex, Gemini CLI) can call scraping tools without bespoke glue. Embedded mode runs the engine in-process — no server, no API key, no setup. The crw Python SDK and the crw-mcp Node binary both shell to the same Rust core.

bash
npm install -g crw-mcp # MCP server (Node wrapper)
pip install crw # Python SDK (auto-downloads binary)
claude mcp add crw -- npx -y crw-mcp # Claude Code, embedded
claude mcp add crw \
-e CRW_API_URL=https://api.fastcrw.com -e CRW_API_KEY=… \
-- npx -y crw-mcp # Claude Code, managed
```

Per-client config recipes (Claude Desktop, Cursor, Windsurf, Cline,
Continue.dev) live under docs.fastcrw.com/mcp-clients/.

---

curl -fsSL https://fastcrw.com/install | CRW_BINARY=crw sh

curl -fsSL https://fastcrw.com/install | CRW_BINARY=crw-server sh

`crw_scrape`

Scrape one URL to markdown, HTML, or links.

`crw_crawl`

Start an async site crawl; returns a job id to poll with crw_check_crawl_status.

`crw_check_crawl_status`

Poll an async crawl job and retrieve its pages.

`crw_map`

Discover URLs on a site via sitemap and/or a short crawl. Returns a URL list only, no page content.

`crw_extract`

Extract structured JSON from URLs via a prompt and/or JSON schema. Async job — poll crw_check_extract_status with the returned id. Needs an LLM.

`crw_check_extract_status`

Poll an extract job; returns status and, when complete, a per-URL results array.

`crw_cancel_extract`

Request cancellation of an extract job. Returns the canonical status; cancelling remains non-terminal until the claimed URL settles.

`crw_parse_file`

Parse a local PDF (base64 in contentBase64) to markdown. No OCR: scanned PDFs return empty markdown with a warning.

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "fastcrw": {
            "crw": {
                "command": "npx",
                "args": [
                    "crw-mcp"
                ],
                "env": {
                    "CRW_API_KEY": "<YOUR_API_KEY>"
                }
            }
        }
    }
}

McpServers

{
    "crw": {
        "command": "npx",
        "args": [
            "crw-mcp"
        ],
        "env": {
            "CRW_API_KEY": "<YOUR_API_KEY>"
        }
    }
}

client = CrwClient()

result = client.scrape("https://example.com", formats=["markdown", "links"])
pages = client.crawl("https://docs.example.com", max_depth=2, max_pages=50)
urls = client.map("https://example.com")
results = client.search("AI news", limit=10, sources=["web", "news"])

Requires Python 3.10+. Local mode auto-downloads crw-mcp on first use. Framework extras:

bash
pip install crw[crewai] # CRW scraping tools for CrewAI agents
pip install crw[langchain] # CRW document loader for LangChain

TypeScript / Node.js

bash
npm install crw-sdk

SDK examples → Frameworks & platforms CrewAI · LangChain · Agno · Dify · n8n · Flowise All integrations → --- Architecture

┌─────────────────────────────────────────────┐
│ crw-server │
│ Axum HTTP API + Auth + MCP │
├──────────┬──────────┬───────────────────────┤
│ crw-crawl│crw-extract│ crw-renderer │
│ BFS crawl│ HTML→MD │ HTTP + CDP(WS) │
│ robots │ LLM/JSON │ LightPanda/Chrome │
│ sitemap │ clean/read│ auto-detect SPA │
├──────────┴──────────┴───────────────────────┤
│ crw-core │
│ Types, Config, Errors │
└─────────────────────────────────────────────┘
``

| Crate | Description | |-------|-------------| |crw-core| Core types, config, and error handling | |crw-renderer| HTTP + CDP browser rendering engine | |crw-extract| HTML → markdown/plaintext extraction | |crw-crawl| Async BFS crawler with robots.txt & sitemap | |crw-server | Axum API server (native /v1 plus Firecrawl /firecrawl/v2compatibility) | |crw-mcp| MCP stdio server (embedded + proxy mode) | |crw-cli | Standalone CLI (crw binary, no server) |


Full architecture docs →
---
Security

- SSRF protection — blocks loopback, private IPs, cloud metadata (169.254.x.x), IPv6 mapped addresses, and non-HTTP schemes (file://, data:) - Auth — optional Bearer token with constant-time comparison - robots.txt — RFC 9309 compliant with wildcard patterns - Rate limiting — token-bucket algorithm, returns 429 witherror_code- Resource limits — max body 1 MB, max crawl depth 10, max pages 1,000


Full security docs →
---
Contributing
Contributions are welcome — issues and PRs both.

1. Fork the repository 2. Install pre-commit hooks:make hooks3. Create your feature branch (git checkout -b feat/my-feature) 4. Commit your changes (git commit -m 'feat: add my feature') 5. Push to the branch (git push origin feat/my-feature) 6. Open a Pull Request

The pre-commit hook runs the same checks as CI (cargo fmt, cargo clippy,cargo test). Run manually with make check.


Contributors
<p>

  <a href="https://github.com/us"></a>

  <a href="https://github.com/adambenhassen"></a>

  <a href="https://github.com/mj520"></a>

</p>
---
License
fastCRW is open source under AGPL-3.0. If you embed fastCRW in

a closed-source product or expose it as a hosted service to third parties

and you can't comply with AGPL's source-availability requirements, the

managed offering at fastcrw.com includes a

commercial carve-out, and standalone commercial licenses are available

on request — write to [email protected].
---
Links

- Documentation: docs.fastcrw.com - API reference: docs.fastcrw.com/#rest-api - MCP setup guide: docs.fastcrw.com/#mcp - Playground: docs.fastcrw.com/playground/ - Benchmarks: fastcrw.com/benchmarks - Marketing site: fastcrw.com - Changelog:CHANGELOG.md- X / Twitter: @fast_crw - LinkedIn: fastcrw - Discord: discord.gg/kkFh2SC8 - MCP Registry: registry.modelcontextprotocol.io


---
Star History
<a href="https://www.star-history.com/?repos=us%2Fcrw&type=timeline&legend=bottom-right">

 <picture>

   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=us/crw&type=timeline&theme=dark&legend=bottom-right" />

   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=us/crw&type=timeline&legend=bottom-right" />

   

 </picture>

</a>
---

It is the sole responsibility of end users to respect websites' policies when scraping. Users are advised to adhere to applicable privacy policies and terms of use. By default, fastCRW respectsrobots.txt`
directives.

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

About

Details

Explore

Setting up with Highlight

From the repository

crw_scrape

crw_crawl

crw_check_crawl_status

crw_map

crw_extract

crw_check_extract_status

crw_cancel_extract

crw_parse_file

Claude Desktop / Cursor

McpServers

client = CrwClient()

TypeScript / Node.js

Frameworks & platforms

Architecture

Security

Contributing

Contributors

License

Links

Star History

More like this

@mendableai/firecrawl Mcp Server

Playwright

Fetch

Puppeteer

`crw_scrape`

`crw_crawl`

`crw_check_crawl_status`

`crw_map`

`crw_extract`

`crw_check_extract_status`

`crw_cancel_extract`

`crw_parse_file`