Crawlbase MCP
About
Enables AI agents to access real-time web data with HTML, markdown, and screenshot support. SDKs: Node.js, Python, Java, PHP, .NET.
Explore
- Fetches live web data for AI agents
- JavaScript rendering for modern web apps
- Proxy rotation and anti‑bot evasion
- Structured outputs: HTML, Markdown, screenshots
- Async crawling with Crawlbase Cloud Storage
- HTTP transport with per‑request token authentication
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
Crawlbase MCPCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Once configured, use these commands inside Claude, Cursor, or Windsurf:
- crawl → Fetch raw HTML
- crawl_markdown → Extract clean Markdown
- crawl_screenshot → Capture screenshots
Example prompts:
- “Crawl Hacker News and return top stories in markdown.”
- “Take a screenshot of TechCrunch homepage.”
- “Fetch Tesla investor relations page as HTML.”
git clone https://github.com/crawlbase/crawlbase-mcp.git
cd crawlbase-mcp
npm install
HTTP mode supports per-request tokens via headers, allowing multiple users to share a single server:
curl -X POST http://localhost:3000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "X-Crawlbase-Token: your_token" \
-H "X-Crawlbase-JS-Token: your_js_token" \
-d '{"jsonrpc": "2.0", "method": "tools/list", "id": 1}'
Headers:
- X-Crawlbase-Token - Normal token for HTML requests
- X-Crawlbase-JS-Token - JavaScript token for JS-rendered pages/screenshots
Headers override environment variables when provided, enabling multi-tenant deployments.
🔑 Get your free tokens at Crawlbase ↗️.
crawl
Crawl a URL and return HTML content
crawl_markdown
Crawl a URL and extract clean markdown content
crawl_screenshot
Take a screenshot of a webpage
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"crawlbase mcp": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": [
"@crawlbase/mcp@latest"
],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}
}McpServers
{
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": [
"@crawlbase/mcp@latest"
],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}What is Crawlbase MCP?
Crawlbase MCP is a Model Context Protocol (MCP) server that bridges AI agents and the live web. Instead of relying on outdated training data, your LLMs can now fetch fresh, structured, real-time content — powered by Crawlbase’s proven crawling infrastructure trusted by 70,000+ developers worldwide.
It handles the complexity of scraping for you:
- JavaScript rendering for modern web apps
- Proxy rotation & anti-bot evasion
- Structured outputs (HTML, Markdown, screenshots)
How It Works
- Get Free Crawlbase Tokens → Sign up at Crawlbase ↗️, get free Normal, and JavaScript tokens.
- Set Up MCP Configuration → Configure the MCP server in your preferred client (Claude, Cursor, or Windsurf) by updating the MCP Servers settings.
- Start Crawling → Use commands like crawl, crawl_markdown, or crawl_screenshot to bring live web data into your AI agent.
Setup & Integration
Claude Desktop
1. Open Claude Desktop → Settings → Developer → Edit Config
2. Add to claude_desktop_config.json:
3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}
Claude Code
Add to your claude.json configuration:
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}
Cursor IDE
1. Open Cursor IDE → File → Preferences → Cursor Settings → Tools and Integrations → Add Custom MCP
2. Add to mcp.json:
3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}
Windsurf IDE
1. Open WindSurf IDE → File → Preferences → WindSurf Settings → General → MCP Servers → Manage MCPs → View raw config
2. Add to mcp_config.json:
3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}
HTTP Transport Mode
For scenarios where you need a shared MCP server accessible over HTTP (e.g., multi-user environments, custom integrations), you can run the server in HTTP mode:
```bash
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
