CrawlForge MCP

One-step setup (v4.6.0+):crawlforge initdetects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Usecrawlforge init --all --yesto configure every detected client non-interactively.

3. Configure Your IDE (if not auto-configured)

Every tool is metered and requires an API key.New accounts get 1,000 free trial credits — no credit card required to start.

- Access to all 27 tools
- Credits never expire and roll over month-to-month
- API access and webhook notifications

# Optional: Set API key via environment export CRAWLFORGE_API_KEY="cf_live_your_api_key_here" # Optional: Custom API endpoint (for enterprise) export CRAWLFORGE_API_URL="https://api.crawlforge.dev" # As of v3.0.18, this variable is validated against an allow-list of CrawlForge backend hosts. # Optional: Local LLM (Ollama) overrides — extract_with_llm defaults to Ollama export OLLAMA_BASE_URL="http://localhost:11434" # default export OLLAMA_DEFAULT_MODEL="llama3.2" # default; any locally-pulled model name works # Optional: Cloud LLM keys — only needed when you pass provider: "openai" or "anthropic" export OPENAI_API_KEY="sk-..." export ANTHROPIC_API_KEY="sk-ant-..." # Optional: limit which tools this client sees — by name, by group, or both (comma-separated) export CRAWLFORGE_TOOLS="scrape,search_web,extract_content" export CRAWLFORGE_TOOL_GROUPS="basic,search,scrape" # unset = all tools; unknown names/groups are ignored with a warning # Optional: deep_research stealth extraction fallback (v4.6.6) — see below export RESEARCH_STEALTH_ENGINE="auto" # auto (default) | camoufox | chromium export RESEARCH_STEALTH_FALLBACK="true" # set to "false" to disable entirely export RESEARCH_MAX_STEALTH_RETRIES="8" # cap on stealth retries per research run

CrawlForge tracks the current MCP spec (2025-06-18) plus select experimental extensions:

- Structured output—scrape,map_site,serp_rank,search_web,extract_structured, andcrawl_deepreturn machine-parseablestructuredContentalongside the usual text, validated against a publishedoutputSchema; legacy clients keep working off the text.
- Self-correctable errors— invalid tool input now comes back as anisError: trueresult the calling model can read and retry from, instead of a raw JSON-RPC protocol error.
- JSON Schema 2020-12tool schemas, deterministictools/listordering (client prompt-cache friendly), and cacheable-result hints on read-only tools.
- Iconson the server, its tools, and its prompts.
- Async tasks(experimental) on the four long-running tools —crawl_deep,batch_scrape,deep_research,agent— for clients that support polling; synchronous results are still returned for clients that don't.

Local-LLM quickstart (extract_with_llm with Ollama)

extract_with_llmdefaults to a local Ollama model — no LLM-provider key, no per-token LLM costs, and no data leaving your machine (the CrawlForge credit cost still applies).

# 1. Install Ollama: https://ollama.com # 2. Pull any model from https://ollama.com/library ollama pull llama3.2 # 3. Discover what's installed (from your MCP client) # list_ollama_models() # 4. Extract — defaults to Ollama with the model from step 2 # extract_with_llm({ url: "https://example.com", prompt: "…", model: "llama3.2" })

Stealth extraction for deep_research (Camoufox)

deep_researchautomatically retries sources that block the normal fetch path (Reddit, Quora, forums, and Cloudflare/DataDome-protected pages return HTTP 403) through areal fingerprinted browser, then re-extracts from the rendered HTML. It's bounded (RESEARCH_MAX_STEALTH_RETRIES, default 8, plus a per-page timeout) and lazy — the browser stack only loads when a source is actually blocked.

Engine selection (RESEARCH_STEALTH_ENGINE):

- auto(default) — preferCamoufox(Firefox anti-detect), fall back to Chromium stealth, then plain fetch.
- camoufox— force Camoufox.
- chromium— force the Chromium stealth engine.

Headless Chromiumcannotclear modern challenges (Cloudflare Turnstile, DataDome) —Camoufox can. In testing it recovered Quora and Trustpilot pages that were otherwise fully blocked. To enable it, install the optional dependency and run its one-time binary fetch:

# Camoufox is declared as an optional dependency, so a normal install already pulls it. # If you installed with --no-optional, add it explicitly: npm install camoufox # One-time download of the Camoufox Firefox binary (~130 MB): npx camoufox fetch

Without the Camoufox binary,deep_researchsilently falls back to Chromium stealth and then to plain fetch — no errors, just lower recovery on heavily-protected sites. Disable the whole fallback withRESEARCH_STEALTH_FALLBACK=false.

Your configuration is stored at~/.crawlforge/config.json:

{ "apiKey": "cf_live_...", "userId": "user_...", "email": "you@example.com" }

Once configured, use these tools in your AI assistant:

"Search for the latest AI news" "Extract all links from example.com" "Crawl the documentation site and summarize it" "Monitor this page for changes" "Extract product prices from this e-commerce site"

- Secure Authentication: API keys required for all metered tools
- Local Storage: API keys stored securely at~/.crawlforge/config.json
- HTTPS Only: All connections use encrypted HTTPS
- No Data Retention: We don't store scraped data, only usage logs
- Rate Limiting: Built-in protection against abuse
- Compliance: Respects robots.txt and GDPR requirements

- SSRF enforcement: Every scraped URL is validated before the request is sent — http/https only; blocks loopback, RFC1918, IPv6 private/link-local ranges, cloud metadata endpoints (GCP, Azure), and dangerous ports (SSH, SMTP, DNS, MySQL, Postgres, Redis, MongoDB, etc.). Redirects are re-validated each hop, capped at 5.
- Backend endpoint guard(v3.0.18): The server's own calls to CrawlForge.dev use a separate fail-closed allow-list ({crawlforge.dev, www.crawlforge.dev, api.crawlforge.dev}, HTTPS required). SettingCRAWLFORGE_API_URLto an arbitrary host is blocked at parse time.
- Action allowlist:scrape_with_actionsaccepts only 7 action types (wait,click,type,press,scroll,screenshot,executeJavaScript). No download, file-write, or arbitrary cross-page navigation primitives exist.
- JavaScript gate: TheexecuteJavaScriptaction throws by default. SetALLOW_JAVASCRIPT_EXECUTION=trueat deploy time to enable (not recommended in production).
- MCP Elicitation(v3.6.0): Four tools request user confirmation before executing expensive operations —deep_research(>50 URLs),batch_scrape(sync mode, >25 URLs),crawl_deep(projected >500 pages),extract_structured(schema has >3 required fields with no LLM configured). Credit-low situations also elicit. Confirmation is best-effort: if the MCP client does not support elicitation the tool proceeds (fail-open).
- Per-tool credit gating: Every tool is wrapped withwithAuth()and is metered — credits are checked and deducted before execution, and a valid API key is required for every tool (fail-closed since v3.0.18).

v3.0.3 (2025-10-01): Removed authentication bypass vulnerability. All users must authenticate with valid API keys.

The observation layer for AI agents. Outrings provides deterministic web intelligence through MCP, transforming websites into structured, evidence-backed signals that machines can inspect and trust.

Enable AI agents to get structured data from unstructured web with AgentQL.

Easy web data access. Simplified retrieval of information from websites and online sources.

Real-time web data, structured for agents

Turn websites into datasets with Scrapezy

Scrapfly MCP Server gives AI agents a simple, unified way to scrape live web data with built-in anti-bot handling.

Interact with WebScraping.AI for web data extraction and scraping.

Zillow property data for AI agents — search listings by city or ZIP, look up any US address, and get 50+ fields per property including prices, Zestimates, price history, and sold data.