TheCrawler

TheCrawler — AI-ready web scraper with validated extraction contracts

Scrape web pages, run LLM-powered structured extraction, or diagnose whether URLs are ready for a built-in extraction contract before spending LLM tokens. Open source engine (AGPL-3.0). $0.005 per successfully scraped page on Apify.

Start with a safe test: run one public URL withdryRun: trueon Apify, or clone the current GitHub source and run the local CLI/MCP build fromengine/. A small proof pack is inexamples/diagnostic-challenge, including a sample readiness report atexamples/diagnostic-challenge/sample-report.md.

- Scope: up to 25 public URLs and one target output shape.
- First step: send a public fit check through thestructured issue formor use the private fit-check path on thesprint page.
- Payment: requested only after the workflow looks like a fit, by one-off $500 payment link or invoice.
- Output: a readiness report with ready, mixed, blocked, or not-worth-automating-yet guidance.
- Credit: if the workflow continues into setup or hosted usage, the $500 is credited toward that next step.

The public offer thread isGitHub issue #1. The proof pack includes asample readiness reportshowing the report shape before a buyer sends URLs.

Workflow type: Public URLs (up to 25): Target output shape / required fields: Known blockers or constraints: Timing:

{ "urls": ["https://example.com"], "extractMarkdown": true, "dryRun": true }

git clone https://github.com/manchittlab/TheCrawler.git cd TheCrawler/engine npm install npm run build node dist/cli.js crawl https://example.com --markdown

{ "urls": ["https://example.com"], "extractMarkdown": true, "rotateUserAgent": true, "requestRetries": 3 }

Returns richPageDataper URL: title, description, language, canonical URL, robots directives, full text, boilerplate-stripped markdown, links (with internal/external flag), images (with lazy-load src), meta tags, OG/Twitter Card, JSON-LD, microdata, commerce data, forms, analytics-detected, optional email-like/phone-like public text fields, social links, hreflang, pagination, redirect chain, response headers + timing, plus structurederrorType+errorRetryableon failure.

{ "urls": ["https://shop.example.com/products/123"], "extractMode": true, "extractJsonSchema": { "type": "object", "properties": { "productName": { "type": "string" }, "price": { "type": "number" }, "currency": { "type": "string" }, "inStock": { "type": "boolean" } }, "required": ["productName"] }, "llmBaseUrl": "https://api.openai.com/v1/chat/completions", "llmModel": "gpt-4o-mini" }

Crawls the URL → cleans to markdown → sends(markdown + schema)to your OpenAI-compatible chat-completions endpoint → returns parsed typed data per URL. Schema-backed extraction uses JSON Schema response format where supported, with fallbacks for endpoints that only support JSON-object or text output. Supports natural-languageextractPromptinstead of/alongside the schema. The actor charges per page like normal; the LLM call cost is whatever your endpoint charges.

Note: extract mode requires a publicly-reachable LLM endpoint. LAN URLs (e.g.http://192.168.x.x) are not reachable from Apify infrastructure. Use OpenAI, hosted vLLM, or expose your local server via a tunnel.

SetTHECRAWLER_LLM_API_KEYas an Actor environment variable so the LLM key never lands in run inputs (visible in run history).

{ "urls": ["https://example.com/listing-1", "https://example.com/listing-2"], "diagnoseMode": true, "extractContract": "real-estate-listing", "diagnosticReport": true }

Runs crawl + readiness scoring without an LLM call. Dataset output includes per-URLverdict,readyForExtraction,score,blockers,warnings, andrecommendedNextStep, plus a workflow summary. WhendiagnosticReportis true, the actor savescontract-diagnostic-reportin the run key-value store as Markdown with a missing-readiness-signal summary. The report intentionally excludes raw extracted contact details.

{ "urls": ["https://example.com/listing-1"], "extractMode": true, "extractContract": "product-page", "llmBaseUrl": "https://api.openai.com/v1/chat/completions", "llmModel": "gpt-4o-mini" }

Uses the selected contract schema and prompt, then appends contract validation to the extraction result. Agents can branch onvalidation.validandvalidation.missingRequiredFieldsinstead of trusting loose markdown. Built-in contracts currently coverreal-estate-listingandproduct-page.

Top-N Google results crawled in one call. Optional SerpAPI key for reliable search.

{ "searchQuery": "best CRM 2026", "searchLimit": 10, "extractMarkdown": true }

Sitemap.xml + sitemap-index files resolved automatically.

{ "sitemapUrl": "https://example.com/sitemap.xml", "maxPages": 50 }

PDF and DOCX URLs are auto-detected and parsed. Returns extracted text + (for PDFs) metadata, page count.

- Crawl mode: $0.005 per page successfully scraped (failed pages don't charge).
- Extract mode / diagnostic mode: still charged per successfully scraped page. LLM endpoint cost is paid by the endpoint owner, not by this actor.
- Extraction readiness sprint: $500 after fit confirmation for one public workflow: up to 25 public URLs, one target output shape, and a ready / mixed / blocked report. Payment is by one-off link or invoice after scope is confirmed. If the workflow continues into setup or hosted usage, the $500 is credited toward that next step. If another stack is a better fit, the report says so.

The current open-source engine source for this actor build is inengine/; drop it into your own Node project, MCP server, CLI, or REST API server. The published npm package is older than this GitHub source until the next npm publish, so use the GitHub-source path below for current validated-contract and MCP tools. Self-hosting avoids Apify per-page charges, while your own infrastructure and LLM endpoint costs still apply.

# Current GitHub source build cd engine npm install npm run build # CLI node dist/cli.js crawl https://example.com --markdown node dist/cli.js extract https://example.com --schema '{...}' # MCP server (Cline, Claude Code, Cursor, Windsurf) node dist/mcp.js # REST API server THECRAWLER_API_KEY=local_test_key node dist/server.js --port 3000 curl -H "Authorization: Bearer local_test_key" \ "http://localhost:3000/v1/contracts?includeSchema=true" curl -X POST "http://localhost:3000/v1/scrape" \ -H "Authorization: Bearer local_test_key" \ -H "Content-Type: application/json" \ -d '{"url":"https://example.com/product","formats":["markdown","metadata","links","structuredData","commerceData"]}' curl -X POST "http://localhost:3000/v1/diagnose" \ -H "Authorization: Bearer local_test_key" \ -H "Content-Type: application/json" \ -d '{"contractName":"product-page","urls":["https://example.com/product"],"reportMarkdown":true}' curl -X POST "http://localhost:3000/v1/map" \ -H "Authorization: Bearer local_test_key" \ -H "Content-Type: application/json" \ -d '{"url":"https://example.com","maxPages":1}' curl -X POST "http://localhost:3000/v1/extract-contract" \ -H "Authorization: Bearer local_test_key" \ -H "Content-Type: application/json" \ -d '{"contractName":"product-page","urls":["https://example.com/product"],"llmBaseUrl":"http://localhost:1234/v1/chat/completions","llmModel":"qwen/qwen3.5-9b"}' # Older npm package; use for plain crawl only until the next publish npm install thecrawler thecrawler crawl https://example.com --markdown

For Cline setup from a GitHub clone, usellms-install.md. The current GitHub source is the review path for validated contracts and MCP tools until npm is updated.

GitHub:https://github.com/manchittlab/TheCrawler· License: AGPL-3.0

Scrapfly MCP Server gives AI agents a simple, unified way to scrape live web data with built-in anti-bot handling.