Web Scraper Service

A Python-based MCP server for headless web scraping. It extracts the main text content from web pages and outputs it as Markdown, text, or HTML.

Web Scrapper Service (MCP Stdin/Stdout & HTTP)

A Python-based MCP server for robust, headless web scraping—extracts main text content from web pages and outputs Markdown, text, or HTML for seamless AI and automation integration.

- Headless browser scraping (Playwright, BeautifulSoup, Markdownify)
- Outputs Markdown, text, or HTML
- Designed for MCP (Model Context Protocol) stdio/JSON-RPC integration
- Dual transport: stdio (default) and Streamable HTTP for shared service mode
- Persistent browser pool: Chromium stays alive across requests for fast scraping
- Smart DOM wait: MutationObserver-based content stabilization instead of fixed sleep
- Dockerized, with pre-built images
- Configurable via environment variables
- Robust error handling (timeouts, HTTP errors, Cloudflare, etc.)
- Per-domain rate limiting
- Easy integration with AI tools and IDEs (Cursor, Claude Desktop, Continue, JetBrains, Zed, etc.)
- One-click install for Cursor, interactive installer for Claude

Run with Docker (stdio mode — one container per client)

docker run -i --rm ghcr.io/justazul/web-scrapper-stdio

Run as Shared HTTP Service (one container, multiple clients)

docker run -d --name web-scraper \ -e MCP_TRANSPORT=streamable-http \ -e MCP_HTTP_PORT=8080 \ -e BROWSER_POOL_SIZE=3 \ -p 8080:8080 \ --shm-size=3gb \ ghcr.io/justazul/web-scrapper-stdio

Each MCP client spawns its own container viadocker run -i. Simple, zero configuration, works with any MCP client.

{ "mcpServers": { "web-scrapper-stdio": { "command": "docker", "args": ["run", "-i", "--rm", "ghcr.io/justazul/web-scrapper-stdio"] } } }

Run one persistent container that serves multiple MCP clients over HTTP. Saves resources when running multiple AI tool instances (e.g., multiple Claude Code sessions).

docker run -d --name web-scraper \ -e MCP_TRANSPORT=streamable-http \ -e MCP_HTTP_PORT=8080 \ -p 8080:8080 \ --shm-size=3gb \ ghcr.io/justazul/web-scrapper-stdio

{ "mcpServers": { "web-scrapper": { "url": "http://localhost:8080/mcp" } } }

This service supports integration with a wide range of AI tools and IDEs that implement the Model Context Protocol (MCP). Below are ready-to-use configuration examples for the most popular environments. Replace the image/tag as needed for custom builds.

Add to your.cursor/mcp.json(project-level) or~/.cursor/mcp.json(global):

{ "mcpServers": { "web-scrapper-stdio": { "command": "docker", "args": [ "run", "-i", "--rm", "ghcr.io/justazul/web-scrapper-stdio" ] } } }

Add to your Claude Desktop MCP config (typicallyclaude_desktop_config.json):

{ "mcpServers": { "web-scrapper-stdio": { "command": "docker", "args": [ "run", "-i", "--rm", "ghcr.io/justazul/web-scrapper-stdio" ] } } }

Add to your.mcp.jsonor global~/.claude.json:

{ "mcpServers": { "web-scrapper-stdio": { "command": "docker", "args": ["run", "-i", "--rm", "ghcr.io/justazul/web-scrapper-stdio"] } } }

HTTP mode(shared service — start the service first):

{ "mcpServers": { "web-scrapper": { "url": "http://localhost:8080/mcp" } } }

Add to yourcontinue.config.jsonor via the Continue plugin MCP settings:

{ "mcpServers": { "web-scrapper-stdio": { "command": "docker", "args": [ "run", "-i", "--rm", "ghcr.io/justazul/web-scrapper-stdio" ] } } }

Go toSettings > Tools > AI Assistant > Model Context Protocol (MCP)and add a new server. Use:

{ "command": "docker", "args": [ "run", "-i", "--rm", "ghcr.io/justazul/web-scrapper-stdio" ] }

Add to your Zed MCP config (see Zed docs for the exact path):

{ "mcpServers": { "web-scrapper-stdio": { "command": "docker", "args": [ "run", "-i", "--rm", "ghcr.io/justazul/web-scrapper-stdio" ] } } }

This web scrapper is used as an MCP (Model Context Protocol) tool, allowing it to be used by AI models or other automation directly.

- url(string, required): The URL to scrape
- max_length(integer, optional): Maximum length of returned content (default: unlimited)
- timeout_seconds(integer, optional): Timeout in seconds for the page load (default: 30)
- user_agent(string, optional): Custom User-Agent string passed directly to the browser (defaults to a random agent)
- wait_for_network_idle(boolean, optional): Wait for network activity to settle before scraping (default: true)
- custom_elements_to_remove(list of strings, optional): Additional HTML elements (CSS selectors) to remove before extraction
- grace_period_seconds(float, optional): Time to wait for JS rendering after navigation. Uses MutationObserver for smart detection. Set to 0 to skip entirely. (default: 0.5)
- output_format(string, optional):markdown,text, orhtml(default:markdown)
- click_selector(string, optional): If provided, click the element matching this selector after navigation and before extraction

- Markdown formatted content extracted from the webpage, as a string
- Errors are reported as strings starting with[ERROR] ...

Example: Usingclick_selectorandcustom_elements_to_remove

{ "url": "http://uitestingplayground.com/clientdelay", "click_selector": "#ajaxButton", "grace_period_seconds": 10, "custom_elements_to_remove": [".ads-banner", "#popup"], "output_format": "markdown" }

- url(string, required): The URL to scrape
- output_format(string, optional):markdown,text, orhtml(default:markdown)

- Content extracted from the webpage in the chosen format

- Markdown is returned by default but text or HTML can be requested viaoutput_format.
- The scrapper does not check robots.txt and will attempt to fetch any URL provided.
- No REST API or CLI tool is included; this is a pure MCP stdio/JSON-RPC tool.
- The scrapper always extracts the full<body>content of web pages, applying only essential noise removal (removing script, style, nav, footer, aside, header, and similar non-content tags). The scrapper detects and handles Cloudflare challenge screens, returning a specific error string.

You can override most configuration options using environment variables:

- DEFAULT_TIMEOUT_SECONDS: Timeout for page loads and navigation (default: 30)
- DEFAULT_MIN_CONTENT_LENGTH: Minimum content length for extracted text (default: 100)
- DEFAULT_MIN_CONTENT_LENGTH_SEARCH_APP: Minimum content length for search.app domains (default: 30)
- DEFAULT_MIN_SECONDS_BETWEEN_REQUESTS: Minimum delay between requests to the same domain (default: 2)
- DEFAULT_GRACE_PERIOD_SECONDS: Default grace period for JS rendering (default: 0.5)
- DEBUG_LOGS_ENABLED: Set totrueto enable debug-level logs (default:false)

- BROWSER_POOL_ENABLED: Enable persistent browser pool (default:true). Set tofalsefor per-request browser launch (original behavior).
- BROWSER_POOL_SIZE: Number of Chromium instances to keep alive (default: 2). Each instance uses ~100-200MB RAM.

- MCP_TRANSPORT: Transport mode —stdioorstreamable-http(default:stdio)
- MCP_HTTP_PORT: HTTP server port when using streamable-http transport (default: 8080)
- MCP_HTTP_HOST: HTTP server bind address (default:0.0.0.0)

- CAPTCHA_API_KEY: API key for captcha solver service. When set, Cloudflare Turnstile challenges are solved automatically. When empty (default), CF-protected pages return an error.
- CAPTCHA_PROVIDER: Captcha solver provider —2captcha,capsolver, orcapmonster(default:2captcha)
- CAPTCHA_BASE_URL: Custom solver API endpoint (default: uses provider's official URL)
- CAPTCHA_TIMEOUT: Timeout in seconds for captcha solving (default: 120)

- DEFAULT_TEST_REQUEST_TIMEOUT: Timeout for test requests (default: 10)
- DEFAULT_TEST_NO_DELAY_THRESHOLD: Threshold for skipping artificial delays in tests (default: 0.5)

- The scrapper detects and returns errors for navigation failures, timeouts, HTTP errors (including 404), and Cloudflare anti-bot challenges.
- Rate limiting is enforced per domain (default: 2 seconds between requests).
- Cloudflare bypass: UsesPatchright(CDP-level anti-detection) for passive evasion. Most CF-protected sites are scraped without triggering a challenge. When a Turnstile challenge is triggered andCAPTCHA_API_KEYis set, it's solved automatically via third-party API.
- Limitations:

docker compose up --build --abort-on-container-exit test

docker compose up --build --abort-on-container-exit test_mcp

docker compose up --build --abort-on-container-exit test_scrapper

This project is licensed under theMIT License.

MCP Server for Xiaohongshu (小红书) — let AI search notes, read content, and publish via browser automation