Web-curl

🚀Open Source Project
🛠️ Built with Node.js & TypeScript (Node.js v18+ required)

Web-curlis a powerful tool for fetching and extracting text content from web pages and APIs. Use it as a standalone CLI or as an MCP (Model Context Protocol) server. Web-curl leverages Puppeteer for robust web scraping and supports advanced features such as resource blocking, custom headers, authentication, and Google Custom Search.

🚀 Deep Research & Automation (v1.4.2)

- Network Monitoring (browser_network_requests)
- Console Logs (browser_console_messages)

- multi_search: Run multiple Google searches at once (only exposed search tool).

- Idle Auto-Close: Browser automatically shuts down after 15 minutes of inactivity to save RAM/CPU.
- Tab Rotation: Automatically replaces the oldest tab when the 10-tab limit is reached.

- Full-Page Screenshots: Capture high-quality screenshots with a 5-day auto-cleanup lifecycle and custom destination support.
- Document Parsing: Extract text from PDF and DOCX files directly from URLs.

- 🗂️ Error log rotation:logs/error-log.txtis rotated when it exceeds ~1MB (renamed toerror-log.txt.bak) to prevent unbounded growth.
- 🧹 Logs & temp cleanup: old temporary files in thelogs/directory are cleaned up at startup.
- 🛑 Browser lifecycle: Puppeteer browser instances are closed in finally blocks to avoid Chromium temp file leaks.
- 🔎 Content extraction:

- Returns raw text, HTML, and Readability "main article" when available. Readability attempts to extract the primary content of a webpage, removing headers, footers, sidebars, and other non-essential elements, providing a cleaner, more focused text.
- Readability output is subject tostartIndex/maxLength/chunkSizeslicing when requested.

- destinationFolderaccepts relative paths (resolved against the project root) or absolute paths.
- The server createsdestinationFolderif it does not exist.
- Downloads are streamed using Node streams +pipelineto minimize memory use and ensure robust writes.
- Filenames are derived from the URL path (e.g.,https://.../path/file.jpg->file.jpg). If no filename is present, the fallback name isdownloaded_file.
- Overwrite semantics: by default the implementation will overwrite an existing file with the same name.

- Auto language detection (franc-min) and optional translation (dynamictranslateimport).
- Query enrichment is heuristic-based; results depend on the detected intent.

This section outlines the high-level architecture of Web-curl.

To integrate web-curl as an MCP server, add the following configuration to yourmcp_settings.json:

- In Google Cloud Console, go toAPIs & Services > Library.
- Search forCustom Search APIand enable it.

ReplaceYOUR_GOOGLE_API_KEYandYOUR_CX_IDin the config above.

# Clone the repository git clone https://github.com/rayss868/MCP-Web-Curl cd web-curl # Install dependencies npm install # Build the project npm run build

- Prerequisites: Ensure you have Node.js (v18+) and Git installed on your system.

Linux / Ubuntu Server:You must install extra dependencies for Chromium to handle rendering and screenshots in a headless environment. Run:

sudo apt-get update && sudo apt-get install -y \ fonts-liberation \ libasound2 \ libatk-bridge2.0-0 \ libatk1.0-0 \ libc6 \ libcairo2 \ libcups2 \ libdbus-1-3 \ libexpat1 \ libfontconfig1 \ libgbm1 \ libgcc1 \ libglib2.0-0 \ libgtk-3-0 \ libnspr4 \ libnss3 \ libpango-1-0-0 \ libpangocairo-1.0-0 \ libstdc++6 \ libx11-6 \ libx11-xcb1 \ libxcb1 \ libxcomposite1 \ libxcursor1 \ libxdamage1 \ libxext6 \ libxfixes3 \ libxi6 \ libxrandr2 \ libxrender1 \ libxss1 \ libxtst6 \ lsb-release \ wget \ xdg-utils

The CLI supports fetching and extracting text content from web pages.

# Basic usage node build/index.js https://example.com # With options node build/index.js --timeout 30000 https://example.com # Save output to a file node build/index.js -o result.json https://example.com

- --timeout <ms>: Set navigation timeout (default: 60000)
- -o <file>: Output result to specified file

Web-curl can be run as an MCP server for integration with Roo Context or other MCP-compatible environments.

Only the tools below are exposed vialist_toolsto reduce tool-chaining in agent clients.

-

browser_flow: One-call browser workflow (optional navigate → optional actions → return ONE result).

browser_configure: Set proxy/user-agent/viewport (session persistence is always on viauser_data/).

browser_close: Close browser and tabs (also auto-closes after 15 minutes of inactivity).

multi_search: Run multiple Google searches in parallel (the only exposed search entrypoint).

fetch_api: REST API request with response truncation (limit).

download_file: Download a file from a URL.

parse_document: Extract text from PDF/DOCX URLs.

🚦 HTML Slicing Example (Recommended for Large Pages)

{ "name": "browser_flow", "arguments": { "result": { "type": "snapshot", "mode": "html", "startIndex": 0, "endIndex": 20000 } } }

{ "mode": "html", "totalLength": 123456, "startIndex": 0, "endIndex": 20000, "remainingCharacters": 103456, "content": "<html>...first slice...</html>" }

{ "name": "fetch_api", "arguments": { "url": "https://api.github.com/repos/nodejs/node", "method": "GET", "headers": { "Accept": "application/vnd.github.v3+json" }, "limit": 10000 } }

{ "name": "download_file", "arguments": { "url": "https://example.com/image.jpg", "destinationFolder": "downloads" } }

Note:destinationFoldercan be either a relative path (resolved against the project root) or an absolute path. The server will create the destination folder if it does not exist.

{ "name": "browser_configure", "arguments": { "proxy": "http://proxy.example.com:8080", "viewport": { "width": 1920, "height": 1080 } } }

- Timeout Errors: Increase thetimeoutparameter if requests are timing out.
- Google Search Fails: EnsureAPIKEY_GOOGLE_SEARCHandCX_GOOGLE_SEARCHare set in your environment.
- Error Logs: Check thelogs/error-log.txtfile for detailed error messages.

🧠 Tips & Best Practices {#tips--best-practices}

🤝 Contributing & Issues {#contributing--issues}

Contributions are welcome! If you want to contribute, fork this repository and submit a pull request.
If you find any issues or have suggestions, please open an issue on the repository page.

📄 License & Attribution {#license--attribution}

This project was developed byRayss.
For questions, improvements, or contributions, please contact the author or open an issue in the repository.

One MCP server for 160+ live web-data APIs (search, social, e-commerce, real estate, jobs, finance) — clean JSON, free tier.

Enables AI agents and tools to access real-time, high-volume search engine data through a unified Model Context Protocol interface.

Get structured search results from Google, Bing, Yandex, DuckDuckGo, and more through one simple API.

One API for Search, Crawling, and Sitemaps

Google, Yandex, Baidu, Bing, Duckduckgo, Ecosia Search and URL data Extraction

20+ pay-per-request APIs for AI agents — screenshots, web scraping, PDF, OCR, search, QR codes, translation & more. No API keys needed. Pay with USDC via x402. npm: @pylonapi/mcp

Web search and webpage scraping using the Serper API.

1GB Free Trial, World's Leading Proxy Service Platform, Efficient Data Collection