Open Crawler MCP Server
About
A web crawler and text extractor with robots.txt compliance, rate limiting, and page size protection.
Details
- Author
- naoto24kawa
- Categories
- Web Scraping, Automation, Other
Jump to
Setup
Install Open Crawler MCP Server in your MCP client (Claude Desktop, Cursor, Windsurf, and others).
Repository: https://github.com/naoto24kawa/open-crawler-mcp-server
Follow the installation instructions in the repository README, then restart your MCP client.
A web crawler and text extractor with robots.txt compliance, rate limiting, and page size protection.
Enable AI agents to get structured data from unstructured web with AgentQL.
Easy web data access. Simplified retrieval of information from websites and online sources.
Turn websites into datasets with Scrapezy
Scrapfly MCP Server gives AI agents a simple, unified way to scrape live web data with built-in anti-bot handling.
Interact with WebScraping.AI for web data extraction and scraping.
Retrieve structured data through natural language conversations
CrawlForge MCP is a production-ready MCP server with 18 web scraping tools for AI agents. It gives Claude, Cursor, and any MCP-compatible client the ability to fetch URLs, extract structured data with CSS/XPath selectors, run deep multi-step research, bypass anti-bot detection with TLS fingerprint randomization, process documents, monitor page changes, and more. Credit-based pricing with a free tier (1,000 credits/month, no credit card required).
A high-performance x402 intelligence gateway providing 20+ social and web endpoints. Powered by 6-stage routing logic—including cloud-rendered JS fetches, residential IP rotation, and automatic settlement refunds for blocked hosts. Built for AI Agents that require reliable, pay-per-request access to LinkedIn, Reddit, Instagram, and beyond without API key management
Scrape, crawl, and extract data from any website using the Firecrawl API.
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.


