MCP Go Colly Crawler

A web crawling framework that integrates the Model Context Protocol (MCP) with the Colly web scraping library.

MCP Go Colly is a sophisticated web crawling framework that integrates the Model Context Protocol (MCP) with the powerful Colly web scraping library. This project aims to provide a flexible and extensible solution for extracting web content for large language model (LLM) applications.

- Concurrent web crawling with configurable depth and domain restrictions
- MCP server integration for tool-based crawling
- Graceful shutdown handling
- Robust error handling and result formatting
- Support for both single URL and batch URL crawling

- Go 1.21 or later
- Make (for using Makefile commands)

git clone https://github.com/yourusername/mcp-go-colly.git cd mcp-go-colly

The project includes a Makefile with several useful commands:

# Build the binary (outputs to bin/mcp-go-colly) make build # Build for all platforms (Linux, Windows, macOS) make build-all # Run tests make test # Clean build artifacts make clean # Format code make fmt # Run linter make lint

All binaries will be generated in thebin/directory.

Then you need to add the following configuration to theclaude_desktop_config.jsonfile:

{ "mcpServers": { "web-scraper": { "command": "<add path here>/mcp-go-colly/bin/mcp-go-colly" } } }

The crawler is implemented as an MCP tool that can be called with the following parameters:

{ "urls": ["https://example.com"], // Single URL or array of URLs "max_depth": 2 // Optional: Maximum crawl depth (default: 2) }

result, err := crawlerTool.Call(ctx, mcp.CallToolRequest{ Params: struct{ Arguments map[string]interface{} }{ Arguments: map[string]interface{}{ "urls": []string{"https://example.com"}, "max_depth": 2, }, }, })

- max_depth: Set maximum crawl depth (default: 2)
- urls: Single URL string or array of URLs to crawl
- Domain restrictions are automatically applied based on the provided URLs
- Fork the repository
- Create your feature branch
- Commit your changes
- Push to the branch
- Create a Pull Request

- Colly Web Scraping Framework
- Mark3 Labs MCP Project

Enable AI agents to get structured data from unstructured web with AgentQL.

Easy web data access. Simplified retrieval of information from websites and online sources.

Turn websites into datasets with Scrapezy

Scrapfly MCP Server gives AI agents a simple, unified way to scrape live web data with built-in anti-bot handling.

Interact with WebScraping.AI for web data extraction and scraping.

Retrieve structured data through natural language conversations

CrawlForge MCP is a production-ready MCP server with 18 web scraping tools for AI agents. It gives Claude, Cursor, and any MCP-compatible client the ability to fetch URLs, extract structured data with CSS/XPath selectors, run deep multi-step research, bypass anti-bot detection with TLS fingerprint randomization, process documents, monitor page changes, and more. Credit-based pricing with a free tier (1,000 credits/month, no credit card required).

A high-performance x402 intelligence gateway providing 20+ social and web endpoints. Powered by 6-stage routing logic—including cloud-rendered JS fetches, residential IP rotation, and automatic settlement refunds for blocked hosts. Built for AI Agents that require reliable, pay-per-request access to LinkedIn, Reddit, Instagram, and beyond without API key management

Scrape, crawl, and extract data from any website using the Firecrawl API.