Crawl4ai Rag
About
<h1 align="center">Crawl4AI RAG MCP Server</h1> <p align="center"> <em>Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants</em> </p> A powerful implementation of the [Model Context Protocol (MCP)](https://modelcontextprotocol.io) integrated with [Crawl4AI](https://crawl4ai.com) and…
Explore
- Smart URL Detection: Automatically detects and handles different URL types (regular webpages, sitemaps, text files)
- Recursive Crawling: Follows internal links to discover content
- Parallel Processing: Efficiently crawls multiple pages simultaneously
- Content Chunking: Intelligently splits content by headers and size for better processing
- Vector Search: Performs RAG over crawled content, optionally filtering by data source for precision
- Source Retrieval: Retrieve sources available for filtering to guide the RAG process
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
Crawl4ai RagCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
- Docker/Docker Desktop if running the MCP server as a container (recommended)
- Python 3.12+ if running the MCP server directly through uv
- Supabase (database for RAG)
- OpenAI API key (for generating embeddings)
- Neo4j (optional, for knowledge graph functionality) - see Knowledge Graph Setup section
Before running the server, you need to set up the database with the pgvector extension:
1. Go to the SQL Editor in your Supabase dashboard (create a new project first if necessary)
2. Create a new query and paste the contents of crawled_pages.sql
3. Run the query to create the necessary tables and functions
To enable AI hallucination detection and repository analysis features, you need to set up Neo4j.
Also, the knowledge graph implementation isn't fully compatible with Docker yet, so I would recommend right now running directly through uv if you want to use the hallucination detection within the MCP server!
For installing Neo4j:
Alternatively, install Neo4j directly:
1. Install Neo4j Desktop: Download from neo4j.com/download
2. Create a new database:
- Open Neo4j Desktop
- Create a new project and database
- Set a password for the neo4j user
- Start the database
3. Note your connection details:
- URI: bolt://localhost:7687 (default)
- Username: neo4j (default)
- Password: Whatever you set during creation
HOST=0.0.0.0
PORT=8051
TRANSPORT=sse
OPENAI_API_KEY=your_openai_api_key
SUPABASE_URL=your_supabase_project_url
SUPABASE_SERVICE_KEY=your_supabase_service_key
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_neo4j_password
For general documentation RAG:
USE_CONTEXTUAL_EMBEDDINGS=false
USE_HYBRID_SEARCH=true
USE_AGENTIC_RAG=false
USE_RERANKING=true
For AI coding assistant with code examples:
USE_CONTEXTUAL_EMBEDDINGS=true
USE_HYBRID_SEARCH=true
USE_AGENTIC_RAG=true
USE_RERANKING=true
USE_KNOWLEDGE_GRAPH=false
For AI coding assistant with hallucination detection:
USE_CONTEXTUAL_EMBEDDINGS=true
USE_HYBRID_SEARCH=true
USE_AGENTIC_RAG=true
USE_RERANKING=true
USE_KNOWLEDGE_GRAPH=true
For fast, basic RAG:
USE_CONTEXTUAL_EMBEDDINGS=false
USE_HYBRID_SEARCH=true
USE_AGENTIC_RAG=false
USE_RERANKING=false
USE_KNOWLEDGE_GRAPH=false
Once you have the server running with SSE transport, you can connect to it using this configuration:
{
"mcpServers": {
"crawl4ai-rag": {
"transport": "sse",
"url": "http://localhost:8051/sse"
}
}
}
> Note for Windsurf users: Use serverUrl instead of url in your configuration:
> > {
> "mcpServers": {
> "crawl4ai-rag": {
> "transport": "sse",
> "serverUrl": "http://localhost:8051/sse"
> }
> }
> }
>
>
> Note for Docker users: Use host.docker.internal instead of localhost if your client is running in a different container. This will apply if you are using this MCP server within n8n!
> Note for Claude Code users:
claude mcp add-json crawl4ai-rag '{"type":"http","url":"http://localhost:8051/sse"}' --scope user
Add this server to your MCP configuration for Claude Desktop, Windsurf, or any other MCP client:
{
"mcpServers": {
"crawl4ai-rag": {
"command": "python",
"args": ["path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"],
"env": {
"TRANSPORT": "stdio",
"OPENAI_API_KEY": "your_openai_api_key",
"SUPABASE_URL": "your_supabase_url",
"SUPABASE_SERVICE_KEY": "your_supabase_service_key",
"USE_KNOWLEDGE_GRAPH": "false",
"NEO4J_URI": "bolt://localhost:7687",
"NEO4J_USER": "neo4j",
"NEO4J_PASSWORD": "your_neo4j_password"
}
}
}
}
{
"mcpServers": {
"crawl4ai-rag": {
"command": "docker",
"args": ["run", "--rm", "-i",
"-e", "TRANSPORT",
"-e", "OPENAI_API_KEY",
"-e", "SUPABASE_URL",
"-e", "SUPABASE_SERVICE_KEY",
"-e", "USE_KNOWLEDGE_GRAPH",
"-e", "NEO4J_URI",
"-e", "NEO4J_USER",
"-e", "NEO4J_PASSWORD",
"mcp/crawl4ai"],
"env": {
"TRANSPORT": "stdio",
"OPENAI_API_KEY": "your_openai_api_key",
"SUPABASE_URL": "your_supabase_url",
"SUPABASE_SERVICE_KEY": "your_supabase_service_key",
"USE_KNOWLEDGE_GRAPH": "false",
"NEO4J_URI": "bolt://localhost:7687",
"NEO4J_USER": "neo4j",
"NEO4J_PASSWORD": "your_neo4j_password"
}
}
}
}
```
The server provides essential web crawling and search tools:
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"crawl4ai rag": {
"crawl4ai-rag": {
"command": "python",
"args": [
"path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"
],
"env": {
"TRANSPORT": "stdio",
"OPENAI_API_KEY": "your_openai_api_key",
"SUPABASE_URL": "your_supabase_url",
"SUPABASE_SERVICE_KEY": "your_supabase_service_key",
"USE_KNOWLEDGE_GRAPH": "false",
"NEO4J_URI": "bolt://localhost:7687",
"NEO4J_USER": "neo4j",
"NEO4J_PASSWORD": "your_neo4j_password"
}
}
}
}
}McpServers
{
"crawl4ai-rag": {
"command": "python",
"args": [
"path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"
],
"env": {
"TRANSPORT": "stdio",
"OPENAI_API_KEY": "your_openai_api_key",
"SUPABASE_URL": "your_supabase_url",
"SUPABASE_SERVICE_KEY": "your_supabase_service_key",
"USE_KNOWLEDGE_GRAPH": "false",
"NEO4J_URI": "bolt://localhost:7687",
"NEO4J_USER": "neo4j",
"NEO4J_PASSWORD": "your_neo4j_password"
}
}
}<h1 align="center">Crawl4AI RAG MCP Server</h1>
<p align="center">
<em>Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants</em>
</p>
A powerful implementation of the Model Context Protocol (MCP) integrated with Crawl4AI and Supabase for providing AI agents and AI coding assistants with advanced web crawling and RAG capabilities.
With this MCP server, you can <b>scrape anything</b> and then <b>use that knowledge anywhere</b> for RAG.
The primary goal is to bring this MCP server into Archon as I evolve it to be more of a knowledge engine for AI coding assistants to build AI agents. This first version of the Crawl4AI/RAG MCP server will be improved upon greatly soon, especially making it more configurable so you can use different embedding models and run everything locally with Ollama.
Consider this GitHub repository a testbed, hence why I haven't been super actively address issues and pull requests yet. I certainly will though as I bring this into Archon V2!
Overview
This MCP server provides tools that enable AI agents to crawl websites, store content in a vector database (Supabase), and perform RAG over the crawled content. It follows the best practices for building MCP servers based on the Mem0 MCP server template I provided on my channel previously.
The server includes several advanced RAG strategies that can be enabled to enhance retrieval quality:
- Contextual Embeddings for enriched semantic understanding
- Hybrid Search combining vector and keyword search
- Agentic RAG for specialized code example extraction
- Reranking for improved result relevance using cross-encoder models
- Knowledge Graph for AI hallucination detection and repository code analysis
See the Configuration section below for details on how to enable and configure these strategies.
Vision
The Crawl4AI RAG MCP server is just the beginning. Here's where we're headed:
1. Integration with Archon: Building this system directly into Archon to create a comprehensive knowledge engine for AI coding assistants to build better AI agents.
2. Multiple Embedding Models: Expanding beyond OpenAI to support a variety of embedding models, including the ability to run everything locally with Ollama for complete control and privacy.
3. Advanced RAG Strategies: Implementing sophisticated retrieval techniques like contextual retrieval, late chunking, and others to move beyond basic "naive lookups" and significantly enhance the power and precision of the RAG system, especially as it integrates with Archon.
4. Enhanced Chunking Strategy: Implementing a Context 7-inspired chunking approach that focuses on examples and creates distinct, semantically meaningful sections for each chunk, improving retrieval precision.
5. Performance Optimization: Increasing crawling and indexing speed to make it more realistic to "quickly" index new documentation to then leverage it within the same prompt in an AI coding assistant.
Features
- Smart URL Detection: Automatically detects and handles different URL types (regular webpages, sitemaps, text files)
- Recursive Crawling: Follows internal links to discover content
- Parallel Processing: Efficiently crawls multiple pages simultaneously
- Content Chunking: Intelligently splits content by headers and size for better processing
- Vector Search: Performs RAG over crawled content, optionally filtering by data source for precision
- Source Retrieval: Retrieve sources available for filtering to guide the RAG process
Tools
The server provides essential web crawling and search tools:
Core Tools (Always Available)
1. crawl_single_page: Quickly crawl a single web page and store its content in the vector database
2. smart_crawl_url: Intelligently crawl a full website based on the type of URL provided (sitemap, llms-full.txt, or a regular webpage that needs to be crawled recursively)
3. get_available_sources: Get a list of all available sources (domains) in the database
4. perform_rag_query: Search for relevant content using semantic search with optional source filtering
Conditional Tools
5. search_code_examples (requires USE_AGENTIC_RAG=true): Search specifically for code examples and their summaries from crawled documentation. This tool provides targeted code snippet retrieval for AI coding assistants.
Knowledge Graph Tools (requires USE_KNOWLEDGE_GRAPH=true, see below)
6. parse_github_repository: Parse a GitHub repository into a Neo4j knowledge graph, extracting classes, methods, functions, and their relationships for hallucination detection
7. check_ai_script_hallucinations: Analyze Python scripts for AI hallucinations by validating imports, method calls, and class usage against the knowledge graph
8. query_knowledge_graph: Explore and query the Neo4j knowledge graph with commands like repos, classes, methods, and custom Cypher queries
Prerequisites
- Docker/Docker Desktop if running the MCP server as a container (recommended)
- Python 3.12+ if running the MCP server directly through uv
- Supabase (database for RAG)
- OpenAI API key (for generating embeddings)
- Neo4j (optional, for knowledge graph functionality) - see Knowledge Graph Setup section
Installation
Using Docker (Recommended)
1. Clone this repository:
git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
cd mcp-crawl4ai-rag
2. Build the Docker image:
docker build -t mcp/crawl4ai-rag --build-arg PORT=8051 .
3. Create a .env file based on the configuration section below
Using uv directly (no Docker)
1. Clone this repository:
git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
cd mcp-crawl4ai-rag
2. Install uv if you don't have it:
pip install uv
3. Create and activate a virtual environment:
uv venv
.venv\Scripts\activate
# on Mac/Linux: source .venv/bin/activate
4. Install dependencies:
uv pip install -e .
crawl4ai-setup
5. Create a .env file based on the configuration section below
Database Setup
Before running the server, you need to set up the database with the pgvector extension:
1. Go to the SQL Editor in your Supabase dashboard (create a new project first if necessary)
2. Create a new query and paste the contents of crawled_pages.sql
3. Run the query to create the necessary tables and functions
Knowledge Graph Setup (Optional)
To enable AI hallucination detection and repository analysis features, you need to set up Neo4j.
Also, the knowledge graph implementation isn't fully compatible with Docker yet, so I would recommend right now running directly through uv if you want to use the hallucination detection within the MCP server!
For installing Neo4j:
Local AI Package (Recommended)
The easiest way to get Neo4j running locally is with the Local AI Package - a curated collection of local AI services including Neo4j:
1. Clone the Local AI Package:
git clone https://github.com/coleam00/local-ai-packaged.git
cd local-ai-packaged
2. Start Neo4j:
Follow the instructions in the Local AI Package repository to start Neo4j with Docker Compose
3. Default connection details:
- URI: bolt://localhost:7687
- Username: neo4j
- Password: Check the Local AI Package documentation for the default password
…
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
