Crawl4AI RAG

With this MCP server, you canscrape anythingand thenuse that knowledge anywherefor RAG.

Consider this GitHub repository a testbed, hence why I haven't been super actively address issues and pull requests yet. I certainly will though as I bring this into Archon V2!

The server includes several advanced RAG strategies that can be enabled to enhance retrieval quality:

- Contextual Embeddingsfor enriched semantic understanding
- Hybrid Searchcombining vector and keyword search
- Agentic RAGfor specialized code example extraction
- Rerankingfor improved result relevance using cross-encoder models
- Knowledge Graphfor AI hallucination detection and repository code analysis

The Crawl4AI RAG MCP server is just the beginning. Here's where we're headed:
-

Multiple Embedding Models: Expanding beyond OpenAI to support a variety of embedding models, including the ability to run everything locally with Ollama for complete control and privacy.

Advanced RAG Strategies: Implementing sophisticated retrieval techniques like contextual retrieval, late chunking, and others to move beyond basic "naive lookups" and significantly enhance the power and precision of the RAG system, especially as it integrates with Archon.

Enhanced Chunking Strategy: Implementing a Context 7-inspired chunking approach that focuses on examples and creates distinct, semantically meaningful sections for each chunk, improving retrieval precision.

Performance Optimization: Increasing crawling and indexing speed to make it more realistic to "quickly" index new documentation to then leverage it within the same prompt in an AI coding assistant.

- Smart URL Detection: Automatically detects and handles different URL types (regular webpages, sitemaps, text files)
- Recursive Crawling: Follows internal links to discover content
- Parallel Processing: Efficiently crawls multiple pages simultaneously
- Content Chunking: Intelligently splits content by headers and size for better processing
- Vector Search: Performs RAG over crawled content, optionally filtering by data source for precision
- Source Retrieval: Retrieve sources available for filtering to guide the RAG process

The server provides essential web crawling and search tools:
- crawl_single_page: Quickly crawl a single web page and store its content in the vector database
- smart_crawl_url: Intelligently crawl a full website based on the type of URL provided (sitemap, llms-full.txt, or a regular webpage that needs to be crawled recursively)
- get_available_sources: Get a list of all available sources (domains) in the database
- perform_rag_query: Search for relevant content using semantic search with optional source filtering
- search_code_examples(requiresUSE_AGENTIC_RAG=true): Search specifically for code examples and their summaries from crawled documentation. This tool provides targeted code snippet retrieval for AI coding assistants.

Knowledge Graph Tools (requiresUSE_KNOWLEDGE_GRAPH=true, see below)

git clone https://github.com/coleam00/mcp-crawl4ai-rag.git cd mcp-crawl4ai-rag

docker build -t mcp/crawl4ai-rag --build-arg PORT=8051 .

Create a.envfile based on the configuration section below

git clone https://github.com/coleam00/mcp-crawl4ai-rag.git cd mcp-crawl4ai-rag

Create and activate a virtual environment:

uv venv .venv\Scripts\activate # on Mac/Linux: source .venv/bin/activate

Create a.envfile based on the configuration section below

Before running the server, you need to set up the database with the pgvector extension:
-

Go to the SQL Editor in your Supabase dashboard (create a new project first if necessary)

Create a new query and paste the contents ofcrawled_pages.sql

Run the query to create the necessary tables and functions

To enable AI hallucination detection and repository analysis features, you need to set up Neo4j.

Also, the knowledge graph implementation isn't fully compatible with Docker yet, so I would recommend right now running directly through uv if you want to use the hallucination detection within the MCP server!

git clone https://github.com/coleam00/local-ai-packaged.git cd local-ai-packaged

Start Neo4j: Follow the instructions in the Local AI Package repository to start Neo4j with Docker Compose

- URI:bolt://localhost:7687
- Username:neo4j
- Password: Check the Local AI Package documentation for the default password
-

- Open Neo4j Desktop
- Create a new project and database
- Set a password for theneo4juser
- Start the database

- URI:bolt://localhost:7687(default)
- Username:neo4j(default)
- Password: Whatever you set during creation

Create a.envfile in the project root with the following variables:

# MCP Server Configuration HOST=0.0.0.0 PORT=8051 TRANSPORT=sse # OpenAI API Configuration OPENAI_API_KEY=your_openai_api_key # LLM for summaries and contextual embeddings MODEL_CHOICE=gpt-4.1-nano # RAG Strategies (set to "true" or "false", default to "false") USE_CONTEXTUAL_EMBEDDINGS=false USE_HYBRID_SEARCH=false USE_AGENTIC_RAG=false USE_RERANKING=false USE_KNOWLEDGE_GRAPH=false # Supabase Configuration SUPABASE_URL=your_supabase_project_url SUPABASE_SERVICE_KEY=your_supabase_service_key # Neo4j Configuration (required for knowledge graph functionality) NEO4J_URI=bolt://localhost:7687 NEO4J_USER=neo4j NEO4J_PASSWORD=your_neo4j_password

The Crawl4AI RAG MCP server supports four powerful RAG strategies that can be enabled independently:

When enabled, this strategy enhances each chunk's embedding with additional context from the entire document. The system passes both the full document and the specific chunk to an LLM (configured viaMODEL_CHOICE) to generate enriched context that gets embedded alongside the chunk content.

- When to use: Enable this when you need high-precision retrieval where context matters, such as technical documentation where terms might have different meanings in different sections.
- Trade-offs: Slower indexing due to LLM calls for each chunk, but significantly better retrieval accuracy.
- Cost: Additional LLM API calls during indexing.

Combines traditional keyword search with semantic vector search to provide more comprehensive results. The system performs both searches in parallel and intelligently merges results, prioritizing documents that appear in both result sets.

- When to use: Enable this when users might search using specific technical terms, function names, or when exact keyword matches are important alongside semantic understanding.
- Trade-offs: Slightly slower search queries but more robust results, especially for technical content.
- Cost: No additional API costs, just computational overhead.

Enables specialized code example extraction and storage. When crawling documentation, the system identifies code blocks (≥300 characters), extracts them with surrounding context, generates summaries, and stores them in a separate vector database table specifically designed for code search.

- When to use: Essential for AI coding assistants that need to find specific code examples, implementation patterns, or usage examples from documentation.
- Trade-offs: Significantly slower crawling due to code extraction and summarization, requires more storage space.
- Cost: Additional LLM API calls for summarizing each code example.
- Benefits: Provides a dedicatedsearch_code_examplestool that AI agents can use to find specific code implementations.

Applies cross-encoder reranking to search results after initial retrieval. Uses a lightweight cross-encoder model (cross-encoder/ms-marco-MiniLM-L-6-v2) to score each result against the original query, then reorders results by relevance.

- When to use: Enable this when search precision is critical and you need the most relevant results at the top. Particularly useful for complex queries where semantic similarity alone might not capture query intent.
- Trade-offs: Adds ~100-200ms to search queries depending on result count, but significantly improves result ordering.
- Cost: No additional API costs - uses a local model that runs on CPU.
- Benefits: Better result relevance, especially for complex queries. Works with both regular RAG search and code example search.

Enables AI hallucination detection and repository analysis using Neo4j knowledge graphs. When enabled, the system can parse GitHub repositories into a graph database and validate AI-generated code against real repository structures. (NOT fully compatible with Docker yet, I'd recommend running through uv)

- When to use: Enable this for AI coding assistants that need to validate generated code against real implementations, or when you want to detect when AI models hallucinate non-existent methods, classes, or incorrect usage patterns.
- Trade-offs: Requires Neo4j setup and additional dependencies. Repository parsing can be slow for large codebases, and validation requires repositories to be pre-indexed.
- Cost: No additional API costs for validation, but requires Neo4j infrastructure (can use free local installation or cloud AuraDB).
- Benefits: Provides three powerful tools:parse_github_repositoryfor indexing codebases,check_ai_script_hallucinationsfor validating AI-generated code, andquery_knowledge_graphfor exploring indexed repositories.

You can now tell the AI coding assistant to add a Python GitHub repository to the knowledge graph like:

You can also have the AI coding assistant check for hallucinations with scripts it just created, or you can manually run the command: