BerryRAG
About
A local RAG system with Playwright MCP integration for Claude and OpenAI embeddings, using local storage.
Details
- Author
- berrydev-ai
- Categories
- Developer Tools, Knowledge Base, Other, Automation
Jump to
Setup
Install BerryRAG in your MCP client (Claude Desktop, Cursor, Windsurf, and others).
Repository: https://github.com/berrydev-ai/berry-rag
Follow the installation instructions in the repository README, then restart your MCP client.
🍓 BerryRAG: Local Vector Database with Playwright MCP Integration
A complete local RAG (Retrieval-Augmented Generation) system that integrates Playwright MCP web scraping with vector database storage for Claude.
- Zero-cost self-hostedvector database
- Playwright MCP integrationfor automated web scraping
- Multiple embedding providers(sentence-transformers, OpenAI, fallback)
- Smart content processingwith quality filters
- Claude-optimizedcontext formatting
- MCP serverfor direct Claude integration
- Command-line toolsfor manual operation
git clone https://github.com/berrydev-ai/berry-rag.git cd berry-rag # Install dependencies npm run install-deps # Setup directories and instructions npm run setup
Add to yourclaude_desktop_config.json:
{ "mcpServers": { "playwright": { "command": "npx", "args": ["@playwright/mcp@latest"] }, "berry-rag": { "command": "node", "args": ["mcp_servers/vector_db_server.js"], "cwd": "/Users/eberry/BerryDev/berry-rag" } } }
# Example workflow: # 1. Scrape with Playwright MCP through Claude # 2. Process into vector DB npm run process-scraped # 3. Search your knowledge base npm run search "React hooks"
berry-rag/ ├── src/ # Python source code │ ├── rag_system.py # Core vector database system │ └── playwright_integration.py # Playwright MCP integration ├── mcp_servers/ # MCP server implementations │ └── vector_db_server.ts # TypeScript MCP server ├── storage/ # Vector database storage │ ├── documents.db # SQLite metadata │ └── vectors/ # NumPy embedding files ├── scraped_content/ # Playwright saves content here └── dist/ # Compiled TypeScript
Launch the web interface for easy interaction with your RAG system:
# Start the Streamlit web interface python run_streamlit.py # Or directly with streamlit streamlit run streamlit_app.py
- 🔍 Search: Interactive document search with similarity controls
- 📄 Context: Generate formatted context for AI assistants
- ➕ Add Document: Upload files or paste content directly
- 📚 List Documents: Browse your document library
- 📊 Statistics: System health and performance metrics
# RAG System Operations python src/rag_system.py search "query" python src/rag_system.py context "query" # Claude-formatted python src/rag_system.py add <url> <title> <file> python src/rag_system.py list python src/rag_system.py stats # Playwright Integration python src/playwright_integration.py process python src/playwright_integration.py setup python src/playwright_integration.py stats
"Use Playwright to scrape the React hooks documentation from https://react.dev/reference/react and save it to the scraped_content directory"
"Process all new scraped files and add them to the BerryRAG vector database"
"Search the BerryRAG database for information about React useState best practices" "Get context from the vector database about implementing custom hooks"
BerryRAG provides two powerful MCP servers for Claude integration:
- add_document- Add content directly to vector DB
- search_documents- Search for similar content
- get_context- Get formatted context for queries
- list_documents- List all stored documents
- get_stats- Vector database statistics
- process_scraped_files- Process Playwright scraped content
- save_scraped_content- Save content for later processing
- crawl_content- Advanced web content extraction with subpage support
- extract_links- Extract internal links for subpage discovery
- get_content_preview- Quick content preview without full processing
📖For complete MCP setup and usage guide, seeBERRY_MCP.md
The system supports multiple embedding providers with automatic fallback:
- sentence-transformers(recommended, free, local)
- OpenAI embeddings(requires API key, setOPENAI_API_KEY)
- Simple hash-based(fallback, not recommended for production)
# Optional: for OpenAI embeddings export OPENAI_API_KEY=your_key_here
- Content shorter than 100 characters
- Navigation-only content
- Repetitive/duplicate content
- Files larger than 500KB
- Default chunk size: 500 characters
- Overlap: 50 characters
- Smart boundary detection (sentences, paragraphs)
# Vector database statistics python src/rag_system.py stats # Processing status python src/playwright_integration.py stats # View recent documents python src/rag_system.py list
- Database:storage/documents.db(SQLite metadata)
- Vectors:storage/vectors/(NumPy arrays)
- Scraped Content:scraped_content/(Markdown files)
- Scrape research papers with Playwright
- Process into vector database
- Query for specific concepts across all papers
- Scrape API documentation from multiple sources
- Build unified searchable knowledge base
- Get contextual answers about implementation details
- Scrape blog posts and articles
- Create topic-based knowledge clusters
- Find related content across sources
# Test RAG system python src/rag_system.py stats # Test integration python src/playwright_integration.py setup # Test MCP server node mcp_servers/vector_db_server.js
Embedding model download slow:The first run downloads sentence-transformers model (~90MB). This is normal.
- Check if documents were processed:python src/rag_system.py list
- Verify content quality filters aren't too strict
- Try broader search terms
- Python logs: Check console output
- MCP server logs: Stderr output
- Processing status:scraped_content/.processed_files.json
MIT License - feel free to modify and extend for your needs.
This is a personal project for Eric Berry, but feel free to fork and adapt for your own use cases.
This is a web browser that enables your coding agent, such as Claude Code, to visit websites on your behalf and assist you in identifying bugs or creating UI test cases.
Local code analysis MCP server with 25+ tools: semantic search, call graph tracing, dependency analysis, and symbol navigation. Built with Tree-sitter and CozoDB. Supports Go, Python, JS, TS.
A local-first code indexer that enhances LLMs with deep code understanding. It integrates with AI assistants via the Model Context Protocol (MCP) and supports AI-powered semantic search.
A server for managing structured project context using SQLite, with support for vector embeddings for semantic search and Retrieval Augmented Generation (RAG).
MCP of MCPs is a meta-server that merges all your MCP servers into a single smart endpoint. It gives AI agents instant tool discovery, selective schema loading, and massively cheaper execution, so you stop wasting tokens and time. With persistent tool metadata, semantic search, and direct code execution between tools, it turns chaotic multi-server setups into a fast, efficient, hallucination-free workflow. It also automatically analyzes the tools output schemas if not exist and preserves them across sessions for consistent behavior.
Persistent memory for AI coding agents with semantic search, contradiction detection, memory decay, and cross-session learning. 25 MCP tools, local-first, #1 on LongMemEval (95.4%).
An MCP server for intelligent semantic search and automatic learning within codebases, allowing AI agents to efficiently query and index project artifacts.
Stdio MCP server that exposes zkShare tools to AI clients: store encrypted context, proofs, semantic search, sharing, and sandbox calls via POST /api/v1/context with ZKSHARE_API_KEY.
Embeddings, vector search, document storage, and full-text search with the open-source AI application database
Semantic search through Dickens' classic tale. Find passages by meaning, theme, or concept - not just keywords.
An MCP server providing semantic search capabilities for APLCart data.
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
