crawl4-mcp
About
本项目是一个crawl4ai 的爬虫MCP SERVER，提供高级网络爬虫。使用此 MCP SERVER，您可以抓取任何内容，将内容保存为本地markdown文件，然后在任何地方将该知识用于 RAG。
Details
- Author
- ShiDuLin
- Downloads
- 359
- Categories
- Other
Jump to
- Crawls any web content with advanced crawl4ai engine
- Saves content as local markdown files
- Output ready for RAG (Retrieval-Augmented Generation)
- SSE-based MCP server integration
- Simple Python 3.12+ setup with uv
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
crawl4-mcpCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Install by cloning the repository, creating a Python 3.12 virtual environment with uv, running uv sync and crawl4ai-setup, then creating a .env file. Start the server with uv run src/crawl4ai_mcp.py. The server listens on the configured host and port via SSE transport. Integrate with any MCP client using transport: sse and url: http://localhost:8051/sse.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"crawl4-mcp": {
"crawl4-mcp": {
"command": "uv",
"args": [
"venv",
"-p",
"3.12"
]
}
}
}
}McpServers
{
"crawl4-mcp": {
"command": "uv",
"args": [
"venv",
"-p",
"3.12"
]
}
}crawl4-mcp
本项目是一个crawl4ai 的爬虫MCP SERVER，提供高级网络爬虫。使用此 MCP SERVER，您可以抓取任何内容，将内容保存为本地markdown文件，然后在任何地方将该知识用于 RAG。
环境要求
- Python 3.12 或更高版本 - uv 包管理器安装步骤
1. 克隆仓库
git clone <仓库地址>
cd crawl4-mcp
2. 使用 uv 创建虚拟环境并安装依赖
uv venv -p 3.12
.\.venv\Scripts\activate
uv sync
crawl4ai-setup
3. 根据下面的配置部分创建一个.env文件
运行
uv run src/crawl4ai_mcp.py
与 MCP client集成
一旦服务器使用 SSE 传输运行，您就可以使用以下配置连接到它：{
"mcpServers": {
"crawl4-mcp": {
"transport": "sse",
"url": "http://localhost:8051/sse"
}
}
}
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
