WebSearch

WebSearch - Advanced Web Search and Content Extraction Tool

A powerful web search and content extraction tool built with Python, leveraging the Firecrawl API for advanced web scraping, searching, and content analysis capabilities.

- Advanced Web Search: Perform intelligent web searches with customizable parameters
- Content Extraction: Extract specific information from web pages using natural language prompts
- Web Crawling: Crawl websites with configurable depth and limits
- Web Scraping: Scrape web pages with support for various output formats
- MCP Integration: Built as a Model Context Protocol (MCP) server for seamless integration

- Python 3.8 or higher
- uv package manager
- Firecrawl API key
- OpenAI API key (optional, for enhanced features)
- Tavily API key (optional, for additional search capabilities)

# On Windows (using pip) pip install uv # On Unix/MacOS curl -LsSf https://astral.sh/uv/install.sh | sh # Add uv to PATH (Unix/MacOS) export PATH="$HOME/.local/bin:$PATH" # Add uv to PATH (Windows - add to Environment Variables) # Add: %USERPROFILE%\.local\bin

git clone https://github.com/yourusername/websearch.git cd websearch

- Create and activate a virtual environment with uv:

# Create virtual environment uv venv # Activate on Windows .\.venv\Scripts\activate.ps1 # Activate on Unix/MacOS source .venv/bin/activate

# Create .env file touch .env # Add your API keys FIRECRAWL_API_KEY=your_firecrawl_api_key OPENAI_API_KEY=your_openai_api_key

Instead of running the server directly, you can configure Claude for Desktop to access the WebSearch tools:
-

Locate or create your Claude for Desktop configuration file:

- Windows:%env:AppData%\Claude\claude_desktop_config.json
- macOS:~/Library/Application Support/Claude/claude_desktop_config.json

Add the WebSearch server configuration to themcpServerssection:

{ "mcpServers": { "websearch": { "command": "uv", "args": [ "--directory", "D:\\ABSOLUTE\\PATH\\TO\\WebSearch", "run", "main.py" ] } } }

-

Make sure to replace the directory path with the absolute path to your WebSearch project folder.

Save the configuration file and restart Claude for Desktop.

Once configured, the WebSearch tools will appear in the tools menu (hammer icon) in Claude for Desktop.

- query(str): The search query
- Returns: Search results in JSON format

- urls(List[str]): List of URLs to extract information from
- prompt(str): Instructions for extraction
- enableWebSearch(bool): Enable supplementary web search
- showSources(bool): Include source references
- Returns: Extracted information in specified format

- url(str): Starting URL
- maxDepth(int): Maximum crawl depth
- limit(int): Maximum pages to crawl
- Returns: Crawled content in markdown/HTML format

- url(str): Target URL
- Returns: Scraped content with optional screenshots

The tool requires certain API keys to function. We provide a.env.examplefile that you can use as a template:

# On Unix/MacOS cp .env.example .env # On Windows copy .env.example .env

# OpenAI API key - Required for AI-powered features OPENAI_API_KEY=your_openai_api_key_here # Firecrawl API key - Required for web scraping and searching FIRECRAWL_API_KEY=your_firecrawl_api_key_here

- VisitOpenAI's platform
- Sign up or log in
- Navigate to API keys section
- Create a new secret key

- VisitFirecrawl's website
- Create an account
- Navigate to your dashboard
- Generate a new API key

This project is licensed under the MIT License - see theLICENSEfile for details.

- Firecrawlfor their powerful web scraping API
- OpenAIfor AI capabilities
- MCPThe MCP community for the protocol specification

José Martín Rodriguez Mortaloni -@m4s1t425-jmrodriguezm13@gmail.com

Interact with the Bilibili video website, enabling actions like searching for videos, retrieving video information, and accessing user data.