Web Search MCP

by pranavms13

Not rated
GitHub

About

Scrapes Google search results using a headless browser. Requires Chrome to be installed.

Details

Author
pranavms13
Categories
Search, Web Scraping, Automation

Setup

Install Web Search MCP in your MCP client (Claude Desktop, Cursor, Windsurf, and others).

Repository: https://github.com/pranavms13/web-search-mcp

Follow the installation instructions in the repository README, then restart your MCP client.

A Model Context Protocol (MCP) server that provides web search functionality by scraping Google, DuckDuckGo, and Bing search results through a headless browser.

- Multi-Engine Search: Searches Google, DuckDuckGo, and Bing with automatic fallback when an engine is blocked or returns nothing
- Structured Results: Titles, URLs, domains, snippets, and rankings — with redirect URLs unwrapped to their real destinations
- Web Page Content: Fetch and extract text content from any webpage
- Pluggable Browser Backends: Selenium + headless Chrome (default) or the lightweightObscuraheadless browser, selected via an environment variable
- MCP Compatible: Fully compatible with Claude Desktop and other MCP clients

Search the web across multiple engines (Google → DuckDuckGo → Bing) with automatic fallback and return structured results.

- query(string): The search query string
- max_results(int, optional): Maximum number of results to return (default: 10, max: 100)
- include_snippets(bool, optional): Whether to include text snippets (default: true)

- List of search results with:

- title: Page title
- url: Full URL (redirect wrappers resolved to the real destination)
- domain: Domain name
- snippet: Text snippet (if enabled)
- rank: Search result ranking
- source_engine: Which engine produced the result (google,duckduckgo, orbing)

Fetch and return the text content of a webpage.

- url(string): The URL of the webpage to fetch
- max_length(int, optional): Maximum content length (default: 5000, max: 20000)

- Dictionary with:

- url: The requested URL
- title: Page title
- content: Extracted text content
- length: Content length in characters

Report the active browser backend and the availability of each search engine.

- Dictionary with abackendkey (seleniumorobscura) plus each engine name mapped toavailableorblocked.

Clear the list of engines marked as blocked during the session (useful if an engine recovers).

# Using uv (recommended) uv sync # Or using pip pip install -e .

- On macOS:brew install --cask google-chrome
- On Ubuntu:sudo apt-get install google-chrome-stable
- On Windows: Download from the Google Chrome website

ChromeDriver is downloaded and managed automatically by webdriver-manager.

Obscura— no Chrome required. See theBrowser Backendssection below.

# Run directly python main.py # Or using the installed script web-search-mcp

The server will start and listen for MCP connections.

Add this configuration to your Claude Desktop MCP settings:

{ "mcpServers": { "web-search-mcp": { "command": "uvx", "args": ["git+https://github.com/pranavms13/web-search-mcp"] } } }

Once connected, you can use the tools like this:

Search for "python web scraping tutorials" and show me the top 5 results. Get the content from this webpage: https://example.com/article

The web searcher uses these Chrome options by default:

- Headless mode (no visible browser window)
- Window size: 1920x1080
- User agent: Modern Chrome browser
- Security flags for running in containers

The server can render pages with one of two interchangeable backends, selected via theBROWSER_BACKENDenvironment variable:

Both backends produce fully JS-rendered HTML, which is then parsed with BeautifulSoup, so search results are identical in shape regardless of backend.
-

Install the Obscura binary (see itsreleases), e.g. on macOS:

curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-macos.tar.gz tar xzf obscura-x86_64-macos.tar.gz

Ifobscurais not on yourPATH, point to it explicitly withOBSCURA_BIN=/full/path/to/obscura. When the binary cannot be found, the server logs a warning and falls back to the Selenium backend automatically.

The active backend is reported by theget_search_engine_statustool under thebackendkey.

The tool includes comprehensive error handling for:

- Network timeouts
- WebDriver failures
- Page parsing errors
- Invalid URLs

Errors are logged and graceful fallbacks are provided.

- Python 3.10+
- A browser backend: Chrome (Selenium)orthe Obscura binary
- Internet connection

- fastmcp: MCP server framework
- selenium: Web browser automation (Selenium backend)
- beautifulsoup4: HTML parsing
- webdriver-manager: Chrome driver management (Selenium backend)
- lxml: XML/HTML parser

The Obscura backend has no Python dependency — it is invoked as an external CLI binary.

- Search engines may rate-limit or block automated access; the multi-engine fallback mitigates but does not eliminate this
- Google in particular often blocks non-stealth headless traffic — DuckDuckGo and Bing typically remain available as fallbacks
- Results may vary based on location and each engine's ranking
- The Selenium backend requires Chrome; the Obscura backend requires the Obscura binary
- Clone the repository
- Install in development mode:uv syncorpip install -e .
- Make your changes
- Run the test suite:uv run pytest test.py -q
- Smoke-test the server:python main.py(orBROWSER_BACKEND=obscura OBSCURA_BIN=/path/to/obscura python main.py)

This project is licensed under MIT License. You can check it out at -LICENSE

Contributions are welcome! Please feel free to submit a Pull Request.

Search global news using natural language. Webz.io News Search API returns the most relevant articles and content, with filters for source, country, language, date, sentiment, and category.

A Google search server using Playwright for parallel keyword searches.

Self-hosted Stealth Scraping & Federated Search for AI Agents. A 100% private, free alternative to Firecrawl, Jina Reader, and Tavily. Featuring Universal Anti-bot Bypass + Semantic Research Memory, Copy-Paste setup

Adds powerful web scraping and search capabilities to LLM clients like Cursor and Claude.

Scrape, crawl, and extract data from any website using the Firecrawl API.

Give Claude, Cursor, ChatGPT, Kilo, and other MCP clients access to scraping, web search, screenshots, and network tools.

A collection of servers for file system operations, Google search, web automation, and executing terminal commands.

Hosted, Stateless & Multitenant Firecrawl MCP server enables AI assistants to crawl, scrape, and extract structured web data through Firecrawl.

Search, extract, crawl, map, and research the web — from any AI agent or terminal.

A server for web scraping, Google searches, and website URL lookups using the Olostep API.

No reviews yet — be the first

Sign in to leave a review

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.