Web Cat API

by Kode-Rex

309 downloads Not rated yet MIT license

GitHub

About

Web Cat API is a collection of Python-based APIs that includes a Model Context Protocol (MCP) server for web search and content extraction. It integrates with Serper.dev for search capabilities and uses the readability library to extract clean text from web pages, making it easy…

Details

License: MIT license

Explore

- Content Extraction: Utilizes the readability library for clean text extraction
- Text Processing: Further processes extracted content for improved usability
- Search Functionality: Integrates with Serper.dev to provide web search capabilities
- MCP Compliance: Follows standardized Model Context Protocol specifications
- Multiple API Styles: Supports both Server-Sent Events (SSE) streaming and RESTful endpoints
- Rate Limiting: Protects the API from abuse with configurable rate limits
- API Versioning: Ensures backward compatibility as the API evolves
- Docker Support: Easy deployment with Docker containers
- Parallel Processing: Faster response times with parallel search result processing

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below

Plugin Name Web Cat API

Command (node, npx, python, etc.)

Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

From the repository


- SERPER_API_KEY: Your Serper API key (required)
- PORT: The port to run the server on (default: 8000)
- RATE_LIMIT_WINDOW: Time window in seconds for rate limiting (default: 60)
- RATE_LIMIT_MAX_REQUESTS: Max requests per window (default: 10)
Here's a quick example of how to test the Azure Functions API locally:

bash
cd customgpt
func start
curl -X POST http://localhost:7071/api/scrape -H "Content-Type: application/json" -d "{\"url\":\"https://example.com\"}" # text only
curl -X POST http://localhost:7071/api/scrape_with_images -H "Content-Type: application/json" -d "{\"url\":\"https://bigmedium.com/speaking/sentient-design-josh-clark-talk.html\"}" #text and images
curl -X POST http://localhost:7071/api/search -H "Content-Type: application/json" -d "{\"query\":\"your search query\"}" # search and get content
```

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "web cat api": {
            "webcat": {
                "command": "docker",
                "args": [
                    "run",
                    "-p",
                    "8000:8000",
                    "-e",
                    "SERPER_API_KEY=your_key",
                    "tmfrisinger/webcat:latest"
                ]
            }
        }
    }
}

McpServers

{
    "webcat": {
        "command": "docker",
        "args": [
            "run",
            "-p",
            "8000:8000",
            "-e",
            "SERPER_API_KEY=your_key",
            "tmfrisinger/webcat:latest"
        ]
    }
}

Introduction

Web Cat is a collection of Python-based APIs designed to enhance AI models with web search and content extraction capabilities. The project includes:

1. A serverless Python-based API hosted on Azure Functions
2. A Model Context Protocol (MCP) server that provides web search capabilities for AI models

Both implementations are designed to responsibly scrape and process website content, making it easy to integrate web content into AI applications like ChatGPT through Custom GPTs.

Components

Azure Functions API

The Azure Functions API leverages the readability library and BeautifulSoup to extract the main body of text and related images from web pages.

MCP Server

The Model Context Protocol (MCP) server is a FastAPI-based implementation that provides web search capabilities with enhanced content extraction. It follows the MCP specification for standardized AI model interactions.

Features

- Content Extraction: Utilizes the readability library for clean text extraction - Text Processing: Further processes extracted content for improved usability - Search Functionality: Integrates with Serper.dev to provide web search capabilities - MCP Compliance: Follows standardized Model Context Protocol specifications - Multiple API Styles: Supports both Server-Sent Events (SSE) streaming and RESTful endpoints - Rate Limiting: Protects the API from abuse with configurable rate limits - API Versioning: Ensures backward compatibility as the API evolves - Docker Support: Easy deployment with Docker containers - Parallel Processing: Faster response times with parallel search result processing

Getting Started

Azure Functions API

- See the customgpt directory for specific documentation

MCP Server (Docker)

- Current version: 1.1.0 - Docker image: tmfrisinger/webcat:1.1.0 or tmfrisinger/webcat:latest

Running with Docker

```bash

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

About

Details

Explore

Setting up with Highlight

From the repository

Claude Desktop / Cursor

McpServers

Introduction

Components

Azure Functions API

MCP Server

Features

Getting Started

Azure Functions API

MCP Server (Docker)

Running with Docker

More like this

Sentry

Github

Brave Search

Git