Better Fetch

by flutterninja9

593 downloads Not rated yet

GitHub

About

A powerful Model Context Protocol (MCP) server that intelligently fetches and processes web content with nested URL crawling capabilities. Transform any documentation site or web resource into clean, structured markdown files perfect for AI consumption and analysis.

Explore

- Smart web crawling with nested URL fetching and configurable depth
- Single page mode for simple content extraction
- Domain filtering and regex pattern matching
- Intelligent content cleaning that removes ads, navigation, and scripts
- Automatic generation of table of contents with proper nesting
- Clean markdown output with code block and link preservation
- Highly configurable crawl depth, page limits, and timeouts

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below

Plugin Name Better Fetch

Command (node, npx, python, etc.)

Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

From the repository

- Node.js 18+
- npm or yarn
- MCP-compatible client (Claude Desktop, VS Code with MCP extension, etc.)

- Crawl Depth Control: Set maximum levels to crawl (default: 2)
- Page Limits: Control maximum pages to process (default: 50)
- Timeout Settings: Configurable request timeouts
- Respectful Crawling: Built-in delays between requests
- Error Handling: Graceful handling of failed requests and invalid URLs

``bash git clone https://github.com/yourusername/better-fetch.git cd better-fetch npm install



npm run dev



Fetch content from the React documentation but exclude any URLs containing 'api' or 'reference' and only process pages containing 'tutorial' or 'guide':



URL: https://react.dev

Max Depth: 2

Exclude Patterns: ["/api/", "/reference/"]

Include Patterns: ["/tutorial/", "/guide/"]

Max Pages: 30

1. Fork the repository 2. Create a feature branch:git checkout -b feature/amazing-feature3. Make your changes and add tests 4. Commit your changes:git commit -m 'Add amazing feature'5. Push to the branch:git push origin feature/amazing-feature`
6. Open a Pull Request

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "better fetch": {
            "better-fetch": {
                "command": "node",
                "args": [
                    "test-mcp.js"
                ]
            }
        }
    }
}

McpServers

{
    "better-fetch": {
        "command": "node",
        "args": [
            "test-mcp.js"
        ]
    }
}

🚀 Key Features

🕸️ Smart Web Crawling

- Nested URL Fetching: Automatically discovers and crawls linked pages up to configurable depth - Single Page Mode: Option for simple single-page content extraction - Domain Filtering: Stay within the same domain or allow cross-domain crawling - Pattern Matching: Include/exclude URLs based on regex patterns

🧠 Intelligent Content Processing

- Content Cleaning: Removes ads, navigation, scripts, and other noise automatically - Smart Section Detection: Identifies main content areas (<main>, <article>, .content) - Automatic Titles: Generates meaningful section headers based on page titles and URL structure - Table of Contents: Creates organized TOC with proper nesting

📝 Advanced Markdown Generation

- Clean Formatting: Converts HTML to well-structured markdown - Code Block Preservation: Maintains formatting for code snippets and technical content - Link Preservation: Keeps all important links with proper markdown syntax - Metadata Integration: Includes source URLs, generation timestamps, and site information

⚙️ Highly Configurable

- Crawl Depth Control: Set maximum levels to crawl (default: 2) - Page Limits: Control maximum pages to process (default: 50) - Timeout Settings: Configurable request timeouts - Respectful Crawling: Built-in delays between requests - Error Handling: Graceful handling of failed requests and invalid URLs

📋 Available Tools

1. `fetch_website_nested`

Comprehensive web crawling with nested URL processing. Parameters: - url (required): Starting URL to crawl - maxDepth (optional, default: 2): Maximum crawl depth - maxPages (optional, default: 50): Maximum pages to process - sameDomainOnly (optional, default: true): Restrict to same domain - excludePatterns (optional): Array of regex patterns to exclude - includePatterns (optional): Array of regex patterns to include - timeout (optional, default: 10000): Request timeout in milliseconds

2. `fetch_website_single`

Simple single-page content extraction. Parameters: - url (required): URL to fetch - timeout (optional, default: 10000): Request timeout in milliseconds

💡 Use Cases

📚 Documentation Processing

- API Documentation: Convert REST API docs, SDK guides, and technical references - Framework Docs: Process React, Vue, Angular, or any framework documentation - Library Guides: Extract comprehensive guides from library documentation sites - Tutorial Series: Gather multi-part tutorials into single organized documents

🔍 Content Analysis & Research

- Competitive Analysis: Gather competitor documentation and feature descriptions - Market Research: Extract product information from multiple related pages - Academic Research: Collect and organize web-based research materials - Knowledge Base Creation: Transform scattered web content into structured knowledge bases

🤖 AI Training & Context

- LLM Context Preparation: Create clean, structured content for AI model training - RAG System Input: Generate high-quality documents for Retrieval-Augmented Generation - Chatbot Knowledge: Build comprehensive knowledge bases for customer service bots - Content Summarization: Prepare web content for automated summarization tasks

🛠️ Installation & Setup

Prerequisites

- Node.js 18+ - npm or yarn - MCP-compatible client (Claude Desktop, VS Code with MCP extension, etc.)

Step 1: Clone and Install

bash
git clone https://github.com/yourusername/better-fetch.git
cd better-fetch
npm install



Step 2: Build the Project

bash
npm run build



Step 3: Test the Server (Optional)

``bash

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

About

Explore

Setting up with Highlight

From the repository

Claude Desktop / Cursor

McpServers

🚀 Key Features

🕸️ Smart Web Crawling

🧠 Intelligent Content Processing

📝 Advanced Markdown Generation

⚙️ Highly Configurable

📋 Available Tools

1. fetch_website_nested

2. fetch_website_single

💡 Use Cases

📚 Documentation Processing

🔍 Content Analysis & Research

🤖 AI Training & Context

🛠️ Installation & Setup

Prerequisites

Step 1: Clone and Install

Step 2: Build the Project

Step 3: Test the Server (Optional)

More like this

@mendableai/firecrawl Mcp Server

Playwright

Aws Kb Retrieval Server

Memory

1. `fetch_website_nested`

2. `fetch_website_single`