Web Crawler

by jitsmaster

1 stars

Not rated

GitHub

About

Configurable web crawler that extracts structured content from websites while respecting robots.txt rules and offering customizable settings for depth, delay, and concurrency.

Details

Author: jitsmaster
Repository: jitsmaster/WebScrapeMCPServer
GitHub stars: 1
Categories: Productivity, Design, Developer Tools, AI, Project Management, Infrastructure, API
Tags: #mobile

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below
Plugin Name Web Crawler

Command (node, npx, python, etc.) node
Arguments
- Argument 1 /path/to/web-crawler/build/index.js
Environment
- TIMEOUT 5000
- MAX_DEPTH 3
- CRAWL_LINKS false
- REQUEST_DELAY 1000
- MAX_CONCURRENT 5
Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

JSON

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

From the repository

The server provides a crawl tool that can be accessed through MCP. Example usage:

{
  "url": "https://example.com",
  "depth": 1
}

Start the MCP server:

npm start

Add the following to your MCP settings file:

{
  "mcpServers": {
    "web-crawler": {
      "command": "node",
      "args": ["/path/to/web-crawler/build/index.js"],
      "env": {
        "CRAWL_LINKS": "false",
        "MAX_DEPTH": "3",
        "REQUEST_DELAY": "1000",
        "TIMEOUT": "5000",
        "MAX_CONCURRENT": "5"
      }
    }
  }
}

| Environment Variable | Default | Description |
|----------------------|---------|-------------|
| CRAWL_LINKS | false | Whether to follow links |
| MAX_DEPTH | 3 | Maximum crawl depth |
| REQUEST_DELAY | 1000 | Delay between requests (ms) |
| TIMEOUT | 5000 | Request timeout (ms) |
| MAX_CONCURRENT | 5 | Maximum concurrent requests |

`crawl`

Crawl a specified URL. Parameters: url (string), depth (integer)

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "web crawler": {
            "env": {
                "TIMEOUT": "5000",
                "MAX_DEPTH": "3",
                "CRAWL_LINKS": "false",
                "REQUEST_DELAY": "1000",
                "MAX_CONCURRENT": "5"
            },
            "args": [
                "/path/to/web-crawler/build/index.js"
            ],
            "command": "node"
        }
    }
}

Linux

{
    "env": {
        "TIMEOUT": "5000",
        "MAX_DEPTH": "3",
        "CRAWL_LINKS": "false",
        "REQUEST_DELAY": "1000",
        "MAX_CONCURRENT": "5"
    },
    "args": [
        "/path/to/web-crawler/build/index.js"
    ],
    "command": "node"
}

Macos

{
    "env": {
        "TIMEOUT": "5000",
        "MAX_DEPTH": "3",
        "CRAWL_LINKS": "false",
        "REQUEST_DELAY": "1000",
        "MAX_CONCURRENT": "5"
    },
    "args": [
        "/path/to/web-crawler/build/index.js"
    ],
    "command": "node"
}

Windows

{
    "env": {
        "TIMEOUT": "5000",
        "MAX_DEPTH": "3",
        "CRAWL_LINKS": "false",
        "REQUEST_DELAY": "1000",
        "MAX_CONCURRENT": "5"
    },
    "args": [
        "/path/to/web-crawler/build/index.js"
    ],
    "command": "node"
}

Web Crawler MCP Server Deployment Guide

Prerequisites

- Node.js (v18+) - npm (v9+)

Installation

1. Clone the repository:

   git clone https://github.com/jitsmaster/web-crawler-mcp.git
   cd web-crawler-mcp

2. Install dependencies:

   npm install

3. Build the project:

   npm run build

Configuration

Create a .env file with the following environment variables:

CRAWL_LINKS=false
MAX_DEPTH=3
REQUEST_DELAY=1000
TIMEOUT=5000
MAX_CONCURRENT=5

Running the Server

Start the MCP server:

npm start

MCP Configuration

Add the following to your MCP settings file:

{
  "mcpServers": {
    "web-crawler": {
      "command": "node",
      "args": ["/path/to/web-crawler/build/index.js"],
      "env": {
        "CRAWL_LINKS": "false",
        "MAX_DEPTH": "3",
        "REQUEST_DELAY": "1000",
        "TIMEOUT": "5000",
        "MAX_CONCURRENT": "5"
      }
    }
  }
}

Usage

The server provides a crawl tool that can be accessed through MCP. Example usage:

{
  "url": "https://example.com",
  "depth": 1
}

Configuration Options

| Environment Variable | Default | Description | |----------------------|---------|-------------| | CRAWL_LINKS | false | Whether to follow links | | MAX_DEPTH | 3 | Maximum crawl depth | | REQUEST_DELAY | 1000 | Delay between requests (ms) | | TIMEOUT | 5000 | Request timeout (ms) | | MAX_CONCURRENT | 5 | Maximum concurrent requests |

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

About

Details

Jump to

Setting up with Highlight

From the repository

crawl

Claude Desktop / Cursor

Linux

Macos

Windows

Web Crawler MCP Server Deployment Guide

Prerequisites

Installation

Configuration

Running the Server

MCP Configuration

Usage

Configuration Options

Install

More like this

Time

Sentry

Github

Git

GitLab

Context 7

`crawl`