Fetch as Markdown MCP Server

Fetches web pages and converts them to clean markdown, focusing on main content extraction.

A Model Context Protocol (MCP) server that fetches web pages and converts them to clean, readable markdown format, focusing on main content extraction while minimizing context overhead.

This MCP server acts as a bridge between AI assistants and the web, specifically designed to:

- Extract Clean Content: Focuses on main article content, removing navigation, ads, and sidebars
- Minimize Context: Strips unnecessary elements to reduce token usage while preserving content structure
- Respectful Scraping: Implements proper rate limiting, user-agent headers, and timeout handling
- Error Resilience: Gracefully handles various web-related errors and edge cases

- Fetch any publicly accessible web page
- Automatic redirect handling with final URL reporting
- Configurable timeouts and proper error handling
- Respectful rate limiting (1-second intervals between requests)

- Removes navigation, ads, sidebars, and other non-essential elements
- Focuses on main content areas using semantic HTML detection
- Strips unnecessary HTML attributes to reduce token usage
- Preserves content structure and readability

- Converts HTML to clean, readable markdown
- Configurable link and image inclusion
- Proper heading hierarchy and formatting
- Post-processing to remove excessive whitespace

- Python 3.12 or higher
- uvpackage manager

uvx git+https://github.com/bhubbb/mcp-fetch-as-markdown

This MCP server is designed to work with AI assistants that support the Model Context Protocol. Configure your AI assistant to connect to this server via stdio.

Example configuration for Claude Desktop:

{ "mcpServers": { "fetch-as-markdown": { "command": "uvx", "args": ["git+https://github.com/bhubbb/mcp-fetch-as-markdown"] } } }

{ "mcpServers": { "fetch-as-markdown": { "command": "uv", "args": ["run", "python", "/path/to/mcp-fetch-as-markdown/main.py"] } } }

Fetch a web page and convert it to clean markdown format.

- url(required): URL of the web page to fetch and convert
- include_links(optional): Whether to preserve links in markdown output (default: true)
- include_images(optional): Whether to include image references (default: false)
- timeout(optional): Request timeout in seconds (5-30, default: 10)

- Fetch metadata (original URL, final URL, title, content length, status code, content type)
- Clean markdown content with proper formatting

{ "name": "fetch", "arguments": { "url": "https://example.com/article", "include_links": true, "include_images": false, "timeout": 15 } }

- Fetch Page: Makes HTTP request with proper headers and timeout handling
- Parse HTML: Uses BeautifulSoup to parse the HTML content
- Remove Unwanted Elements: Strips scripts, styles, navigation, ads, sidebars, footers
- Find Main Content: Looks for semantic elements like<main>,<article>, or common content classes
- Clean Attributes: Removes unnecessary HTML attributes to reduce size
- Convert to Markdown: Uses configurable markdown conversion with proper formatting
- Post-process: Removes excessive whitespace and blank lines

- Rate Limiting: Minimum 1-second interval between requests
- User Agent: Proper identification as "MCP-Fetch-As-Markdown" tool
- Timeout Handling: Configurable timeouts to avoid hanging requests
- Error Handling: Graceful handling of network issues, HTTP errors, and malformed content
- Redirect Support: Follows redirects and reports final URLs

- Metadata Block: Original URL, final URL, page title, content statistics, HTTP status
- Content Block: Clean markdown conversion of the main page content

This structure makes responses both human-readable and machine-parseable while minimizing token usage.

- Invalid URLs: Clear validation and error messages
- Network Issues: Timeout, connection error, and DNS failure handling
- HTTP Errors: Proper handling of 404, 500, and other HTTP status codes
- Malformed Content: Graceful handling of broken HTML and encoding issues

- Convert articles and blog posts to clean markdown for analysis
- Extract main content from news articles and research papers
- Gather information while minimizing irrelevant context

- Prepare web content for further AI processing
- Extract clean text from web pages for summarization
- Convert HTML content to markdown for documentation

- Fetch and process web content with minimal token overhead
- Extract relevant information while filtering out noise
- Provide clean, structured content for AI reasoning

Ask your AI assistant:"Fetch the content from this article URL as markdown"
- Fetch the web page with proper headers and rate limiting
- Extract the main content area, removing navigation and ads
- Convert to clean markdown format
- Return structured metadata and content

Ask your AI assistant:"Fetch this page but keep all the links intact"
- Fetch and process the page normally
- Preserve all hyperlinks in markdown formattext
- Maintain link structure while cleaning other elements

Ask your AI assistant:"Try to fetch content from this broken URL"
- Validate the URL format
- Attempt the request with proper timeout
- Return a structured error message if the request fails
- Provide helpful information about what went wrong

mcp-fetch-as-markdown/ ├── main.py # Main MCP server implementation ├── pyproject.toml # Project dependencies and metadata ├── AGENT.md # Development rules and guidelines ├── example.py # Usage examples and demonstrations └── .venv/ # Virtual environment (created by uv)

- mcp: Model Context Protocol framework
- requests: HTTP request handling
- beautifulsoup4: HTML parsing and content extraction
- markdownify: HTML to markdown conversion

The server can be easily customized by modifyingmain.py:

- Content Selectors: Modify the CSS selectors used to find main content
- Rate Limiting: Adjust the minimum interval between requests
- Timeout Settings: Change default and maximum timeout values
- Content Filtering: Add custom content processing or filtering rules
- Markdown Options: Customize markdown conversion settings

uvx git+https://github.com/bhubbb/mcp-fetch-as-markdown

cd mcp-fetch-as-markdown uv run python main.py

For interactive testing, use the example script:
- Import Errors: Make sure all dependencies are installed withuv sync
- Connection Timeouts: Some websites may be slow; try increasing the timeout parameter
- Rate Limiting: The server enforces 1-second intervals between requests
- Blocked Requests: Some websites may block automated requests; this is expected behavior

Enable debug logging by modifying the logging level inmain.py:

logging.basicConfig(level=logging.DEBUG)

- Modern Websites: Works best with standard HTML structure
- JavaScript-heavy Sites: Cannot execute JavaScript; fetches initial HTML only
- Protected Content: Respects robots.txt and website access restrictions
- Rate Limits: Implements respectful scraping practices

This tool is designed for legitimate research, analysis, and content processing. Please:

- Respect Terms of Service: Always check and comply with website terms of service
- Avoid Overloading: The built-in rate limiting helps, but be mindful of request frequency
- Attribution: Give proper credit to original sources when using extracted content
- Legal Compliance: Ensure your use case complies with applicable laws and regulations

This is a simple, single-file implementation designed for clarity and ease of modification. Feel free to:

- Add support for additional content extraction strategies
- Implement custom filtering for specific website types
- Add caching for better performance
- Extend with additional markdown formatting options

This project uses the same license as its dependencies. Content fetched from websites remains subject to the original website's terms of service and copyright.

Fetches web content from a URL and converts it from HTML to markdown for easier consumption by LLMs.

Fetch web pages and convert HTML to clean, formatted Markdown. Handles large pages with automatic file saving to bypass token limits.

Local-first MCP server that captures web URLs (X, Reddit, YouTube, Wikipedia, articles) as typed data + Markdown into a self-hosted capture/store/recall substrate, with offline semantic recall. Six tools over a local khiipd daemon; run khiipd serve first.

Download webpages as markdown files using the r.jina.ai service, with configurable directories and persistent settings.

Fetches content from any URL and converts it to HTML, JSON, Markdown, or plain text.

Extracts and transforms webpage content into clean, LLM-optimized Markdown using Mozilla's Readability algorithm.

Convert any URL to clean, LLM-ready Markdown

Extracts and transforms webpage content into clean, LLM-optimized Markdown using the Readability algorithm.

Extracts web content from a URL and converts it to clean Markdown format.

An MCP server for the Urlbox Screenshot API. It enables your client to take screenshots, generate PDFs, extract HTML/markdown, and more from websites.