MCP Web Scraper

Aproduction-ready global content extraction platformwith ML-powered automation, international site support, and intelligent optimization. Featurescomplete browser automationwith 29 tools,21+ supported sitesacross 4 regions,6 content platformswith specialized optimization, andpersistent SQLite cachingwith cross-session learning. Built with TypeScript using the officialMCP TypeScript SDKand Playwright.

# Pull and run the latest version docker run -p 3001:3001 descoped/mcp-web-scraper # Or with docker-compose curl -O https://raw.githubusercontent.com/descoped/mcp-web-scraper/main/docker-compose.yml docker-compose up

# Clone and build git clone https://github.com/descoped/mcp-web-scraper.git cd mcp-web-scraper npm install npm run build npm start

{ "url": "https://example.com/article", "outputFormats": [ "text", "html", "markdown" ], "correlation_id": "task_085668b2-8f3d-418e", "extractSelectors": { "title": "h1", "content": "article", "author": ".author" } }

Returns: Title, content, author, date, summary + full text + consent verification

{ "url": "https://example.com", "fullPage": true }

Returns: PNG screenshot (base64) + consent status + metadata

{ "url": "https://example.com", "timeout": 5000 }

Returns: Detailed consent verification + method used + performance metrics

Complete Playwright navigation and interaction capabilities with session-based operations:

- browser_navigate- Navigate to URLs with wait conditions
- browser_click- Click elements with multiple targeting strategies
- browser_type- Type text with clear/delay options
- browser_hover- Mouse hover interactions
- browser_select_option- Dropdown and select element handling
- browser_press_key- Keyboard input simulation
- browser_handle_dialog- Alert, confirm, prompt dialog management
- browser_file_upload- File upload functionality
- browser_close- Page lifecycle management

Specialized browser capabilities for testing and analysis:

- browser_pdf_save- PDF generation with formatting options
- browser_console_messages- Console log monitoring and filtering
- browser_resize- Viewport control for responsive testing
- browser_snapshot- Accessibility tree snapshots
- browser_install- Browser installation management
- browser_generate_playwright_test- Automated test script generation

Create, switch between, close, and list browser tabs.

{ "action": "new", "url": "https://example.com" }

Actions:list,new,switch,close
Returns: Tab management results with metadata

Track HTTP requests and responses with detailed analysis.

{ "sessionId": "session_123", "action": "start", "filterUrl": "api.example.com" }

Actions:start,stop,get
Returns: Network analysis with timing, status codes, domains

Perform drag and drop operations between elements.

{ "sessionId": "session_123", "sourceSelector": ".drag-item", "targetSelector": ".drop-zone" }

Returns: Drag operation results with position validation

Navigate browser history (back/forward) with step control.

{ "sessionId": "session_123", "direction": "back", "steps": 2 }

Returns: Navigation results with URL changes and history info

Advanced automation with intelligent element discovery and analysis:

- browser_find_text- Advanced text search and location
- browser_find_element- Element discovery by description
- browser_describe_element- Element analysis and description
- browser_annotate_page- Visual page annotation system
- browser_get_element_text- Enhanced text extraction with analysis
- browser_wait_for_page_state- Advanced page state monitoring
- browser_execute_javascript- Custom JavaScript execution with safety

Track requests across your system using correlation IDs for batch processing, user sessions, or distributed tracing:

// Track multiple articles in a batch analysis const batchId = 'analysis_batch_2025_001'; const articles = [ {url: 'https://example.com/article1', id: 'item_001'}, {url: 'https://example.com/article2', id: 'item_002'} ]; // Process with correlation tracking for (const article of articles) { const result = await client.callTool('scrape_article_content', { url: article.url, correlation_id: ${batchId}_${article.id}, outputFormats: ['text', 'markdown'] }); } // Monitor progress via SSE with correlation const eventSource = new EventSource('http://localhost:3001/mcp'); eventSource.onmessage = (event) => { const data = JSON.parse(event.data); if (data.params?.correlationId?.startsWith(batchId)) { console.log(Progress for ${data.params.correlationId}: ${data.params.progress}%); } };

Add to~/Library/Application Support/Claude/claude_desktop_config.json(macOS):

{ "mcpServers": { "mcp-web-scraper": { "command": "node", "args": [ "/path/to/mcp-web-scraper/dist/server.js" ], "env": { "BROWSER_POOL_SIZE": "3", "DEBUG_LOGGING": "true" } } } }

version: '3.8' services: mcp-web-scraper: image: descoped/mcp-web-scraper:latest ports: - "3001:3001" environment: - BROWSER_POOL_SIZE=5 - REQUEST_TIMEOUT=30000 - CONSENT_TIMEOUT=3000 volumes: - ./logs:/app/logs restart: unless-stopped healthcheck: test: [ "CMD", "curl", "-f", "http://localhost:3001/health" ] interval: 30s timeout: 10s retries: 3

# 1. Build and start server npm run build npm start # 2. Test health endpoint curl http://localhost:3001/health # 3. Test MCP protocol curl -X POST http://localhost:3001/mcp-request \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' # 4. Run comprehensive test suite npm test # 5. Run system validation (Phase 3.6) npx tsx tests/run-system-validation.ts --help

Built with modern technologies for production reliability:

- TypeScript: Full type safety with path mapping (@/imports) and excellent developer experience
- Playwright: Industry-standard browser automation with 100% MCP parity
- MCP SDK: Official Model Context Protocol implementation
- Express.js: Robust HTTP server with middleware support
- Docker: Production-ready containerization
- Zod: Runtime schema validation for all inputs
- SQLite: Persistent caching and cross-session intelligence

Recent Technical Improvements (January 2025)

- 🔧 TypeScript Path Mapping: Eliminated deeply nested relative imports with clean@/paths
- 🗑️ Legacy Code Cleanup: Removed deprecated Phase 3.5/3.5.1 systems and artifacts
- 🔒 Proper Encapsulation: Fixed private property access violations with public API methods
- 🧪 Enhanced Testing: Improved test suite performance and maintainability
- 🎯 System Validation: Phase 3.6 unified validation pipeline using production MCP tools
- 📁 Clean Architecture: Clear separation between production (src/) and testing (tests/) code

- Multi-Tier Detection: International → Norwegian → Universal → Emergency fallback system
- Regional Optimization: Adaptive strategies for Scandinavian, European, American, and International content
- Quality Assurance: 15+ metrics with frontpage detection and content validation
- Rate Limiting: Token bucket algorithm with per-connection throttling
- Health Monitoring: Comprehensive status, metrics, and analytics endpoints
- Graceful Shutdown: SIGTERM handling with resource cleanup and browser management
- Error Recovery: Automatic browser restart, connection resilience, and emergency fallback

- ML-Powered Optimization: Automatic rule generation with 83% success rate and 88% confidence
- A/B Testing Framework: Statistical validation with two-sample t-tests and significance analysis
- Cross-Session Learning: 89% recommendation accuracy with SQLite-based persistent intelligence
- Performance Baselines: Continuous calibration and optimization recommendations
- Cache Intelligence: 73% hit rate with HTML signature detection and automatic optimization

- Real-Time Dashboard: Live analytics at/dashboardwith 30+ metrics and performance tracking
- Analytics API: 6 comprehensive endpoints for rule performance, cache statistics, and quality trends
- Correlation Tracking: Client-provided correlation IDs flow through all events and analytics

- SSE progress events include correlation_id for request tracking
- Supports batch processing and distributed tracing scenarios
- Perfect for correlating frontend UI updates with backend operations

- Model Context Protocol- Official MCP specification and documentation
- MCP TypeScript SDK- Official TypeScript SDK used in this project
- MCP Client Libraries- Official client implementations for various languages

- Playwright- The browser automation framework powering our implementation
- Playwright Documentation- Comprehensive automation guides and API reference

For questions, issues, or contributions, visit ourGitHub repository.

This project is licensed under the MIT License - see theLICENSEfile for details.

A server providing Playwright tools for browser automation and web scraping.