MCP Server Tester
About
Automated testing tool for Model Context Protocol (MCP) servers - WORK IN PROGRESS
Explore
- 🔍 Automatically discovers available tools from any MCP server
- 🧪 Generates realistic test cases for each tool using Claude AI
- ⚡ Executes tests and validates responses
- 📊 Provides detailed test reports
- 🔑 Supports multiple connection methods through configuration
- Configuration-Based: Simple JSON configuration for defining MCP servers to test
- Multiple Server Support: Test multiple MCP servers at once
- Comprehensive Testing: Tests all tools exposed by each server
- Natural Language Context: Includes the user query that would trigger each tool, providing real-world context
- Detailed Reports: Generate reports in console, JSON, HTML, or Markdown formats
- Secure: Keeps API keys in environment variables, not in configuration files
- Node.js 18 or higher
- An Anthropic API key for generating test cases
npm install
The MCP Server Tester is designed to be driven entirely through configuration files. This approach offers several advantages:
- Reusability: Define your servers once, test them repeatedly
- Version control: Check in your test configurations alongside your code
- Sharing: Easily share server test configurations with team members
mcp-server-tester
mcp-server-tester path/to/my-config.json
The configuration file (mcp-servers.json) controls all aspects of testing:
{
"numTestsPerTool": 3,
"timeoutMs": 10000,
"outputFormat": "console",
"outputPath": "./reports/results.json",
"verbose": false,
"mcpServers": {
"filesystem": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-filesystem", "./"],
"env": {
"DEBUG": "true"
}
},
"github": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "your-github-token-here"
}
},
"dev-server": {
"command": "node",
"args": ["/absolute/path/to/your/dev-server.js"],
"env": {
"DEBUG": "true",
"NODE_ENV": "development"
}
}
}
}
By default, the tool will test all servers defined in the mcpServers section. If you want to test only specific servers, you can add an optional servers array:
{
"servers": ["filesystem", "dev-server"],
"numTestsPerTool": 3,
// other settings...
"mcpServers": {
// server definitions...
}
}
1. Create a default configuration file:
mcp-server-tester --init
2. Edit the mcp-servers.json file to add your own servers and settings
3. Create a .env file with your Anthropic API key:
echo "ANTHROPIC_API_KEY=your-api-key-here" > .env
4. Run the tests:
mcp-server-tester
You can maintain different configuration files for different testing scenarios:
cp mcp-servers.json config-dev.json
cp mcp-servers.json config-prod.json
mcp-server-tester ./config-dev.json
mcp-server-tester ./config-prod.json
npm install
1. Install Node.js 18 or newer.
2. Clone the repository and install dependencies:
git clone https://github.com/r-huijts/mcp-server-tester.git
cd mcp-server-tester
npm install
npm run build
3. Optionally link the package globally so the mcp-server-tester command is available system-wide:
npm link
All behaviour is controlled by a JSON configuration file (by default mcp-servers.json). The configuration lists which MCP servers to test and defines options such as timeouts and report formats.
Create the file with --init or copy the provided example:
mcp-server-tester --init
With the configuration and environment variables in place, run:
mcp-server-tester
``
Use
mcp-server-tester path/to/config.json to specify a different configuration or --servers filesystem,github` to test only certain servers.If tool executions are failing:
1. Ensure your server implements the MCP protocol correctly
2. Check the server logs for errors
3. Verify the tool parameters are valid
4. Increase the timeout if the tool takes longer to execute
> ⚠️ WORK IN PROGRESS: This project is under active development and has not been thoroughly tested yet. Features may be incomplete, contain bugs, or change significantly. Use at your own risk in non-production environments only.
A powerful, configuration-driven testing tool for Model Context Protocol (MCP) servers. This project provides a comprehensive solution for validating, benchmarking, and ensuring reliability of MCP servers that integrate with AI models like Claude.
Current Status
This tool is moving toward an alpha release and currently offers:
- ✅ Basic configuration framework
- ✅ MCP server connection and CLI support
- ✅ Test generation using Claude AI
- ✅ Natural language query generation for tests
- ✅ Comprehensive response validation with multiple rules
- ✅ Report generation in console, JSON, HTML, and Markdown formats
- 🚧 Broader automated test coverage of the tester
- 🚧 Production hardening and packaging improvements
If you're interested in contributing, please feel free to open issues and submit pull requests.
Introduction
The Model Context Protocol (MCP) enables AI models to access external tools and data sources through standardized interfaces. As MCP servers grow in complexity and importance, ensuring their correct functionality becomes critical. The MCP Server Tester addresses this need by:
- Automating tests for all tools exposed by an MCP server
- Leveraging Claude AI to generate intelligent, contextually-relevant test cases
- Validating responses against expected outcomes and schemas
- Providing detailed reports to identify issues and performance bottlenecks
This tool is designed for MCP server developers, AI integration teams, and quality assurance professionals who need to ensure their MCP implementations are robust, reliable, and correctly follow the protocol specifications.
Concept
The Model Context Protocol is a standard that allows AI models to call external tools. An MCP server exposes one or more tools through a simple HTTP interface. Each tool describes its name, parameters, and response schema so that a model can invoke it safely.
mcp-server-tester automates the process of checking that an MCP server and its tools work correctly:
1. Discovery – it queries the server for all available tools.
2. Test generation – it uses Claude AI to create realistic test cases for each tool.
3. Execution – it runs those tests against the server.
4. Validation – it verifies the responses using configurable rules.
5. Reporting – it summarizes the results in the console or in JSON, HTML, or Markdown formats.
The goal is to quickly spot mismatches between expected and actual behaviour so that you can fix issues before exposing the tools to production models.
Purpose
- Reliability – catch bugs or inconsistent behaviour in your MCP server.
- Regression testing – run the same set of tests whenever the server changes.
- Documentation – generated reports describe the queries and expected outcomes for each tool.
- Automation – integrate the tester into CI pipelines to ensure ongoing quality.
Repository
- GitHub: https://github.com/r-huijts/mcp-server-tester
- Issues: https://github.com/r-huijts/mcp-server-tester/issues
- License: MIT
Features
- 🔍 Automatically discovers available tools from any MCP server
- 🧪 Generates realistic test cases for each tool using Claude AI
- ⚡ Executes tests and validates responses
- 📊 Provides detailed test reports
- 🔑 Supports multiple connection methods through configuration
- Configuration-Based: Simple JSON configuration for defining MCP servers to test
- Multiple Server Support: Test multiple MCP servers at once
- Comprehensive Testing: Tests all tools exposed by each server
- Natural Language Context: Includes the user query that would trigger each tool, providing real-world context
- Detailed Reports: Generate reports in console, JSON, HTML, or Markdown formats
- Secure: Keeps API keys in environment variables, not in configuration files
Prerequisites
- Node.js 18 or higher
- An Anthropic API key for generating test cases
Installation
Since this project is still in development, installation is done by cloning the repository:
```bash
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
