MCP Eval Runner

A standardized testing harness for MCP servers and agent workflows. Define test cases as YAML fixtures (steps → expected tool calls → expected outputs), run regression suites directly from your MCP client, and get pass/fail results with diffs — without leaving Claude Code or Cursor.

Tool reference|Configuration|Fixture format|Contributing|Troubleshooting|Design principles

{ "mcpServers": { "eval-runner": { "command": "npx", "args": ["-y", "mcp-eval-runner@latest"] } } }

By default, eval fixtures are loaded from./evals/in the current working directory. To use a different path:

{ "mcpServers": { "eval-runner": { "command": "npx", "args": ["-y", "mcp-eval-runner@latest", "--fixtures=~/my-project/evals"] } } }

Amp · Claude Code · Cline · Cursor · VS Code · Windsurf · Zed

Create a file atevals/smoke.yaml. Uselive mode(recommended) by including aserverblock:

name: smoke description: "Verify eval runner itself is working" server: command: node args: ["dist/index.js"] steps: - id: list_check description: "List available test cases" tool: list_cases input: {} expect: output_contains: "smoke"

Then enter the following in your MCP client:

Your client should return a pass/fail result for the smoke test.

Fixtures are YAML (or JSON) files placed in the fixtures directory. Each file defines one test case.

server: command: node # executable to spawn args: ["dist/index.js"] # arguments env: # optional environment variables MY_VAR: "value"

Whenserveris present the eval runner spawns the server as a child process, connects via MCP stdio transport, and calls each step's tool against the live server.

- The server is spawned and each step calls the named tool via MCP stdio.
- Assertions run against the real tool response.
- Errors from the server cause the step (and by default the case) to fail immediately.

- No server is started.
- Each step's output is taken fromexpected_output(or empty string if absent).
- Assertions run against that static output.
- Useful for authoring and CI dry-runs, butoutput_containsassertions will always fail ifexpected_outputis not set.

All assertions go inside a step'sexpectblock:

expect: output_contains: "substring" # output includes this text output_not_contains: "error" # output must NOT include this text output_equals: "exact string" # output exactly matches output_matches: "regex pattern" # output matches a regular expression tool_called: "tool_name" # verifies which tool was called latency_under: 500 # latency in ms must be below this threshold schema_match: # output (parsed as JSON) matches JSON Schema type: object required: [id] properties: id: type: number

Multiple assertions in oneexpectblock are all evaluated; the step fails if any assertion fails.

Reference the output of a previous step in a downstream step'sinputusing{{steps.<step_id>.output}}:

steps: - id: search_step tool: search input: query: "mcp eval runner" expected_output: "result: mcp-eval-runner v1.0" expect: output_contains: "mcp-eval-runner" - id: summarize_step tool: summarize input: text: "{{steps.search_step.output}}" expected_output: "Summary: mcp-eval-runner v1.0" expect: output_contains: "Summary"

Piping works in both live mode and simulation mode.

Fixtures created with thecreate_test_casetool do not include aserverblock. They always run in simulation mode. To use live mode, add aserverblock manually to the generated YAML file.

- run_suite— execute all fixtures in the fixtures directory; returns a pass/fail summary
- run_case— run a single named fixture by name
- list_cases— enumerate available fixtures with step counts and descriptions

- create_test_case— create a new YAML fixture file (simulation mode; noserverblock)
- scaffold_fixture— generate a boilerplate fixture with placeholder steps and pre-filled assertion comments

- regression_report— compare the current fixture state to the last run; surfaces regressions and fixes
- compare_results— diff two specific runs by run ID
- generate_html_report— generate a single-file HTML report for a completed run

- evaluate_deployment_gate— CI gate; fails if recent pass rate drops below a configurable threshold
- discover_fixtures— discover fixture files across one or more directories (respectsFIXTURE_LIBRARY_DIRS)

Directory to load YAML/JSON eval fixture files from.

Path to the SQLite database file used to store run history.

Maximum time in milliseconds to wait for a single step before marking it as failed.

Watch the fixtures directory and rerun the affected fixture automatically when files change.

Type:stringChoices:console,json,htmlDefault:console

Number of test cases to run in parallel.

Start an HTTP server on this port instead of stdio transport.

Type:numberDefault: disabled (uses stdio)

Pass flags via theargsproperty in your JSON config:

{ "mcpServers": { "eval-runner": { "command": "npx", "args": ["-y", "mcp-eval-runner@latest", "--watch", "--timeout=60000"] } } }

- No mocking: Live mode evals run against real servers. Correctness is non-negotiable.
- Fixtures are text: YAML/JSON in version control; no proprietary formats or databases.
- Dogfood-first: The eval runner's own smoke fixture tests the eval runner itself.

Before publishing a new version, verify the server with MCP Inspector to confirm all tools are exposed correctly and the protocol handshake succeeds.

npm run build && npm run inspect

# List all tools npx @modelcontextprotocol/inspector --cli node dist/index.js --method tools/list # List resources and prompts npx @modelcontextprotocol/inspector --cli node dist/index.js --method resources/list npx @modelcontextprotocol/inspector --cli node dist/index.js --method prompts/list # Call a tool (example — replace with a relevant read-only tool for this plugin) npx @modelcontextprotocol/inspector --cli node dist/index.js \ --method tools/call --tool-name list_cases # Call a tool with arguments npx @modelcontextprotocol/inspector --cli node dist/index.js \ --method tools/call --tool-name run_case --tool-arg name=smoke

Run before publishing to catch regressions in tool registration and runtime startup.

New assertion types go insrc/assertions.ts— implement theAssertioninterface and add a test. Integration tests live undertests/as unit tests and underevals/as eval fixtures.

This is a web browser that enables your coding agent, such as Claude Code, to visit websites on your behalf and assist you in identifying bugs or creating UI test cases.

Agents make claims. Reelier writes receipts — record an agent's tool-call workflow once, replay it deterministically at 0 tokens, and diff runs to catch drift.

fable-discipline is a Claude Code plugin that makes agentic software work follow repeatable working patterns: design before code, verify after edits, separate author from reviewer, preserve verified state between sessions, and report uncertainty honestly.

Preflight checks for paid x402 and MCP tools before marketplace listing.

An intelligent MCP server that automates the reproduction of GitHub issues for AWS CDK projects.

Modern Open source Test Management with MCP Server integration

Read-first Jenkins MCP server in Go for agent-driven build debugging. 20 tools including compare_builds, flaky-test detection, JUnit/Ginkgo failure parsing, and disk-cached console logs with on-disk path handoff. Write tools (trigger/stop/cancel) gated by JENKINS_MCP_READONLY env var.

An MCP server for Nextflow development and testing, which requires a local clone of the Nextflow Git repository.

Provides a sleep/wait tool to add delays between operations, such as waiting between API calls or testing eventually consistent systems.

Pauses the execution of an agent for a specified duration.

Source-grounded launch review for x402, MCP, paid API, and agent-tool listings.