Patronus MCP Server

An MCP server implementation for the Patronus SDK, providing a standardized interface for running powerful LLM system optimizations, evaluations, and experiments.

Features

- Initialize Patronus with API key and project settings
- Run single evaluations with configurable evaluators
- Run batch evaluations with multiple evaluators
- Run experiments with datasets

Installation

1. Clone the repository:

git clone https://github.com/yourusername/patronus-mcp-server.git

cd patronus-mcp-server

2. Create and activate a virtual environment:

python -m venv .venv

source .venv/bin/activate  # On Windows: .venv\Scripts\activate

3. Install main and dev dependencies:

uv pip install -e .

uv pip install -e ".[dev]"

Usage

Running the Server

The server can be run with an API key provided in two ways:

1. Command line argument:

python src/patronus_mcp/server.py --api-key your_api_key_here

2. Environment variable:

export PATRONUS_API_KEY=your_api_key_here

python src/patronus_mcp/server.py

Interactive Testing

The test script (tests/test_live.py) provides an interactive way to test different evaluation endpoints. You can run it in several ways:

1. With API key in command line:

python -m tests.test_live src/patronus_mcp/server.py --api-key your_api_key_here

2. With API key in environment:

export PATRONUS_API_KEY=your_api_key_here

python -m tests.test_live src/patronus_mcp/server.py

3. Without API key (will prompt):

python -m tests.test_live src/patronus_mcp/server.py

The test script provides three test options:
1. Single evaluation test
2. Batch evaluation test

Each test will display the results in a nicely formatted JSON output.

API Usage

Initialize

from patronus_mcp.server import mcp, Request, InitRequest
request = Request(data=InitRequest(

    project_name="MyProject",

    api_key="your-api-key",

    app="my-app"

))

response = await mcp.call_tool("initialize", {"request": request.model_dump()})

Single Evaluation

from patronus_mcp.server import Request, EvaluationRequest, RemoteEvaluatorConfig
request = Request(data=EvaluationRequest(

    evaluator=RemoteEvaluatorConfig(

        name="lynx",

        criteria="patronus:hallucination",

        explain_strategy="always"

    ),

    task_input="What is the capital of France?",

    task_output="Paris is the capital of France."

    task_context=["The capital of France is Paris."],

))

response = await mcp.call_tool("evaluate", {"request": request.model_dump()})

Batch Evaluation

from patronus_mcp.server import Request, BatchEvaluationRequest, RemoteEvaluatorConfig
request = Request(data=BatchEvaluationRequest(

    evaluators=[

        AsyncRemoteEvaluatorConfig(

            name="lynx",

            criteria="patronus:hallucination",

            explain_strategy="always"

        ),

        AsyncRemoteEvaluatorConfig(

            name="judge",

            criteria="patronus:is-concise",

            explain_strategy="always"

        )

    ],

    task_input="What is the capital of France?",

    task_output="Paris is the capital of France."

    task_context=["The capital of France is Paris."],

))

response = await mcp.call_tool("batch_evaluate", {"request": request.model_dump()})

Run Experiment

```python
from patronus_mcp import Request, ExperimentRequest, RemoteEvaluatorConfig, CustomEvaluatorConfig