agent-eval

Statistical regression testing for LLM agents: run version A and version B 50x each and get a p-value, an effect size, and a 95% confidence interval on whether behavior actually shifted.

(uv addand the npm/npx wrapper are covered inInstallbelow.)

p=0.003, Cohen's d=-0.61 -> REGRESSED (deploy blocked) p=0.410, Cohen's d=0.021 -> STABLE (safe to ship)

[!NOTE] Promptfoo, one of the most widely used open-source LLM eval frameworks, wasacquired by OpenAI in March 2026, staying open source but folding its team into OpenAI's Frontier platform. agent-eval is Apache 2.0-licensed, self-hostable, and has no commercial dependency. The statistical core (Mann-Whitney U, bootstrap CI, Cohen's d) will never be paywalled.

pip install agent-regress-cli # or uv add agent-regress-cli # or, from Node/npx (thin wrapper around the same Python CLI) npx agent-regress-cli

pip install agent-regress-cli agent-regress compare \ --version-a-results v1_scores.json \ --version-b-results v2_scores.json \ --metric tool_accuracy # ============================================================ # agent-regress Report -- tool_accuracy # ============================================================ # Verdict: REGRESSED # p-value: 0.0000 # Cohen's d: -2.193 # 95% CI: [-0.213, -0.148] # # Version A: 0.8470 +/- 0.0525 (n=50) # Version B: 0.6685 +/- 0.1025 (n=50) # Delta: -0.1786 # ============================================================

Add--json --fail-on-regressionto get clean, parseable output and a non-zero exit code onREGRESSED, for wiring straight into CI.

The top-levelagent-regress --versionflag prints the installed version and exits.

Driving the agent yourself instead of pre-computing scores? Use the Python API:

from agent_regress import compare # Any callable that takes a test case dict and returns a score 0.0-1.0 def agent_v1(test_case: dict) -> float: ... # your existing agent def agent_v2(test_case: dict) -> float: ... # your updated agent test_suite = [ {"query": "find SKU for order 8823", "expected": "SKU-4492"}, # ... more test cases ] report = compare( version_a=agent_v1, version_b=agent_v2, test_suite=test_suite, n_runs=50, metric="tool_accuracy", # use any name except "accuracy" when agents return floats ) print(report) # structured output with p-value, CI, effect size report.assert_stable() # raises AssertionError if behavior regressed

Agent returns text? Pass a scorer or use the built-ins:

from agent_regress import compare, exact_match_scorer, f1_scorer # exact_match_scorer: 1.0 if str(output).strip() == str(expected).strip() # f1_scorer: token-level F1 (multiset — handles repeated tokens correctly) report = compare( version_a=agent_v1, version_b=agent_v2, test_suite=test_suite, n_runs=50, scorer=exact_match_scorer, # test_case must have an "expected" key )

def my_scorer(output: str, test_case: dict) -> float: return 1.0 if output.strip() == test_case["expected"] else 0.0 report = compare(..., scorer=my_scorer)

agent-eval ships aModel Context Protocolserver so an AI agent (Claude, Cursor, or any MCP-compatible client) can run statistical regression tests directly, without a human invoking the CLI by hand.

pip install "agent-regress-cli[mcp]"

Add it to your MCP client's config (for Claude Desktop,claude_desktop_config.json):

{ "mcpServers": { "agent-eval": { "command": "uvx", "args": ["--from", "agent-regress-cli", "agent-regress-mcp"] } } }

The server exposes one tool,run, that shells out to theagent-regressCLI with the given subcommand and arguments plus--json, and returns the parsed JSON result:

run(["compare", "--version-a-results", "a.json", "--version-b-results", "b.json", "--metric", "accuracy"])

Transport is stdio, so there is nothing to host: the MCP client spawns the server as a local subprocess. Source:src/agent_regress/mcp_server.py.

- #5243: a new typedcontext=API replaced untypedconfig['configurable']. A single run on either invocation style still clears a threshold check; only a version-A-vs-B comparison shows whether the switch changed measured behavior.
- #4486: node/task-level result caching can silently mask repeated-sampling variance. Threshold checks don't care whether a result came from cache; a statistical comparison depends on genuinely independent samples, soagent-evaladded cache-busting to protect that assumption.

- #2463: agent-as-tool calls were silently dropping the parent run'sRunConfig. The nested call still returns a normal-looking response, so a single-response check clears; only inspecting config propagation across runs reveals the regression.
- #2214: image/audio/file tool outputs were silently downgraded to text-only. A text-only threshold scorer has no way to notice a dropped attachment.

- #6134: a security fix for file tools leaking absolute filesystem paths in responses. A quality scorer checks whether the answer is correct, not whether it also leaks a path, so the leak clears the bar.
- #6236: tools gained an optional Pydanticoutput_schema, moving from unstructuredstr()output to structured JSON. Both the old and new format can look "reasonable" to a threshold scorer even though the schema changed underneath.

These are the regressions that motivated this project. Full detail on all 14 individually-documented PRs (drawn from a 29-PR, 239-row validation campaign across LangGraph, CrewAI, and the OpenAI Agents SDK) is indocs/pr-analysis.md.

============================================================ agent-regress Report -- tool_accuracy ============================================================ Verdict: REGRESSED p-value: 0.0031 Cohen's d: -0.610 95% CI: [-0.221, -0.067] Version A: 0.8400 +/- 0.0601 (n=50) Version B: 0.7000 +/- 0.0903 (n=50) Delta: -0.1400 ============================================================

When CI fails, the assertion error gives the deploy-blocking message:

AssertionError: REGRESSED: tool_accuracy dropped 16.7% (p=0.003, Cohen's d=-0.61, 95% CI [-0.22, -0.07]) Version A: 0.840 +/- 0.060 (n=50) Version B: 0.700 +/- 0.090 (n=50)

Verdict: STABLE p-value: 0.4100 Cohen's d: 0.021

DeepEval, Promptfoo, and Braintrust test whether individual responses meet thresholds. None of them answer whether a version's behavior distribution shifted significantly from the last. Agent Evaluation addresses that specific statistical question, which threshold testing cannot answer.

report.assert_stable()— inline, after you've already calledcompare():

# test_regression.py -- add to your existing test suite from agent_regress import compare def test_no_regression(): report = compare( version_a=production_agent, version_b=staging_agent, test_suite=load_test_suite(), n_runs=50, ) report.assert_stable( p_threshold=0.05, # act on changes at p < 0.05 min_effect=0.2, # Cohen's d threshold -- ignore noise below 0.2 )

RegressionGate— reusable gate object, useful when you run multiple comparisons with the same thresholds:

from agent_regress import compare, RegressionGate gate = RegressionGate(p_threshold=0.05, min_effect=0.2) def test_tool_accuracy(): report = compare(version_a=prod, version_b=staging, test_suite=suite, n_runs=50) gate.check(report) # raises AssertionError on regression; warns if n < 30 def test_routing_accuracy(): report = compare(version_a=prod, version_b=staging, test_suite=routing_suite, n_runs=50) gate.check(report)

Both patterns: warn (not fail) whenn < 30per version, and treatn < 10as insufficient data and skip the gate. This CI-gate threshold (30) is intentionally lower thancompare()'s own general low-power warning (n < 50, seeFAQ) — it exists to stop a genuinely too-small sample from silently gating a build, not to guarantee 80% statistical power the way the 50-run recommendation does.

Bootstrap confidence intervals(1,000 resamples, seed=42) give a 95% CI on the mean score delta. The CI tells you how large the shift was: a CI of [-0.22, -0.07] means you can be 95% confident the true per-run accuracy drop is between 7 and 22 percentage points.

Cohen's d(pooled standard deviation) separates statistical significance from operational significance. A shift at p=0.001 with d=0.04 is real but meaningless. A shift at p=0.06 with d=0.5 is operationally large but requires more data to confirm. The default CI gate acts only when both p < 0.05 and d >= 0.2.

Seedocs/statistical-methods.mdfor the full methodology.

Comparing twoinstalled versionsof the same framework (rather than two in-process configurations)? Seedocs/cross-version-comparison.mdfor thesubprocess_runner()pattern.

[!WARNING] The[crewai]extra: CrewAI's own memory/knowledge/RAG backend can pull in ChromaDB, which currently has an unpatched critical CVE (GHSA-f4j7-r4q5-qw2c) affecting any ChromaDB server run withtrust_remote_code=Trueand exposed to the network.agent-evalnever starts, configures, or exposes a ChromaDB server itself, so this only matters if your ownCrewdoes — don't run a network-exposed ChromaDB instance withtrust_remote_code=Trueuntil a fix ships.

from agent_regress.benchmarks.tau_bench import TauBenchHarness harness = TauBenchHarness(agent=my_agent, dataset=tau_bench_dataset) results = harness.evaluate(k_values=[1, 4, 8])

GAIA Level 1-3 splitstratifies by task difficulty. Overall accuracy hides per-difficulty regressions: a prompt change that helps Level 1 often hurts Level 3.

from agent_regress.benchmarks.gaia import GAIAHarness harness = GAIAHarness(agent=my_agent, dataset=gaia_dataset) results = harness.evaluate() # returns list[GAIALevelResult], one per level for r in results: print(f"Level {r.level}: {r.accuracy:.3f} ({r.n_correct}/{r.n_questions})")

SWE-bench scaffold scoreisolates framework contribution from model contribution.

from agent_regress.benchmarks.swebench import SWEBenchHarness harness = SWEBenchHarness(agent=my_agent, dataset=swe_dataset) result = harness.evaluate() print(f"scaffold pass rate: {result.scaffold_pass_rate:.3f} ({result.n_resolved}/{result.n_instances})")

Seeleaderboard/README.mdto submit results.

git clone https://github.com/RudrenduPaul/agent-eval cd agent-eval docker compose up

- Supply chain:SLSA Level 2 via GitHub Actions provenance. All releases signed with Sigstore. SBOM attached to every GitHub Release.
- Vulnerability scanning:Trivy scans on every CI run (HIGH/CRITICAL only, exit on unfixed). CodeQL static analysis on every push.
- Dependency pinning:Dependabot keeps all GitHub Actions and Python dependencies current.
- Disclosure:SECURITY.md— report vulnerabilities privately via GitHub Security Advisories.

The full breakdown is in thecomparison tableabove. In short: DeepEval, Promptfoo, and Braintrust all test whether an individual response clears a fixed quality bar. None of the three report a p-value, an effect size, or a bootstrap confidence interval on whether behavior shifted between two versions, which is the specific statistical question agent-eval is built to answer.

No.compare()takes two callables you provide,version_aandversion_b, and runs your existing agent code against your test suite. Agent Evaluation never makes a model call itself, and the stats module (src/agent_regress/stats/) is required to stay pure Python and scipy with no LLM calls, so the statistical core has no network dependency and nothing to configure credentials for.