flakiness-knowledge-graph-mcp

Builds a knowledge graph of flaky tests from Playwright run history

A Playwright custom reporter + MCP server that builds a local flakiness knowledge graph from your test run history. Ask your AI agent which tests are unreliable, on which browser, and whether they're getting worse.

A single Playwright trace tells youwhatfailed right now. It doesn't tell you whether this test has been silently flaking for two weeks, or only fails on Firefox in CI, or is getting slower with every release.

This tool fixes that by accumulating run history into a SQLite database and exposing it to AI agents via MCP.

npm install -g flakiness-knowledge-graph-mcp

git clone https://github.com/vola-trebla/flakiness-knowledge-graph-mcp.git cd flakiness-knowledge-graph-mcp npm install && npm run build

2. Add the reporter to your Playwright project

// playwright.config.ts export default defineConfig({ reporter: [["html"], ["flakiness-knowledge-graph-mcp/reporter", { dbPath: "./flakiness.db" }]], });

Run your tests normally — the reporter writes every result toflakiness.dbautomatically.

Cursor / VS Code (.cursor/mcp.jsonor.vscode/mcp.json)

{ "mcpServers": { "flakiness-knowledge-graph": { "command": "flakiness-knowledge-graph-mcp" } } }

claude mcp add flakiness-knowledge-graph flakiness-knowledge-graph-mcp

No Playwright project yet? Generate 30 days of realistic sample data:

Then point your AI agent at./demo.dbto explore all 8 tools.

I've been running my Playwright suite for two weeks. The DB is at /my-project/flakiness.db. 1. get_flaky_tests — which tests are most unreliable? Show last 7 days only. 2. get_test_history for the top flaky test — is it getting worse? 3. get_flakiness_trend for the same test over 14 days — plot the daily rate. 4. get_failure_patterns — does it only fail on a specific browser or OS? 5. cluster_semantic_error_trees — are multiple tests failing with semantically identical errors? 6. correlate_git_commit_flakiness — which commit introduced the flakiness? 7. get_slow_tests — which tests should I optimize for CI speed?

Grouping errors that look different but aren't

get_error_groupsclusters by raw string prefix — if the error contains a UUID or element ID it creates separate groups for what is really one root cause.cluster_semantic_error_treesstrips dynamic values first:

{ "total_clusters": 2, "clusters": [ { "cluster_id": "cluster-1", "canonical_message": "TimeoutError: locator.click: Timeout 30000ms exceeded\n waiting for locator('#submit-btn')", "normalized_message": "TimeoutError: locator.click: Timeout <num>ms exceeded waiting for locator", "error_taxonomy": "TimeoutError", "instance_count": 14, "affected_tests": 3, "sample_test_ids": ["checkout > submit order", "cart > add item", "checkout > apply coupon"] }, { "cluster_id": "cluster-2", "canonical_message": "Error: 2 requests to https://api.example.com/orders/8f3a1c were made. Expected 1", "normalized_message": "Error: <num> requests to <url> were made. Expected <num>", "error_taxonomy": "AssertionError", "instance_count": 6, "affected_tests": 1, "sample_test_ids": ["api-mock > intercept order"] } ] }

correlate_git_commit_flakinessuses a state machine — it looks for runs where a test was stable for ≥3 consecutive passes, then failed. The transition record includes the SHA from the CI environment:

{ "total_transitions": 1, "transitions": [ { "test_id": "auth > login > should redirect after login", "title": "should redirect after login", "transition_type": "stable_to_flaky", "git_commit_sha": "a3f8c1d9e2b54f6a", "git_branch": "main", "git_author": "dev-handle", "transition_date": "2025-04-14" } ] }

The reporter readsGITHUB_SHA/CI_COMMIT_SHA/CIRCLE_SHA1/GIT_COMMITautomatically — no reporter config changes needed beyond upgrading to v0.2.0.

🔗 Works great with playwright-trace-decoder-mcp

These two MCP servers are designed to complement each other:

- flakiness-knowledge-graph-mcpanswers "is this test flaky historically, and which commit caused it?"
- playwright-trace-decoder-mcpanswers "what exactly failed in this specific run?"

playwright.config.ts └── FlakinessReporter → flakiness.db (SQLite via sql.js) flakiness.db └── test_runs table id, test_id, title, suite, file, status, duration_ms, browser, os, timestamp, error, retry, git_commit_sha, git_branch, git_author ← added in v0.2.0 MCP server └── reads flakiness.db on demand (in-process handle reuse)

npm run build # compile TypeScript → dist/ npm run lint # ESLint npm run format # Prettier --write npm run format:check # Prettier check (used in CI) npm run seed # populate flakiness.db with 30 days of demo data

Create and manage end-to-end tests using the Octomind platform.