llmprobe

This is the intended use case: repeated synthetic probes that make LLM latency, TTFT regressions, throughput drops, and provider degradation visible before users report them.

go install github.com/Jwrede/llmprobe@latest llmprobe version

Install as a Claude Code plugin for/llmprobeskill and MCP tools:

claude mcp add --transport stdio llmprobe -- llmprobe mcp

llmprobe works with OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock, and OpenAI-compatible endpoints such as vLLM, Ollama, OpenRouter, Groq, Together AI, Fireworks, DeepSeek, and Mistral.

Create aprobes.yml(or copy the included example):

providers: - name: openai api_key: ${OPENAI_API_KEY} models: - name: gpt-4o thresholds: max_ttft: 2s - name: gpt-4o-mini thresholds: max_ttft: 500ms - name: anthropic api_key: ${ANTHROPIC_API_KEY} models: - name: claude-sonnet-4-20250514 thresholds: max_ttft: 1s

$ llmprobe probe Provider Model Status TTFT Latency Tok/s Tokens Error -------- ----- ------ ---- ------- ----- ------ ----- openai gpt-4o healthy 312ms 2100ms 68.4 42 openai gpt-4o-mini healthy 98ms 814ms 112.3 56 anthropic claude-sonnet-4-20250514 healthy 420ms 2831ms 52.1 38 azure gpt-4o healthy 289ms 1950ms 71.2 44 bedrock anthropic.claude-3-5... degraded 1820ms 4510ms 28.1 38 4 healthy, 1 degraded, 0 errors

One-off health check. Probes all configured endpoints and prints results.

llmprobe probe # table output llmprobe probe -f json # JSON output llmprobe probe --fail-on degraded # exit 1 if any endpoint is degraded llmprobe probe -c custom-config.yml # custom config path

Continuous monitoring. Probes all endpoints on an interval and prints a summary line per iteration.

llmprobe watch # default 60s interval llmprobe watch --interval 30s # custom interval llmprobe watch --tui # live terminal dashboard with TTFT chart llmprobe watch --tui --load data.jsonl # load historical data into the dashboard llmprobe watch -f json # JSONL output (one line per result) llmprobe watch --prometheus :9090 # expose Prometheus metrics llmprobe watch --otel localhost:4317 # export OpenTelemetry metrics via OTLP/gRPC

The--tuiflag launches a live terminal dashboard with a TTFT chart, color legend, and statistics table. Use--loadto import historical JSONL data (fromllmprobe watch -f json > data.jsonl).

claude mcp add --transport stdio llmprobe -- llmprobe mcp

Once registered, Claude Code can call llmprobe tools during any conversation.

Example use case:An agent callslist_providersto see what models are configured, thenprobe_allto verify they are healthy before deploying changes.

defaults: prompt: "Hello" # probe prompt max_tokens: 20 # max output tokens timeout: 30s # per-probe timeout concurrency: 5 # max parallel probes providers: - name: openai # openai, anthropic, google, azure, bedrock label: openai-prod # optional display name; useful for multiple OpenAI-compatible endpoints api_key: ${OPENAI_API_KEY} # env var expansion base_url: https://custom.api # optional, override endpoint models: - name: gpt-4o prompt: "Say hello." # override default prompt max_tokens: 10 # override default max_tokens response_format: json # optional; OpenAI-compatible JSON mode validate_json: true # optional; mark degraded if returned content is not valid JSON thresholds: max_ttft: 2s # alert if TTFT exceeds this max_latency: 10s # alert if total latency exceeds this min_tokens_per_sec: 20 # alert if throughput drops below this max_ttft_multiplier: 2.0 # optional; compare against baseline p50 max_latency_multiplier: 2.5 - name: azure api_key: ${AZURE_OPENAI_API_KEY} base_url: https://your-resource.openai.azure.com api_version: "2024-10-21" # optional, defaults to 2024-10-21 models: - name: gpt-4o # deployment name - name: bedrock access_key: ${AWS_ACCESS_KEY_ID} secret_key: ${AWS_SECRET_ACCESS_KEY} region: us-east-1 models: - name: anthropic.claude-3-5-sonnet-20241022-v2:0

API keys and AWS credentials support${ENV_VAR}syntax. Only credential fields are expanded, so env var references in prompts or model names are left as-is.

Many providers (Groq, Together AI, Fireworks, DeepSeek, Mistral, OpenRouter, Ollama, vLLM) expose an OpenAI-compatible API. These work out of the box by settingbase_url. Use thelabelfield to distinguish multiple OpenAI-compatible blocks:

providers: # Groq - name: openai label: groq api_key: ${GROQ_API_KEY} base_url: https://api.groq.com/openai models: - name: llama-3.3-70b-versatile # DeepSeek - name: openai label: deepseek api_key: ${DEEPSEEK_API_KEY} base_url: https://api.deepseek.com models: - name: deepseek-chat # Together AI - name: openai label: together api_key: ${TOGETHER_API_KEY} base_url: https://api.together.xyz models: - name: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo # Local Ollama - name: openai label: ollama api_key: unused base_url: http://localhost:11434 models: - name: llama3.2

For OpenAI-compatible endpoints, setresponse_format: jsonto request JSON mode andvalidate_json: trueto mark the probe asdegradedif the streamed content is not valid JSON.

providers: - name: openai label: vllm-json api_key: unused base_url: http://localhost:8000 models: - name: meta-llama/Llama-3.1-8B-Instruct prompt: 'Return {"ok": true} as JSON.' response_format: json validate_json: true

Run with--prometheusto expose metrics for scraping:

llmprobe watch --interval 30s --prometheus :9090

Thellmprobe_statusgauge encodes health as: 1 = healthy, 0.5 = degraded, 0 = error. Use this for alerting in Grafana or Alertmanager.

Run with--otelto export probe metrics to an OTLP/gRPC collector.

llmprobe watch --interval 30s --otel localhost:4317

All metrics includeproviderandmodelattributes.

probes.yml -> Config loader (YAML + env var expansion) -> Probe engine (concurrent goroutines per provider/model) -> Provider clients (raw HTTP + SSE parsing, no SDKs) -> Results (TTFT, latency, tokens/sec, status) -> Output (table, JSON, JSONL)

Each provider client is a thin HTTP wrapper that sends a streaming request and parses the response. No LLM SDKs are imported. The SSE parser handles both data-only events (OpenAI, Google) and named events (Anthropic). The Bedrock client implements SigV4 signing and AWS binary event stream parsing from scratch.

TTFT is measured from the moment the HTTP request is sent to the first event that contains actual content text (not role assignments or metadata).

OpenAI-compatible covers: Groq, Together AI, Fireworks, DeepSeek, Mistral, OpenRouter, Ollama, vLLM, and any endpoint that speaks the OpenAI chat completions API.

- More provider-specific examples for self-hosted OpenAI-compatible endpoints
- More report formats for long-running monitoring windows
- Optional runbook templates for common LLM endpoint failures

This is a web browser that enables your coding agent, such as Claude Code, to visit websites on your behalf and assist you in identifying bugs or creating UI test cases.

Diagnoses token waste in Claude Code sessions with 6 anomaly types and severity scoring. Fully local.

AI-powered live runtime debugging with Lightrun production context.

Analyze your MCP setup: measure token cost per tool, find bloated schemas, detect duplicates, and get optimization tips to reclaim your context window.

MCP server for .NET runtime diagnostics — memory, GC, threads, and more.

MCP server for .NET runtime diagnostics — memory, GC, threads, and more.

JVM diagnostics MCP server for thread dump and GC log analysis. Detects deadlocks, lock contention, and GC pressure. Runs via npx on Node.js — no JVM or Docker required.

Reticle intercepts, visualizes, and profiles JSON-RPC traffic between your LLM and MCP servers in real-time, with zero latency overhead. Stop debugging blind. Start seeing everything.

MCP server that predicts your monthly LLM spend from local Claude Code / Cursor / Codex logs. Forecasts end-of-month $, ranks spike drivers, warns before budget breach. Free tier + paid tier ($19/mo), MIT licensed.

Turn natural language into powerful Perfetto trace analysis. Quickly explain jank, diagnose ANRs, spot CPU hot threads, uncover lock contention, and find memory leaks.