WorkloadTruth
About
Classifies GPU workloads as inference or training from telemetry alone via MCP tools.
Details
- Author
- rudrendupaul
- Categories
- Cloud Service, Other, Infrastructure
Jump to
Setup
Install WorkloadTruth in your MCP client (Claude Desktop, Cursor, Windsurf, and others).
Repository: https://github.com/rudrendupaul/WorkloadTruth
Follow the installation instructions in the repository README, then restart your MCP client.
What is WorkloadTruth, and why does it exist
WorkloadTruth is an open-source command-line tool and MCP server that classifies a running GPU workload asTRAINING,INFERENCE, orIDLEusing only GPU-level telemetry (utilization, memory pattern, power draw), with no changes to the workload's own code and no reliance on a self-reported job label.
It exists because every mainstream GPU scheduler asks the job's owner to declare its type at submission time and never checks that declaration against what the hardware is actually doing. That gap has two real consequences: cost misallocation (a job scheduled at low-priority "inference" pricing that is actually running full training) and unauthorized workload changes (an inference endpoint that quietly starts training on live traffic). WorkloadTruth closes that verification gap today, and doubles as the first open, installable implementation of a real academic research thread on verifying AI training runs from hardware telemetry (see below).
WorkloadTruth's core technique, classifying training vs. non-training GPU activity from telemetry, is not novel. It's the direct application of a real, active research thread:
- Yonadav Shavit (Harvard),](https://docs.wandb.ai/models/ref/python/experiments/system-metrics)"What does it take to catch a Chinchilla?"(2023): proposed hardware-level "training transcripts" for verifying large training runs.
- GovAI,"Computing Power and the Governance of AI"(2024): surveyed compute-governance mechanisms, explicitly framed as exploratory, not endorsed policy.
- "Hardware-Enabled Mechanisms for Verifying Responsible AI Development"(2025): hardware-security researchers proposing on-chip attestation.
- Rahman & Tajdari,"Detecting Hidden ML Training With Zero-Overhead Telemetry"(ICML 2026 Technical AI Governance workshop): a working NVML-telemetry classifier, 98.2% accurate on unobfuscated workloads, the direct prior art for this project's core classification technique.
What WorkloadTruth adds:as of this project's own research (2026-07-19), no open-source, installable implementation of this research thread existed, only academic prototypes. WorkloadTruth is that packaging: a real CLI, an MCP server, a hash-chained audit log, and a reproducible evasion-robustness benchmark, built in the open. It does not claim to improve on the paper's classification technique. See the benchmark section above: the current rule-based classifier is considerably more evadable than the paper's ML approach on the one axis it measures.
- Not a compliance or regulatory-audit tool.No law currently requires inference/training classification or reporting. Any future claim otherwise will name the specific enacted regulation; none exists as of this writing.
- Not a content inspector.WorkloadTruth reads GPU-level signals only (utilization, memory, power). It never inspects model weights, training data, prompts, or completions.
- Not proof of "compliant" or "safe" operation.The audit log proves what was classified and when, and that the record wasn't altered afterward, not that the classification was correct or that any policy was followed.
Does this need an NVIDIA GPU?Only for thenvmlbackend.--backend syntheticruns the full classifier and CLI against documented synthetic traces, no GPU required. Useful for trying the tool or for CI.
Can it classify AMD or Intel GPU workloads?Not yet. The telemetry layer is a pluggable interface (TelemetryBackend) specifically so a new vendor backend (AMD ROCm, Intel Level Zero) can be added without touching the classifier. SeeCONTRIBUTING.md.
Is the classifier accurate enough to bill or penalize someone based on its output?Not yet, and the benchmark section above is the honest reason why: 0% accuracy on evasive training workloads today. Treatworkload_typeas a signal to investigate, not a verdict.
Why not just use the ML classifier from the paper?Its trained weights and dataset were never published. Reimplementing an ML classifier without real training data would produce an unvalidated accuracy claim, not a measured one. SeeHow classification works.
How is this different from run:ai or NVIDIA DCGM?run:ai and DCGM both expose or use GPU telemetry, but neither classifies workload type from that telemetry. run:ai relies entirely on the label the job's owner declares at submission; DCGM just exposes raw utilization and memory metrics for something else to interpret. WorkloadTruth is the layer that actually looks at the telemetry and answers the question. See thecomparison table.
Does this work on Windows, macOS, and Linux?Thesyntheticbackend runs anywhere Python 3.9+ runs, including this project's own macOS build environment (which has no NVIDIA GPU). Thenvmlbackend requires an NVIDIA GPU and driver, which in practice means Linux or Windows with NVIDIA hardware; NVML itself is not available on macOS.
What license is this under, and can I use it commercially?Apache 2.0. Commercial use, modification, and redistribution are all permitted under its terms; seeLICENSE.
SeeCONTRIBUTING.md. Security issues: seeSECURITY.md.
Access your Fitbit health and fitness data for personalized insights.
Access Garmin Connect running data and training plan information.
Connects to Garmin Connect to expose your fitness and health data to MCP-compatible clients.
A fitness data platform that aggregates data from providers like Strava and Fitbit, offering secure B2B API access.
Fetch health and fitness metrics like heart rate, sleep, and steps from the Ultrahuman API.
Physics-based anomaly detection via MCP — send any data, get anomalies back using wave-equation dynamics. No training pipelines, no model files.
Read-only access to your Alma nutrition data, meals, goals, scores, workouts, weight history, pantry, and grocery list.
AI workout coach MCP server. Access training data, workout history, personal records, and body progress through Claude Desktop.
AI-powered cycling training coach: personalized training plans, daily workouts, nutrition, strength training, fitness metrics (FTP, VO2Max), and Strava, Garmin, Wahoo, MyWhoosh, and Rouvy integrations.
Free remote MCP server giving AI assistants direct access to data from a mentoring practice: 3,400+ paid 1:1 sessions with 300+ engineering leaders since 2019.
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
