Loss Curves Lie: Building a Deterministic Linter for ML Training Runs

Loss Curves Lie: Building a Deterministic Linter for ML Training Runs A run learning pure noise reduced its loss by 62% and looked textbook-healthy. trainproof catches the training failures that can be caught — and is honest about the ones that can't. trainproof v0.18.0active Deterministic PASS/WARN/FAIL verdicts across a training run's whole life — before it starts (dataset, tokenizer, and an environment preflight that catches a stack which will not import, an unreadable checkpoint, or too little system RAM), a live guardian during training, a post-run autopsy after, and a baseline compare. Reads five log formats including TensorBoard event files, decoded from the wire format with no torch, tensorflow or tensorboard installed. No ML judging ML: every verdict is a rule that fires or doesn't, every finding cites its exact numbers, and a check that could not run is reported as NOT-CHECKED rather than as a pass. Install: pip install trainproof Latest Documented Release: v0.18.0 · 2026-08-09 Tests: 258 passing tests Releases: 18 releases License: MIT Built on: ttsproof

Repo (GitHub)PyPI Objective checks: catches a cross-entropy ignore_index that is also a valid output class - a collision that silently deletes a target from the loss while leaving the loss curve indistinguishable from a correct run (0.0035 vs 0.0034) trainproof doctor . — zero-config autopsy: discovers every training log in a directory, judges them, prints a triage-sorted summary Live guardian: a one-line Hugging Face callback; opt-in stop_on_fail aborted a diverging Qwen2.5-3B QLoRA at step 20 of 300 in a real demo Preflight: lints dataset + tokenizer before a single GPU-second (malformed JSONL, empty rows, duplicates, missing eos/pad, over-length) compare catches shuffled-labels corruption that single-run loss rules cannot — the relative loss floor sits ~6× a known-good baseline Stable rule IDs (TP-DIVERGE, TP-OVERFIT, TP-DEAD-RUN, …) documented in RULES.md, plus --json output built for CI and AI coding agents CONTRACTS.md pins the promises — exit codes, JSON schema policy, rule-ID stability and a verdict-stability guarantee; SARIF 2.1.0 output turns findings into GitHub PR annotations A PASS names every check that ran and every check it skipped, with the reason — a skipped check is never reported as a passed one Grounded in an 18-run fault-injection study (Qwen2.5-3B QLoRA, RTX 5080) — six configurations at three seeds each, every real log shipped in-repo — plus a 9.8-hour Coqui XTTS fine-tune that diverged on its own, the first shipped failure nobody injected Reads five log formats including TensorBoard event files, decoded from the wire format with no tensorflow, tensorboard, protobuf, torch or numpy — validated byte-exact against tensorboard's own EventAccumulator, and the rules are exercised on real runs from three frameworks: HuggingFace, Coqui XTTS and PyTorch Lightning Recent Releases v0.18.0 2026-08-09 — v0.18.0 - the objective release v0.17.0 2026-08-02 — v0.17.0 — the lint gate, and one log that used to vanish v0.16.0 2026-08-02 — v0.16.0 — the rule registry (no behaviour change) v0.15.0 2026-08-01 — v0.15.0 — the before-the-GPU release Full changelog on GitHub →

I kept losing GPU hours to runs that were already dead. A fine-tune that had quietly gone to NaN somewhere after step 300 and trained on garbage the rest of the night. A run that was training at learning-rate-zero the whole time and updating nothing. A dataset with a few hundred broken rows I only discovered at the end. Every time, the pattern was the same: the run looked like it was working, and nothing told me otherwise until the hours were already spent. So I built a linter for training runs — trainproof — and then I spent days trying to prove it wrong. pip install trainproof trainproof doctor . # zero-config: discovers and judges every training log it findsRepo: github.com/Mormolykos/trainproof (MIT) The philosophy: no ML judging ML The most important decision in trainproof is what it isn't. There's no model scoring your run. No "87%-confidence this looks unhealthy." Every verdict is a deterministic rule that either fires or it doesn't, and every finding cites the exact numbers behind it. When it can't be sure, it says so instead of guessing. That constraint is the whole point. A reliability tool that hallucinates is worse than no tool, because now you don't trust your own alarms. trainproof's engine is IO-free and doesn't even import torch — it reads plain logs and applies rules. You can read every rule and know exactly why it fired. Validating the rules: controlled fault injection You validate a detector by feeding it faults whose answer you already know, then measuring what it catches. One base setup — a Qwen2.5-3B QLoRA on an RTX 5080 — run six ways: healthy — the control. lr_hot — learning rate cranked 100× too high. lr_zero — learning rate at zero. fp16_nan — an fp16 overflow that NaNs the...

Loss Curves Lie: Building a Deterministic Linter for ML Training Runs

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

NHS apologises and admits Palantir have access to identifiable patient data

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills