stillsane · PyPI
Skip to main content<br>Switch to mobile version
Warning
You are using an unsupported browser, upgrade to a newer version.
Warning
Some features may not work without JavaScript. Please try enabling it if you encounter problems.
Search PyPI
Search
stillsane 0.0.6
Know when your LLM app quietly stops working.
pip install stillsane
Copy PIP instructions
stillsane
A drift canary for deployed LLM apps and agents.
Your LLM app does not crash when it gets worse. It returns 200, latency looks<br>normal, the error rate is zero, and the output is quietly less correct than it was<br>last month. You find out when a user complains.
stillsane runs a small set of prompts against your live endpoint on a schedule,<br>compares each response to a stored baseline, and tells you when behaviour has<br>moved outside the range that probe normally varies by. It observes from outside,<br>over plain HTTP. There is nothing to instrument, no SDK to import, no account, and<br>no hosted service.
Status: early, v0.0.6. Everything described below works. The config format<br>may still change before 0.1. See Status.
What it looks like
A probe that extracts a total and a due date as JSON. The model was updated, and<br>it started explaining itself. The numbers are still correct and still present,<br>but every caller doing json.loads(response) is now throwing.
DRIFT extract_invoice @ prod<br>semantic_distance 0.133 band =1<br>5 other signal(s) unchanged
baseline (v1, 2026-07-31):<br>{"due_date": "2026-07-01", "total": 1240.50}<br>now:<br>Sure! I found the following:<br>{"total": 1240.5, "due_date": "2026-07-01"}<br>Happy to help with more invoices.
1 drift -> DRIFT
Exit code 1, so this fails a build. Nothing errored, nothing was slower, and no<br>conventional monitor would have noticed.
That band was not a number anyone chose. The baseline watched this<br>probe vary in whitespace, number formatting and key order, and learned how much<br>that is worth. The prose-wrapped version sits nearly nine times outside it.
The two bands still marked (floor) are the tool being honest about how it got<br>those: this example runs against a tidy mock whose lengths and token counts barely<br>move, so there was too little spread to measure and they fell back to a built-in<br>floor. stillsane says which of the two happened rather than presenting a defaulted<br>number as a measured one. stillsane bands reports it in full.
You can run exactly this in about thirty seconds, with no API key, from<br>examples/invoice-extract/.
Quickstart
pip install stillsane
stillsane init # write a starter config, then point it at your endpoint
stillsane baseline # capture what "normal" looks like. Explicit, never automatic.
stillsane check # compare against it. Non-zero exit on drift.
Put stillsane check on a schedule in CI and you are done. See<br>In CI for a workflow you can copy.
The problem
Three ways an LLM app degrades without ever erroring:
The provider changed the model. Hosted providers update the model behind an<br>endpoint without changing the version string. The same call can return<br>meaningfully different output months later, and this is entirely outside your<br>control.
Someone edited a prompt. A tweak to a system prompt or a tool description<br>breaks a behaviour three steps downstream that no test covers.
Retrieval drifted. RAG context shifts, retrieval quality slides, answers get<br>vaguer.
In all three cases the app keeps working. Latency is fine. Error rate is zero.<br>Quality is worse. Nobody gets paged.
This is the same failure mode as a drifting sensor on unattended infrastructure: a<br>crash is loud and you find out immediately, but a plausible-looking wrong number<br>gets believed. The fix there was synthetic monitoring: walk the whole pipeline on<br>a schedule, validate what comes back, alert before a human notices. stillsane is<br>that, pointed at an LLM.
Should you use this?
Probably not, if you already have something:
You want to know whether a prompt is good before you ship it. Use a<br>pre-ship eval framework. There are several good open-source ones, and stillsane<br>will not help you. That is not false modesty. Pre-ship evaluation is a different<br>problem, and tools built for it solve it better than a tool of this scope ever<br>will.
You already run a tracing or eval platform. You have evaluator scores on<br>real production traffic. Watch those. Adding stillsane buys you<br>provider-fingerprint watching and not much else.
You are willing to instrument your app. Then instrument it. Tracing sees<br>every real request; stillsane only ever sees the handful of probes you wrote.<br>That is a genuine and permanent disadvantage.
stillsane is for the case none of those cover: you shipped an agent or an LLM<br>pipeline, quite possibly mostly AI-assisted, you have no evals and no<br>observability, you are never going to add a tracing SDK, and right now you would<br>find out about degradation from a user complaint.
If that is you, this is a config file and one command.
How it compares
stillsane<br>Pre-ship eval frameworks<br>Tracing...