GitHub - Wasserpuncher/readme-check: Run the console blocks in your README and check they still tell the truth -- I audited eight of my own repos and four of them lied · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Wasserpuncher
readme-check
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>1 Commit<br>1 Commit
.github/workflows
.github/workflows
readmecheck
readmecheck
tests
tests
.gitignore
.gitignore
LICENSE
LICENSE
README.md
README.md
pyproject.toml
pyproject.toml
View all files
Repository files navigation
readme-check
Your README makes promises. This runs them.
I audited the READMEs of eight of my own repositories, by hand, in one night.<br>Four of them lied.
Not maliciously. One said 7 passed when there were six. One pasted a console<br>transcript straight out of a test fixture — it advertised 1.2.3.4 as a real<br>nameserver, which is a placeholder constant, which means nobody had ever run the<br>tool they were documenting. One quoted a benchmark 2.5× more flattering than the<br>machine actually produces. One reported a binary size that was true when it was<br>written and stale within the hour, because the code got better and the README<br>didn't notice.
Every one of those was a number a reader could have checked in thirty seconds. So<br>now a machine checks them.
$ readme-check<br>✗ README.md:73 $ python -m pytest tests/test_elf.py -q<br>the README promises '7 passed'; nothing in the output says that<br>--- the README promises<br>| 7 passed<br>--- what actually happened<br>| 6 passed in 0.26s
1 of 4 promise(s) broken, 6 skipped
What it does
It finds the console blocks in your README, runs the commands , and checks<br>that what they print still contains what you said they would.
A console block is the only part of documentation that is falsifiable. Prose<br>("it's fast", "it handles Unicode") needs a human. But
```console<br>$ python -m pytest -q<br>39 passed<br>```
is a promise, and a promise can be broken.
What it does not do
It does not demand a byte-exact transcript. A README abbreviates, and it is<br>right to. pytest -q prints a line of dots, a percentage and a duration; your<br>README shows 39 passed, because that is the part that is a claim. A tool that<br>flagged that would flag every honest README in existence — and a tool that cries<br>wolf gets switched off, at which point it protects nothing at all.
So the question it asks is the narrow, useful one: does every line you promised<br>still appear, in order? Extra output is fine. Missing output is not.
Numbers that move
Timings are not promises. A laptop CPU throttles; the same benchmark swings 2–3×<br>between runs. Say so, with a ~:
$ python -m tinyjit bench<br>Native fib(30): ~19 ms<br>Interpreter: ~25000 ms
A ~19 accepts anything within 3× (--tolerance changes it). It will not accept<br>300. And the reader sees the tilde too, which is the whole point: a number the<br>author will not stand behind should not look like one they will. A README that<br>hides its uncertainty in a config file is still lying, just quietly.
For output that is genuinely unpredictable — a random root nameserver, a<br>timestamp — use ...:
delegation">$ python -m recursivedns example.com<br>resolving example.com from the root<br>... [.] example.com -> delegation
... on its own line skips any number of lines.
Blocks that should not run
Some blocks are instructions, not promises: they need root, or a tool the reader<br>has to install, or they talk to the network. Mark them, in an HTML comment that<br>renders as nothing:
```console<br>$ sudo ip tuntap add dev tun0 mode tun"> readme-check: skip=needs-root --><br>```console<br>$ sudo ip tuntap add dev tun0 mode tun
Other directives: `timeout=120`, `cwd=examples`.
## Verified
```console<br>$ python -m pytest -q<br>32 passed
The test suite is the audit. Every lie found that night is pinned as a test —<br>the drifting test count, the transcript from the fixture, the flattering<br>benchmark, the stale byte count — and so is the honest thing each one must not<br>be confused with: the abbreviated pytest...