Asked to scan Erlang/OTP, cloc kept breaking — what it took to measure it properly

The job

Measure six and a half years of erlang/otp — seven windows, 2020 to July 2026, each defined by two public commit hashes. OTP is close to worst-case by construction: two million statements of Erlang, a C/C++ virtual machine with a JIT whose vendored, machine-generated tables hold quarter-megabyte statements, leex/yecc grammars, SNMP MIBs, and test suites deliberately built to be hostile.

Where cloc breaks

cloc 2.10 is the standard answer, so it ran first, on the exact window hashes. Three things happened, all reproducible:

Three of seven windows abort outright. The 2020, 2023 and 2025 diff runs fail with “Failed to create tarfile of files from git” — six runs out of six. The trigger is legitimate repository content: OTP’s test suites contain Unicode-named directories (sftp_tar_test_data_高兴, an emoji-named directory under erts/test/) that cloc’s git-archive step cannot package unless you know to disable git’s core.quotepath.

4,046 files are set aside silently — 2,764 of them “language unknown”. You only find out if you know to ask for the --ignored log. There is no coverage line telling you what fraction of the tree you actually measured.

Its Erlang is two regular expressions, and real Erlang defeats them. Erlang’s % appears inside strings, quoted atoms, character literals, and (since EEP-64) triple-quoted doc strings. On a 16-line golden file with known-by-construction truth (14 code lines, 6 comment lines) cloc reads 14/2; on OTP’s real argparse.erl it classifies code inside -doc """…""" blocks as comments.

Where it does run, the numbers aren’t churn. It diffs physical lines in changed files only: no whole-codebase denominator, and no way to tell an edited statement from a reformatted one — a pure reformat of one real file produced 80 lines of “churn” and exactly 0 changed statements.

None of this is a criticism of cloc at its own job — counting lines, quickly, in 397 languages. The problem is that teams quote its diff numbers as churn, and on a real platform that substitution fails.

Measuring it properly broke our tool too

We build a churn engine (CodeDelta), so we scanned OTP with that instead — and the honest version of this story is that OTP beat us up as well, twice.

Our first scan was wrong. It predated our Erlang support, measured the C virtual machine and documentation markup, and its own coverage line reported the verdict: 55.6% of files measured. A churn report on 55.6% of a codebase is not a churn report on the codebase. We withdrew it, built Erlang support against the language as OTP actually writes it (the %-in-strings problem above, EEP-64, the .app/.appup/grammar/MIB family), and re-ran at 93.0% coverage , with the remaining 7% named in the output: images, compiled .beam artifacts, test certificates.

Then the JIT tables found a bug in our engine. OTP vendors asmjit, whose machine-generated instruction database serialises the complete x86 instruction set as one C++ array initializer — a single logical statement of 220KB. Past our 64KB statement buffer, two defects were hiding: a change beyond the cap in an oversized statement went undetected, and a brace-classification state variable froze when the buffer filled, so an oversized initializer could vanish from statement-level churn entirely. Both fixed, regression-guarded, and every figure re-derived: three windows moved, by exactly one statement each. We publish the correction rather than hide it.

What came out

One command per window, whole-platform accounting, logical statements as the unit. REWORK is the share of churn that edits an existing statement in place, rather than adding or deleting one.

WindowChurn (LLOC)REWORKCodebase at end (LLOC)

2020722,6715.9%2,868,225<br>2021262,5069.7%3,065,399<br>2022137,82814.7%3,083,293<br>2023178,8238.4%3,193,943<br>2024362,5437.1%3,124,040<br>2025224,7009.3%3,302,795<br>2026 (Jan–Jul) 133,428 22.6% 3,355,831

Expansion years run 5.9–9.3%; the 2022 consolidation ran 14.7%. 2026 stands alone at 22.6% — nearly one churned statement in four is an in-place edit of existing code. That is OTP&rsquo;s deepest repair regime in the series, on a platform that grew +17% over the six and a half years: the numeric signature of intensive, hands-on maintenance of a mature codebase. It is also a number no line counter can produce, because producing it requires statement identity, whole-tree coverage, and a parser that reads Erlang as written.

Reproduce it

All seven window hash pairs, the full method, per-window cloc output (including the completed runs with core.quotepath disabled) and the engine-bug write-up are in the full report. Nothing is sampled and no ML is involved in the measurement — exact, repeatable diff arithmetic on statements.