CHAOSS Metrics in 2026 | Andrew Nesbitt

The CHAOSS project has spent the last eight years writing down careful, implementation-agnostic definitions for the things people measure about open source projects: how many issues get opened, how long they take to close, how many distinct people commit, how stale the dependencies are. The point of writing them down is that two dashboards computing “issue response time” should at least be computing the same thing, and the released metric set has become the closest thing the field has to a shared vocabulary.

Most of those definitions were drafted between roughly 2018 and 2023, against a background where contributions were produced at human speed, security advisories arrived a few times a year per ecosystem, and a maintainer who stopped maintaining usually stopped committing too. All three of those conditions are changing, and a lot of the metric definitions encode them without saying so. The common thread is that most of the catalogue counts repository events, and event counts were informative because producing an event used to cost a person something. An issue cost someone ten minutes to write up, a PR cost someone an afternoon. That cost is being removed from one side of the interaction and not the other, so the counts increasingly measure how much of the cheap thing is being pointed at a project rather than anything about the community behind it.

I went through the current released set, in roughly the order the catalogue groups them, and tried to sort out which metrics depend most on those assumptions. This is the same territory as The Mismeasure of Open Source and Centrality is not vitality, but applied to a specific published metric set rather than to research practice in general.

Activity counts

The largest group of CHAOSS metrics are counts and rates of repository events: Issues New, Issues Closed, Change Requests, Change Requests Accepted, Code Changes Commits, Code Changes Lines. These were always proxies, but the thing they were a proxy for was reasonably stable: an issue meant a person had a problem and took the time to write it up, a change request meant a person had done some work and wanted it reviewed.

When a measurable share of new issues are model-generated bug reports of varying accuracy, and a measurable share of change requests are agents proposing fixes that may or may not compile, the counts decouple from the thing they were standing in for. Daniel Stenberg has been writing up what this looks like from the receiving end on curl. The top-level numbers go up, the maintainer time per item goes up, and the proportion of items that represent something a user actually needs goes down. None of the count metrics can distinguish those three movements.

The original framing of Change Request Acceptance Ratio reads a falling ratio as a sign the project is becoming less welcoming to contributors. On a project receiving a stream of low-effort generated PRs the reading inverts: a falling acceptance ratio is the maintainers doing their job, and a high one might mean they’ve given up reviewing.

Responsiveness

Issue Response Time, Time to First Response, Issue Resolution Duration, and Time to Close are framed as measures of how attentive the community is, and they’re affected on both the input and output sides. A project that previously received twenty issues a month and now receives two hundred will see its median response time rise even if the maintainers are doing exactly the same amount of work, because the denominator is being set by whoever is filing issues rather than by anything the project controls. Going the other way, projects that wire up an AI triage bot to post an initial classification on every new issue will see Time to First Response drop to seconds. The metric definitions do say to filter out bot responses, but that filter assumes bots are labelled, and the whole difficulty with the current generation is that the account is a normal user account and the response reads like prose.

Issue Age trends upwards on any project that decides the right response to a flood of generated issues is to leave them open rather than spend maintainer time closing them one by one. That’s a defensible triage decision and the metric reads it as decline.

Contributor identity

Contributors, New Contributors, Occasional Contributors, Conversion Rate, and Contribution Attribution all rest on the assumption that a contributor identity maps to a person, and that a new identity appearing means the community has reached someone it hadn’t reached before.

An agent that opens PRs from freshly created accounts, or a single person running a fleet of agents each with its own token, registers as a burst of new contributors. A project’s New Contributors chart can spike in the same month its only actual maintainer stops reading notifications. Conversion Rate, which tracks how many first-time contributors come back for a second contribution, was designed to measure whether...