Who Comments on Rules That Change? | Federal Regulatory Tracker

Skip to main content

Here is a study that would be easy to write and also wrong.

Take every proposed rule you can pair with its final version. Record which kinds of organizations filed comments — companies, trade associations, advocacy groups, unions, state agencies. Then record whether the rule changed on its way to finalization. Cross-tabulate and then announce which constituencies comment on rules that change. Nothing causal suggested, to be clear, but interesting.

I ran that study on 572 paired rulemakings across 31 federal agencies, and the raw numbers cooperate beautifully. Rules where a state or local government filed a comment are 13.1 percentage points more likely to show a measured change between proposal and finalization. Rules where a regulated business commented are 8.1 points more likely. Academic commenters, 9.3 points. Every single constituency I could measure points the same direction: where they show up, rules change. Hmm.

It is a tidy result and it dissolves completely upon first contact with a control variable.

What is actually being measured

FRTracker extracts obligations — actor, deontic, action, object — from the regulatory text of proposed and final rules, then matches each proposed obligation forward into the final rule. Every proposed obligation ends up labelled unchanged, modified, removed, or split. For each rulemaking I have two outcomes: whether anything changed, and what share of the proposal's obligations changed.

The universe here is a frozen one. The obligation matcher was last regenerated as a single audited run, and every rulemaking it touched left a receipt recording the inputs, the code commit, and the resulting counts. I built the cohort from those receipts rather than from the current database, and verified that the delta counts still reconcile exactly — 1,486 of 1,486 rulemakings, 28,601 matched obligations on both sides. Twenty-one rulemakings whose obligation match produced nothing at all were dropped as undefined, rather than being silently recoded as "no change." Of what remained, 572 had organizational commenter data. That is/was the universe of this back of the envelope "study." I use that term very loosely.

Commenters were classified into broad constituencies (with lots of room for edge-case argumentation) — regulated business, trade association, public-interest NGO, labor, government, professional association, academic, individual — using a classifier built for a different project months earlier. A hand audit of thirty classified organizations found twenty-nine correct. Neat.

That taxonomy has a real limit though, and it bounds what the null below can mean. About 29 percent of organizational commenters remain unclassified (8,554 organization rows in total), and on 54.7 percent of rules the majority of identified commenters are unclassified. So, to be clear, the defensible claim is not that constituency does not matter. It is that constituency does not separate at the resolution this taxonomy can currently achieve — a statement as much about my measurement precision as about the regulatory world. A sharper classifier, a better PI, or something built from filer identities rather than organization names, could find structure this one cannot see. But you go to war with the Andrew you've got, not the Andrew you want.

The variable that eats the result

The problem with "did anything change" as an outcome is that it is very nearly a function of how much there was to actually change.

matched proposed obligations1–23–56–1011–2526–100101+share of rules where something changed0.3910.7360.6600.9000.9451.000 rulemakings871109413012823

The gradient is not perfect — 3–5 obligations sits above 6–10 — and the 1.000 rests on only 23 rulemakings, so read it as a strong tendency at the top of the range rather than a law or anything approaching a law.

A rulemaking that proposes more than a hundred obligations always changes at least one of them, in the universe of this study. A rulemaking that proposes two, frequently changes neither. And organized constituencies participate disproportionately in the large ones. Where a regulated business commented, the median rulemaking had 14 matched obligations; where none did, 8. Where a state or local government commented, 22 against 8. Where an academic commented, 21 against 9, on rulemakings with a median of 4,957 public comments against 31 (47 rulemakings with an academic commenter, 525 without).

So the raw finding is not "these constituencies comment on rules that change." It is "these constituencies comment on the big rules, and big rules always change something." Once I hold constant the agency, the year, the number of proposed obligations, the total comment volume, the length of the proposed regulatory text, and whether the rule was designated significant, every one of those associations stops being distinguishable from zero:

constituency presentraw gap...