Measuring Personal Data Accuracy | Schema-Based Classification Insights | Regixo

Skip to content

GitHub<br>Read the docs

GitHub<br>Read the docs

Somebody asks for "a list of everywhere we store personal data."

If you are the engineer that request lands on, you know the week that follows: a spreadsheet, a<br>tour of every database you have credentials for, and a document that is out of date by the next<br>migration.

Here is the premise I built on: you rarely need the data for this job. You need to know<br>where personal data lives, and most of that is already written down — in the schema. A<br>column called iban in a table called payroll tells you what you need<br>without reading a row. So the tool reads structure only — table and column names and types, never<br>the values — and the scan talks to nothing but the sources you configure.

That constraint is not a privacy pose. A mapping tool that reads production data becomes the<br>thing your security review is about — one more system with read access to everything. A scan that<br>touches only information_schema is one you can run without a meeting, and running it<br>without a meeting is the only way the map stays current.

The interesting question is how far names alone actually get you. So I measured it.

The naive version fails loudly

The obvious implementation is substring matching against a word list, and it is why this<br>approach gets dismissed. trace_id contains "race". A substring matcher flags it as<br>ethnicity — and that is not an ordinary mistake. Ethnicity is special-category data under GDPR<br>Article 9, the kind of flag a compliance team has to stop and clear. A tool that cries "ethnicity"<br>at a tracing header costs them a week instead of saving one.

Substring vs whole-token — three trap columns

Substring<br>trace_id<br>✗ flagged as ethnicity<br>"race" found inside "trace"

Whole-token<br>traceid<br>✓ clean<br>no rule matches either token

Whole-token<br>raceconditioncount<br>✓ clean<br>the rule is =race — a column named exactly race

Whole-token<br>geneticalgorithmgeneration<br>✓ clean<br>the genetic rules are sequences: genetic_data, genetic_test

Tokenization is the whole trick. A column name is split on underscores,<br>dashes, digits and camelCase; rule terms match whole tokens or consecutive sequences — never<br>fragments inside a word.

So matching is whole-token, never substring. A column name is tokenized — split on underscores,<br>dashes, digits and camelCase — and a rule term matches whole tokens or consecutive token<br>sequences.

There are four term forms in the rule syntax, and the loosest one is deliberately caged.<br>name matches a whole token. date_of_birth matches a consecutive<br>sequence. =race matches an exact full name. And religio* matches a token<br>stem — the form closest to substring matching, allowed on the deny list only : the<br>special-category and criminal-offence packs, where missing<br>religionszugehoerigkeit is worse than over-flagging it. Ordinary personal data never<br>gets a stem rule. Precision is the point of the whole exercise, and the one place recall is bought<br>at its expense is the place a miss would be an Article 9 miss.

Rules, not a model

The classifier is a rule engine, not a model, and that is a choice with an argument behind<br>it.

Every flag says how the column was recognised — the matched term, by field name — so a reviewer<br>can audit the decision in one look. When it is wrong, the fix is a one-line rule change with a<br>test, not a retraining run. It runs locally in milliseconds with no model call, which the<br>metadata-only architecture requires anyway. And a rule list can be read, end to end, by the person<br>accountable for the result — which matters when the output feeds a record someone signs.

The cost of rules is recall on creative names. That cost has to be measured, not asserted.

The benchmark — with the misses printed

The benchmark is held out from development: schemas the rules were not written against,<br>labelled by hand. The English set is drawn from real open-source products — OpenMRS<br>(healthcare), Odoo and ERPNext (ERP and HR), Spree (e-commerce) — 349 labelled columns of<br>real-world naming. The German, French and Dutch sets are authored, because the approach collapses<br>if it only works in English and I wanted that failure on paper if it existed.

Recall on ordinary personal data, by benchmarked language

100<br>75<br>50<br>25

floor 0.85

92.5%<br>87.2%<br>95.0%<br>94.1%

German<br>English<br>French<br>Dutch

The gold line is the CI floor: 0.85 recall on ordinary personal data per<br>benchmarked language. A rule change that drops any language below it — or puts one false<br>special-category flag on the trap set — fails the build.

LanguageLabelled columnsOrdinary personal dataCaughtRecallPrecision

German55403792.5%100%<br>English34914112387.2%94.4%<br>French30201995.0%100%<br>Dutch35171694.1%100%

And the trap set: 34 infrastructure and telemetry columns with no personal data by<br>construction, where the required score is zero false special-category flags. The current run:<br>zero.

The trap set — infrastructure columns that must...