Our PII model comparison reversed when we changed the dataset | Velum

Privacy-first AI infrastructure.

Back to Blog

We build a tool that finds personal data in text and masks it before it reaches a model. In April, OpenAI released Privacy Filter, an open-weight model that does the detection half of that job. We already run GLiNER for the same purpose, so we measured one against the other.

The first answer we got was clear and wrong. This is how it went wrong, because the mistake is easy to make and we came within about a day of publishing it.

The setup

Two token-level detectors, same harness, same machine, same metric.

The metric matters, so: we score surface leak . Run the whole pipeline end to end, then ask whether each thing that should have been masked is still sitting in the output as a literal substring. Not span F1, not per-token accuracy. The reason is that our pipeline merges overlapping detections into one span, and span-level scoring undercounts that: when a merged span covers two adjacent expected spans it can only be credited with one, so the other reads as a miss even though the text is fully masked. Span scoring told us one configuration was leaking 2.9% on a corpus where nothing leaked at all. We tune on span F1 because it is more sensitive; we report surface leak because it is what a user could check.

The first result

Our own evaluation corpus is 1,800 examples across nine languages, generated by our own builder and validated by our own oracle. On it, our existing setup leaks 0.04% and swapping in Privacy Filter made things worse, 3.28%, concentrated in Catalan and Spanish person names.

That is a home-field result and we knew it, so we needed a corpus we had not written. We picked the obvious one: PII-Masking-300k , the standard reference in this space.

On that corpus the ordering flipped hard. Privacy Filter alone beat our existing setup by roughly three to one. Same code, same harness, same metric, opposite conclusion.

We drafted that up. A model from a large lab, beating what we ship, on a neutral dataset. It is a good story and it was nearly a public one.

Why it was wrong

PII-Masking-300k is the corpus OpenAI reports Privacy Filter against.

That is not an accusation of anything. Nobody outside OpenAI can say whether it is in the training data, and we are not claiming it is. The point is narrower and it does not require contamination: a corpus that a model's authors selected as their reference is a corpus that model was steered toward , through architecture choices, label taxonomy, threshold calibration and everything else that gets tuned while looking at a benchmark. It was Privacy Filter's home field in precisely the way our corpus is ours.

We had swapped one biased corpus for another and called the second one neutral.

So we picked a third: Gretel's synthetic PII corpus. Nine hundred contracts and statements across six languages, averaging about 1.2k characters, chosen by neither party.

On that corpus, Privacy Filter alone is worse than what we already ran: 26.6% against 23.5%.

The three-to-one win did not survive the dataset change. Nothing else changed.

The licence problem, which we should have hit first

There is a second reason not to use PII-Masking-300k, and it is the one we should have checked before running anything.

Its licence grants access "exclusively for academic research and non-commercial purposes," requires written permission for redistribution or derivative works, and states that no licence is available to companies without prior discussion. We are a commercial product. Benchmark figures derived from it, published on a company blog, are not ours to publish.

That is why this post describes the reversal in prose and gives you exact numbers only from the Apache-2.0 corpus. It is an awkward shape for an article and it is the correct one.

If you are evaluating PII models, check this before you build an argument on a dataset. The permissively licensed options we found usable were the Gretel corpus above and beki/privy under MIT.

What our own corpus was hiding

The uncomfortable part is not what the third corpus said about Privacy Filter. It is what it said about our benchmark.

On our corpus, every configuration we ship scores between 0.0% and 1.1% leak. That reads like a strong result. It is closer to a measurement failure: the corpus cannot distinguish our configurations from each other. On the Gretel corpus the same three score 51.2%, 23.5% and 15.1%.

Both are real numbers. The first describes clean text in languages we have specifically tuned for, the second describes document formats nobody tuned for. But only one of them can tell you whether a change you just made was an improvement, and it is not the one we had been using alone.

Two bugs that needed a second model to surface

Running two detectors over the same corpora exposed problems one detector had been hiding.

A policy interaction that leaked every date of birth. Privacy Filter has no...