The web’s newest weapon against AI scrapers is a font - Ars Technica
Skip to content
AI
Biz & IT
Cars
Culture
Gaming
Health
Policy
Science
Security
Space
Tech
Forum
Subscribe
Story text
Size
Small<br>Standard<br>Large
Width
Standard<br>Wide
Links
Standard<br>Orange
* Subscribers only
Learn more
Pin to story
Theme
Search
Sign In
Sign in dialog...
Text<br>settings
Story text
Size
Small<br>Standard<br>Large
Width
Standard<br>Wide
Links
Standard<br>Orange
* Subscribers only
Learn more
Minimize to nav
AI companies’ penchant for scraping through large swaths of the public web in search of valuable training data has already led to lawsuits and technical fixes aimed at stopping the practice. Now, a pair of designers is hoping to stymie these scrapers with a new font designed to offer people a perfectly readable webpage while serving scrapers a subtly edited, nonsensical version in the underlying HTML.
ShieldFont, as designers Isaque Seneda and Gabriel Abrucio write in a recent white paper, was made to offer web publishers “a practical opt-out from unauthorized AI training and [to] disrupt what is collected when that choice is ignored.”
When is a horse a potato?
The font is based around ligatures, a long-standing feature of many fonts that is usually used to replace certain letter pairs with a more readable version when they’re smushed up next to each other. With ShieldFont, though, those ligatures are instead used to replace entire words with others in an attempt to destroy the text’s value to scrapers. This substitution only happens when the font engine draws the page onscreen, meaning scrapers that simply download plaintext source code get an altered version that end users never see.
When it comes to fooling AI scrapers, though, not all ligature-based word replacements are created equal. Simply replacing common words with synonyms or antonyms would be too easy for a smart scraper to reverse. On the other end, replacing words with completely unrelated gibberish could lead to easier detection (and potentially circumvention) by a smart scraping filter.
The original content, as seen by the end user.
ShieldFont
The original content, as seen by the end user.
ShieldFont
The raw HTML, before being processed by ShieldFont, which gets scraped by bots.
The raw HTML, before being processed by ShieldFont, which gets scraped by bots.
The original content, as seen by the end user.
ShieldFont
The raw HTML, before being processed by ShieldFont, which gets scraped by bots.
So ShieldFont replaces words with similar parts of speech that occupy a completely different informational context—swapping “horse” with “potato,” for instance. The result is a scrapable sentence that looks semantically correct but has a completely altered meaning. Thus, even altered pages that get through a scraper’s quality filter will contain scrambled informational content that can poison a training data set.
After refining their word-swapping dictionary over three months, the ShieldFont creators ended up with a list of nearly 12,000 common words that can be replaced with ligatures. To avoid easy detection, the font lets publishers increase the underlying chaos by choosing from three different potential mappings for each word replacement, with the ability to encode their own and/or swap mappings from paragraph to paragraph.
On average, ShieldFont ends up replacing 24.5 percent of all words on a page, including 45.8 percent of all “content words,” marring the meaning of anywhere from 31 to 56 percent of individual passages (depending on the corpus studied). In testing on six publicly available scraper pipelines, the ShieldFont authors say that over 90 percent of pages that would otherwise be accepted by scrapers are rejected by the quality filter after these word replacements.
Of the small subset of pages that still get accepted after ShieldFont is applied, nearly 20 percent of the component words are what the authors refer to as “training-time garbage: real English, correctly spelled, asserting nothing true.” This means that both dropped and kept ShieldFont pages can both be useful in stopping AI scrapers: “Dropped means they did not get your work. Kept means they got something wrong,” the authors write.
A game of scrape and mouse
While ShieldFont pages can still be read perfectly well by average humans, there can be some side effects when using the font on published webpages. Search engines, screen readers, copy/paste tools, and translation software can all get tripped up by the altered HTML, making the page a little less useful to your intended audience.
Tell me more about the very interstate southern engineer with the sofa car…
Credit:<br>ShieldFont
Tell me more about the very interstate southern engineer with the sofa car…
Credit:
ShieldFont
ShieldFont isn’t a foolproof defense, either. Any page that’s readable by a human could also be correctly interpreted by an AI scraping tool that simply renders the full webpage...