ShieldFont: Bludgeoning AI Scrapers That Disrespect Robots.txt | Hackaday

Skip to content

In the more innocent days of the World Wide Web you could simply put a robots.txt file in the root of your website that search engine indexing bots and similar would consult for the indexing wishes of the site owner. In this brave new world of LLM training data indexing such pleasantries are however rarely respected, leaving site owners to resort to increasingly more involved ways to bludgeon so-called AI scrapers, with ShieldFont being one of the most recent methods.

Its basic functioning is detailed in the white paper, explaining their use of ligatures. These are normally used to join multiple graphemes or letters into a single glyph which are rendered in the final text. By substituting about a quarter of the words in a text with such ligature-based versions in an intelligent, dictionary-based manner, the HTML version – as typically parsed by a scraper – will read as grammatically valid but nonsensical text, while the rendered font version will look normal.

Naturally, there are some disadvantages to this, such as screen readers for the visually impaired needing to also use the rendered font version, and it’s just as effective on legitimate search engine indexing bots. That said, if you apply this to static, archived content, or content marked as ‘do not follow’ in said robots.txt, then it might just be one way to make ChatGPT and friends spit out really funny output in the future now that the novelty of wood glue on pizza and eating rocks has somewhat worn off.

While LLM scrapers can adapt to this by also parsing the rendered text, this makes the scraping effort significantly more expensive. Together with maze traps like Nepenthes and Cloudflare’s offerings that seek to keep these scrapers busy scraping dynamically generated content through infinite linked pages, the tools available to combat the menace of these scrapers keep developing.

37 thoughts on “ShieldFont: Bludgeoning AI Scrapers That Disrespect Robots.txt”

This is a "screw the lot of you sub humans with disability" solution. This is terrible and I hope being so accessibility unfriendly it is never used.

Not just the b/Blind are affected. It spreads over a LOT of accessability "hacks" making the internet even more only suitable for the abled.

Anyone who was set a locked theme font is especially hit, so all the dyslexia font users, you’re SOL.

My own adjustments for poor eyesight needing high magnification and locked sans fonts is corrupted text. I also see the corrupted version on my tuned dark mode browser extensions, hear them in my reader, can’t use the reader version on my phone in safari, or see them in Links which I often use at CLI too.

Horrible. Unethical. Non-accessable friction that locks the web away from more and more with difficulty who try to function in an already hostile environment. The web has already become a mess of endless captchas after capthchas, but this is the worst possible thought out implementation for fixed font users, I can only hope this fails fast and dies in a fire in a huge way.

The worst part, those of us not fully abled see the corrupt shit-text with no understanding why such stupid words are there. There is nothing to explain – you are seeing shittext because we think the disabled are shit.

Report comment

Reply

And I’m pretty sure I could bypass this in a few minutes by having the bot OCR screenshots from a headless browser and correlate the bounding boxes with the HTML elements.

So it’ll only affect disabled people, bots will get past it fine.

Report comment

Reply

I’m sure that any decent programmer (or even a suitable LLM) could code around this new font. Their trick, apparently, is in the font description, which is part of the font. Obtain the font, and you can work out the substitutions (and, because you don’t have the font natively on your system, web pages that use it /have/ to force you to download it).

Even the authors of TFP say

[blockquote]"The font must be sent to the browser so it can display the original words. Anyone who downloads it can therefore inspect it and work out the substitutions."[/blockquote]

Report comment

Reply

They can, but it is computationally expensive to do that on-scale.

Being able to suck up all these sites for pennies is what makes this current model make any sense.

Normal users don’t pull down and process 100k pages a second.

Making it 10x or 100x more work for any individual user is meaningless in their eyes.

Making it 100x more computationally expensive to scrape content is HUGE.

Report comment

Reply

…and it’ll also break copy/paste, and search on the page.

Yeah, no thanks.

Report comment

Reply

Yes and no – its pretty trivial to correct, so other than the additional hurdles faced being difficult, and something you might require able assistance to fix it isn’t really that big a deal – its also breaking normal able users from using the...