ShieldFont: Bludgeoning AI Scrapers That Disrespect Robots.txt | Hackaday

Skip to content

In the more innocent days of the World Wide Web you could simply put a robots.txt file in the root of your website that search engine indexing bots and similar would consult for the indexing wishes of the site owner. In this brave new world of LLM training data indexing such pleasantries are however rarely respected, leaving site owners to resort to increasingly more involved ways to bludgeon so-called AI scrapers, with ShieldFont being one of the most recent methods.

Its basic functioning is detailed in the white paper, explaining their use of ligatures. These are normally used to join multiple graphemes or letters into a single glyph which are rendered in the final text. By substituting about a quarter of the words in a text with such ligature-based versions in an intelligent, dictionary-based manner, the HTML version – as typically parsed by a scraper – will read as grammatically valid but nonsensical text, while the rendered font version will look normal.

Naturally, there are some disadvantages to this, such as screen readers for the visually impaired needing to also use the rendered font version, and it’s just as effective on legitimate search engine indexing bots. That said, if you apply this to static, archived content, or content marked as ‘do not follow’ in said robots.txt, then it might just be one way to make ChatGPT and friends spit out really funny output in the future now that the novelty of wood glue on pizza and eating rocks has somewhat worn off.

While LLM scrapers can adapt to this by also parsing the rendered text, this makes the scraping effort significantly more expensive. Together with maze traps like Nepenthes and Cloudflare’s offerings that seek to keep these scrapers busy scraping dynamically generated content through infinite linked pages, the tools available to combat the menace of these scrapers keep developing.

44 thoughts on “ShieldFont: Bludgeoning AI Scrapers That Disrespect Robots.txt”

This is a "screw the lot of you sub humans with disability" solution. This is terrible and I hope being so accessibility unfriendly it is never used.

Not just the b/Blind are affected. It spreads over a LOT of accessability "hacks" making the internet even more only suitable for the abled.

Anyone who was set a locked theme font is especially hit, so all the dyslexia font users, you’re SOL.

My own adjustments for poor eyesight needing high magnification and locked sans fonts is corrupted text. I also see the corrupted version on my tuned dark mode browser extensions, hear them in my reader, can’t use the reader version on my phone in safari, or see them in Links which I often use at CLI too.

Horrible. Unethical. Non-accessable friction that locks the web away from more and more with difficulty who try to function in an already hostile environment. The web has already become a mess of endless captchas after capthchas, but this is the worst possible thought out implementation for fixed font users, I can only hope this fails fast and dies in a fire in a huge way.

The worst part, those of us not fully abled see the corrupt shit-text with no understanding why such stupid words are there. There is nothing to explain – you are seeing shittext because we think the disabled are shit.

Report comment

Reply

Angry much??? :)

Report comment

Reply

Accessibility can be a difficult thing to achieve.

Intentionally reducing accessibility is infuriating.

Maybe if we plucked your eyes out for an afternoon you’d understand how hostile the world can be.

Report comment

Reply

In so many ways.

Report comment

Reply

Yeah, this decrease in accessibility is mentioned in passing in the article, but it should in the headline and the first, last, and most frequent thing addressed in the article.

I hate what AI is doing to the web. This is not how we change that.

Report comment

Reply

Not angry enough.

Report comment

Reply

Ah Tom, likely not enough so that people going "wow, that’s awesome" stop for a second and consider the deeper problems here around accessibility that’s taken decades to get in place being stripped away.

I didn’t even touch on how it potentially could hit YOU once the crims work out how utterly awesome it is for prompt injection attacks via email on your AI laden phone now that white-on-white text is mostly dealt with. Do we now need (insert company logo) to add ligature fonts into their end user protection guardrails?

Have you considered how ads using WebGL/WASM on other sites could abuse this to get around content moderation tools?

Not a competent English speaker? Maybe you Speak Spanish or you’re Canadian and don’t speak English. You reach a shittext page and you will see the bad translation. Here, hover over this google link to see https://shieldfont-org.translate.goog/?_x_tr_sl=auto&_x_tr_tl=es&_x_tr_hl=en-US&_x_tr_hist=true or...