Summary
In the more innocent days of the World Wide Web you could simply put a robots.txt file in the root of your website that search engine indexing bots and similar would consult for the indexing wishes of the site owner. In this brave new world of LLM training data indexing such pleasantries are however rarely respected, leaving site owners to resort to increasingly more involved ways to bludgeon so-called AI scrapers, with ShieldFont being one of the most recent methods. Its basic functioning is detailed in the white paper, explaining their use of ligatures. These are normally used to join multiple graphemes or letters into a single glyph which are rendered in the final text. By substituting about a quarter of the words in a text with such ligature-based versions in an intelligent, dictionary-based manner, the HTML version – as typically parsed by a scraper – will read as grammatically valid but nonsensical text, while the rendered font version will look normal. Naturally, there are some disadvantages to this, such as screen readers for the visually impaired needing to also use the rendered font version, and it’s just as effective on legitimate search engine indexing bots. That said, if you apply this to static, archived content, or content marked as ‘do not follow’ in said robots.txt , then it might just be one way to make ChatGPT and friends spit out really funny output in the future now that the novelty of wood glue on pizza and eating rocks has somewhat worn off. While LLM scrapers can adapt to this by also parsing the rendered text, this makes the scraping effort significantly more expensive. Together with maze traps like Nepenthes and Cloudflare’s offerings that seek to keep these scrapers busy scraping dynamically generated content through infinite linked pages, the tools available to combat the menace of these scrapers keep developing. This is a “screw the lot of you sub humans with disability” solution. This is terrible and I hope being so accessibility unfriendly it is never used. Not just the b/Blind are affected. It spreads over a LOT of accessability “hacks” making the internet even more only suitable for the abled. Anyone who was set a locked theme font is especially hit, so all the dyslexia font users, you’re SOL. My own adjustments for poor eyesight needing high magnification and locked sans fonts is corrupted text. I also see the corrupted version on my tuned dark mode browser extensions, hear them in my reader, can’t use the reader version on my phone in safari, or see them in Links which I often use at CLI too. Horrible. Unethical. Non-accessable friction that locks the web away from more and more with difficulty who try to function in an already hostile environment. The web has already become a mess of endless captchas after capthchas, but this is the worst possible thought out implementation for fixed font users, I can only hope this fails fast and dies in a fire in a huge way. The worst part, those of us not fully abled see the corrupt shit-text with no understanding why such stupid words are there. There is nothing to explain – you are seeing shittext because we think the disabled are shit. Angry much??? :) Accessibility can be a difficult thing to achieve. Intentionally reducing accessibility is infuriating. Maybe if we plucked your eyes out for an afternoon you’d understand how hostile the world can be. In so many ways. Not angry enough. And I’m pretty sure I could bypass this in a few minutes by having the bot OCR screenshots from a headless browser and correlate the bounding boxes with the HTML elements. So it’ll only affect disabled people, bots will get past it fine. “This is a “screw the lot of you sub humans with disability” solution.” As another one with vision issues, I agree. I finally had my cataracts removed but retina problems are mostly not fixable. I would include blurry phone optimized images, along with hard coded background colors, cough HAD. I heard at one time from a designer I know that contrast and color trade offs were taught in schools. Line has to go up, quality are not our job. The world of cyberpunk can’t come soon enough. Back then it as only six million and one’s implants came with cool sound effects. BTW OCR breaks their model.