AI crawler wars threaten to make the web more closed for everyone
1 min read
Summary
The popularity of AI has thrown the internet’s ecosystem of crawlers into disarray, creating what some sources refer to as ‘the AI crawler crisis’
Websites now fear that using AI crawlers that mine data indiscriminately for content may allow competitors to undercut their economic interests, and are increasingly restricting access to this data source for AI companies
In response, major AI companies like OpenAI and Anthropic have been accused of overwhelming websites and ignoring restrictions placed on their data collection, in order to gain as much data as possible
To counter this, websites are now fighting back with a trifeca of lawsuits, legislation and computer science, tightening the data faucet at the source; web crawlers, by erecting barriers that make sites harder to navigate, or which require crawlers to register to access.
The end result is a fragmented internet poised to restrict the open and transparent access that has until now been taken for granted.