Devs say AI crawlers dominate traffic, forcing blocks on entire countries
1 min read
Summary
Software developer Xe Iaso publicised their problems with aggressive traffic to their Git repository service, mainly using Amazon bots that appeared to automate the process of accessing their site.
The developer attempted to put in place standard defensive mechanisms, but these were unsuccessful in curbing the traffic, so the site ultimately had to move its server to a VPN.
This highlighted a more significant problem for the open-source community, as aggressive AI crawlers are increasingly overloading community-maintained infrastructure, leading to perpetual distributed denial of service attacks on public resources.
Members of the Fedora Pagure project reported blocks had to be put in place stopping traffic from Brazil, after attempts to mitigate the number of bots failed.
GNOME GitLab also implemented the “Anubis” system which required bots to solve computational puzzles before they could access content, and found that only 3.2% of requests made it through this challenge.
KDE’s GitLab infrastructure was also affected by traffic from Alibaba, leading to it being temporarily taken offline.
