abadideathe recent burst of companies flagrantly disregarding robots.txt are eating their seed corn — the entire point is that as long as everyone shows some basic manners to other internet hosts, there’s no need for some international regulatory body to get involved and make everyday matters difficult and expensive for everyone https://arstechnica.com/information-technology/2025/08/ai-site-perplexity-uses-stealth-tactics-to-flout-no-crawl-edicts-cloudflare-says/
abadideaone time I was on vacation and a Microsoft employee contacted me personally with an emergency request to get ahold of my coworkers and tell them to knock off whatever they were doing because it was overwhelming a certain chunk of Microsoft infrastructure.
This has always been how the internet has stayed online — direct interpersonal manners and the understanding that being a “you can’t MAKE me show restraint” dick about it will have dire long term consequences for everyone. The AI upstarts who think they’re too special to show manners to their peers are fucking up the common good for everyone.