Mar 28, 2025

🪤 Devs vs. AI Crawlers: Anubis, Bleach Articles & Vengeful Vibes

FOSS devs are fighting back against aggressive AI web crawlers with clever, chaotic, and sometimes vengeful tricks.

Rafelia

AIWebCrawlersOpenSourceAnubisNepenthesFOSS

386

2025-03-28 05:30 +0530

🤖 AI bots: “We’re here to learn!”

Open-source devs: “You’re about to learn PAIN.” 🪓🔥

Welcome to the AI Crawler Wars, where bots don’t respect boundaries (or robot.txt files), and developers have had enough.

🚫 The Problem: Crawlers Gone Wild

AI bots are crawling FOSS (Free and Open Source Software) sites like termites on a sugar cube.

  • Ignore robot.txt 🤥
  • Pretend to be human 👨
  • Hit servers like a DDoS hurricane 🌪️
  • Click every. single. link. over and over and over… 🤯

Xe Iaso described AmazonBot hammering their Git server so hard it caused outages. And it lied while doing it. Rude.

“They will scrape your site until it falls over, then scrape it some more.” — Xe, crying into terminal logs 🧵

🐕 Enter: Anubis (aka Internet Gatekeeper Supreme)

Xe built a tool called Anubis — named after the Egyptian god who weighs your soul.

  • Human? ✅ You pass the challenge, see a cute anime Anubis pic 🐶
  • Bot? ❌ Denied. Your soul’s too heavy, get wrecked 👻

Since March 19:

  • ⭐ 2,000+ GitHub stars
  • 👨‍💻 20+ contributors
  • 🍴 39 forks

FOSS devs are LOVING it.

😤 Dev Pain, Global Scale:

  • SourceHut CEO: spends up to 100% of the week just fighting LLM crawlers.
  • LWN.net: slowed down by crawler-driven DDoS.
  • Fedora’s sysadmin: had to block Brazil (yes, the entire country).
  • Others: blocked China, blocked entire continents, rage posted on Hacker News.

🪰 Meet Nepenthes: The Trap Plant 🌿

A dev named “Aaron” built Nepenthes, a tool that:

  • Creates a fake maze of pages
  • Wastes bot compute time
  • Feeds it garbage data 🗑️

Goal: Make bots regret everything.

“Bots shouldn’t just get zero value. They should get negative value.” — Hacker News user xyzal 🧪

🧪 Cloudflare’s Turn: AI Labyrinth

Cloudflare joined the party with a bot trap of their own:

  • Slows AI crawlers
  • Feeds them useless nonsense
  • Wastes their compute budget 💸

Big tech playing tower defense now.

🛑 Devs Say: Enough Already

SourceHut’s founder Drew DeVault:

“Please stop using LLMs. Stop making them. Just… stop.” 🙏

Odds of that? Roughly 0.00001%. So instead:

  • Devs are building traps 🪤
  • Naming them after gods & carnivorous plants 🌿
  • Dropping anime pics as rewards 🎀

TL;DR:

AI crawlers came for open source. Open source devs came back with ✨vengeance✨.

#AI #Anubis #Nepenthes #WebCrawlers #OpenSource #FOSS #DevHumor #BotWars