Why I blocked bots on my site
You got to be wondering why there is a captcha on my site. The reason is simple: language models.
Figure 1 - I’m under attack mode
Even though my site is statically generated I am protecting my content from bots. There is lots of crawlers feeding proprietary AIs using my articles and I simply do not want to be part of it, my articles are meant for human beings and not for language models.
I have been spotting lots of traffic coming from the US lately and what’s funny about it is that my site is not even in English.
Figure 2 - Amount of traffic by country
The American traffic was responsible for 79% of all site traffic and it was mostly created by bots that were stealing content from the articles. How do I know that? Simple. Google is not showing my sites, nor was Bing. There is a service called diff.blog that was reading my RSS feeds in order to publish my site on their site, I asked them politely to stop it and they did it, but the traffic did not go down as expected.
Figure 3 - Hubspot crawler
HubSpot crawler from hubspot.com is a bot that was looking through my articles to feed their language models. In their site you can see clearly that they have a service called AI Content Writer specifically designed to generate blog posts. How do they train this AI? By stealing content from real blogs.
Figure 4 - Hubspot AI content writer
Cloudflare’s “under attack mode” with CAPTCHAs stopped the bots by filtering traffic.
Figure 5 - Traffic going down
The traffic drastically dropped, and the captcha has been kept enabled since then. Now very few people view my site, but I can rest assure that the traffic is coming from humans.
What else did I do?
I ended up removing the RSS Feed from the website, only a handful of hackers know how to use it and nowadays, no one really cares about it. I know some boomers will start to complain:
Figure 6 - Boomer rants
I agree, the problem is, RSS feeds made it quite easy for bots to steal your content and what’s worse, they use it to train proprietary AIs and are making money on behalf of your work. So after all the work you’ve put into writing blog posts, doing research, making free software, all of this is being used to train language models and is being marketed as the “future” of software engineering. Therefore, by prohibiting bots I am not only postponing AI growth but also helping build a better internet by providing an artisanal site for real people.
I also decided to remove my site from search engines like DuckDuckGo, Google and Bing and also updated the robots file.
What are the downsides?
Blocking bots affects SEO and Opengraph. Search engine crawlers require access to the site for updates, but captchas prevent them from viewing the content. Additionally, prohibiting bots hinders Opengraph, preventing the display of thumbnails when sharing article links.
Figure 7 - No thumbnail when sharing articles
This is a small price to pay in order to keep the traffic organic. Search engines are basically dead, and I don’t care about SEO anymore. Google and Bing never gave me clicks anyway. You, too, can help me build a great internet by banning bots from your site. Let’s make internet great again!
I might want to disable under attack mode in the future.
- https://blog.barracuda.com/2021/09/01/report-insights-into-the-growing-number-of-automated-attacks
- https://blog.cloudflare.com/introducing-im-under-attack-mode/
Peace.