Skip to main content

Kari Silvennoinen

A web log

Chatbots Scraping an Ancient Forum

Among other projects, I maintain[1] a phpBB based forum about old freeware games, Curly’s World (of Freeware). It’s way past its active days, the forum was originally started in February of 2006, so over 20 years ago. The world has changed a bit from there.

And this wasn’t even the original start of the community; it was a continuation of another freeware/abandonware community. This was the age of forums and boards on the internet, before social media and mobile apps really took over the way we use the internet. But at least it was humans.

The site isn’t monetized in any way, because it really doesn’t see much activity. Due to costs of hosting files on the internet[2], already a decade ago we donated the freeware game collection that was part of that site to the Internet Archive. The only reason the site was really up was it didn’t really require any maintenance except the occasional version bump to keep SEO spammers away[3].

So, imagine my surprise in 2024 when I randomly checked the Cloudflare dashboard for the site and saw that the site was getting over 15 000 unique visitors per month. That audience would be worth it to integrate Google AdSense and the forum would easily cover its VPS and maintenance costs! Finally!

The reader can probably guess where that the uptick of the traffic was coming from (the title of this post is a hint). No payday. Just insane amount of bot traffic from especially the likes of Huawei and ByteDance that didn’t bother to look at nor click on the ads. Then Meta joined the party as well slamming the forums hard. However, most of the UA strings continue to date to look like “real” browsers[4].

Let’s fast forward to 2026.

Screenshot of Cloudflare's dashboard for Curly's World showing visitor and page request stats
Last 30 days of traffic to the forum

This site hasn’t really seen forum activity for the last decade or so, but while it was active, it has amassed a lot of (unique) information about obscure PC games[5] and of course a lot of off-topic discussion ranging from philosophy to operating system wars[6].

The traffic is up 10x from two years back[7] and it continues to have some wild spikes on some days. To be honest, if it wasn’t for Cloudflare’s free services, it’s very unlikely I would have the energy or the knowledge to keep the site up from these sporadic and intense DDOS attacks from various bots.

And this is an ancient, low-traffic, obscure freeware gaming site and so far, the traffic has been manageable. Some other sites haven’t been as lucky, as Ars Technica reported already last year.

The content on that forum doesn’t really change, there hasn’t been a new thread or reply in years. Yet it is constantly hammered by companies with billion and trillion dollar valuations. It’s all very silly.

And I fear it will one day mean more sites like that forum will go dark, forever.


  1. Essentially just keeps the lights on. ↩︎

  2. There were less and less webhosts that had generous enough plans to really let us share the games to the masses of the internet. Athough services like AWS S3 and CDNs were pretty cheap, it wasn’t really worth it as the community around the games had dwinled down. ↩︎

  3. They still attempt to link farm the forums to this day, but they are easy block: all new posts go to moderation queue, which is a very easy solution when there are no real new users to speak of. ↩︎

  4. This is especially a problem for a phpBB forum, because its session tracking can really suck, and its often necessary to whitelist all the bots and crawlers so they can index the site properly. ↩︎

  5. There were even some games that were exclusively available through us thanks to the preservation efforts of the dedicated community members (the “game hunters”). ↩︎

  6. As was inevitable on any forum on the net in the old days. ↩︎

  7. And this is with all Cloudflare’s AI bot crawl protections on ↩︎