In case anyone has wondered about the various downtime/slowness events on Textkit this year, this article describes the state of affairs pretty well. As the article states, user agent blocks are the primary means I’ve been using to deal with the scrapers, but I have to dig through httpd logs and resort to IP blocks as well whenever scrapers disguise their User Agent.
We also get a number of AI-submitted join requests these days, but they mostly get blocked, I hope. If that ever starts to fail, I guess I can ask people to video chat with someone for an account, although that would be incredibly suboptimal.
Thanks for all your hard work in maintaining the site Joel, much appreciated. HNY 2025 to you and your family and here’s to more great conversations on Textkit.
Textkit underwent a DoS beginning Monday night, I assume scraping related. It was enough to cause the server to run out of memory after 5-10 seconds of uptime each time I restarted it. So I banned a number of IPs (7000 /16 ip blocks, or about 10% of the planet’s IP space), which seems to have fixed things. I hope to turn off most of the blocking as soon as I can, as I expect that quite a few normal users are caught up in it.
Joel just a small suggestion: create a new subforum called Annoucements or System Updates or something similar under Textkit Community and post stuff like this there instead of adding it to a thread in the Open Board subforum. That way it will be easier for members to find out what happened when there’s been system down time for some reason
