Meta not only hasn't noticed, but is currently sending about 11 requests per second to my site. I've also seemingly trapped one of those TV proxy scraper nets as I'm getting absolutely hammered by requests from all over the place now. I get maybe 10 legit visitors per day, and I'm currently blocking 406,787 IPs from things that have fallen into my honeypot.
I've tweaked my site to return empty status responses a configurable amount of time but the traffic has been so intense that Traefik is now struggling, so I'm going to have to figure out something else. I was returning over-capacity errors and I think that was a mistake, I've swapped to 400 range status codes now. I don't want to use Cloudflare so I'm not sure what to do after this.
The people at these companies are either incompetent or malicious.
Most distributed bots won't follow a 301. If 404/400 don't work, just 444 them, they're not worth giving back a response, especially on a personal website.
Haha I did debate going much worse with the junk images but wanted to err on the side of caution in case I subject possible clients to something like goatse.
Just on the off-chance anyone should see this: if you were one of the brothers involved in running the Blinkys "R" Us forum back in the early 2000s - I still think of it regularly. I can't even remember what my username was these days, and Archive.org never archived it at the time, but I still remember it very fondly.
Yeah I've been really surprised by how inept they are. Lots of others have fallen for the trap but extract themselves fairly quickly and probably blacklist my site. Archive.org got stuck for a while, which was unfortunate, so I had to exclude them manually. There's something on an Oracle network that keeps coming back but it's very slow compared to the ridiculous rate of requests from Meta's scraper.
It's up to 1.5 million requests now, sigh. My guess is that Meta has too much money and is in the "move fast and break things" stage where they're just throwing money at a problem. Not setting a max crawl depth is hilarious though.
I've swapped over to it in the past two weeks, it's been really good. It does what I ask and doesn't think it knows better than me, which so far has made it the most pleasing experience I've had when slop-coding.
My only wish is it were somewhat cheaper, as it tends to balloon pretty quickly when I'm using it in Opencode. I'm currently trying to offload a lot of work to subagents to stop the context expanding so rapidly. But on the upside, I rarely have to correct it - I've spent far less time arguing with this than with anything else so far.
I usually use OpenCode for all open weight models but for Gemini I use Google’s agy coding harness (or my own).
Venders coupling coding harnesses with their own models is usually a good thing. Poolside.ai has a combined harness with their own models that works well locally, and the DeepSeek harness with their models is very interesting.
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it.
Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.
Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.
My team runs a quite popular website, #1 or #2 in the market depending on the region. Several million visits per day.
Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down.
We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got complaints on Reddit from day one, pretty much. At the same time, it got a bit expensive having a WAF.
In the end the only solution that worked was blocking every single Chinese IP.
Traffic instantly got down to 10x less, same as it was before. Same conversion percentage as before.
In an odd sequence of events, I've sorta got a weird tiny following in China due to having a vintage camera stall in an antiques market here in Wales that gets shared on Xiaohongshu (https://en.wikipedia.org/wiki/Xiaohongshu) sometimes when I have Chinese customers. So I'd feel bad blocking the entire country myself, though thankfully I'm not being inundated with traffic from there yet - the most egregious bots I've caught so far are actually ones from Meta which ignore robots.txt.
Yeah Meta's bots are well and truly stuck in the blackhole on this site at the moment. I have no idea what they're supposed to be doing, but in case they are using them for AI work I've just added a token multipler based on subnet totals so they can have some more fun. I might come up with some other fun response types for them too.
The same trick worked for me 15 years ago to drop a lot of log noise out of constant connection attempts. Felt a bit bad but in the end it turned out fairly well.
That was somewhat helpful, thanks, but the most interesting aspect for me is how iocaine tries to determine whether a request comes from a bot (since if this can be done reliably, we're done) -- but on that, all the link says is that it "uses a set of logical rules".
Gymnastics trophy two years late, not two years wrote down on the other side of the two thousand millions. Stop all the time was now a.
Anything of this real London, these actual civilized men and women; in uniform batches. The whole incident could not control the beating of a telescreen. Folly, folly, folly! He.
One thing I've thought of trying is, if I ever get around to putting things back online, loading the pages with altered versions of themselves in a manner that human users are unlikely to see (though they could if they wanted to) but scrapers are unlikely to ignore (because they look enough like something users might see, perhaps by opening a <detail> tag).
The altered versions would be the page or paragraphs of it, with sentences and words swapped arbitrarily so that training on the data adds as many iffy token sequences as correct ones.
To be clear, making pages with bad training data for bots won't make the bots go away.
It'll just punish the bad actors running the scrapers.
As the original poster mentions that they are using TVs as proxies to get residential IPs, one really can't think of these bots as criminal enterprises.
Sadly, if the bad actors has two cents for brain, they'll limit how much importance each domain name can have on training data. To mitigate impact of bad data like this.
(Note I'd suggest only linking to them from robot.txt as pages to not be indexed, that way no human or well behaved not ever will see them, which is kind of the point).
> … making pages with bad training data for bots won't make the bots go away.
It'll just punish the bad actors running the scrapers.
Exactly. I can't hope to keep them all at bay, but I can at least have the petty little victory of making their visit less convenient than it might otherwise be.
> if the bad actors has two cents for brain
I suspect that a majority of them are little better than the script kiddies of yore, running tools with minimal understanding of what is actually going on.
> I'd suggest only linking to them from robot.txt as pages to not be indexed
Agreed. Blocking all bots from all pages, well those that bother to listen to robots.txt. All bots because pretty much all of them are scraping for AI and similar these days, even googlebot. If I want people to see my stuff they'll get a link, and maybe they'll pass it on further, but all indexers/trainers can get stuffed. I'll likely make an exception for archive.org and similar.
Thankfully this is my photography website so what they're mostly after is images, which they have to request. I'm also intercepting all my legit images once the bot is flagged, which will no doubt have unintended consequences but I'll find out at some point I guess.
Have you considered releasing any of this as a plug for phoenix or similar?
I’ve also toyed with this approach and have a similar labyrinth on my website, but it just serves back huge amounts of meaningless text one word at a time.
If I get some time to un-LLM-ify it then sure! I build all my sites in Phoenix these days anyway so I'll likely want to port it to my other sites at some point too.
I do want to add some more gibberish generation to mine. Once a visitor hits the honeypot route it does replace the rest of the site content with nonsense but I think it'd be fun to have it spewing really daft, really long LLM-generated essays on toasters.
If you know the IP, it's easy to tar scraper's requests.
However the article mentions that now they are being scraped 4-5 requests at a time from random IPs that are then never reused. It still probably possible to detect scrapers if they start by requesting obscure pages, but in general it's a much more difficult case.
Currently not really an issue on this site but connections aren't an issue on Elixir usually anyway, unless you get up to about 1 million on one machine IIRC.
Elixir's really fun to do this in because the BEAM will let you have hundreds of thousands of processes sat around doing nothing quite happily, so slow IO and such is something it's weirdly well-suited to.
This is on my photography site so I'm less fussed about them harvesting my writing, and more about causing problems for image harvesters. I'm sure they'll get some stuff anyway but at least one bot got stuck in a 400 page hellhole earlier today so I'm overall very pleased :D
Gemini's surprisingly bad at it but so far I have some voluptuous curvy toasters, toasters in the shape of a butt, and robot girls with toasters for tits. I have a Gemini subscription for the month after mistakenly thinking I'd get cheap Opencode usage through it, so gotta use it for something.
Not op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers
Yep this is how I'm doing it. It's marked as my super exciting client archive full of high-res RAW photos that all bots are told not to index in robots.txt, only for some weird reason the link never gets rendered to the page :>
I also explicitly tell all crawlers not to index the images on my site, actually, so I feel even better about serving the ones that do complete junk.
Not sure why I get more crawler traffic than I do legitimate traffic tbh - it's a relatively new website that I've been trying to sort out the SEO for as I'm trying to get photography work, so it's probably started cropping up on the radar of scrapers looking for training data.
It's mostly about luring them into URLs they're explicitly told in robots.txt that they shouldn't index. I do some identification via reverse DNS of known crawlers I actually want like Googlebot, though they respect robots.txt, in case something goes wrong and they accidentally get flagged.
If it's anything like the PS4, you might be able to fix this by running a HTTP proxy using Squid and configuring that on the PlayStation. There's something really funky about Sony's networking on them, and configuring a squid proxy for my PS4 vastly sped up my downloads.
I tried to find an article covering this but I've had no luck, so I'm just going off memory here, but I'm pretty certain that's all that was required.
Sigh, frustrating for me but I guess I understand the rationale, if not the implementation.
I'm retired from programming professionally and I liked using Sourcehut for my personal projects because it just got out of my way, and I could put my mostly-old open source projects up there without worrying about people trying to contribute to them thanks to the friction of the system. I do slop-code hobby projects because I'm pretty much done with programming for fun now, but they're always private and I know that my relative load is low as I don't do much different from how I would personally use it.
Personally I'm in favour of blocking LLM pull requests/merge requests/patches to open source repos - the sheer volume of crap is out of this world. I saw a repo with 1.1k unmerged PRs yesterday! The code can be from an LLM but the request to merge it damn well better be written by a human. I'm not going to put effort into reviewing something someone else couldn't be bothered to even write a blurb for.
But for private projects, just shove a rate limiter on the endpoints and be done with it.
Lately I'm very conflicted about software development as a whole. I've always aspired to be good, both in my work and in my relationship with others in the industry. But I can't be arsed any more - the joy has gone out of software, and I just want it to be done and out of my way. And now I have to move my photography portfolio repo that has fewer commits per day than I could write by hand, because there can't be grey areas or compromise on the modern internet.
I used to work for GitLab, for what it's worth (and they've gone quite far down the AI toilet), and I honestly don't think I can be arsed setting up that or anything else. I might as well just shove bare repos on my NAS and go do something I actually enjoy.
Yeah I test drove a Honda E. I don't know why but they somehow used a webcam from a 1999 laptop for the cameras. It was a shame as the interior of that car is gorgeous, but the range is absolutely appalling and those mirrors are awful.
Meta not only hasn't noticed, but is currently sending about 11 requests per second to my site. I've also seemingly trapped one of those TV proxy scraper nets as I'm getting absolutely hammered by requests from all over the place now. I get maybe 10 legit visitors per day, and I'm currently blocking 406,787 IPs from things that have fallen into my honeypot.
I've tweaked my site to return empty status responses a configurable amount of time but the traffic has been so intense that Traefik is now struggling, so I'm going to have to figure out something else. I was returning over-capacity errors and I think that was a mistake, I've swapped to 400 range status codes now. I don't want to use Cloudflare so I'm not sure what to do after this.
The people at these companies are either incompetent or malicious.
reply