Every server with port 80/443 open has thousands of hits a day from random boxes looking for wordpress login pages. The only new thing is that they're pretending to be a different type of annoying bot. There's a new layer of sophistication and subterfuge, but it's the same junk traffic we've always dealt with.
People or their agents must be accidentally committing or publishing their repository level secrets and configs with enough regularity that it’s worth scanning.
There are a few novel ones but I’ve been seeing most of them in my logs for longer than generative AI has existed. This isn’t remotely new, the vector is just getting bigger.
Think about how many webmaster and business owners' egos are stroked by all the traffic they are getting, when in actuality they are often just serving thousands of bots.
I recently brought up a website on a never-before-seen .com domain. Within about 10 mins of bringing it up with a SSL certificate, Anthropic came knocking on the door requesting the front page. (Almost certainty due to them watching the Public Certificate Transparency logs)
Had a bit setup a new Wordpress install before I could lock it down. I was very confused why a brand new install didn’t give me the setup page before seeing in the logs someone had automated it. Pure evil to be scraping new renewals and dns changes to look for this kinda stuff.
For the record I thought I had this site behind basic auth.
When I stood up some sites last year, I used codenames for the subdomains thinking I was obfuscating a little. I didn't know about the transparency logs until months later.
Yeah... you have to remember to setup and fully secure the site before LE certs are issued or you're going to have a bad time. Learned that the hard way when I popped a couple dozen wordpress sites in one go.
On one hand yes, but on the other hand just configuring your server to refuse connections by IP address rather than server name seemed to drop roughly half the bots I ever see.
Could be because of number of pages you have. At work we have several million public content pages, so a few badly behaving bots can already do a lot of damage.
For my personal website it’s 10x more bots but I barely notice because it’s a few pages.
Humans never visit but I have some websites with ancient cms's. When I got bored playing with them I download the html, change the extensions to php and replace the site with a static copy. People then tirelessly try to hack it since they have laundry lists of known vulnerabilities. (Now that I think about it I regret not making the admin area public for added nostalgia.)
Not that I follow my own advice, but a popped server can still be a liability for you. If you don't need it online I would take it down. Don't want to suddenly realise you've been an email spam node for several months.
It's fine, I am just being a nit, but static doesn't mean zero code. There is still a server and program turning a web request into a response. But in your case it's not your problem, it's cloudflare's.
I only mentioned it in the assumption they had a VM or shared hosting, in which case it's worth thinking about.
If it's a computer publicly visible on the internet then it has an attack surface, not having a scripting language enabled reduces it a lot but not entirely. There is still a web server involved in a static site being served, amongst other services that computers often have by default.
If it's shared hosting, still a problem, just not theirs.
I never had this issue really… until two years ago, new website new host… boom, hugged to death by thousands of bots per second. This is on an alicloud vps.
I always had a decent bit of background noise, I think hosting on AWS comes with that. But after someone linked to me on hackaday the bot traffic went through the roof. One link to you from a noteworthy website, and all bets are off forever.
Is that basically the delta between Cloudflare’s Pages analytics and Google Analytics? CF says I get thousands of visits a day and GA says it’s like 3-4 users.
You pretty quickly learn to qualify incoming traffic since it directly impacts how you track conversions. But definitely to begin with, you see thousands of "people" view a page and no one bites, it can be disheartening until you realise it was mostly bots.
I have basically 180d entirely on view metrics, they are more or less noise to a small business owner. Did someone buy or not, that's all you actually need to care about.
Even big retail stores are pushing back on crap like KEPLAR/foot traffic tracking, since it doesn't actually change what you do, or impact sales.
Measure sales, measure customer delight, make those the targets.
This is so true. Every junior sysadmin I have trained over the years (including myself) has had a “are we being attacked?!” moment when tasked with WAF report analysis, monitoring fail2ban logs, etc.
Monitoring WAN traffic really gets the paranoia juices flowing.
I remember when you could stand up a website and no bots would scrape it or scan it. It was a lovely time. No one had firewalls or antivirus and things were working fine until the worms and viruses started coming. You could be confident that your guests were real, so much so we had guest counters on many public sites.
Just build your website yourself as deep in the stack as you can instead of piling up 50 abstractions on top of each other. Some decisions like having your page be accessible by IP can only happen if you use technology like generic http servers (like apache or nginx) from the 2000s instead of implementing the lower stacks and actually thinking about whether that makes sense for a second.
If when you build a website or a backend, your server responds to requests by IP address (for example), you are building a bottom 90% product, and considering most software markets are super top-heavy, (say 1% win), that's ngmi land.
I am always surprised that its considered legal.
IMO that's the same as going on the street door by door and checking if one is left open to steal everything inside the house...
That sort of vulnerability scanning is at best legally dubious, and almost certainly illegal under CFAA and similar state statues when there's clear criminal intent. That's why the 2022 DOJ guidance regarding non-prosecution good faith security research was such a big deal at the time.
> IMO that's the same as going on the street door by door and checking if one is left open to steal everything inside the house...
From experience: this does happen regularly in some neighborhoods of some cities in the US, and even that isn't always an enforcement priority. So lack of enforcement on the internet, where most the perpetrators probably aren't even in a jurisdiction with an extradition treaty, isn't exactly surprising.
Most servers with port 25565 open get hits from either Minecraft griefer bots, or from a bot that looks for that port and warns anyone on that server about the risks of leaving that port open. It doesn't take a huge scale operation to spam every IPv4 address in the world, there are only 2^32 of them, and even then many of those addresses are reserved.
Exactly this. It's no different from a bot pretending to be Googlebot. I've tried reporting abusive IPs to various foreign hosts, but nothing every comes to it. I've settled for just blacklisting excessively abusive IP ranges.
Almost lol: grep, sort and uniq. If I notice someone is hammering my employer's ecommerce site, I'll block them. It isn't required often so I've been reluctant to spend the time setting up fail2ban.
Google (and other “legitimate” scrapers) publish the ip ranges they crawl from, anyone claiming to be googlebot (or whatever) but not in the ip range can safely be black holed.
Years ago, I built a sort of camo-proxy like, in Elixir. It was doing full passthrough of the User-Agent because the project that I originally built it for needed it for the upstreams (i don't remember why). Anyway, I ended up pulling it into pleroma -- a couple of months later, we had been informed by some instance owners that google itself were sending them DMCA notices, because it was proxying googlebot's request, with its user-agent!
Yeah, It started bothering me enough that I recently put together a system where when a application detects a bad actor(a bot enumerating too fast, a random scan for vulnerabilities, etc) it notifies the firewall. Right now I am just shutting them down, But have plans for a honeypot/tarpit system, something real slow that takes up all their time. Something like "have fun at 300 baud"
I'm doing the same thing. Set up a honeypot API and having it log everything anything tries to access outside that home page. Collected 15,000 hits in a week from 300 unique IPs doing the usual WordPress and .env scans.
I'm just collecting the data now to be used to secure some of my upcoming projects, but I would absolutely also like to take it in a direction where it sends the bots into an infinite slow loop, or preferably something that burns as many tokens as possible for them.
I don't really care about the morality of that. I'm a big fan of fighting fire with fire.
> something real slow that takes up all their time
Not to throw water on your plan, but the bots I've written intentionally run very slow with respect to each target. When done in parallel, across a wide range of targets, it doesn't slow down the effort at all.
I don’t really have a reason to use WordPress anymore. I’m about halfway through switching my site over to something else that I can control with github and AI. Now these boats are mostly getting static files unless there’s a reason to show something else and even then the footprint is very small. There’s just not that much to hack.
I've done the same by scraping the HTML/JS/CSS off our own Wordpress site and sticking it in S3 (behind CloudFront, of course). It was a mild hassle at the time, but would take seconds with today's LLMs. It was a great decision and has been no hassle since, as we didn't really need a CMS, it was just the default for the person who did our design.
Definitely gotten worse from where I'm standing — you used to be able to just filter on the UA string and move on, now you can't even trust that. Same junk traffic, but harder to sweep away.
On average about 100 (TCP) requests hit my home router per minute doing various probing and scanning. Lots of checking for the telnet port obviously. Sometimes you can see a swarm of entirely different IPs scanning the full port range (probing the ports one-by-one).
You'll see a lot of deepfield, censys-scanner, visionheight.com, shadowserver.io, etc., but also the usual suspects of Chinese or Russian IPs.
With OpenWRT I use something like this: `tcpdump -i pppoe-wan 'inbound and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn'`, or alternatively `tcpdump -i pppoe-wan 'inbound and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and not port 44000'`, if we have some torrent client running (e.g. here at port 44000) which would mess up the result. I'm not sure it's the best way to handle this, but it's definitely enlightening what bounces off on the router.
The easiest way to deal with the usual suspects is to just block the entire countries network range(s). There really is no reason they should be connecting to your home router anyway, and you lose nothing from blocking them.
Sure their packets will still hit your router, but if they are dropped immediately at least you're not wasting a syn-ack on them.
I, temporarly, banned some ip range.
I didn't find a source for pinpointing countries; though I am interested. Could you point me to some sources which, deterministically, resolve to some countries? To my knowledge you can not reliably identify countries by ip since this would be dependent on DNS servers. Though I am just a application programmer!
Your router doesn't care about their DNS settings. IP addresses are very easy to tie back to countries. The reason they say it's not reliable is because it's trivial to spoof the country, but even so, a lot of attackers don't even bother. It's sort of like the Nigerian prince scam calls: if you're wise enough to block Russia, you're not worth their time.
Your firewall vendor should supply you with country lists, just select the known bad ones and drop their traffic. If you have a consumer grade router, you will probably have to configure the blocklists manually.
Unless you're serving something, you should probably drop all incoming requests. If you are serving something, the US is a pretty big market to lock yourself out of.
> The reason they say it's not reliable is because it's trivial to spoof the country
ISPs sometimes do trade IPv4 blocks and countries to which it belongs do change occasionally. That can become a problem if you were like literally Netflix and someone few nation states over started an ISP.
I'm happy to unblock China from accessing my website, once China does something about the millions of daily hack attempts originating from that country.
> The easiest way to deal with the usual suspects is to just block the entire countries network range(s).
Keep in mind that this should be paired with an ASN blacklist - MaxMind also has an ASN mmdb for convenience - because IP address to country maps are almost entirely self-declared[0].
For example, Tencent (AS132203), which you almost certainly want to block, has ranges in 73 different countries per [1].
I have ubiquity UniFi for this reason (amongst others). OpenWRT is a good choice as well. Most home router software is such junk, might as well leave the door open..
Many of those user-agents listed are often faked. Look up which ASN owns their IP. If I block most VPS providers most of the faked bots vanish. There are still some running from residential and phones using hijacked code (readers that are not really just readers but really multipurpose proxies). On that note, do not trust the linked source code but rather decompile the live code your phone is running and have AI analyze it.
Yeah, that's exactly what these visits are: faked user agents that fail IP verification or Web Bot Auth. What's interesting is the surge across so many websites in the last week.
There are many possibilities but one of them could be some new vuln was released and they are looking for it. That would require looking at the URL's they are requesting. Botters run their own purpose built campaigns. Do you also have a summary of URL's requested by unique counts?
I've had a similar bump in scanners in the past week, more than half of it is coming from MS and Google owned IPs and all of them are spoofing AI agents.
Same for the origin IP address. The fiber leaving your country is tapped, and those people can inject packets with any origin IP that they want. Your ISP has no way to check if their peer actually received a certain packet from a certain country or not.
From a technical perspective, all this "china/russia" attribution is built on a quite shaky foundation. As a sysadmin you'd never know if it would be the British crown attacking your European company instead.
Not minimizing nation state cyber crime here, but the packet goes through many hands with different incentives.
Problem here is there are not single fibers attaching (most) countries, but a bunch of them. If you control both the ingress and egress for some particular users it's possible, but if you don't then your probing packing may end up back in China with a lot of evidence of backscatter.
Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.
I second this. When I have tested blocking VPS/data-centers to my silly blog there were about a dozen people on HN [1] that could not view my site out of the roughly ~17,000 (not counting bots) that could. It's not a big number but those are real people and they count.
I am going to move full blocking to a test node that people can play with but I have to finish working with Claude to revise someones repo is is no longer maintained because one does not simply put an anonymous chan board on the great wide open internets without some critical thinking.
Yes but it’s not cheap. Maxmind and ipinfo etc sell a tier that tells you this information, then you can 403 based on it.
But the price is nuts like $40,000 a year.
I work for IPinfo. We offer IPinfo Lite for free. With a little bit of time in identifying the ASNs, you can implement a decent way to block a good number of bots fairly easily using the free data alone.
You can block entire ASNs. If you are frustrated with bots, blocking Tencent's entire IP address space would have very few downsides.
If you have fail2ban or NGINX logs, you can use our CLI to summarize those IPs and identify the ASNs you want to block. But before you block entire ASNs, make sure they are not classified as "ISP" type. For that, visit our website's ASN page first.
Cool site. I was curious and dropped your 100k list into a reverse DNS lookup site [1]. They may still have some of the records cached. I recognized quite a few of the scanner nodes and some other usual suspects.
Yeah, that’s pretty interesting! You can also see a live view of the ASN/ISP leaderboard by going to https://knock-knock.net and choosing ISP from the carousel. That’s ordered by bot transaction count rather than IP count though.
It never ceases to amaze me that these ISPs don’t bother to shut down the botnets. They could do so very easily. For example, they could identify the IP address of every bot that hit this honeypot with their ASN with one API call: https://api.knock-knock.net/check-asn?asn=<asn number>. (See https://knock-knock.net/api). They just don’t care!
The ISP's do not have a financial incentive to shut them down. To them that's a paying customer. The feds will go after the big botnets if they are touching financial networks or siphoning enough money from people because there is usually a few big bank accounts and virtual currency exchange accounts they can seize once big enough to look good in the media. That's why it's on us and a few big CDN's to block some of them.
Not the person you are asking but site operators can not tell intent. It could be something nasty on the network or a botter feigning ignorance.
I'd say its probably an acceptable casualty in the battleground that is the internet especially for little one-off sites hosting blogs, forums, chat servers, etc... For a bigger site I would expect that person may have to open a ticket with the platform such as Amazon accepting that some CDN's and firewalls may be harder to get the block removed. This is why we can't have nice things.
thats the hard part, right - my 76 year old dad is on his banking app while his samsung TV is allowing a bot to try and take over other accounts at the same bank on the same IP.
IP Blacklists, no matter how good can't stop this. You have to start using stats or deep-diving telemetry.
thats the hard part, right - my 76 year old dad is on his banking app while his samsung TV is allowing a bot to try and take over other accounts at the same bank on the same IP.
So appeal to emotion doesn't fly with me. If grandpa is 76 in the year of our lord 2026 that means he was 50 when the internet was getting popular and 59 when cell phones became very popular on the internet. He's not much older than I. He knows what's up.
God help the makers of that television if he finds out it has been spying on him and dorking around with his traffic. If they are lucky he will just take a baseball bat to it. If they are unlucky he will fly to their headquarters and end up on a viral bodycam video likely with a lot of supporters that will bail him out of jail.
IP Blacklists, no matter how good can't stop this. You have to start using stats or deep-diving telemetry.
I use a myriad of methods including IP blacklists. That's my choice and every site operators choice. I do not have to use deep-diving telemetry but you are free to do so.
There are several methods. [1] The most aggressive method-02 and method-03 on my document will block VPS and some data-centers but that also means it will block some legit users that are on a VPN. Most VPNs transit a data-center. If experimenting with these methods use a test server that you do not care about and set up a dummy site and ask people in your circle of friends to test it. I have to step away for a bit but if you have questions I will try to answer.
I've noticed they strip away a header [1] in private browsing mode but I don't know why they do it since it does not disclose anything about the person. I think that may be the same thing that causes some people grief on Cloudflare as well.
By default they use 2.0 [1] unless someone or an addon disables it or unless the person is on a really old version. OperaMini however will use 1.1. No idea if anyone here uses OperaMini.
There are some reader apps that act as a proxy that only support http/1.1. Be careful, some of those are not just readers and do not trust what they claim to be the source code. Some of them are created by cute and fuzzy bunnies.
There are a number of botters on HN, some that control residential and phone browser-hijacked systems. One was sending me playful messages the other day. I enjoyed the bot block-jousting with them.
No idiocy, sometimes defaults change or don't get updated by a deployment script. Different distributions may have a slightly different default configuration file depending on how involved the artifact maintainer is with the project. Or put another way, I've done far sillier things.
No, because the bad guys use residential proxies if you block DCs. They just prefer not to because it's more expensive. Meanwhile that silly HN project that scrapes and reformats your site is now dead before it began. Who are you actually defending and from what?
I did just this. Using a $2k a year database from a smaller provider that isn't maxmind, claude and I built a pretty slick ASN based categorization system. I can categorize an ASN as a residential IP, a service provider, a legit crawler/scraper, etc. For anything that is suspicious, I dynamically use turnstile to gate access to our service. Turns out there's no ISP for any VPN, they just contract with a shitload of mom and pop shady colocation services across the world.
We collect signals that help determine good vs bad networks. For example, large amounts of requests to .php endpoints, large amounts of empty accounts from the same /24 subnet, etc etc. All these signals let us automatically determine risk, and then put up a challenge. Authenticated users never see the challenge even if they are on a risky network (VPN 99.9% of the time), unless the network has been identified as 100% malicious, then it gets a full block.
This was probably a total of 3-4 days of work, spread out over a couple months of iterative claude led hacking. I didn't know exactly what to build, but had some of the key architectural ideas in my head. Opus+Faable made easy work of it all, and ended up guiding some really slick improvements for performance.
I would say this has dropped about 20% of all traffic to our service, though it turns out turnstile is a massive target for bots, so replacing that with something custom is next on the list.
Contracting with their colocation facilities is exactly how that's supposed to work. If you don't actually operate a wide area network then you aren't supposed to be registered in these databases and have IP blocks. The exception is people who do anycast, but VPN companies don't.
You know all these guys just switch to residential proxies if they detect a site is blocking data centers, right? Because that's a very common thing to do.
Not sure what you mean by your first comment - there is no technical reason that I know of that prevents a VPN provider from having their own ASN and address space.
As for the latter comment....not sure what your implication is. Yes, bot/spam mitigation is whackamole, but there are consequences for not playing the game of whackamole. Luckily residential proxies are few and far between so far, but they will grow in popularity. When they do, and I can't get by with the occasional individual residential IP ban, we'll come up with other methods to handle.
Luckily the signal is strong with vulnerability scanning, which makes it pretty easy to automate. The only reason to put up whole ASN mitigation (captcha/turnstile, outright bans) is just efficiency. Nothing stopping individual IP banning. The scrapers are the tricky ones, since they more easily hide in legit traffic. However legit traffic has patterns that scrapers do not emulate (at least for a service like ours with millions of pieces of user generated content that's easily walkable), so you can still pull out the signal. It's just a little trickier.
That’s not the pattern I am seeing, but I might be outside the norm. The majority of bot action (scraper, spam) comes from vpn providers by a long shot, ignoring (Chinese ASNs, Indian mobile ISPs etc). I see very little consumer isp action except cheap international providers, which are still swamped by vpn traffic.
I remember when 'code red' spread and it had the effect of crapping up the contents of my apache server logs. Fun times.
such as:
GET /default.ida?NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a HTTP/1.0
Worth saying out loud: user-agent is not identity. Verify AI crawlers by reverse DNS or the provider's published IP ranges - the ones worth letting in all publish them.
Why would you voluntarily pretend to be a AI bot, when those have already a much higher chance of being blocked? Seems holly unproductive.
Best hypothesis I can come up with is to somehow make the AI companies look bad, but they seem to be doing an excellent job at that themselves already by scraping everyone hundreds of times per hour over and over.
Because businesses dont want them blocked, that would be a very stupid thing for most of them to do given its becoming a vital traffic source now that people are using chatbots instead of google.
From what I have seen at work, everyone is using chat bots but no one is visiting websites through them. We still get almost all traffic through social media and google search.
the numbers are much smaller than traditional search but they are trending up, and they convert at almost 2x the rate of organic search inbounds. sure you can play catchup later, but the trend is quite clear. if there's one thing the last two years have taught me, my prior heuristics on now vs future don't work in 2026.
Very similar experience here. Started July 30, sustained through August 6, when it started a significant ramp-up in volume (5x or so).
Most of the traffic is originating in GCP. We're seeing ~70k req/min sustained from Google Cloud IP space (AS396982). Reported to GCP Abuse, they've been non-responsive so far.
The main distinguishing factor is the reuse of a bunch of legit AI-training bot UserAgent strings. It's clear that the traffic is under the same centralized control because of how it changes volume across thousands of IP addresses simultaneously.
Seems like you’re part of the group represented in this dataset trend then, many of these visits are also from (compromised) Google servers in that same ASN.
In my haste to initially get a blog started and published, I created it as a single HTML file with fragment identifiers / links for each post. Earlier today, sethops1’s valid feedback prompted me to restructure it as an index page with separate pages for each post.
Links to posts no longer contain URL fragments, but I kept the fragment-style links working with JavaScript.
For those suggesting fail2ban as a solution, that's dinosaur software from the palaeolithic. If you have a website of any size then the number of bots will overwhelm the block list in days with their millions of unique IPs.
Besides, projects with lots of dependencies are taking on more risk than they realize. If one dependency gets compromised, you have no idea how many projects are affected until it's too late.
Fake Googlebot visits are #1 in website logs I've been working on. At the beginning I was fighting with them using Cloudflare ASN block rules or their managed Bot Fight mode but it appeared to be not only pointless, but also harmful for my websites.
Bot Fight mode randomly started blocking real Bing / Google / OpenAI crawlers what wasted crawling budget and discouraged crawlers to revisit updated pages.
Sometimes it's better to not fight with bots actively but harden environment and only react for the worst offenders.
For Google it's pretty straight forward to throw away fake crawlers by just only allowing their published list of crawler IPs so you don't accidentally allow someone from a random GCP IP to crawl you if unwanted (https://developers.google.com/crawling/docs/crawlers-fetcher...).
Just in case any of the authors read HN, I'm getting a pretty crazy rendering bug on this page, where a bunch of the contents are redrawing up and down by a few pixels. It seemed to go away with resizing the width a few times, but I didn't look into it too hard. My page width was probably small on first draw. Incredibly distracting though and hard to read with the text moving. Using latest chrome, and it occurred on more than one page refresh. I didn't dig in beyond that though.
I'm seeing the same thing on Firefox on Linux. It almost looks like the page scroll is jiggling up and down a tiny amount constantly when it's supposed to be stationary.
How can I attract more of these bots to my server? I want to test my Apache bad bot blocker. It uses basic header fingerprinting and h2 support to filter them. I get less than 5000 hits on an average day, and want a lot more.
I have a domain with an SSL cert. https://solarsi.ca/
I tried posting on /r/sysadmin asking for help testing a bot blocker, but a moderator quickly deleted the post, claiming it was marketing/promoting.
Thanks for the github suggestion; I'll add a repo for the bot blocker Apache config, and include the above server URL that I'm using to test the blocker.
Definitely, the point here though is there is a stat sig surge in the chart in the last week, across thousands of websites. At least a surge in this particular spoofing pattern.
This is a random sample of completely unrelated websites, which indicates that the total scale is much larger. This is not saying that it is difficult to make thousands of requests.
Is this your company? If it is, your cheerleading makes you very hard to trust. If it’s not, I’m sure that everyone gets the point - you adore everything about this research and can’t see any possible problems.
I think this is more of a "if you left your AI tools exposed someone is looking for them" change. Hacking someone else's agents sounds like a great way to spend less on your own tokens.
This kind of stuff is getting pretty wild, even for using something like Cloudflare it seems like a good idea to have another layer behind it that's non-cloudflare for when vulnerabilities are discovered.
Every server with port 80/443 open has thousands of hits a day from random boxes looking for wordpress login pages. The only new thing is that they're pretending to be a different type of annoying bot. There's a new layer of sophistication and subterfuge, but it's the same junk traffic we've always dealt with.
Another interesting thing here is the paths they're targeting, many are for newish AI coding tools
People or their agents must be accidentally committing or publishing their repository level secrets and configs with enough regularity that it’s worth scanning.
Totally. I'm sure this campaign was inspired by sloppy vibe coding
There are a few novel ones but I’ve been seeing most of them in my logs for longer than generative AI has existed. This isn’t remotely new, the vector is just getting bigger.
Think about how many webmaster and business owners' egos are stroked by all the traffic they are getting, when in actuality they are often just serving thousands of bots.
I was insulted recently reading about the apparently thousands of hits per second the rest of you guys are getting. Even the bots are shunning me!
I don't even have a domain name and I'm getting lots of hits.
Old and busted: serverless
New hotness: DNS-less
Yeah it's even superior to smol web in some circles. It's not that much longer than a phone number.
Ah yes my phone number is 2602 1337 ABCD CAFE 3542 77FF FE12 3456
See? It's quite short.
Some people still use ipv4
I recently brought up a website on a never-before-seen .com domain. Within about 10 mins of bringing it up with a SSL certificate, Anthropic came knocking on the door requesting the front page. (Almost certainty due to them watching the Public Certificate Transparency logs)
How do you know who's visiting? Reverse IP lookups? Or do they announce it in the headers?
I saw it via the User-agent header + confirmed via IP ownership lookup.
Had a bit setup a new Wordpress install before I could lock it down. I was very confused why a brand new install didn’t give me the setup page before seeing in the logs someone had automated it. Pure evil to be scraping new renewals and dns changes to look for this kinda stuff.
For the record I thought I had this site behind basic auth.
"Public Certificate Transparency logs"...
The scam of "everyone should have SSL" right here, ladies and gentlemen.
Get a letsencrypt cert, I always get a massive wave of vulnerability probers after renewing.
Because your certificate shows up in the global chain, which triggers all kinds of automated things including bots
It's all TLS certs, because they show up in the Transparency Log[1]
You can watch a live stream of it here: https://bencevans.io/security/certificate-stream
[1] https://en.wikipedia.org/wiki/Certificate_Transparency
I use subdomains and a wildcard cert to partly obfuscate this.
I do the same, but I switched from cert-per-subdomain a couple of years ago.
They're either using Passive DNS logs or a historical dataset.
When I stood up some sites last year, I used codenames for the subdomains thinking I was obfuscating a little. I didn't know about the transparency logs until months later.
TIL about Certificate Transparency (they didn't teach that in security school)
Yeah... you have to remember to setup and fully secure the site before LE certs are issued or you're going to have a bad time. Learned that the hard way when I popped a couple dozen wordpress sites in one go.
On one hand yes, but on the other hand just configuring your server to refuse connections by IP address rather than server name seemed to drop roughly half the bots I ever see.
Could be because of number of pages you have. At work we have several million public content pages, so a few badly behaving bots can already do a lot of damage.
For my personal website it’s 10x more bots but I barely notice because it’s a few pages.
Humans never visit but I have some websites with ancient cms's. When I got bored playing with them I download the html, change the extensions to php and replace the site with a static copy. People then tirelessly try to hack it since they have laundry lists of known vulnerabilities. (Now that I think about it I regret not making the admin area public for added nostalgia.)
Not that I follow my own advice, but a popped server can still be a liability for you. If you don't need it online I would take it down. Don't want to suddenly realise you've been an email spam node for several months.
Re-read the post. He converted the sites to static html.
To GP's point, any connected server can become a liability, even if it is intended to just host static HTML (or for some other purpose).
I'm serving static sites via Cloudflare pages. Pretty sure that's gonna be actually static.
You're running Cloudflare Pages on a home server?
It's fine, I am just being a nit, but static doesn't mean zero code. There is still a server and program turning a web request into a response. But in your case it's not your problem, it's cloudflare's.
I only mentioned it in the assumption they had a VM or shared hosting, in which case it's worth thinking about.
If it's a computer publicly visible on the internet then it has an attack surface, not having a scripting language enabled reduces it a lot but not entirely. There is still a web server involved in a static site being served, amongst other services that computers often have by default.
If it's shared hosting, still a problem, just not theirs.
I never had this issue really… until two years ago, new website new host… boom, hugged to death by thousands of bots per second. This is on an alicloud vps.
yeah someone told me he had a single static HTML homepage and used up his entire 10TB/month traffic limit from bots hitting it.
I always had a decent bit of background noise, I think hosting on AWS comes with that. But after someone linked to me on hackaday the bot traffic went through the roof. One link to you from a noteworthy website, and all bets are off forever.
Maybe you just have fail2ban properly set up.
> I was insulted recently reading about the apparently thousands of hits per second the rest of you guys are getting. Even the bots are shunning me!
You're running the wrong stack - I, myself, find that simply having a static file website is enough to cut down on the traffic.
You need to run something other than static file serving to get bot attention.
imagine having a KPI tied to cash bonuses based on that...
2004 was a weird time.
Is that basically the delta between Cloudflare’s Pages analytics and Google Analytics? CF says I get thousands of visits a day and GA says it’s like 3-4 users.
Basically, unless CF is counting static asset network requests etc. For what it's worth, GA also miscategorizes some bots as humans as well.
Cloudflare analytics is great, but you need to filter by edge status code 200 to see the actual real traffic that was allowed through their firewall.
You pretty quickly learn to qualify incoming traffic since it directly impacts how you track conversions. But definitely to begin with, you see thousands of "people" view a page and no one bites, it can be disheartening until you realise it was mostly bots.
I have basically 180d entirely on view metrics, they are more or less noise to a small business owner. Did someone buy or not, that's all you actually need to care about.
Even big retail stores are pushing back on crap like KEPLAR/foot traffic tracking, since it doesn't actually change what you do, or impact sales.
Measure sales, measure customer delight, make those the targets.
What are you talking about? Are you saying 80% of my loyal visitors aren’t from Singapore?
/s
Since most analytics is done with JS (Google Analytics, etc.) very little of this shows up in site visit stats.
Opening port 80 and realizing the world is an anarchic warzone is a canonic rite of initiation for otherwise innocent backend devs and sysadmins.
This is so true. Every junior sysadmin I have trained over the years (including myself) has had a “are we being attacked?!” moment when tasked with WAF report analysis, monitoring fail2ban logs, etc.
Monitoring WAN traffic really gets the paranoia juices flowing.
I remember when you could stand up a website and no bots would scrape it or scan it. It was a lovely time. No one had firewalls or antivirus and things were working fine until the worms and viruses started coming. You could be confident that your guests were real, so much so we had guest counters on many public sites.
Remember when you had to submit a request for google to scan your site?
You still can.
Just build your website yourself as deep in the stack as you can instead of piling up 50 abstractions on top of each other. Some decisions like having your page be accessible by IP can only happen if you use technology like generic http servers (like apache or nginx) from the 2000s instead of implementing the lower stacks and actually thinking about whether that makes sense for a second.
If when you build a website or a backend, your server responds to requests by IP address (for example), you are building a bottom 90% product, and considering most software markets are super top-heavy, (say 1% win), that's ngmi land.
It's so bad the ISPs with CGNAT actually protect their users by acting as a sort of firewall...
So glad wireguard exists. It just drops all packets unless I authenticate with my cryptographic keys. It's like the computer is not even there.
I am always surprised that its considered legal. IMO that's the same as going on the street door by door and checking if one is left open to steal everything inside the house...
Unauthorized access is not legal, it's just not enforced in the slightest.
Illegal and actively enforced are different things. Report it to the hosting company originating the abuse and they usually don't care.
You can also make up fake reports to the hosting companies that do care, to terminate their customers at random.
That sort of vulnerability scanning is at best legally dubious, and almost certainly illegal under CFAA and similar state statues when there's clear criminal intent. That's why the 2022 DOJ guidance regarding non-prosecution good faith security research was such a big deal at the time.
> IMO that's the same as going on the street door by door and checking if one is left open to steal everything inside the house...
From experience: this does happen regularly in some neighborhoods of some cities in the US, and even that isn't always an enforcement priority. So lack of enforcement on the internet, where most the perpetrators probably aren't even in a jurisdiction with an extradition treaty, isn't exactly surprising.
It's not legal.
Most servers with port 25565 open get hits from either Minecraft griefer bots, or from a bot that looks for that port and warns anyone on that server about the risks of leaving that port open. It doesn't take a huge scale operation to spam every IPv4 address in the world, there are only 2^32 of them, and even then many of those addresses are reserved.
There's just two or three such operations btw.
Exactly this. It's no different from a bot pretending to be Googlebot. I've tried reporting abusive IPs to various foreign hosts, but nothing every comes to it. I've settled for just blacklisting excessively abusive IP ranges.
What is your way of detecting them? Just cat your way through your logs?
fail2ban
Almost lol: grep, sort and uniq. If I notice someone is hammering my employer's ecommerce site, I'll block them. It isn't required often so I've been reluctant to spend the time setting up fail2ban.
Is it a multi-server setup? If so, do you ssh into each machine and look at the logs?
It's 3 servers so it's not too much hassle to ssh into them and check it manually.
Google (and other “legitimate” scrapers) publish the ip ranges they crawl from, anyone claiming to be googlebot (or whatever) but not in the ip range can safely be black holed.
Years ago, I built a sort of camo-proxy like, in Elixir. It was doing full passthrough of the User-Agent because the project that I originally built it for needed it for the upstreams (i don't remember why). Anyway, I ended up pulling it into pleroma -- a couple of months later, we had been informed by some instance owners that google itself were sending them DMCA notices, because it was proxying googlebot's request, with its user-agent!
Yeah, It started bothering me enough that I recently put together a system where when a application detects a bad actor(a bot enumerating too fast, a random scan for vulnerabilities, etc) it notifies the firewall. Right now I am just shutting them down, But have plans for a honeypot/tarpit system, something real slow that takes up all their time. Something like "have fun at 300 baud"
I'm doing the same thing. Set up a honeypot API and having it log everything anything tries to access outside that home page. Collected 15,000 hits in a week from 300 unique IPs doing the usual WordPress and .env scans.
I'm just collecting the data now to be used to secure some of my upcoming projects, but I would absolutely also like to take it in a direction where it sends the bots into an infinite slow loop, or preferably something that burns as many tokens as possible for them.
I don't really care about the morality of that. I'm a big fan of fighting fire with fire.
> something real slow that takes up all their time
Not to throw water on your plan, but the bots I've written intentionally run very slow with respect to each target. When done in parallel, across a wide range of targets, it doesn't slow down the effort at all.
Serve them a zip bomb? LLM poison text?
I don’t really have a reason to use WordPress anymore. I’m about halfway through switching my site over to something else that I can control with github and AI. Now these boats are mostly getting static files unless there’s a reason to show something else and even then the footprint is very small. There’s just not that much to hack.
I've done the same by scraping the HTML/JS/CSS off our own Wordpress site and sticking it in S3 (behind CloudFront, of course). It was a mild hassle at the time, but would take seconds with today's LLMs. It was a great decision and has been no hassle since, as we didn't really need a CMS, it was just the default for the person who did our design.
Definitely gotten worse from where I'm standing — you used to be able to just filter on the UA string and move on, now you can't even trust that. Same junk traffic, but harder to sweep away.
If I had a nickel for every time my blog gets a reuqest probing some wordpress exploit, I'd have paid off my student loans years ago.
Numbers say that the number of attacks are increasing: https://radar.cloudflare.com/security/application-layer?date...
Where do you think all the AI investment goes? They expand AI data centers.
Yes, probably, and when a new web-related CVE drops the number of attacks targeted that CVE increases too. It's just the new orange...
On average about 100 (TCP) requests hit my home router per minute doing various probing and scanning. Lots of checking for the telnet port obviously. Sometimes you can see a swarm of entirely different IPs scanning the full port range (probing the ports one-by-one).
You'll see a lot of deepfield, censys-scanner, visionheight.com, shadowserver.io, etc., but also the usual suspects of Chinese or Russian IPs.
With OpenWRT I use something like this: `tcpdump -i pppoe-wan 'inbound and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn'`, or alternatively `tcpdump -i pppoe-wan 'inbound and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and not port 44000'`, if we have some torrent client running (e.g. here at port 44000) which would mess up the result. I'm not sure it's the best way to handle this, but it's definitely enlightening what bounces off on the router.
The easiest way to deal with the usual suspects is to just block the entire countries network range(s). There really is no reason they should be connecting to your home router anyway, and you lose nothing from blocking them.
Sure their packets will still hit your router, but if they are dropped immediately at least you're not wasting a syn-ack on them.
I, temporarly, banned some ip range. I didn't find a source for pinpointing countries; though I am interested. Could you point me to some sources which, deterministically, resolve to some countries? To my knowledge you can not reliably identify countries by ip since this would be dependent on DNS servers. Though I am just a application programmer!
Thanks in advance.
Routers got such a thing build in nowadays, just gotta enable it (not the ones from your ISP of course)
Your router doesn't care about their DNS settings. IP addresses are very easy to tie back to countries. The reason they say it's not reliable is because it's trivial to spoof the country, but even so, a lot of attackers don't even bother. It's sort of like the Nigerian prince scam calls: if you're wise enough to block Russia, you're not worth their time.
Your firewall vendor should supply you with country lists, just select the known bad ones and drop their traffic. If you have a consumer grade router, you will probably have to configure the blocklists manually.
I wonder if adding the US is now sensible.
Unless you're serving something, you should probably drop all incoming requests. If you are serving something, the US is a pretty big market to lock yourself out of.
Almost all spam traffic comes from the US, it's not even close. But for many people so does much of their human traffic.
FWIW, I blocked the US for a while.
Eventually, my lets encrypt cert expired and it turns out certbot is run from USA, so the auto renewal failed me.
> The reason they say it's not reliable is because it's trivial to spoof the country
ISPs sometimes do trade IPv4 blocks and countries to which it belongs do change occasionally. That can become a problem if you were like literally Netflix and someone few nation states over started an ISP.
Getting it directly from IANA is always the best approach.
Here is a "simplified" version in various formats.
https://github.com/HotCakeX/Official-IANA-IP-blocks
Gross - you're going to block countries just because the US government doesn't like those countries?
Do you think you get excluded from mass scans for disagreeing with your government?
I'm happy to unblock China from accessing my website, once China does something about the millions of daily hack attempts originating from that country.
But you pointed to a list of OFAC sanctioned countries, not China.
> just block the entire countries network range(s)
Why not just block all the inbound connections you don't need? Is there a particular reason your firewall policy needs to be xenophobic?
> The easiest way to deal with the usual suspects is to just block the entire countries network range(s).
Keep in mind that this should be paired with an ASN blacklist - MaxMind also has an ASN mmdb for convenience - because IP address to country maps are almost entirely self-declared[0].
For example, Tencent (AS132203), which you almost certainly want to block, has ranges in 73 different countries per [1].
[0]: https://datatracker.ietf.org/doc/html/rfc8805
[1]: https://bgp.tools/as/132203#prefixes
I can corroborate visionheight and shadowserver from my firewall logs.
I have ubiquity UniFi for this reason (amongst others). OpenWRT is a good choice as well. Most home router software is such junk, might as well leave the door open..
This is one reason I don’t mind that I’m behind CGNAT.
That's the only advantage of CGNAT I can think of. :D
(You still could get poked from the other users' hosts behind the ISP's NAT, of course.)
an angry skid in a game can't (D)DoS your router might be one. I'd still prefer to not have CGNAT
Depends if the ISP has client isolation of one method or another enabled.
fail2ban?
Many of those user-agents listed are often faked. Look up which ASN owns their IP. If I block most VPS providers most of the faked bots vanish. There are still some running from residential and phones using hijacked code (readers that are not really just readers but really multipurpose proxies). On that note, do not trust the linked source code but rather decompile the live code your phone is running and have AI analyze it.
Yeah, that's exactly what these visits are: faked user agents that fail IP verification or Web Bot Auth. What's interesting is the surge across so many websites in the last week.
There are many possibilities but one of them could be some new vuln was released and they are looking for it. That would require looking at the URL's they are requesting. Botters run their own purpose built campaigns. Do you also have a summary of URL's requested by unique counts?
Looks like many of the paths relate to AI coding tools. There are some examples below the chart
You keep repeating this about a small minority of the tools that were posted.
I've had a similar bump in scanners in the past week, more than half of it is coming from MS and Google owned IPs and all of them are spoofing AI agents.
Same for the origin IP address. The fiber leaving your country is tapped, and those people can inject packets with any origin IP that they want. Your ISP has no way to check if their peer actually received a certain packet from a certain country or not.
From a technical perspective, all this "china/russia" attribution is built on a quite shaky foundation. As a sysadmin you'd never know if it would be the British crown attacking your European company instead.
Not minimizing nation state cyber crime here, but the packet goes through many hands with different incentives.
Problem here is there are not single fibers attaching (most) countries, but a bunch of them. If you control both the ingress and egress for some particular users it's possible, but if you don't then your probing packing may end up back in China with a lot of evidence of backscatter.
I'd be surprised if there is a single route from EU to non-EU countries which does not pass through British control.
Does Britain own all fiber links between Switzerland and France?
Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.
> /commercial IP from legitimate users
No, because legitimate users do not just use residential and "commercial" IPs. Like me, right now
You are the 0.001%
Much more than 0.001% of people care about their privacy or (the larger portion) do not have unfiltered access to the internet.
I second this. When I have tested blocking VPS/data-centers to my silly blog there were about a dozen people on HN [1] that could not view my site out of the roughly ~17,000 (not counting bots) that could. It's not a big number but those are real people and they count.
I am going to move full blocking to a test node that people can play with but I have to finish working with Claude to revise someones repo is is no longer maintained because one does not simply put an anonymous chan board on the great wide open internets without some critical thinking.
[1] - https://news.ycombinator.com/item?id=49060945
What did you use for detection?
The complaints in the thread. I am aware the lurkers would not have said anything.
There were a dozen people who reported not being able to access your site.
The complete hysteria people go to over the near-non-issue of bots is, well, completely hysterical. Don't be that guy.
Yes but it’s not cheap. Maxmind and ipinfo etc sell a tier that tells you this information, then you can 403 based on it. But the price is nuts like $40,000 a year.
You don't need to spend anywhere near $40k a year to get that info... You don't even need to spend $1
What's your strategy?
Focsec.com IP database for offline use (datacenters, VPNs, proxies, bots) runs around $1k/month for internal-only use.
I work for IPinfo. We offer IPinfo Lite for free. With a little bit of time in identifying the ASNs, you can implement a decent way to block a good number of bots fairly easily using the free data alone.
Good to know, thank you. Would you do this by fully blocking particular ASNs? Or something more granular?
You can block entire ASNs. If you are frustrated with bots, blocking Tencent's entire IP address space would have very few downsides.
If you have fail2ban or NGINX logs, you can use our CLI to summarize those IPs and identify the ASNs you want to block. But before you block entire ASNs, make sure they are not classified as "ISP" type. For that, visit our website's ASN page first.
I have quite a few community posts around this approach. https://community.ipinfo.io/
If you have raw logs, you can send them to me as well, and I can review them and provide some guidance.
Appreciate it, I'll check out your posts.
Open source works just fine for this - take a look at geoacumen-country
https://github.com/geoacumen/geoacumen-country
Isn’t that only countries though? They were talking about detecting VPS
See my comment in the parent - there are cheaper options, don't use maxmind!
I run an open source honeypot that collects these botnet scans and produces blocklists.
Blocklist download and configuration: https://knock-knock.net/blocklist
Honeypot dashboard, where you can see attempted attacks in realtime: http://knock-knock.net
API: http://knock-knock.net/api
This looks cool, where can I find the source?
It's on github with an MIT license: https://github.com/djkurlander/knock-knock. Have fun!
Thank you!!
Thank you for sharing. I will take a look.
Cool site. I was curious and dropped your 100k list into a reverse DNS lookup site [1]. They may still have some of the records cached. I recognized quite a few of the scanner nodes and some other usual suspects.
[1] - https://adver.tools/reverse-dns-lookup/
Yeah, that’s pretty interesting! You can also see a live view of the ASN/ISP leaderboard by going to https://knock-knock.net and choosing ISP from the carousel. That’s ordered by bot transaction count rather than IP count though.
It never ceases to amaze me that these ISPs don’t bother to shut down the botnets. They could do so very easily. For example, they could identify the IP address of every bot that hit this honeypot with their ASN with one API call: https://api.knock-knock.net/check-asn?asn=<asn number>. (See https://knock-knock.net/api). They just don’t care!
The ISP's do not have a financial incentive to shut them down. To them that's a paying customer. The feds will go after the big botnets if they are touching financial networks or siphoning enough money from people because there is usually a few big bank accounts and virtual currency exchange accounts they can seize once big enough to look good in the media. That's why it's on us and a few big CDN's to block some of them.
What if I have a slimy TV box or nasty on my phone, living on my network?
You take IP down, you kill the cancer but you also end up killing the patient.
Not the person you are asking but site operators can not tell intent. It could be something nasty on the network or a botter feigning ignorance.
I'd say its probably an acceptable casualty in the battleground that is the internet especially for little one-off sites hosting blogs, forums, chat servers, etc... For a bigger site I would expect that person may have to open a ticket with the platform such as Amazon accepting that some CDN's and firewalls may be harder to get the block removed. This is why we can't have nice things.
thats the hard part, right - my 76 year old dad is on his banking app while his samsung TV is allowing a bot to try and take over other accounts at the same bank on the same IP.
IP Blacklists, no matter how good can't stop this. You have to start using stats or deep-diving telemetry.
https://darknetdiaries.com/episode/172/
thats the hard part, right - my 76 year old dad is on his banking app while his samsung TV is allowing a bot to try and take over other accounts at the same bank on the same IP.
So appeal to emotion doesn't fly with me. If grandpa is 76 in the year of our lord 2026 that means he was 50 when the internet was getting popular and 59 when cell phones became very popular on the internet. He's not much older than I. He knows what's up.
God help the makers of that television if he finds out it has been spying on him and dorking around with his traffic. If they are lucky he will just take a baseball bat to it. If they are unlucky he will fly to their headquarters and end up on a viral bodycam video likely with a lot of supporters that will bail him out of jail.
IP Blacklists, no matter how good can't stop this. You have to start using stats or deep-diving telemetry.
I use a myriad of methods including IP blacklists. That's my choice and every site operators choice. I do not have to use deep-diving telemetry but you are free to do so.
It doesn't dork around with his traffic, it makes its own traffic. In exchange for a discount.
Nothing wrong with contacting the customer, and taking down the IP briefly until the customer can deal with their malware hosting toaster.
However, I can see the argument for giving the customer 24-48 hours to resolve the problem.
Better yet, it's the only CGNAT address for a little town in Brazil. You just banned an entire town from your site.
There are several methods. [1] The most aggressive method-02 and method-03 on my document will block VPS and some data-centers but that also means it will block some legit users that are on a VPN. Most VPNs transit a data-center. If experimenting with these methods use a test server that you do not care about and set up a dummy site and ask people in your circle of friends to test it. I have to step away for a bit but if you have questions I will try to answer.
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
Your site does not currently seem to be reachable / responsive when I try to reach it from a US Comcast IPv4 address - you are not advertising IPv6.
Edit / Update: It was Apple's Private browsing mode that causes it not to respond. I can now see it when this is disabled.
I've noticed they strip away a header [1] in private browsing mode but I don't know why they do it since it does not disclose anything about the person. I think that may be the same thing that causes some people grief on Cloudflare as well.
[1] - https://caniuse.com/?search=sec-fetch
Per your link...
> block http 1.1, real users only use 2.0
Chrome on android and Firefox on linux both appear to use 1.1 still...
By default they use 2.0 [1] unless someone or an addon disables it or unless the person is on a really old version. OperaMini however will use 1.1. No idea if anyone here uses OperaMini.
There are some reader apps that act as a proxy that only support http/1.1. Be careful, some of those are not just readers and do not trust what they claim to be the source code. Some of them are created by cute and fuzzy bunnies.
There are a number of botters on HN, some that control residential and phone browser-hijacked systems. One was sending me playful messages the other day. I enjoyed the bot block-jousting with them.
[1] - https://caniuse.com/http2
Apparently nginx didn't enable it on my new server... why it's not enabled by default, I don't know... I retract my idiocy :P
No idiocy, sometimes defaults change or don't get updated by a deployment script. Different distributions may have a slightly different default configuration file depending on how involved the artifact maintainer is with the project. Or put another way, I've done far sillier things.
No, because the bad guys use residential proxies if you block DCs. They just prefer not to because it's more expensive. Meanwhile that silly HN project that scrapes and reformats your site is now dead before it began. Who are you actually defending and from what?
I did just this. Using a $2k a year database from a smaller provider that isn't maxmind, claude and I built a pretty slick ASN based categorization system. I can categorize an ASN as a residential IP, a service provider, a legit crawler/scraper, etc. For anything that is suspicious, I dynamically use turnstile to gate access to our service. Turns out there's no ISP for any VPN, they just contract with a shitload of mom and pop shady colocation services across the world.
We collect signals that help determine good vs bad networks. For example, large amounts of requests to .php endpoints, large amounts of empty accounts from the same /24 subnet, etc etc. All these signals let us automatically determine risk, and then put up a challenge. Authenticated users never see the challenge even if they are on a risky network (VPN 99.9% of the time), unless the network has been identified as 100% malicious, then it gets a full block.
Here's a small snapshot of the dashboard:
https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358...
This was probably a total of 3-4 days of work, spread out over a couple months of iterative claude led hacking. I didn't know exactly what to build, but had some of the key architectural ideas in my head. Opus+Faable made easy work of it all, and ended up guiding some really slick improvements for performance.
I would say this has dropped about 20% of all traffic to our service, though it turns out turnstile is a massive target for bots, so replacing that with something custom is next on the list.
Contracting with their colocation facilities is exactly how that's supposed to work. If you don't actually operate a wide area network then you aren't supposed to be registered in these databases and have IP blocks. The exception is people who do anycast, but VPN companies don't.
You know all these guys just switch to residential proxies if they detect a site is blocking data centers, right? Because that's a very common thing to do.
Not sure what you mean by your first comment - there is no technical reason that I know of that prevents a VPN provider from having their own ASN and address space.
As for the latter comment....not sure what your implication is. Yes, bot/spam mitigation is whackamole, but there are consequences for not playing the game of whackamole. Luckily residential proxies are few and far between so far, but they will grow in popularity. When they do, and I can't get by with the occasional individual residential IP ban, we'll come up with other methods to handle.
Luckily the signal is strong with vulnerability scanning, which makes it pretty easy to automate. The only reason to put up whole ASN mitigation (captcha/turnstile, outright bans) is just efficiency. Nothing stopping individual IP banning. The scrapers are the tricky ones, since they more easily hide in legit traffic. However legit traffic has patterns that scrapers do not emulate (at least for a service like ours with millions of pieces of user generated content that's easily walkable), so you can still pull out the signal. It's just a little trickier.
Definitely a continual arms race though.
Majority of scraper traffic right now is from residential proxies
That’s not the pattern I am seeing, but I might be outside the norm. The majority of bot action (scraper, spam) comes from vpn providers by a long shot, ignoring (Chinese ASNs, Indian mobile ISPs etc). I see very little consumer isp action except cheap international providers, which are still swamped by vpn traffic.
Mass automated vulnerability scans have been a very common thing since years before the advent of this in 2001:
https://en.wikipedia.org/wiki/Code_Red_(computer_worm)
I remember when 'code red' spread and it had the effect of crapping up the contents of my apache server logs. Fun times.
such as:
GET /default.ida?NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a HTTP/1.0
Worth saying out loud: user-agent is not identity. Verify AI crawlers by reverse DNS or the provider's published IP ranges - the ones worth letting in all publish them.
Why would you voluntarily pretend to be a AI bot, when those have already a much higher chance of being blocked? Seems holly unproductive.
Best hypothesis I can come up with is to somehow make the AI companies look bad, but they seem to be doing an excellent job at that themselves already by scraping everyone hundreds of times per hour over and over.
Most websites don't have an incentive to block AI bots to their main sites. Think businesses, government and community websites, nonprofits, etc.
Because businesses dont want them blocked, that would be a very stupid thing for most of them to do given its becoming a vital traffic source now that people are using chatbots instead of google.
From what I have seen at work, everyone is using chat bots but no one is visiting websites through them. We still get almost all traffic through social media and google search.
the numbers are much smaller than traditional search but they are trending up, and they convert at almost 2x the rate of organic search inbounds. sure you can play catchup later, but the trend is quite clear. if there's one thing the last two years have taught me, my prior heuristics on now vs future don't work in 2026.
Yeah. Do you want the bot to buy the product from your website or your competitors?
How many from a direct link without a tracker? That’s usually how I use it for search, get the answer, find the sources and copy the link to a new tab
Googlebot and other web scrapers are how you get visible, how you get adwords, ways to boost your traffic, etc.
bots hitting your site aren't problems per se
Very similar experience here. Started July 30, sustained through August 6, when it started a significant ramp-up in volume (5x or so).
Most of the traffic is originating in GCP. We're seeing ~70k req/min sustained from Google Cloud IP space (AS396982). Reported to GCP Abuse, they've been non-responsive so far.
The main distinguishing factor is the reuse of a bunch of legit AI-training bot UserAgent strings. It's clear that the traffic is under the same centralized control because of how it changes volume across thousands of IP addresses simultaneously.
Seems like you’re part of the group represented in this dataset trend then, many of these visits are also from (compromised) Google servers in that same ASN.
If how they’ve handled Gmail abuse is any indicator, they’re not likely to do anything. They’re still getting paid for the server time
I recently blogged about some Cloudflare Workers I developed to combat this type of traffic: https://code.backwater.systems/blog/#2026-06-29T23:40:00.000...
Using a normal page per blog entry would go a long way to making your site more indexable, readable, shareable and seo-able. (Good article btw).
That is a “normal page” for a single blog entry. What are you talking about?
Oh, the link is entirely contained in the fragment. So it’s some SPA blog thing. I get it.
In my haste to initially get a blog started and published, I created it as a single HTML file with fragment identifiers / links for each post. Earlier today, sethops1’s valid feedback prompted me to restructure it as an index page with separate pages for each post.
Links to posts no longer contain URL fragments, but I kept the fragment-style links working with JavaScript.
For those suggesting fail2ban as a solution, that's dinosaur software from the palaeolithic. If you have a website of any size then the number of bots will overwhelm the block list in days with their millions of unique IPs.
Unique IPs is not something AI can generate easily, and blocking IPs in general is not efficient.
The big problem with AIs is that they can try new paths/payloads very easily, adapt quickly, I'd be more worried about that.
fail2ban has always just been a log cleaner.
Besides, projects with lots of dependencies are taking on more risk than they realize. If one dependency gets compromised, you have no idea how many projects are affected until it's too late.
Fake Googlebot visits are #1 in website logs I've been working on. At the beginning I was fighting with them using Cloudflare ASN block rules or their managed Bot Fight mode but it appeared to be not only pointless, but also harmful for my websites. Bot Fight mode randomly started blocking real Bing / Google / OpenAI crawlers what wasted crawling budget and discouraged crawlers to revisit updated pages.
Sometimes it's better to not fight with bots actively but harden environment and only react for the worst offenders.
For Google it's pretty straight forward to throw away fake crawlers by just only allowing their published list of crawler IPs so you don't accidentally allow someone from a random GCP IP to crawl you if unwanted (https://developers.google.com/crawling/docs/crawlers-fetcher...).
Also Googlebot is an AI training crawler so if you block AI training crawlers you should just block it.
With Cloudflare you can set a rule to block traffic that identifies as Googlebot but is not a "verified bot", ie is not from the proper IP range.
Interesting thought: what if the idea of an open internet is over.
What if we're now moving into a world of strictly KYC. The same way "The Facebook" generated massive revenue by creating a KYC world.
Just in case any of the authors read HN, I'm getting a pretty crazy rendering bug on this page, where a bunch of the contents are redrawing up and down by a few pixels. It seemed to go away with resizing the width a few times, but I didn't look into it too hard. My page width was probably small on first draw. Incredibly distracting though and hard to read with the text moving. Using latest chrome, and it occurred on more than one page refresh. I didn't dig in beyond that though.
I'm seeing the same thing on Firefox on Linux. It almost looks like the page scroll is jiggling up and down a tiny amount constantly when it's supposed to be stationary.
How can I attract more of these bots to my server? I want to test my Apache bad bot blocker. It uses basic header fingerprinting and h2 support to filter them. I get less than 5000 hits on an average day, and want a lot more.
run a redlib instance, bots hit these hard: https://github.com/redlib-org/redlib
I went from 2k hits a day to 15k in the past week.
Point a domain at your IP, use letsencrypt, post your domain on Reddit, github, x, etc. The bots will find you.
How letsencrypt "helps"?
Certificate Transparency logs
Whenever a new certificate is issued the bots seem to all monitor that and immediately start scanning the new domain.
I have a domain with an SSL cert. https://solarsi.ca/ I tried posting on /r/sysadmin asking for help testing a bot blocker, but a moderator quickly deleted the post, claiming it was marketing/promoting.
Thanks for the github suggestion; I'll add a repo for the bot blocker Apache config, and include the above server URL that I'm using to test the blocker.
Someone is always running mass vulnerability scans. That's a "water is wet" state of the Internet.
Definitely, the point here though is there is a stat sig surge in the chart in the last week, across thousands of websites. At least a surge in this particular spoofing pattern.
It's still not really anything special. Thousands isn't even large scale.
Any random bozo can trigger that.
This is a random sample of completely unrelated websites, which indicates that the total scale is much larger. This is not saying that it is difficult to make thousands of requests.
Is this your company? If it is, your cheerleading makes you very hard to trust. If it’s not, I’m sure that everyone gets the point - you adore everything about this research and can’t see any possible problems.
Not asking for trust, just sharing the data/math
You forgot the "Yes, that's my company" part in your reply (https://ghking.co)
That doesn't change the data/math brother
It's still good etiquette to disclose affiliation in online discussions.
Fair enough
It changes our evaluation of the likely reasons that you are ignoring the reasons that you are wrong.
I think this is more of a "if you left your AI tools exposed someone is looking for them" change. Hacking someone else's agents sounds like a great way to spend less on your own tokens.
Nah these are just the typical vuln scans that try all kinda basic shit, looking for anything commonly used AI tools included.
I'm not sure what to blame yet, but here's traffic on a tiny side site, all from JS-capable clients: https://i.imgur.com/tdexrEI.png
How about them apples... ai bots use faked browser user agents, so people start pretending to be ai instead...
(Insert spider man meme)
Fork found in kitchen. This is nothing new. Don't have mass vulnerabilities and you have nothing to fear from mass vulnerability scanners.
Hah, what fools! I've been serving empty responses to AI scraper UAs for over a year now.
Man, this "someone" guy sounds like a real jerk!
Why would some of these ignore robots.txt some of the time?
No Small Actors.
Sorry, my bad.
Or some thing!
Looks like Google has started rolling out this Web Bot Auth thing which seems like something that should gain adoption or become an open standard. https://developers.google.com/crawling/docs/crawlers-fetcher...
Seems like the crawler companies would be incentivized to not want to take responsibility for people spoofing their user agents.
Yup, in fact most of them are already. That's one of the ways this data is verifying whether the visits are spoofed or not: https://knownagents.com/insights#spoofing-and-security
This kind of stuff is getting pretty wild, even for using something like Cloudflare it seems like a good idea to have another layer behind it that's non-cloudflare for when vulnerabilities are discovered.
What, like some kind of firewall?