A year of fighting scrapers on my 1.5 million-page website (patronview.com)
397 points by petercooper 13 hours ago
jwr 12 hours ago
The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see.
A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.
And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.
hk__2 12 hours ago
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user.
No; in this case you are not a user, you are a bot user.
matsemann 12 hours ago
I don't disagree, but there is a sliding scale here.
For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself.
However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.
weego 9 hours ago
hluska 12 hours ago
Terretta 12 hours ago
you know what a browser is called by the web site? check the header that tells the version.
USER AGENT
not user, an agent on behalf of the user.
the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT.
for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
the alternative is all your in home information owned by corpos when the info should be from user agents not corpo feeds. if you want to vote this idea down, you might be a corpo. :-)
jt2190 12 hours ago
zamadatix 12 hours ago
shakna 2 hours ago
pixelat3d 7 hours ago
deathbob 11 hours ago
pessimizer 9 hours ago
dspillett 12 hours ago
asgraham 12 hours ago
charcircuit 12 hours ago
ako 12 hours ago
The best way to read the information on the internet today is via a LLM.
Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.
wulfmann 10 hours ago
roboror 2 hours ago
ballooney 12 hours ago
logn 3 hours ago
Writing a script to fetch HTML is no different than writing a web browser. I think it's the scale of the operation that distinguishes bots vs human. The browser is the user's agent, but not the only one.
godwinson__4-8 8 hours ago
"Bot" access on behalf of users should be fine, even preferred. The missing piece is some micro transaction layer and some sort of attestation somewhere in the layer that the person driving the bot is not a bad actor.
Equating bot with bad actor in 2026 is Luddite behavior. Driving people to your site so you can serve them adspam or just make whatever operation they want to do 10x more difficult is the same.
MCP style APIs should eat the web. This doesn't mean the "open" web goes away. > 99% of the time I don't care for a domains particular FE at all. It's a complete waste not only of time, but resources and bloat. Pushing the contract into the agent should become good UX. Making things harder for good faith users should never be the goal.
pwillia7 12 hours ago
There is bot traffic initiated by a human and bot traffic not initiated by a human. I would want to serve the first but not the second if it impacted my cost/performance at all.
lefra 8 hours ago
skinfaxi 12 hours ago
I assume you are unfamiliar with the concept of user agents? Otherwise your browser would count as a bot user no?
And if not, what if it was a custom browser and not Chrome/Firefox/Edge?
andai 3 hours ago
A web browser is an entity that acts on your behalf. That's why it's called a user agent.
They're just better at English now.
Buttons840 9 hours ago
I think it's time for people to build a local database of every site they've ever visited, and then they can give their LLMs access to that. I might be willing to pass this data off to a company to store for me. Companies already store all my emails and money--why not trust them with this too?
Like, all the comments of this post would go into my personal database simply because I loaded the page, and it would help me find old information I've read, and could also inform LLMs I use.
This should be built into browsers.
elorant 12 hours ago
What if I’m a web alerts company? I crawl your content but my clients are all end users who actually see your content at your site.
hk__2 12 hours ago
pessimizer 12 hours ago
criley2 12 hours ago
Scenario one: you use software to connect to their server and download a webpage. You are a user.
Scenario two: you use software to connect to their server and download a webpage. You are a "bot".
Make it make sense
hk__2 8 hours ago
--_-- 12 hours ago
selckin 12 hours ago
scotty79 6 hours ago
Are you gonna police the means through which I'm browsing the internet? If you want to, you are free to put your stuff behind the paywall and give the key only to people who agree to obey your conditions. If you put it in the open you can't make conditions. That's what publishing means. Author can't make demands in what manner their book should be read.
ted_dunning 4 hours ago
eigencoder 12 hours ago
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.
aomix 12 hours ago
The social contract of accepting scraping for visibility was always tenuous and is now fully dead. But it’s shocking the number of people who are essentially victim blaming here. You should spend YOUR time to optimize your free website so MY use case is not impacted. How is that not incredibly selfish on its face?
autoexec 11 hours ago
andai 3 hours ago
...because they want to make it harder for people to get the information they need?
I think I must be missing something here.
paul7986 12 hours ago
Exactly and Cloudflare (there should be others out there too - open source options) has proposed that AI bots need to pay for access to our websites. If they and or others could pull off blocking AI access until it pays creators then AI is forced to pay as it should and always should've!
Overall systems and different business models need to be created that forces AI to pay it's fair share! Elon says there will be an abundance thanks to AI and we wont have to work -- ok how's that gonna work without different systems in place paying us?
I feel strongly about this topic and proposed some systems in this Substack post https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans...
m463 9 hours ago
I am denied by cloudflare CONSTANTLY on one system.
I have an old os (macos 10.11), running the highest firefox esr I can run, and I get denied by cloudflare.
But not always immediately - I get to enable javascript/cookies sometimes just to be denied.
they are not the folks we want gatekeeping the internet, they are opportunists increasing their OKRs
qingcharles 9 hours ago
And here's the rub -- the bots are "running" the latest "MacOS" and have no problem accessing the site.
autoexec 12 hours ago
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user.
People trying to block bots end up keeping out all kinds of users. I get blocked frequently for using a regular browser, just with JS disabled. 99% of the time, I just close the browser tab and move on with my life.
BLKNSLVR 4 hours ago
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
And this is a complaint? I find nothing in this statement to sympathize with whatsoever. Isn't the contract of, essentially everything, that effort is required to obtain something worthwhile?
Please let me know where I'm getting this long-standing, fairly fundamental understanding of the world, wrong.
andersmurphy 10 hours ago
Agreed! Cloudflare absolutely destroys user experience and honestly doesn't seem that effective in practice.
What's worked for me is I block any client that don't support brotli compression and http2. Seems to work well enough for stopping scrapers.
binaryturtle 8 hours ago
I'm getting a "browser not supported" by the Cloudflare check. So I guess the "job is well done", and the user is lost.
aeddZX_0 12 hours ago
How do you monetise bot traffic?
subscribed 7 hours ago
Ibwas looking for something specific late last year, and I made three bookings (ca €500 each) for places found on the $site with the $bot.
I didn't ask the not to use this site, my query was broad and complex and would take me days to do it myself. I wouldn't
I'm quite certain they earn a hearty commission off it, and I think it was mostly possible because the site was "friendly" to bots. Otherwise I probably wouldn't choose the site because it's never any of my top choices when I look for this myself.
So, maybe you monetise it like this? You asked, I answered. Doesn't fit every site or business profile.
autoexec 11 hours ago
You take your site off the public internet and paywall it off. Websites on the open internet should be accessible and ideally the goal should be to share something cool with the world, not just to make yourself rich.
drnick1 9 hours ago
testing22321 11 hours ago
paytonjjones 12 hours ago
dspillett 12 hours ago
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user
You are mistaking yourself, well your bot, as his target audience.
You might as well say “If I want to send you my commercial email, and you block it, you hurt me, the email user.”.
While your point of being concerned about cloudflair becoming a global arbiter of who gets in and who does not (which may at times not just mean blocking bots, intentionally or through technical issues), the need to block the deluge of bot traffic is very real for many sites and that is one of the easy options for them to deal with that. There are other methods like directives in robots.txt and nofollow attributes on links, but so many bots simply ignore those that they are not really useful.
> Bot traffic is not necessarily bad.
Nor is it necessarily wanted. In fact, it often isn't. Unfortunately practically all bot runners seem to either assume that their traffic is the special good kind or not care either way.
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
I WANT! I WANT!! I WANT!!!
Well, that site runner wants you to access the site as a human, if at all, not via bots. Sorry to be the one to break it to you, but what you want isn't always the most important factor for the rest of us.
> but the real problem is that the website is poorly built and should be improved anyway
Firstly: just no.
Secondly: if you and your bot don't like our badly built sites, feel free to go get your information from those that you consider to be better built. Problem solved.
BLKNSLVR 3 hours ago
My understanding would be that, if an API is available, then bots are essentially welcome. If no API is available and the website has to be 'visited' then it seems that it's intended for human consumption 'the old fashioned way'.
TheRealPomax 2 hours ago
No one was hurt, you were at most inconvenienced. And that's perfectly fine.
Also, using jargon that doesn't apply: a knee-jerk reaction is one that does way too much to address a small problem. Objectively, this is the literal opposite: targeted reactions to different aspects of a huge problem.
testing22321 11 hours ago
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
Substitute the word “website” for book, or training course, or documentary or published paper, or patent or one of hundreds of examples.
Now you see the problem.
ethin 8 hours ago
And if you believe that blocking bots is bad, then by all means provide a better alternative that doesn't increase costs for sysadmins. If you believe that the website should just be improved, then by all means feel free to provide instructions on what should be improved and exactly how so that we don't ever have to block bots anymore. I'm sure all the sysadmins having to deal with issues like this one will thank you
buzer 12 hours ago
> If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse.
If the the processing is subject to GDPR (e.g. if controller is in EU) then you do have recourse. You can complain to DPA or sue the company. The company is ultimately responsible for the decision to block you, at least in cases where you personally tried to access the site.
arcrevenant 9 hours ago
Jokes on them, the second I see that “Verifying you are human…” redirect I leave and never return.
tonyhart7 12 hours ago
why you blaming cloudflare that try to solve botting issue and not the Botters ???
you literally can turn off cloudflare and use your own solution
skinfaxi 12 hours ago
Sometimes the cure is worse than the disease.
compiler-guy 12 hours ago
moralestapia 12 hours ago
>That is not the open web that I would like to see.
Cloudflare is opt-in so I don't see that being an issue (yet).
skinfaxi 12 hours ago
Being opt-in doesn't negate the fact that it is closing the web.
moralestapia 11 hours ago
IsTom 11 hours ago
Internet walled gardens are all opt-in and still they've made the web a worse place.
hluska 12 hours ago
This is a whole lot of things you want and feel entitled to. Nobody has to cater to you - we can block whatever we want to block. And if our poorly designed sites bother you, that’s too bad for you. But your wants are not my problem. If you want someone to cater to you, pay them. You aren’t entitled to anything.
BLKNSLVR 4 hours ago
The attitude you're describing seems startlingly common, as if it's not the actual site owner that has made the active choice to block something that's causing them trouble.
They all sound as if they've been logic-twisted by some product idea they think is going to make them rich, and these blocks on bots are costing them access to the raw materials for their magnificently worthwhile project.
wolrah 12 hours ago
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
The article you're replying to describes in explicit detail how the bots and their operators have directly caused and continue to knowingly cause real harm to the author and others in similar positions, both financial costs and administrative/maintenance burdens that would not have otherwise been required.
You then respond "but if you block the bots then I won't be able to use the bots, and that harms me because I might have to read your web site myself..."
Are you serious?
> That might or might not be what you expected, but it's worth taking into account.
I would wager that for almost everyone who is blocking bots after getting functionally DDoSed by them this is absolutely an expected and desired outcome.
> And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway.
Both can be true. If you operate a git repository with a public-facing web interface for example there are going to be a lot of possible operations that are inherently expensive but also incredibly rarely used by normal users so it doesn't really matter, but the bots now ignore your robots.txt and are programmed to go after every link they can find, so they trigger every single possible expensive operation more times in a night than your actual users ever have in the history of the site while dividing requests across so many different IP addresses that rate limiting becomes impossible at the individual scale. These days they're even feeding the discovered URLs back in to their models to have them invent new possible URLs and trying those in hopes of finding content never publicly linked. They will send you thousands of requests for URLs that they literally made up.
Sometimes the site is in fact badly coded and operations that should be simple have higher costs due to bad design but you don't have to look very far to find situations where legitimately high-cost resources are exposed to the public because they're expected to be used in a non-abusive way. We should always be standing up against abuse of public resources, unless we want to lose them altogether.
> Bot traffic is not necessarily bad.
You are right, but whether it's good or bad more or less comes down to a cost/benefit analysis. As we've already covered infinite times, these bots being used to train LLMs cause significant real costs to the operators of these sites. What benefits do they offer in return? We know the clickthrough rates are terrible, so what other reasons would site operators have to make those real costs worth it? So someone can get a response back from an algorithm that confidently misinterprets or even entirely misreports what the data actually meant?
Even the most die-hard "information wants to be free" types who absolutely want their datasets trained on would probably prefer that the bots accessed the data directly via an API or downloaded a database dump rather than spidering and scraping a web interface intended for humans.
bob1029 12 hours ago
> the real problem is that the website is poorly built
This is almost always the problem.
This is currently the problem that GitHub is having too. If they had remained with their crusty old rails architecture, very little of this mess [0] would be occurring right now. They would have been able to focus all their engineering talent on scaling the product rather than inventing elaborate client side state synchronization mechanisms.
throw93003838 12 hours ago
We tried to deploy private cloud github enterprise node back in 2018. It was pure garbage without CI integration. I am happy microsoft took harder long term decision!
GodelNumbering 12 hours ago
I just checked Cloudflare for SignalBloom (https://www.signalbloom.ai, which I own and operate).
Over the last 72 hours, Claude-searchbot [1] alone fetched ~205,000 pages. Sent exactly 1 referral. There is a lot of free financial data on the site, hoping for real users to benefit from it. It is hard to not feel a little cheated out that Claude gets to claim "Found it!" to its users without me getting no credits or compensation whatsoever.
[1] Exact user agent `Claude-SearchBot/1.0; +searchbot@anthropic.com)`
Proof: https://i.postimg.cc/Pqc3SS8T/Screenshot-2026-08-07-at-5-33-...
markdown 11 minutes ago
FYI, your proof is hosted on a NSFW page. You should have mentioned that.
johnorourke 13 hours ago
Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software.
basilikum 13 hours ago
It doesn't detect "real browsers". It simply adds some friction and is niche enough that AI scrapers have not bothered to bypass it yet.
leros 12 hours ago
Yeah this is one of those things that hurts some real users and AI scrapers have figured out how to bypass. Not worth it IMO.
mostlysimilar 10 hours ago
axus 10 hours ago
There is enough visceral hatred for the Anubis branding, I am surprised an AI skill for bypassing it hasn't been broadcast yet.
randomblock1 10 hours ago
Most of the friction is just JS overhead for the computations, a compiled solver is like 1000x faster. If Anubis ever gets popular enough that scrapers care, it would be trivial to defeat. And last I checked you could bypass it by just modifying the user agent
RattlesnakeJake 13 hours ago
I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
vablings 12 hours ago
It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.
duskwuff 7 hours ago
zuzululu 9 hours ago
altairprime 12 hours ago
> is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
Most people either don’t know or don’t care about “those subcultures”. I bet a lot of older people think it’s a cartoon figure of Betty Boop (nurse) and miss the furry bit since it appears and disappears quickly. Most people also don’t have a brand.
So it seems like you’re describing a concern that only affects a tiny fraction of people:
- Not interested in paying for custom branding, so obviously not a corporation or influencer
- Dislikes cartoons
- Aware of, and hostile towards, “furry” subculture
That has to be an exceedingly small fraction of potential users of Anubis, and given how much businesses and branders will pay to custom-brand something, I’d counsel them to stay the course. Sure, a few never-payers will never pay, but they wouldn’t have anyways, so they can cope with Nurse Betty or look elsewhere for a competing free product.
If you think about this in physical market square terms — in other words, a bazaar — it seems horrendously rude to complain about a shop logo sticker on a free product handed out to anyone that walks up and asks for it. If you want it white-labeled so you can write your own name/logo on it, you pay for the privilege of displacing their name with yours. But you don’t stand there and loudly complain that their shop mascot has dog ears while holding a freebie bag of product, without losing the respect of everyone who hears you doing so.
teddyh 13 hours ago
If you have a brand or personal image that you are investing in, you can surely afford to invest in paying for a branded version of Anubis.
miladyincontrol 11 hours ago
I'd argue its rather functioning exactly as intended with regards to obtaining paid users.
Also this view seems a bit elderly. For most towards the end of the millennial curve and younger, anime is no longer subculture, its just general culture at this point. Although I would agree it isnt necessarily what you'd want for every platform and web presence.
pibaker 9 hours ago
How hard is it to maintain a fork that changes nothing except the logo?
If you can't be bothered to maintain a trivial fork, then why should the author of anubis be bothered to serve your branding needs? It's not like you have a service contract or anything do you?
gfaster 13 hours ago
I think that's partly why they do it? If you care about that, you should pay?
smolder 13 hours ago
ivanjermakov 11 hours ago
Pay to change logo??? Just fork Anubis and remove one div... It's not SaaS first software, you're expected to deploy it on your web server.
inigyou 13 hours ago
They're not stopping you. They're asking you politely not to.
xena 12 hours ago
I wish my rent would stop going up year over year.
ryan_n 12 hours ago
You just discovered their business model congrats.
pc86 12 hours ago
It's MIT licensed, you are free to do whatever you want to it, including removing the logo. They're basically just saying "we'd prefer you didn't do this, but we're not preventing you from doing it."
RobotToaster 12 hours ago
haproxy-protection is an alternative.
peter_stokes 12 hours ago
you really revved the weebs with this one
duskdozer 11 hours ago
RattlesnakeJake 7 hours ago
brazukadev 11 hours ago
that's really smart of them. Just pay to use the service you need.
bakugo 12 hours ago
It's open source, just ask your AI agent to change the logo.
xena 11 hours ago
righthand 13 hours ago
You wish they'd give your preferable imagery for free and not make you feel bad for using their free software for personal gain. Your subculture is irrelevant and not special.
1bpp 13 hours ago
If you 'don't mesh' with that then you are not worth protecting anyway :)
tsunamifury 13 hours ago
D2OQZG8l5BI1S06 11 hours ago
Because of this wasteful crap the internet is so slow nowadays...
Try opening gcc bug tracker on your phone: https://gcc.gnu.org/bugzilla/
esperent 11 hours ago
This is like getting angry at cookie banners instead of all the companies tracking and selling your data.
You're complaining about the symptom (needing to have these checks) not the cause (if they don't, 99% of their traffic will be bots, the site will slow to a crawl and be unusable anyway).
In any case, I saw the dumb anime girl for about 2s then the site loaded. Not a big deal.
xena 11 hours ago
The GCC bug tracker uses the meta-refresh challenge, which does not require JavaScript. Due to the fact that the server makes sure the client has waited at least 75% as long as it should, the HTML has to add one second to the meta-refresh wait. Patches welcome. Meta refresh granularity is in single digit seconds.
cuu508 11 hours ago
The Anubis challenge took ~5 seconds.
What do you propose instead?
marklar423 13 hours ago
I'm assuming a bot running a headless browser instance can still get past it?
It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.
drum55 13 hours ago
People tried this in 2013 or so, there's no point to it. Doing proof of work in javascript in a browser is so crushingly, pointlessly slow that there's no value at all. Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely.
RobotToaster 12 hours ago
wbl 12 hours ago
Anubis sucks because CPU is cheap for scrapers and hard for humans.
sethops1 10 hours ago
How is it hard for humans?
czk 13 hours ago
make your only legit users mine fake crypto to access your site, only costs them 5% battery on an android device
econ 12 hours ago
They think scraping = money so you can just ask them to scrape (copy paste websites into a text area) and it will feel like payment.
drum55 13 hours ago
Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, it's security by obscurity being dressed up as something more.
pow_server http://127.0.0.1:8080 backend avx512-x16
──────────────────────────────────────────────────────────
uptime 00:03:12
solver ● BUSY difficulty 9, 0.3s
queue [####################............] 5/8 peak 12
──────────────────────────────────────────────────────────
accepted 1240 solved 1180
503 shed 48 504 timeout 2 4xx/5xx 10
──────────────────────────────────────────────────────────
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
hashes 3.90GH total avg 65.3MH/s
Ctrl-C to stop
harshreality 13 hours ago
It is not absolutely useless, empirically, which you'd discover if you had a website getting hammered by bots and experimented with anubis as a countermeasure.
While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.
The main author is aware that the algorithm is far from ideal for this purpose. See https://news.ycombinator.com/item?id=48869064 . If more bots start to answer the primitive challenge anubis uses now, that'll hasten implementation of a different algorithm.
Don't let the perfect be the enemy of the good enough. For now, the algorithm or challenge scheme almost doesn't matter. Since it's much smaller-scale than cloudflare's challenges, that's probably why very few scrapers and botnets bother to solve anubis's trivial sha2 pow.
Targeted attacks may not be repelled at all. That's not the point.
kro 13 hours ago
inigyou 13 hours ago
But the people you're defending against don't do that.
They also don't load CSS but for some reason the security theater PoW won the mindshare.
econ 12 hours ago
antonvs 11 hours ago
rokkamokka 12 hours ago
Like any lock, it's mainly to deter less determined adversaries (which account for the vast majority)
Galanwe 13 hours ago
The point of PoW access is not that its hard to bypass, it's that you cannot bypass it at scale.
gruez 12 hours ago
petu 13 hours ago
gum_wobble 13 hours ago
how so, can you link to any sources?
drum55 13 hours ago
tarr11 13 hours ago
> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%.
This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.
inigyou 13 hours ago
There's a lot of people who host their site at extremely expensive places and then do everything they can to minimise unneeded traffic - instead of just moving to a cheaper host. Vercel is another popular extremely expensive host.
creshal 12 hours ago
I'm always flabbergasted when I see what people pay and how much effort they need to invest to keep their cloud websites from eating them alive.
My allegedly more complicated VPS stack needs an afternoon of attention every two years when a new Debian major release is necessary, and costs have been predictable for 15 years, no matter what happened traffic wise.
inigyou 12 hours ago
nickgray 13 hours ago
Thank you! I need to tighten up my KV compression, which is actually carrying a lot of D1's load otherwise. We also had some bad queries some months, as the pages and database grew, that were counting the wrong things (or extremely inefficiently) and those have since been fixed.
maxclark 4 hours ago
How often does your data change?
If infrequent you could cache the generated page and not query D1 for subsequent requests at all.
qbane 13 hours ago
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
ihuman 13 hours ago
There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again
Lalabadie 12 hours ago
The author's website is responsible for storing its own data. AI services currently treat the entire web as their storage and cache layer.
qbane 12 hours ago
This is an important context: the site is more likely to be targeted by scrapers because it is a curated collection of scraped information.
0cf8612b2e1e 12 hours ago
everybodyknows 12 hours ago
aeturnum 13 hours ago
Similar to the dose making the poison - the thing that jumped out at me in this blog was the ratio of scraping to visits. Unless OP is scraping thousands of times a day I don't really think they're in the same class as the bots they are blocking.
nickgray 13 hours ago
OP here: I'm not scraping thousands of times per day! Usually just a few times per year.
jader201 10 hours ago
ethersteeds 13 hours ago
Who scrapes the scrapemen?
alansaber 13 hours ago
Live by the scraper, die by the scraper
varenc 12 hours ago
Can someone help me understand the underlying motivation behind this?
It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.
What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?
bigbuppo 12 hours ago
They are poorly implemented by the "fuck you I got mine" crowd. They will get stuck doing things like trying to run through a calendar that could theoretically go back to the beginning of time and all the way to the end of time. And because that calendar might change, it gets scraped for every inquiry made to the poorly implemented AI system.
BLKNSLVR 2 hours ago
My laymans understanding is that poorly written scripts get executed and then owner comes back occasionally to check that there is 'content' in their database. They're not sitting there troubleshooting their thing beyond 'it's getting data' because, for similar reasons to their scripts being poorly written, their strategy is 'get data'.
Nuance, complexity, and an awareness of 'other people' do not exist in their worlds.
phillmv 12 hours ago
Similarly, I've never understood the economics behind the constant rescraping that is flooding the internet or really what's triggering it.
It can't all be agents reacting to user queries. It's confounding how much CPU and bandwidth is getting flushed down the drain.
esseph 12 hours ago
The most recent data on the internet for advertising, intelligence, etc.
And a lot of bad scrapers.
itake 17 minutes ago
I live in Vietnam and was annoyed I can't access the website. You still allow Mullvad VPN users though
ddxv 13 hours ago
I'm in a similar boat. Probably 99.999% is bots. I have nearly 1m unique "visitors" according to cloudflare and my real users are in the dozens a day. That being said, I love the open internet and am holding on to keeping as much open as I can.
fooey 10 hours ago
I shutdown all my little informational hobby projects this year that I've tinkered with for decades
They all shifted from mostly paying for themselves (or being so cheap it didn't matter), to essentially producing zero income while resource usage leapt up in magnitudes
I couldn't justify the stress and hassle of making sites, that a few dozen people a day might find useful, into some complex hyperscalable obligations just to feed the bots
hskalin 12 hours ago
But what do these bots gain from this?
ddxv 12 hours ago
It's all OpenAI / Anthropic / Singapore/China based crawlers. I guess they gain data that I provide for free. I get very little crawling by Googlebot / Bing etc which barely even index my site, I only have a single page "indexed" by Google.
bredren 12 hours ago
Possibly content that appears only briefly? I'm not sure otherwise. It seems wasteful.
AlienRobot 11 hours ago
Have you ever use an AI chatbot? It's basically a democratized scraper.
Type something stupid, it "visits" 200 websites and regurgitates some random gibberish.
esseph 12 hours ago
The most recent info of every single fucking thing on the web. Everything. Data locusts.
mgbmtl 13 hours ago
I run scripts on my servers on an hourly basis to check which are the top 25 IPs visiting the server (aggregated by /24). If anyone in those top 25 IPs are from China, Vietnam, etc, or from Alibaba/Amazon/etc, the /24 gets blocked by iptables.
It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.
However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).
IMSAI8080 12 hours ago
My solution was similar. Anything coming from the ASN of a major cloud provider gets a CAPTCHA with a little nuance to allow Google and Bing to index. That seems to do a pretty good job. Also, anything coming out of China or Singapore also gets a CAPTCHA as my site is not popular in those regions and many Chinese bots seem to show up as a Chinese mobile provider. So far, the bots have never attempted to solve the CAPTCHA.
andai 13 hours ago
> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.
Kinda sounds like we're missing a peer to peer network here.
Instead of downloading the same data over and over again we can just download it once and then share it.
Wouldn't that be better for everyone involved?
It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)
inigyou 13 hours ago
Anyway, Google doesn't send traffic to your site any more. Only important sites and obvious scams seem to get indexed.
NicuCalcea 13 hours ago
Common Crawl?
amazingamazing 13 hours ago
The problem there is trust
econ 12 hours ago
You can validate by repeating the work and sign your content.
The domain name system is now just a rent seeking scheme. It was great as a temp solution but over time it has deleted more content than preserved. I might in theory be billed for having a country name but I don't pay for a city, street name, house number or postal code. Online you should be able to move your widget shop to widget street. Can bill people who want to live on real estate street or used car street and/or set some requirements.
thomashabets2 11 hours ago
Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly.
Is it really a bot if it's in response to a human asking for some aggregate information about charities, triggering a web search and then following the result links to get details for the human? Well, clearly yes it is, but it's a very different proposition from this article's implication that "they have no throttling on their scrapers"[1].
> Challenge 46 datacenter ASNs. Humans don't browse from AWS.
People who have workstations in the cloud do.
> The bots use 99% of the bill and I pay 100% of it.
Running a site this way is always a wallet-DDoS risk.
[1] though yes, by far most will be pure automation with no human in the loop. It's an assumption on my part, but feels like a safe one.
sethops1 10 hours ago
> Running a site this way
What do you mean "this way". What other way is there to run the site?
bakugo 10 hours ago
> Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly.
That's not absent from the article, it's right there in the section titled "The Claude ratio". ChatGPT, Claude, etc. use different user-agents for scraping vs user-initiated requests, and the author notes that user-initiated requests were an absolutely miniscule fraction of the total traffic.
thomashabets2 10 hours ago
Oh, that's what "Anthropic's search crawler, had requested 420,680 pages in one week. That same week, Claude sent me 12 human visitors" meant?
Yeah, googling it does seem like "Claude-User" is for user-initiated requests.
By "Claude sent me" I thought the author meant referer header in real browser requests showed that they came from.
I mean, that's what the section "Pages crawled per visitor referred" refers to, right?
fooey 10 hours ago
cloudflares lack of spending caps and and usage alerts is a seriously unnerving approach
Bender 13 hours ago
Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count:
2 access.log [1]
40 botpoop.log [2]
spockz 13 hours ago
Is there some existing mechanism already that counts how often an ip only scrapes the page and not the css and then block those origin IPs if it occurs “too often”?
Unfortunately, the best practice is to make css cacheable so you need to keep long histories.
Bender 12 hours ago
I thought about that but to your point CSS is cachable. In fact I made mine immutable. No I just visually spot patterns and use that to study other facets of the agent, other client headers or lack thereof, supported protocol, accepted encoding and so on.
spockz 12 hours ago
Bender 3 hours ago
More realistic numbers now that HN and others have been visiting all day. Line count:
104 access.log
366 botpoop.log
54 access.log
181 botpoop.logjoshspankit 3 hours ago
How long until Cloudflare is the data broker for websites like this?
“For a low $/GB, we’ll give you everything from this site and 1000 others as (structured data/a database)!”
(yes there are lots of good counter arguments to this, but before you reply think ahead a couple extra steps)
storus 13 hours ago
Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.
jerf 13 hours ago
"A $5/month VPS won't cut it anymore."
Are you speaking from experience, or inferring from articles like this?
I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second.
Now, if you have some sort of dynamically-computed website that is generated by a slow scripting language that is poorly optimized and hits the database too many times for a single page, yeah, it doesn't take many RPS to take you out. But that isn't the only option; it's the slowest of the slow options. Realistic, there are plenty of sites that match that description, but I concatenated that many clauses on purpose. Drop any one of them and your personal site will be fine.
marginalia_nu 13 hours ago
It's almost always the DBMS that's the bottleneck when websites drop from traffic. As long as you don't do anything fancier than primary key look-ups you're probably fine though.
I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to one of my blog posts, and that was just a short temporary blip.
allthetime 12 hours ago
jerf 13 hours ago
marginalia_nu 13 hours ago
Static files on literally any hardware from the last 15 years on modern server software simply won't get cooked by bots. The network switch will bottleneck you before the server will. Your ephemeral port range will run out before the server will.
throwaway219450 12 hours ago
Bandwidth is still not free, if you cloud host.
I had a client ask me to look into why their bill spiked 5x (admittedly not by much - from a few dollars to tens). They run a static site, and it turned out someone had inadvertently replaced an image with a massive uncompressed photo. Every unique visitor was caching O(25MB) when they loaded the homepage. Also funny because now everyone has 100Mb+ internet, nobody noticed a few extra seconds on load.
There are lots of solutions to this if you actually need to serve a lot of bytes, but you can get stung even with very simple deployments. Not everyone wants to put Anubis or Cloudflare in front of their site.
kube-system 12 hours ago
speak_plainly 13 hours ago
Cloudflare offers a free plan that's fantastic. The free plan gives you effectively unlimited DNS/CDN traffic for a normal site, while the main practical cap is 100,000 Worker invocations per day, with 10 ms CPU per Worker request and a 100 MB request body limit. (If that means anything to you).
The next tier up from free is $25/month or $240 per year.
https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/free/
matsemann 12 hours ago
I use this. Buy a domain, set up a cloud flare pages that builds an Astro page on git push, served statically for free.
everybodyknows 11 hours ago
Does the free tier support geo-blocks? I'm experimenting with it -- running only a "hello, world" site thus far -- but haven't come across any offer of geo-block capability.
Athas 13 hours ago
I run my personal website (and a bunch of other websites and services) off a somewhat more expensive but still reasonable VPS (I think 20€ at TransIP - it's so little that I forgot). Load is basically nil most of the time anyway. I think the bot problem is not so bad for personal websites.
inigyou 13 hours ago
Yeah, you just make sure your site is fast enough to handle more than 1 RPS.
But if you want them to actually stop, you can also just serve a little JavaScript page that sets a cookie and refreshes, to anyone who hasn't set the cookie. The DDOS attacker doesn't run JavaScript.
__turbobrew__ 4 hours ago
I host a static site on S3 with cloudfront on top and my bill is usually a few dollars. It was $2.88 last month. The site could also probably handle many orders of magnitude more traffic than it currently does because cloudfront scales to that level without work needed on my side.
npilk 12 hours ago
GitHub Pages, GitLab Pages, and Cloudflare Pages are all free. If you have a static site, you don't need to worry about lock-in, since your files are easily portable to a different provider.
I made this tool to try and make it easier for beginners to get something up on the public web for free: https://weejur.com
coldpie 12 hours ago
I host a couple WordPress blogs on Lithium Hosting. It's like $5/mo, fixed cost, and I've never had any issues. If I somehow hit a transfer usage cap, they would just shut my site down until the next month, but I've never had that happen.
somehnguy 13 hours ago
A $5/month VPS should cut it completely fine unless you're doing something very complicated.
hk__2 12 hours ago
All my personal websites are on a single $5/mo OVH VPS and it works perfectly fine.
cdrnsf 12 hours ago
My site's built in Go on a reasonably priced VPS and holds up well. I do geoblock traffic from China and a few other countries as well as IP ranges maintained by Spamhaus.
strenholme 13 hours ago
I have a low cost VPS (actually two in two different pre-AI datacenters) and it runs fine. I use nginx to serve the web pages, and the content is about 99% static content.
The vBulletin and PHPbb style forums have issues with slowdown (I haven’t had a forum since 2015; even back then those forums were overrun with spambots), but static content on a nginx site can be served lightning fast.
bigbuppo 12 hours ago
Static site. If you do go for a WCM of some sort you should make sure that if it has a calendar view of some sort it isn't unbounded but that it only goes as far back as your first post, and only goes as far forward as your latest. That's one of the many ways the poorly-implemented AI systems get stuck.
kube-system 13 hours ago
If you have a static site, GitHub pages is free
strenholme 13 hours ago
As someone who is moving my static sites over to GitHub pages: While they are free and work really nice, the problem is that GitHub frequently doesn’t deploy updates to the pages.
I have frequently have had to update a GitHub page, push the change, and then GitHub’s actions puke instead of deploying the change. The workaround is that I have a .txt file with a list of GitHub actions which failed, and when GitHub actions fails, I update that .txt file and push the updated site, which GitHub actions will hopefully successfully deploy.
GitHub pages are OK for pages which aren’t updated very frequently, but they are not OK for pages which update frequently.
allthetime 12 hours ago
A $5/month VPS is fine for mostly static content. I’ve got a site that has been pulling millions of viewers and sits at the top of google for its niche for a decade. It is now hammered with bot traffic constantly. But everything is cached so it’s fine.
rglover 13 hours ago
All of my boxes are cheap VPS. Highly recommend people throw Cloudflare in front of their stuff. I switched all of my load balancers over to there and all of that bot crap went away. That combined with a proper UFW setup keeps the weather clear for me.
inigyou 13 hours ago
Highly do not recommend centralising the internet.
esseph 12 hours ago
ashton314 13 hours ago
I'm on a $4/mo droplet on Digital Ocean and my static site has been just fine. I'm using Caddy and it seems to handle the load like a champ. My site is very lightweight though, so ymmv.
ashu1461 13 hours ago
Cloudfare has a very generous plan usually for hobby projects. The article did mention that cloudfare did not work for them, but they have recently introduced few features to block AI crawling as well.
HaloZero 11 hours ago
I have a very low traffic blog and I just use the nearlyfreespeech to host it. So far it’s been less than $1
archerx 13 hours ago
A lot of my sites are on $5 vps and run very well.
yoursred 13 hours ago
Sef-host with tailscale or something similar
dmux 12 hours ago
Exactly. I've been hosting a site from a spare M1 Macbook Pro with Ngrok. An equivalent bare metal server would be much more expensive.
esseph 12 hours ago
My sites run on cloudflare workers and static pages for free
tananaev 12 hours ago
I also see quite a bit of traffic from China and Singapore. I wonder if it's some scraping for AI training. It doesn't really bother me too much because traffic is still fairly low, but it skews all the analytics for me.
rsolva 7 hours ago
I made a small booking site for a local dutch canal boat, which has a calender function. A simple PHP app. I checked the Apache logs recently, and it had THOUSANDS of claudebot and other AI UserAgents flooding the logs every day, apparently because the scrapers keep hitting the 'next month' button on the calendar in a an infinite loop, all day, everyday!
This is a small booking app without any useful information at all, it surprises me that the AI boots have no discernment about what the are scraping, just wasting their own and other peoples resources. And their own reputation! You would thing they could spare a few tokens on a classifier model to do a quick evaluation of their scraping efforts, but apparently they do not.
Anyway, I have done my best to block these UAs and so far it seems to have improved the situation.
falcor84 12 hours ago
It's a sequence of thin lines, between using Chrome out of the box, to using something like Brave, to using a highly customized Zen Browser, to having ChatGPT look up a particular page for you, to using a small BeautifulSoup/scrapy script to scrape dozens of pages, to scraping the entire web. And at every point on this spectrum it's humans driving a "user agent" tool to make requests and process responses on their behalf.
ashu1461 13 hours ago
I wonder if the author tried out the recently released feature by cloudfare to block ai bots
https://developers.cloudflare.com/bots/additional-configurat...
smolder 13 hours ago
Some people don't want to use cloudflare on principle. Like that putting the whole internet behind cloudflare or AWS is a bad thing, in principle.
dzonga 13 hours ago
blocking by geo yeah might work - but what happens when someone is traveling abroad ? they've to use a VPN to access your site ?
my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.
yoursred 13 hours ago
What happens when someone is from a shit country?
mcraiha 13 hours ago
AFAIK you already have to use VPN if you are e.g. westerner visiting China or Belarus.
dzonga 11 hours ago
good question ? I worry about that too since I'm from a poor/shit country that I travel to constantly & live for half the time !!
thorsson12 12 hours ago
The experience of browsing the web has really suffered lately. The mandatory 3-4 second "verifying that you're a human" block from Cloudflare seem to show up on more and more websites. Seems like a questionable choice from Cloudflare to teach everyone to associate Cloudflare's logo with high latency...
duskdozer 11 hours ago
A 3-4 second cloudflare wait seems good to me at this point. I had to just block any cloudflare requests because some pages that used it would just rev up a cpu core indefinitely and hang the browser.
brownieman1325 12 hours ago
I recently got a surge from Singapore and heard a lot more peeps in my circle of friends saying the same thing...
IMSAI8080 12 hours ago
It'll be the Chinese. The big Chinese cloud providers have data centres in Singapore for some reason. My Chinese mobile phone sends all it's telemetry back to Singapore, not mainland China.
Bender 11 hours ago
for some reason
It's outside the great firewall. No requirements to hand over SSL keys to China. I worked for companies that ran into these challenges and solutions.
r0b0tan 12 hours ago
I think it’s wild how AI and data bots are putting certain business models under pressure. We’ve already seen the same thing happen with Tailwind.
peter_d_sherman 2 hours ago
An interesting article, and an interesting problem to have!
The problem, to recap the title, is that "99% of My Website Traffic Is Bots".
How to fight all of those bots, all of those web-scrapers...
If the sole underlying issue is bandwidth for one's website (someone doesn't have enough of it and/or they pay too much for it), then one possible solution to this might be for a single tech company to create an online cache of web pages, accessible to all bots as an alternate route to fetch the desired web pages.
If a tech company (or consortium thereof) decided to do this, they could certainly charge money to each AI / bot utilizing the service...
Which means that it might be an investable idea as a for-profit service...
Now, if the underlying issue is the privacy of individual web pages (or relative privacy, as the case may be!) from the public internet, then perhaps the solution is to simply put those behind user logins and/or a gauntlet of tests (which could also be tied to login!) designed to exclude AI/bot web traffic from human web traffic.
Consider what some BBS'es and the online services of yesteryear used to do (Compuserve, The Source, Prodigy, QuantumLink, AOL, etc.)... basically they'd get a user's mailing address, and then physically snail mail them their login and password to their physical home address...
Oh sure, user registration wasn't as fast back then as it is today... it might take several days to receive your username and password via postal mail -- but as a system operator you were 99.999999999% guaranteed that when you saw such a login on your BBS, that it was an actual live human user...
There may be a market for a service like that, too...
That is, validate that actual users are actual users, and give them some kind of credential that can be checked by an actual web site... find a way to do this at scale...
Anyway, just thinking aloud... a very interesting article, and a very interesting problem to have!
somebudyelse 12 hours ago
I visited someone's blog and didn't have to solve a cloudflare challenge! That's amazing work
travisgriggs 11 hours ago
I wonder what the web would be like if we priced bandwidth at the requester point and as you go.
PeterHolzwarth 9 hours ago
Your comment kind of reminds me of the old slashdot spam-email-solution copypasta - the purpose of it is to highlight how so many sensible sounding ideas just can't work in practice.
butz 10 hours ago
Cloudflare's "Verify you are human" captcha is the new cookie banner.
luciana1u 11 hours ago
at this point the web is mostly machines politely asking other machines for permission to read each other's content, and humans are the ones triggering the captchas.
dabbz 9 hours ago
Machines don't pay my bills though. If the content was free to share widely, then yea, who cares how it's accessed.
luciana1u 2 hours ago
the captchas aren't for the machines though, they're for us — the machines are the ones deciding we're not robots.
Aeolun 4 hours ago
Did he really say that the VPS would be in trouble if it received 5 requests per second of mostly static pages? Something that currently costs $90 per month to serve through CloudFlare?
FerretFred 13 hours ago
Sigh .. same here. I don't write blog posts often (enough) but the ones I do write are from personal experiences and I take a lot of care with them. I look at my logs snd see bots everywhere, but now I just let them get on with it. AI scrapers are different though; they get to read my content which, just for them contains a smsttering of finest digital toxin. A pox on your datasets!
grigio 6 hours ago
LLM are the best way to read the web. right to the point, no ADS
scotty79 6 hours ago
> My visitor stats got so polluted I couldn't trust my own numbers. This was the one that hurt. I'm trying to run a business here. I want to know what real people read on my site so I know what to build next. I couldn't see them through the bots.
Are bots the solution to nosy websites that want to know things about their visitors?
DaveZale 12 hours ago
This is like technological cannibalism.
I stopped posting to my website. Why should it be so much work to stop this theft?
Would it be helpful to have geofencing and regulation?
wandr 12 hours ago
I'm working on a web app right now, with the intention of it going to be 100% paywalled. It's 95% complete, but the remaining 5% is just implementing the paywall. In the meantime, the app is live and operational with a fully functional signup. I am deleting about 100 new bot signups per day right now, it is crazy out there.
Venn1 13 hours ago
I'm blocking the Amazon search crawler, anything coming from Googleusercontent, and limiting AI crawlers to search rather than allowing AI assistants. The residential proxy waves are something to behold, but Cloudflare does an okay job catching those in the AI labyrinth. Still, it's all a bit silly, and I can't imagine what large sites deal with when I'm tangoing with this much nonsense on a small tech blog.
inigyou 13 hours ago
Don't. Just serve the page unless it's an unusually expensive page to serve (like search).
DoesntMatter22 7 hours ago
I'm surprised that the solution wasn't to poison the data so they will train on junk data
1vuio0pswjnm7 8 hours ago
Original HN title: "99% of My Website Traffic Is Bots"
Not clear how the author arrived at the precise 99% figure; perhaps "99%" is a figure of speech
"And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers."
"I'm trying to run a business here."
What's the business
(where "business" is defined as "buying and selling")
From https://patronview.com/robots.txt
# As a condition of accessing this website, you agree to abide by the following
# content signals:
# (a) If a Content-Signal = yes, you may collect content for the corresponding
# use.
# (b) If a Content-Signal = no, you may not collect content for the
# corresponding use.
# (c) If the website operator does not include a Content-Signal for a
# corresponding use, the website operator neither grants nor restricts
# permission via Content-Signal with respect to the corresponding use.
# The content signals and their meanings are:
# search: building a search index and providing search results (e.g., returning
# hyperlinks and short excerpts from your website's contents). Search does not
# include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
# augmented generation, grounding, or other real-time taking of content for
# generative AI search answers).
# ai-train: training or fine-tuning AI models.
# use: how AI systems may consume the content (immediate, reference, or full).
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
If, for example, the website operator had enforceable intellectual property rights in the "content", such as copyrights, then perhaps the operator could restrict access to the "content" under the threat of litigation to enforce those rights
Basic questions
1. Is the "content" protected by intellectual property rights, e.g., copyrights
2. Does the website operator have intellectual property rights in the "content", e.g., copyrights
3. Does the website operator have agreements with the rights holders, e.g., granting the operator authorization to restrict access to the "content"
sp1982 13 hours ago
The annoying part is a large percentage of misbehaving bots (not obeying robots.txt for example) are via end user proxies across the world. However most of these aren't doing full-browser loop, so if you are behind cloudflare, you can do non-interactive challenge and that can help quite a bit.
nickgray 13 hours ago
Yes! OP here. I did the non-interactive challenge, and yet all those Chinese bots in my article got through (which surprised me).
l72 11 hours ago
I don't look at the logs of my personal site very often as it is a static site, but just went to check, and yeah, it's almost all ai crawlers. Note sure what is going on here, but I hope this isn't really anthropic:
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /secrets.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /credentials.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /secrets.yml HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /service-account.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /key.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /config/.env HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /service_account.json HTTP/2.0" 404 366 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /serviceAccountKey.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /firebase-adminsdk.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /Dockerfile HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.github/.env HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /firebase-service-account.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.docker/config.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.npmrc HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.boto HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.s3cfg HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.svn/entries HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.htpasswd HTTP/2.0" 404 346 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /terraform.tfstate HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /docker-compose.yaml HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.vscode/launch.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_rsa HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_ed25519 HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_ecdsa HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/authorized_keys HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/known_hosts HTTP/2.0" 404 343 "-" "anthropic-ai"darksim905 9 hours ago
so they broke into your server?
knuckleheads 13 hours ago
Previously, I had done a fair amount of research into how Google's monopoly on web crawling further entrenches their monopoly in the search engine market. You can read more about this here, https://knuckleheads.club, there is a long report from ~2020 or so that explains how it worked at the time. The club is mothballed, I am doing other things with my life, and I'm happy to say that we played a very small role in the DOJ ordering Google to share their crawl data with qualified competitors (a work in progress, but it's progressing).
Chatbots have super charged this dynamic though, to the point that it is showing up in the robots.txt data. The last few weeks I've been having Claude rerun some old analysis of Common Crawl from back then, when I have spare usage and time. What I've found is that you can see pretty clearly the rise in people outright blocking AI chatbot related crawlers likely because of how aggressive they have become.
Quarter Crawl GPT Claude CC G-Ext Byte Bing Google
2023 Q1 2023-06 0.00% 0.00% 0.16% 0.00% 0.06% 0.47% 0.39%
2023 Q2 2023-14 0.00% 0.00% 0.18% 0.00% 0.06% 0.45% 0.38%
2023 Q3 none — — — — — — —
2023 Q4 2023-40 2.21% 0.00% 2.12% 0.04% 0.11% 0.39% 0.27%
2024 Q1 2024-10 0.53% 0.05% 0.31% 0.09% 0.18% 0.34% 0.31%
2024 Q2 2024-18 0.55% 0.09% 0.32% 0.11% 0.24% 0.32% 0.31%
2024 Q3 2024-30 0.68% 0.22% 0.36% 0.20% 0.38% 0.24% 0.33%
2024 Q4 2024-42 1.10% 0.50% 0.44% 0.32% 0.50% 0.25% 0.40%
2025 Q1 2025-05 1.14% 0.66% 0.54% 0.42% 0.66% 0.25% 0.44%
2025 Q2 2025-18 1.37% 0.93% 0.63% 0.70% 0.92% 0.29% 0.19%
2025 Q3 2025-30 1.42% 1.07% 0.74% 0.62% 1.01% 0.31% 0.27%
2025 Q4 2025-43 1.92% 1.51% 1.23% 1.15% 1.52% 0.27% 0.19%
2026 Q1 2026-04 2.13% 1.76% 1.68% 1.58% 1.77% 0.22% 0.15%
2026 Q2 2026-17 2.80% 2.38% 2.26% 2.13% 2.50% 0.22% 0.14%
2026 Q3 2026-30 3.45% 3.01% 2.89% 2.71% 3.16% 0.21% 0.14%
It's very clear that website owners do not like getting their content scraped and are indicating to GPTBot et al. that they are not welcome. It's a shame that CCBot is caught in the cross fire, but that's life. Bing and Google are doing just fine though, almost like having significant power in the search engine market gives you an advantage in other markets too. Who knew!
nromiun 13 hours ago
This is a static website running on Cloudflare infra. What on earth costs $90 per month? First optimize your infra before throwing up rules in front of your visitors. I have several websites on Cloudflare too and I don't even check how many million requests I get. Because it does not cost me anything.
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
Being self aware does not make it okey. Either you are okey with scraping (like me) or against it. Don't use it yourself and block your site at the same time.
These same people will be crying about how Cloudflare ruins the internet because they get these captchas.
AdrianB1 13 hours ago
I checked the comments to see if anyone pointed to this: I can imagine so many memes with this line :)
oaw93j4oij 12 hours ago
I despise cloudflare. They've decided that my home IP address is bad, so I have to capchas for most websites. Sometimes on infinite loop and I never get to the website. I even reset my home IP address more than once, but it instantly continues. Especially if I use any VPN, even my work VPN.
bediger4000 12 hours ago
I think that asymmetry is why scraping keeps getting worse. The scrapers' costs fell faster than everyone's defenses improved.
There it is. Just like the fckn spammers who ruined SMTP email, scrapers externalize the costs.
Who finances the effort to use residential proxies? That takes a lot of effort, even if it's shoddy
IMSAI8080 11 hours ago
There was a popular pirate streaming stick sold on Amazon that ran a residential proxy by day and did ad fraud clicks by night.
throwaway63467 11 hours ago
Most people don’t know they’re acting as a residential proxy, lots of devices and apps and free tools install spyware which often includes a proxy script. It’s a really shady market.
righthand 13 hours ago
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer. That's usually the easiest way I spot a bot.
> And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch.
> But those 4,000 bots were just the warm-up.
I just hate this style of writing like you're on Twitter. Why does the above need to be 3 different paragraphs? A paragraph break indicates a separate thought but the author is still talking about the same data and still making their point. The sentence "But those 4,000 bots were just the warm-up." is effective when still the last line of a paragraph and it signals respect for your readers. I stopped reading after this because it's just a terrible reading experience.
Here's a correct version that doesn't read like the author left for a week to think about what the next sentence would be or having some sort of anxiety-induced mental pause:
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer, which is usually the easiest way I spot a bot. They were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. But those 4,000 bots were just the warm-up.
nickgray 13 hours ago
Hey! I'm the OP - thanks for feedback on my writing style. I went ahead and fixed this in the article. It should be updated by the time you read this:
https://patronview.com/news/99-percent-of-my-website-traffic...
And you're totally right: I mostly post on X (nee Twitter) and I probably have ADHD or just a low attention span, so I prefer to read things broken up into paragraphs. But for a smarter audience like this, and that reads long-form blog posts, I should tighten it up.
Thank you for the suggestion. LMK any other edits and I'll be happy to tighten it up.
righthand 12 hours ago
Glad to hear you’re willing to accept feedback. You maybe don’t have ADHD and sorry for continued advice but you shouldn’t assume you have undiagnosed conditions IMO as it allows you to defer your mistakes from the self. Even if you do have ADHD you can still correct and understand good article structure. I highlighted the last sentence of that paragraph because what is clear from the split sentence style is that you’re writing for impact, this lends well to 140 characters but falls apart in longer form writing but as I stated the sentence is still impactful as you’re saying “but wait…there’s more to this!” Which is very intriguing. I think if you’re comfortable writing that way and it helps you split your ideas and sentences up so each one is impactful, that’s a good thing. But consider that style as a draft and then you can go back and group up your impactful ideas into paragraphs very easily.
For other edits I think my qualm applies to other parts of the article but I found that specific paragraph the most impactful way to illustrate what I was talking about. I leave the rest to you as a challenge. Don’t lose sleep over it, there will be more writing in the future to apply it to.
As for the ADHD stuff and the urge to self diagnose consider something less severe but similar symptoms. Have you considered VAST? Here is a good HN comment briefly detailing it and mentioning a book (titled ADHD 2.0 I believe) that may be more in line. I am not a doctor of course and VAST is rather new.
https://news.ycombinator.com/item?id=49035436
Anyways I will finish reading your article now since you’re so wonderful to take a bit of feedback and be proactive.
DANmode 3 hours ago
I write a lot of my comments and emails like this.
It’s a learned habit - I’m more apt to receive a thoughtful reply (or any) in this format.
zuzululu 13 hours ago
pretty crazy that this article is seemingly written by an AI used a detector and it is 95% confident its generated
r0b0tan 12 hours ago
Most AI detectors are dogshit, though. AI is trained on human writing and tries to imitate it, while these detectors basically flag certain writing styles as "AI-generated."