We implemented CrowdSec for bot/scraping mitigation. The architecture is sound, but it ended up having an unacceptable false positive rate for us. This may be an issue with any kind of IP reputation approach. After a couple of months of work getting it ready to go I had to turn it off after a couple of days.
We had the main community blocklist and several of their pricey paid blocklists enabled in a PoC capacity. We had a lot of legitimate users end up blocked. In some cases these may have been VPN exit nodes, or users on CG-NAT, or devices on a shared network with some other compromised / bot device. I didn't get 100% of the details, just that we were inundated with support requests from real users that ended up blocked.
Given the number of residential proxies I see scraping the couple of sites I have responsibility for I don't find IP address-based blocking useful anymore. That ship has sailed.
Behavioral and client fingerprint analysis (ugh-- having to run Javascript just to view a static site) is the only way (at least until we get user "age" and identity attestation rammed down our throats).
What kind of fingerprinting are you thinking of? JA4? I haven't found a way to do that inexpensively at our scale, but we may have to go that route - looking at CloudFront bot mitigation.
For behavioral, we have Anubis honeypot functionality turned on, but it doesn't seem to be effective for 99% of scrapers. Anubis is also running behind TLS termination, so I don't think it can do full JA4. It does have the less robust JA4H apparently, but I'm not sure how effective that will be.
Edit: Oh yeah, forgot to mention - it's almost 100% residential proxies. Primarily China Telecom and China Unicom. Unfortunately those providers are HUGE and also host a ton of legitimate users all over Asia.
> What kind of fingerprinting are you thinking of?
I'm minding a proprietary application where this is being discussed. The vendor is talking about running Javascript in the client to detect headless browsers, bots, etc. I'm guessing they're going to license something off-the-shelf. The technical specifics aren't being discussed. They're at the "wow the stakeholders" phase.
I'd like to do something ourselves but, at the size of the org I'm talking about (with virtually no in-house dev resources), COTS rules the day.
itintheory · · focus · HN ↗
Jnr · · focus · HN ↗
They provide several IP blacklists. None of those seem to be false positives. You can also add custom 3rd party blocklists.
They also provide several different rulesets. It is up to you to choose which ones to use and fine tune. LLMs can be very helpful with that.
And there are 3rd party dashboards and tools that help you manage it more easily.
I use the free version as a simple WAF on multiple servers and it blocks a lot of bots. It did require some initial finetuning though.
Are there any better open source solutions?
itintheory · · focus · HN ↗
EvanAnderson · · focus · HN ↗
Behavioral and client fingerprint analysis (ugh-- having to run Javascript just to view a static site) is the only way (at least until we get user "age" and identity attestation rammed down our throats).
itintheory · · focus · HN ↗
For behavioral, we have Anubis honeypot functionality turned on, but it doesn't seem to be effective for 99% of scrapers. Anubis is also running behind TLS termination, so I don't think it can do full JA4. It does have the less robust JA4H apparently, but I'm not sure how effective that will be.
Edit: Oh yeah, forgot to mention - it's almost 100% residential proxies. Primarily China Telecom and China Unicom. Unfortunately those providers are HUGE and also host a ton of legitimate users all over Asia.
EvanAnderson · · focus · HN ↗
I'm minding a proprietary application where this is being discussed. The vendor is talking about running Javascript in the client to detect headless browsers, bots, etc. I'm guessing they're going to license something off-the-shelf. The technical specifics aren't being discussed. They're at the "wow the stakeholders" phase.
I'd like to do something ourselves but, at the size of the org I'm talking about (with virtually no in-house dev resources), COTS rules the day.