Social Rocket
We had Cloudflare Bot Fight Mode enabled. Suspicious traffic still reached the website.
According to our operator experience, browser challenges were not reliably effective either. The same went for an approach we had tried earlier called "precursor". In the end, what made the difference was not yet another generic bot feature, but a rule tailored precisely to the request patterns we actually observed.
Our AI agents identified two distinct scraping patterns in the traffic and translated them into a targeted Cloudflare block strategy. The rules combine browser characteristics, Fetch Metadata, source networks, referrers, entry pages and Cloudflare's flag for verified crawlers. Matching requests are terminated directly at the edge with HTTP 403.
We deliberately do not publish the complete fingerprints. An anti-scraping rule is not a good place for a public changelog for scrapers.
What first caught our attention
The first hint did not come from a single security event, but from the combination of several measurement points.
A Minilytics snapshot showed 544 direct sessions with zero measured outbound conversions. Between two completed hours, direct sessions rose from 24 to 163.
At the same time, Cloudflare showed roughly 17.700 HTTP requests in an examined 30-minute window, of which about 15.400 reached the origin. A five-minute origin excerpt contained 1.980 requests from 1.238 different IP addresses.
It is important to keep the quantities apart: an HTTP request is not a session. And "zero conversions" in our context means zero triggered configured outbound events – not proven zero sales or zero revenue.
We nevertheless had enough signal to break the traffic down more precisely.
Pattern 1: the browser claimed Chrome – the request metadata did not match
The first large group rotated massively across IP addresses and countries. The clients presented themselves as Windows Chrome browsers and switched between several Chrome identities.
In the examined excerpt, 1.628 requests from 1.120 IP addresses in 100 countries belonged to this pattern. A large share of them additionally requested Next.js page data.
That alone is not proof of a bot. RSC requests also arise during normal browsing of Next.js pages.
The contradiction became interesting at another point: the requests claimed a modern browser identity, but a Fetch Metadata header that we would have expected for this request class was missing entirely.
Our first new block logic therefore couples several conditions:
- request to a Srocket host
- GET
- matching Windows Chrome identity from the observed pattern
- missing Fetch Metadata header
- not a Cloudflare-verified bot
- no internal Next.js or API paths
We do not publish the concrete list of versions.
This is the decisive difference from a crude "block Chrome version X" rule: it is not a single property that triggers the block, but the contradiction between the claimed browser identity and the request context actually sent.
Cloudflare provides a field with `cf.client.bot` that flags known good bots and crawlers. We exclude these verified crawlers from the block logic.
Pattern 2: this time the browser request looked plausible
Shortly afterwards, a second pattern emerged.
These requests had browser-like headers and also used a normal `Sec-Fetch-Site` value. They therefore did not fall under the first fingerprint.
Here the signal lay in the combination of origin, browser profile and target page.
We saw direct HTML entries from a rotating pool of source networks. The requests used only a few exactly recurring browser profiles, arrived without a referrer and specifically hit a small group of our comparison pages.
Such changing source networks are also why IP addresses alone are a poor fingerprint. If you want a better technical picture of what lies behind this, our proxy comparison explains the differences between common proxy types and providers.
Our second block logic therefore combines:
- an observed source pool
- one of the exactly recurring browser profiles
- GET
- empty referrer
- a small defined group of affected entry pages
- not a Cloudflare-verified bot
Important: we do not block an entire ASN across the board, nor do we block all visitors without a referrer across the board. Only the combination of characteristics leads to the block.
The concrete networks, user agents and page combinations also remain internal.
Why `Sec-Fetch-Site: none` is not a bot signal
The second pattern illustrates well why individual headers can be dangerously poor blocking criteria.
According to MDN, `Sec-Fetch-Site: none` is entirely normal browser behaviour, for example when someone types a URL directly into the address bar or opens a bookmark.
A request with this value is therefore not automatically suspicious.
That is precisely why we only block this group when source pool, exact browser profile, empty referrer and affected entry page all match at the same time.
That is narrower than a generic bot rule – and deliberately so.
The AI did not "detect a bot" – it built two rule models from it
The part we found interesting was not that a model looks at traffic and says "bot".
The AI became useful only where many individual requests turned into two distinguishable technical patterns:
1. Identity contradiction: browser profile and Fetch Metadata do not fit together. 2. Source fingerprint: origin, browser profile, direct entry and target page repeatedly occur together.
From these patterns, our AI agents derived a Cloudflare strategy, which we then checked against real requests and normal browser access.
For us, this is the far more interesting form of AI in operations: not a chatbot that explains security, but agents that translate real telemetry into a controlled edge rule.
We have already described elsewhere why bot traffic can distort attribution and retargeting. At the same time, the current debate about transparency obligations for AI crawlers shows how difficult it has become to cleanly attribute automated traffic.
No further deployment: the rule sits directly at Cloudflare
The final solution runs as an existing Cloudflare Custom Rule with the action Block.
For this we neither had to deploy the application nor use up another rule slot. The existing scanner/flood rule was extended with the two additional patterns.
The logic can deliberately only be summarised at the principle level:
- Already confirmed scanners remain blocked.
- Pattern 1 only applies if browser identity and Fetch Metadata contradict each other in the observed way, the request is a GET, no verified bot is detected, and no internal framework or API paths are affected.
- Pattern 2 only applies to the combination of observed source pool, exactly recurring browser profile, direct entry without referrer and one of the affected entry pages.
- Only when a branch matches completely does the action Block follow.
We do not publish the concrete networks, browser versions, user agents or the productive Cloudflare expression. A scraper does not need to know more than that about the rule.
Cloudflare itself recommends Custom Rules when several request fields must be combined. A terminating block action already takes effect at the level of the Custom Rules and does not let matching requests continue into downstream bot mechanisms.
What happened after activation
We did not only check whether Cloudflare accepted the rule. We verified in a controlled way whether it actually hit the intended pattern and whether normal requests continued to work.
The control tests yielded:
- test request with the first fingerprint: Cloudflare 403
- same browser family with normal navigation characteristics: 200
- public comparison page with normal access: 200
- real event from the second pattern: Custom rules / Block
- displayed rule matches during the test: about 1.300
The origin trend was particularly clear: for the first pattern, matching requests fell from 626 in a full minute to zero in the final 18-second interval of the examined excerpt.
In a later Minilytics check, 156 seconds passed with no new direct sessions without outbound conversion.
This does not prove that unwanted traffic has disappeared for good. It does show that the new rule actually intercepted the observed patterns at the edge.
Why we do not claim there can be no false positives
We tested the rule with normal browser-like access and deliberately narrowed it.
Still, "guaranteed free of false alarms" would be irresponsible.
A real direct visitor could in theory also meet several characteristics of the second pattern at once – for example from an affected source network, with one of the recurring browser profiles and on one of the affected pages.
For that reason we do not treat the rule as a universal anti-bot solution, but as an incident-specific fingerprint with a documented rollback option.
The same applies to the first pattern. If scrapers change their request identity, a fingerprint can also lose its effect.
The advantage lies precisely in not claiming a magic permanent solution: we can observe, reclassify and adjust the edge rule in a targeted way.
What we take away from the incident
Bot Fight Mode was not useless. But generic bot defence could not stop these two traffic patterns sufficiently for us.
Only the combination of our own telemetry, AI-supported pattern analysis and a precise Cloudflare Custom Rule made the difference.
Our concrete fingerprints cannot be transferred to other websites. The method, however, can:
Do not chase IP addresses. Do not block individual headers in isolation. Combine several independent request characteristics, exclude verified crawlers, test normal browser access and measure the effect afterwards.
That is how our AI agents turned a traffic problem into a verifiable edge rule – without handing scrapers a universal manual.