High-level map of the crawler’s anti-block stack. Not a how-to for abuse — a team brief on what the production Procore harvest already does, and why “throw more workers” is not the same as “look more human.”
Cloudflare doesn’t score one field. It scores a coherent story: browser, network, timing, and content outcome.
TLS ClientHello, HTTP/2 settings, header order, User-Agent, client hints — one measured Chrome profile, never hand-mixed.
One cookie jar + one browser identity + one proxy exit for one bounded crawl round. Rotate together between rounds.
Rayobyte residential IPs. Fresh exit per session/round, not a datacenter ASN screaming “bot farm.”
Non-metronomic delays, burst pauses, random long idles. One in-flight request per worker process.
Reject challenge/empty “success.” If many workers see blocks together, auto-pause the whole source.
What Cloudflare’s bot management actually sees first.
We use curl_cffi Chrome impersonation profiles
(chrome136–chrome146. The ClientHello is a real browser shape, not Python-default OpenSSL.
Rollout verified JA4 t13d1516h2_8daaf6152771_d8a2da3f94cd (Chrome-like).
Settings, priority, and frame behavior come from the same impersonation bundle as TLS — so H2 doesn’t contradict the ClientHello.
UA is not a string we invent. It ships with the impersonation profile. We only add safe extras: Accept-Language, Referer, Sec-Fetch-Site.
No franken-headers. No “Chrome UA + Python TLS.” No random UA rotation under a sticky TLS session. That mismatch is a classic ban signal.
Network reputation and session hygiene.
| Topic | What we do | Why CF cares |
|---|---|---|
| Egress class | Residential proxy (Rayobyte), not DO datacenter IPs hitting Procore. | Datacenter ASN + scrape volume = easy deny. |
| IP rotation | New session ≈ new exit. Verified multi-IP diversity at rollout. | One residential IP thrashing is still a bot; diversity spreads heat. |
| Geo targeting | Code can suffix region; this account currently ignores region (country-US works). We don’t depend on true state geo. | Fake geo claims aren’t required; lying about geo can hurt if it mismatches. |
| Session scope | One identity + cookies + proxy for one bounded round, then rotate all three together. | IP churn under one cookie jar looks broken/malicious. |
| Compute host | Workers run on our droplet; only browser traffic exits via residential proxy. | Separates “our infra” from “what Procore sees.” |
Procore gets the most conservative politeness of any source.
Base 5–11s between requests, cubic-tail jitter, 10% chance of an extra pause,
every 25 requests a long idle 45–150s. Expected ~14s average delay/request/worker.
concurrency = 1 per worker process. N workers = N independent politeness budgets — the real risk lever.
Drain loop: max ~2 list pages + 12 details per round. Controls (pause/reassign) apply between rounds so we never hard-kill mid-fetch.
Data from Next.js __NEXT_DATA__ / pageProps.business — no headless browser thrashing the DOM, no random clicking.
Retry-After (capped). Not infinite hammering.Looking human is half the job. Not poisoning ourselves is the other half.
Status in 401 / 403 / 407 / 429 / 503, or interstitial body markers
(“just a moment”, “attention required”, “access denied”, …).
HTTP 200 but body lacks expected shape. For details: must have real
pageProps.business object — empty challenge JSON is not “done.”
Passive cdn-cgi/challenge scripts ride on normal pages. We never flag on that alone.
≥5 counting block events from ≥2 workers in 5 minutes → auto source.enabled = false.
HTTP 407 (proxy auth) is logged but does not count toward pause.
Team guardrails — keep these out of “clever” experiments.
| Don’t | Why it burns us |
|---|---|
| Raise per-worker concurrency or slash delays | Classic bot signature; undermines the whole stack. |
| Hand-edit UA / headers against the TLS profile | Fingerprint mismatch — easy ML deny. |
| Rotate IP every request under one cookie jar | Looks like session hijack or broken client. |
| Hit Procore from the droplet’s public IP | Datacenter reputation; burns the host ASN. |
| Seed all 50 states + max workers same hour | Burst of list traffic + detail flood; origin heat + our own queue chaos. |
| Treat 200 empty as success | Silent data poison; “success” that is actually a challenge. |
| Ignore auto-pause and re-enable harder | Turns a soft warning into a hard ban. |
Several slow Chrome users on residential IPs, coherent TLS/H2/UA, cookies that match the exit, reading public directory JSON.
A Python script with a stolen Chrome UA, datacenter IP, metronome timing, or 20 parallel tabs from one identity.