Proxy tier benchmark

We publish a measured answer to the question customers ask before they buy: which class of proxy does this target actually require? The harness, the target list and every raw request record are on GitHub under an MIT licence, so the numbers can be checked, disputed, or re-run against your own targets.

Results for all 32 targets  ·  github.com/botproxy/proxy-tier-benchmark  ·  Seeing BotProxyBenchmark/1.0 in your server logs?

How it is measured

Thirty-two public-data targets are fetched through six arms that differ by one variable at a time — no proxy at all, a datacenter IP pinned to one exit, a datacenter IP rotating per request, each of those with Browser Impersonation Mode enabled, and a residential exit — interleaved by round so the hour of day cannot be confounded with the arm. A request counts as a success only if the expected content is present. A 200 carrying a challenge page or an empty body is a failure, because it is not a page you can use.

Every target is gated on robots.txt before and during the run, targets whose robots.txt cannot be read are excluded rather than assumed permitted, and each exclusion is published with its reason.

What it found

The results do not flatter us, which is the point — a vendor benchmark the vendor wins is not evidence:

• No proxied arm beat the unproxied control. On a modest number of fetches from a clean cloud host, plain direct access scored highest.
• 19 of 32 targets needed no proxy at all at that volume.
• 12 of 32 were unreachable at any tier, because the block is an interactive challenge that no plain HTTP client can solve. Those targets need a real browser — see the Browser API — not a different proxy.
• Exactly one target was decided by the proxy tier.
• Where an official API or bulk download exists, the report recommends it over scraping.

Looking for one particular target? Every benchmarked target has its own page — the exact URL fetched, all six arms with success rate, latency and page weight, what protects it, and whether an official API makes scraping it unnecessary.

All 32 targets, with the minimum viable tier for each
What protects US public-data portals — Cloudflare, Imperva, DataDome, reCAPTCHA, CloudFront, JavaScript shells and tarpits
The 15 targets we excluded, and why

What a low-volume benchmark cannot show

Read the headline with its scope attached. A rotating pool is rarely needed because a target blocks datacenter ranges outright — it is needed because of per-IP rate and volume limits and geographic restrictions. Ten polite requests to a target never approach a rate limit, so the benchmark is structurally unable to measure the thing rotation exists to defeat.

Two effects sit outside a single-day run and matter in production:

• Per-IP throttling. On targets that limit per address, the same workload fails far more often when concentrated onto few exits than when spread across many. On one measured storefront the failure rate moves from 2.4% to 73.3% purely with concentration.
• Daily pool replacement. Our datacenter nodes cycle continuously; roughly 40% of the exit addresses a target saw yesterday are gone today. IP-level bans therefore have a short horizon and per-IP budgets reset with the address, which no one-day test can observe.

If you are evaluating for sustained collection rather than occasional fetches, those two dimensions matter more than the headline pass rate, and the harness is published so you can measure them on your own targets.

Benchmark traffic in your logs

If you found this page from a BotProxyBenchmark/1.0 entry in your server logs: that traffic is the run described above. It fetches a handful of pages per target, never faster than your robots.txt asks, obeys robots.txt at both target selection and request time, and excludes any target whose robots.txt we cannot read rather than assuming permission. It touches public, unauthenticated pages only, and where a site offers an official API or bulk download the report recommends that instead of scraping.

To have your site excluded from future runs, or to ask anything about this traffic, email [email protected] and we will action it.