Proxy tier benchmark › Targets

Do you need a proxy to scrape Steam store pages?

High-defence commercial sites · minimum viable tier: no proxy needed · measured 2026-08-18

Not for occasional fetches, and yes for sustained collection — this is the one target in the study where those two answers differ sharply. The Counter-Strike 2 store page returned the expected content on 60 of 60 requests: 100% unproxied and 100% on every one of the six configurations tested. But this benchmark fetches 10 pages per arm, and Steam's defence is a per-IP rate limit that 10 requests never approach. Measured 2026-08-18.

What was measured

A request counts as a success only if the expected content is present. An HTTP 200 carrying a challenge page, an empty body or the wrong page is a failure, because it is not a page you can use.

This measures whether a tier works, not long-run stability under sustained load. Ten requests per arm, at a two-second floor, never approach a per-IP rate limit — which is the thing a rotating pool exists to defeat.

Results by arm

What protects this target

Nothing visible at this volume — no challenge signature, no 403, no rate-limit status across all 60 requests, with a median unproxied response of 375 ms.

What actually protects it appears only under load, and BotProxy's own production traffic measures it. DIMENSIONS.md reports 5.4 million requests against steampowered.com grouped into day-long observation windows: when the same workload is spread thinly across many exit addresses the failure rate is 2.4%, and when it is concentrated onto few addresses it is 73.3% — a 71-point swing, with a Spearman rank correlation of 0.78 between requests-per-exit-IP and failure. Same requests, same rate, differing only in how many IP addresses carried them.

That is the clearest per-IP throttle in the corpus, and this benchmark cannot see it. A page reporting only the 100% figure would give the wrong answer to anyone collecting the Steam catalogue.

robots.txt status

The URL measured is allowed by the host's robots.txt.

Practical setup

If you are fetching a few store pages, fetch them directly. If you are collecting the catalogue, the variable that matters is not the proxy tier -- every tier scored 100% here — but how many exit addresses the requests are spread across. Use a fresh session id per request so each takes a different exit (see sessions); a fixed session id pins you to one address and reproduces the 73.3% case. Pace the work with daily limits rather than bursting.

Limits of this result

• The 2.4% / 73.3% figures come from production traffic, not from this benchmark, and are derived from response-size distributions rather than status codes; they are directional evidence about the target's behaviour, not precise block rates.
• Ten rounds per arm in a single three-hour window on 2026-08-18. Anti-bot behaviour drifts; a result older than a few months is a historical record, not a current one.
• One exit geography. Every request left from a US datacenter host or a US exit, so nothing here describes how this target answers traffic from elsewhere.
• The unproxied arm ran from a clean, previously unused cloud IP with no accumulated reputation. It is a best case a production scraper will not sustain.

Related

• E-commerce price monitoring
• Do you need a proxy to scrape AliExpress product pages?
• Can you scrape Canva template pages with a proxy?
• Proxy tier benchmark: full report

---

Run date: 2026-08-18 · Full benchmark report · Harness and raw data · All 32 targets

The 60 rows behind this page are in results/six-arm/raw.csv; filter them with grep '^[^,]*,steam_app,' raw.csv.

---

Test your own target

The result above describes one URL on one day. Paste your own URL into the live tester and see the real response before creating an account.

Test your target See plans