Proxy tier benchmark › Targets

Do you need a proxy to scrape SEC EDGAR filings?

Control targets · minimum viable tier: no proxy needed · measured 2026-08-18

No — and you should not scrape it at all. A plain unproxied request from a cloud host returned the Apple 10-K filing index on 10 of 10 attempts (100%), and every one of the six proxy configurations tested scored the same 100%, for 60 successful requests out of 60. The SEC publishes a documented API for exactly this data, which is the route to use. Measured 2026-08-18.

Use the official API instead

This target publishes an official API: https://www.sec.gov/edgar/sec-api-documentation. It returns the same data as structured records, needs no HTML parsing, will not break when the page is restyled, and is the route this study recommends over scraping at any proxy tier.

The SEC's fair-access terms ask automated clients to identify themselves with a contactable User-Agent. The API and the bulk archives serve the same filings as structured data, without HTML parsing and without the ambiguity of scraping a page that may be restyled.

What was measured
URL https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0000320193&type=10-K&dateb=&owner=include&count=10
Content assertion Apple Inc
Arms 6
Rounds per arm 10
Requests 60
Measured 2026-08-18, 15:38–18:09 UTC
Benchmark host DigitalOcean droplet, New York (nyc3)

A request counts as a success only if the expected content is present. An HTTP 200 carrying a challenge page, an empty body or the wrong page is a failure, because it is not a page you can use.

This measures whether a tier works, not long-run stability under sustained load. Ten requests per arm, at a two-second floor, never approach a per-IP rate limit — which is the thing a rotating pool exists to defeat.

Results by arm

Arm Success p50 p95 Bytes per success Outcomes
direct (no proxy) 100% 147 ms 5,277 ms 16,501 success 10
datacenter, one pinned exit 100% 789 ms 13,731 ms 16,501 success 10
datacenter, rotating exit 100% 1,650 ms 11,901 ms 16,501 success 10
anti-detect, one pinned exit 100% 680 ms 1,698 ms 16,683 success 10
anti-detect, rotating exit 100% 535 ms 13,578 ms 16,646 success 10
residential, rotating exit 100% 1,381 ms 17,977 ms 16,646 success 10

What protects this target

Nothing. EDGAR returned HTTP 200 with the expected filing index on all 60 requests, across all six arms, with no challenge signature, no rate limiting and no soft block. It is the study's control target: if the harness had a defect, this is where it would have shown.

At 16,501 bytes per successful page this is also the lightest successful response in the run — a fifth of the weight of the licensing-board pages measured in the same study.

robots.txt status

The URL measured is allowed by the host's robots.txt.
Status allowed
File https://www.sec.gov/robots.txt
Crawl-delay not declared
Fetched as the benchmark's own User-Agent
Note robots.txt 2622 bytes via benchmark_ua

Limits of this result

  • Ten rounds per arm in a single three-hour window on 2026-08-18. Anti-bot behaviour drifts; a result older than a few months is a historical record, not a current one.
  • One exit geography. Every request left from a US datacenter host or a US exit, so nothing here describes how this target answers traffic from elsewhere.
  • The unproxied arm ran from a clean, previously unused cloud IP with no accumulated reputation. It is a best case a production scraper will not sustain.

Run date: 2026-08-18 · Full benchmark report · Harness and raw data · All 32 targets

The 60 rows behind this page are in results/six-arm/raw.csv; filter them with grep '^[^,]*,us_sec_edgar,' raw.csv.

Test your own target

The result above describes one URL on one day. Paste your own URL into the live tester and see the real response before creating an account.

Test your target See plans