Proxy tier benchmark › Targets
Do you need a proxy to scrape UK Companies House?
Control targets · minimum viable tier: no proxy needed · measured 2026-08-18
No. The public company record for Tesco returned the expected content on 10 of 10 unproxied requests (100%), and on every datacenter and anti-detect configuration tested; the only loss in 60 requests was a single connection error on the residential arm (90%). Companies House publishes a free REST API that returns the same record as JSON, and that is the route to use. Measured 2026-08-18.
Use the official API instead
This target publishes an official API: https://developer.company-information.service.gov.uk/. It returns the same data as structured records, needs no HTML parsing, will not break when the page is restyled, and is the route this study recommends over scraping at any proxy tier.
The API is free, requires only a registered key, and returns the company profile, filing history, officers and charges as JSON. Scraping the HTML page to obtain data the same organisation publishes as an API is slower, more fragile, and provides no additional information.
What was measured
| URL | https://find-and-update.company-information.service.gov.uk/company/00445790 |
| Content assertion | TESCO |
| Arms | 6 |
| Rounds per arm | 10 |
| Requests | 60 |
| Measured | 2026-08-18, 15:38–18:09 UTC |
| Benchmark host | DigitalOcean droplet, New York (nyc3) |
A request counts as a success only if the expected content is present. An HTTP 200 carrying a challenge page, an empty body or the wrong page is a failure, because it is not a page you can use.
This measures whether a tier works, not long-run stability under sustained load. Ten requests per arm, at a two-second floor, never approach a per-IP rate limit — which is the thing a rotating pool exists to defeat.
Results by arm
| Arm | Success | p50 | p95 | Bytes per success | Outcomes |
|---|---|---|---|---|---|
| direct (no proxy) | 100% | 539 ms | 674 ms | 36,793 | success 10 |
| datacenter, one pinned exit | 100% | 1,239 ms | 3,020 ms | 36,793 | success 10 |
| datacenter, rotating exit | 100% | 2,019 ms | 2,428 ms | 36,793 | success 10 |
| anti-detect, one pinned exit | 100% | 910 ms | 2,442 ms | 36,793 | success 10 |
| anti-detect, rotating exit | 100% | 1,007 ms | 2,455 ms | 36,793 | success 10 |
| residential, rotating exit | 90% | 2,217 ms | 34,504 ms | 36,793 | success 9, error 1 |
What protects this target
Nothing. No challenge signature appeared on any of the 60 requests. The one failure in the run was a transport-level connection error on the residential arm, not a refusal by the service.
The residential arm is worth a line on its own: it matched the others on content but its 95th-percentile response time was 34,504 ms against 674 ms unproxied. On a target that needs no proxy, routing through the most expensive tier bought a fifty-fold tail-latency penalty and nothing else.
robots.txt status
The host serves no robots.txt — the request returns HTTP 404. Nothing is disallowed, and nothing is granted either.
| Status | allowed |
| File | https://find-and-update.company-information.service.gov.uk/robots.txt |
| Crawl-delay | not declared |
| Fetched as | the benchmark's own User-Agent |
| Note | no robots.txt (HTTP 404) |
find-and-update.company-information.service.gov.uk serves no
robots.txt at all — the request returns HTTP 404. The compliance gate treats an
HTTP 404 as "no policy published, nothing disallowed", which is a different
decision from the one it makes for a robots.txt it cannot read: an unreadable
file is treated as a refusal and the target is dropped. Eleven candidates were
dropped on that rule.
Limits of this result
- Ten rounds per arm in a single three-hour window on 2026-08-18. Anti-bot behaviour drifts; a result older than a few months is a historical record, not a current one.
- One exit geography. Every request left from a US datacenter host or a US exit, so nothing here describes how this target answers traffic from elsewhere.
- The unproxied arm ran from a clean, previously unused cloud IP with no accumulated reputation. It is a best case a production scraper will not sustain.
Related
- B2B data scraping
- Do you need a proxy to scrape UK food hygiene ratings?
- Do you need a proxy to scrape SEC EDGAR filings?
- Proxy tier benchmark: full report
Run date: 2026-08-18 · Full benchmark report · Harness and raw data · All 32 targets
The 60 rows behind this page are in results/six-arm/raw.csv; filter them with grep '^[^,]*,uk_companies_house,' raw.csv.
Test your own target
The result above describes one URL on one day. Paste your own URL into the live tester and see the real response before creating an account.