Solutions › Public records & government registries
Government registries and public records work best with rotating datacenter IPs in the right country, a browser-like TLS fingerprint, and a rotation strategy based on per-IP rate limits rather than outright IP bans. That combination — not residential proxies — is what these targets actually need. They're public documents behind a generic WAF and a request budget, not behind consumer-grade bot detection.
This is the widest and most fragmented class of public data on the web. In the US alone it's spread across roughly 1,450 separate government domains, because every state, every professional board, and every county runs its own infrastructure:
There's no shared platform behind them. Two neighbouring counties can run a modern React front end and a 2003 ASP.NET postback form. A scraper that works against one state is a rewrite for the next, and the transport layer is the one part you shouldn't have to rebuild each time.
Harder than "public data" makes it sound. We tested 50 US state and federal public-data URLs directly from a clean US datacenter IP, with no proxy tricks. Only 36% returned usable content. Of 22 state business registry portals, just three served their content directly.
The rest failed like this:
| What we hit | Where |
|---|---|
| Cloudflare interstitial challenge | FL, CO, NC, IA, GA |
| Cloudflare hard block | UT |
| Imperva / Incapsula | MA, NV, and the national nurse licence verification system |
| DataDome | AK |
| reCAPTCHA gate | LA |
| CloudFront 403 | AR |
| HTTP 202 with an empty body | IN, and one state real-estate commission |
| Connection timeout | NE, MO, TN, WI, OK, OH |
Two takeaways from that table. First, the blocking is almost always an off-the-shelf managed WAF product, not a custom anti-bot system — which is why a browser-shaped TLS fingerprint makes such a big difference. Second, a 2xx response doesn't mean success. Several of these portals answer HTTP 202 with a zero-byte body when they don't like the traffic. If your monitoring only checks status codes, that looks like a success while your dataset quietly goes empty. Always check the actual content, not just the status line.
Before any fingerprint check runs, many US government sites decide whether to respond based on where the connection comes from. In our test, two state sites served a US datacenter IP normally and returned 403 to a European IP for the exact same request.
This is the cheapest fix on the list and the one people miss most often: if the target is a US agency, exit from the US. A scraper debugging "bot detection" from a Frankfurt server is often just hitting a geo-block. BotProxy includes every location on every plan, so pinning a country is a change to the proxy login, not an upgrade:
curl --proxy x.botproxy.net:8080 \
--proxy-user "pxu1000-0+US:password" \
--insecure "https://example.gov/licence/search?last=smith"
State-level exits work the same way (us-ny, us-ca, and so on) when a
portal is regional enough to care.
The common assumption is that datacenter IPs get blocked on sight. That's not what the traffic shows. Across 268 targets in production traffic, 28 show failure rates climbing sharply when the same volume of requests is concentrated onto fewer exit IPs — the same scraper, the same pages, the same time of day, failing far more often just because the requests were spread across a smaller pool.
In other words, the real constraint is a per-IP request budget. Registries and licence lookups are cheap to serve one at a time and expensive to serve in bulk, so the operator caps how much any one address can ask for. Rotation isn't a disguise — it's how you stay inside a budget that's counted per address. The practical question for a public-records crawl is requests per IP per minute, not "is this IP residential" — and that's something you can tune.
Rotation doesn't happen by itself, and this trips up more first crawls than any anti-bot system. Every request through BotProxy belongs to a session, and a session holds one exit IP for its whole life. If you never set a session ID, all your traffic runs in one default session and leaves from one IP — exactly the concentration that hurts in the numbers above.
The session is the proxy login itself:
login + location + SESSIONID
# three workers, three exit IPs, all in the US
pxu1000-0+US+w01
pxu1000-0+US+w02
pxu1000-0+US+w03Change the session ID and you change the IP. Generate one per worker thread at startup; rotate it when you want a fresh address — per county, per result page, per N requests, or at the first sign of throttling.
Two timeouts control a session, and both matter when you're debugging:
limit_session_age caps how long a session can live — 60 seconds on most credentials. Set it to 0 on the proxy user if you need a long-lived sticky IP for a multi-step form.Sticky and rotating are the same mechanism with different discipline. A multi-page registry search that carries server-side state needs one session for the whole sequence; a flat list of 200,000 licence numbers needs a new session every few requests.
The right IP with a Python-shaped TLS handshake still won't work. Imperva, AWS WAF, and
Cloudflare all inspect the TLS and HTTP/2 handshake, and a default requests or
curl client gives itself away long before your headers are read.
Anti-Detect Mode rewrites that handshake to match a real browser, and it's what gets through the WAF-based blocking in the table above. It's on by default. Two things to know:
--insecure in curl, verify=False in requests, or install the CA).Fingerprint spoofing and rotation solve different halves of the problem: the fingerprint decides whether a request looks like a browser; the IP decides whether you've already used up that address's budget. Public-records work usually needs both.
A lot of state portals only expose their data through a JavaScript search form, an ASP.NET postback sequence, or pagination tied to a server-side session. There's no JSON endpoint because the page state is the API.
For those, every plan includes cloud Chrome browsers you control over HTTP: navigate, fill the form, paginate, screenshot, export the PDF the portal insists on. It runs through the same account, the same IP pool, and the same bill, so your pipeline can use plain HTTP for the 90% of targets that allow it and a real browser only for the ones that don't. See the Browser API reference.
The fastest public-records scraper is the one you don't write. Before building anything, check whether the agency already publishes the data:
A bulk file is cheaper, more complete, and more reliable than any crawl. Use the proxy for sources that don't have one, for fields the extract leaves out, and for daily changes a quarterly dump can't give you.
A portal behind a Cloudflare managed challenge, or one that requires a logged-in account, isn't going to be solved by rotating datacenter IPs with a spoofed fingerprint — no matter how the pool is priced. Same goes for targets that only serve residential address space. We'd rather say that up front than sell you a plan that fails on your first crawl.
Everything else in the list above — the timeouts, the 403s, the fingerprint checks, the geo-blocks, the rate limits — is ordinary work, and it's what this service is built for.
Paste your URL into the live tester and see the real response before creating an account. Every plan includes all locations, Anti-Detect Mode, and the Browser API. You're billed on traffic volume only.
Test your target See plans Read the documentation
From our published proxy tier benchmark: what each of these targets actually required, measured rather than asserted.
Licensing boards, Secretary of State business registries, county assessors and recorders. About 1,450 separate government domains, each with its own stack and its own WAF.
Company and professional profiles, directories, and firmographic enrichment — long-running pipelines that re-check the same records on a schedule.
Marketplace and retailer product pages, price history, and stock tracking across many sites at once.
Search result pages by keyword and locale, rank monitoring, and share-of-voice reporting for agencies and in-house teams.
Airline and OTA fare and availability checks, where the price depends on the country the request comes from.
Listing portals plus the county assessor, recorder, and land-record sources that hold the official ownership and tax data.
Marketplace listings and product images collected on a schedule to find copies of your products and document them for takedowns.