Solutions › Public records & government registries
Government registries and public records are scraped with rotating datacenter IPs placed in the right country, a browser-like TLS fingerprint, and a rotation strategy driven by per-IP rate limits rather than by outright IP bans. That combination — not residential exits — is what these targets actually require, because they are public documents protected by a generic WAF and a request budget, not by consumer-grade bot detection.
This is the widest and most fragmented class of public data on the web. In the United States alone it is spread across roughly 1,450 separate government domains, because every state, every professional board and every county publishes on its own infrastructure:
There is no shared platform behind them. Two neighbouring counties can run a modern React front end and a 2003 ASP.NET postback form. A scraper that works against one state is a rewrite for the next, and the transport layer is the one part you should not also have to rebuild each time.
Harder than their "public data" label suggests. We probed 50 US state and federal public-data URLs directly from a clean US datacenter IP, with no proxy tricks applied. Only 36% returned usable content. Of 22 state business registry portals, just three served their content directly.
The rest failed in these ways:
| What we hit | Where |
|---|---|
| Cloudflare interstitial challenge | FL, CO, NC, IA, GA |
| Cloudflare hard block | UT |
| Imperva / Incapsula | MA, NV, and the national nurse licence verification system |
| DataDome | AK |
| reCAPTCHA gate | LA |
| CloudFront 403 | AR |
| HTTP 202 with an empty body | IN, and one state real-estate commission |
| Connection timeout | NE, MO, TN, WI, OK, OH |
Two things follow from that table. First, the failure is usually a managed WAF product bought off the shelf, not a bespoke anti-bot system — which is why a browser-shaped TLS fingerprint moves the needle so much. Second, a 2xx response is not proof of success. Several of these portals answer HTTP 202 with a zero-byte body to traffic they dislike. If your monitoring counts status codes, that is filed as a success and your dataset quietly goes empty. Assert on content, never on the status line.
Before any fingerprint check runs, many US government sites decide whether to talk to you at all based on where the connection comes from. In our probe, two state sites served a US datacenter IP normally and returned 403 to a European IP for the identical request.
This is the single cheapest fix on the list and the one most often missed: if the target is a US agency, exit from the US. A scraper debugging "bot detection" from a Frankfurt box is frequently debugging a geo-block instead. BotProxy includes every location on every plan, so pinning a country is a change to the proxy login, not a change to your plan:
curl --proxy proxy.botproxy.net:8080 \
--proxy-user "pxu1000-0+US:password" \
--insecure "https://example.gov/licence/search?last=smith"
State-level exits work the same way (us-ny, us-ca, and so on) when a
portal is regional enough to care.
The usual story is that datacenter IPs get blocked on sight. That is not what the traffic shows. Measured across 268 targets in production traffic, 28 show failure rates climbing sharply as the same volume of requests is concentrated onto fewer exit IPs — the same scraper, the same pages, the same time of day, failing far more often simply because it spread its requests across a smaller pool.
In other words the binding constraint is a per-IP request budget. Registries and licence lookups are cheap to serve individually and expensive to serve in bulk, so the operator caps how much any one address may ask for. Rotation is not a disguise, it is how you stay inside a budget that is counted per address. It also means the practical question for a public-records crawl is requests per IP per minute, not "is this IP residential" — and that is a question you can tune.
Rotation does not happen by itself, and this trips up more first crawls than any anti-bot system. Every request through BotProxy belongs to a session, and a session holds one exit IP for its whole life. If you never supply a session id, all your traffic runs in one default session and therefore leaves from one IP — exactly the concentration that the numbers above say hurts.
The session is the proxy login itself:
login + location + SESSIONID
# three workers, three exit IPs, all in the US
pxu1000-0+US+w01
pxu1000-0+US+w02
pxu1000-0+US+w03Change the session id and you change the IP. Generate one per worker thread at startup; rotate it when you want a fresh address — per county, per result page, per N requests, or on the first sign of throttling.
Two lifetimes bound a session, and both are useful to know when you are debugging:
limit_session_age caps how long a session may live at all — 60 seconds on most credentials. Set it to 0 on the proxy user if you need a genuinely long-lived sticky IP for a multi-step form.Sticky and rotating are therefore the same mechanism with different discipline. A multi-page registry search that carries server-side state wants one session for the whole sequence; a flat list of 200,000 licence numbers wants a new session every few requests.
A correct IP with a Python-shaped TLS handshake still loses. Imperva, AWS WAF and Cloudflare all
inspect the TLS and HTTP/2 handshake, and a default requests or curl
client announces itself there long before your headers are read.
Anti-Detect Mode rewrites that handshake to match a real browser, and it is what gets through the WAF-based blocking in the table above. It is on by default. Two consequences to design around:
--insecure in curl, verify=False in requests, or install the CA).Fingerprint spoofing and rotation solve different halves of the problem: the fingerprint decides whether a request looks like a browser, the IP decides whether you have already used up this address's budget. Public-records work generally needs both.
A large share of state portals expose their data only through a JavaScript search form, an ASP.NET postback sequence, or pagination bound to a server-side session. There is no JSON endpoint to find because the page state is the API.
For those, every plan includes cloud Chrome workers driven over HTTP: navigate, fill the form, paginate, screenshot, export the PDF that the portal insists on. It runs through the same account, the same IP pool and the same bill, so a pipeline can use plain HTTP for the 90% of targets that allow it and a real browser only for the ones that do not. See the Browser API reference.
The fastest public-records scraper is the one you do not write. Before building anything, check whether the agency already publishes the data:
A bulk file is cheaper, more complete and more stable than any crawl. Use the proxy for the sources that have no such option, for the fields the extract leaves out, and for the daily deltas that a quarterly dump cannot give you.
A portal sitting behind a Cloudflare managed challenge, or one that requires an account and a logged-in session, is not solved by rotating datacenter IPs with a spoofed fingerprint — no matter how the pool is priced. Neither is a target that has decided to serve only residential address space. We would rather say that up front than sell a plan that fails on your first crawl.
Everything else in the list above — the timeouts, the 403s, the fingerprint checks, the geo-gates, the rate limits — is ordinary work, and it is what this service is built for.
Paste your URL into the live tester and see the real response before you create an account. Every plan includes all locations, Anti-Detect Mode and the Browser API, and is billed on traffic volume alone.
Test your target See plans Read the documentation
Licensing boards, Secretary of State business registries, county assessors and recorders. Roughly 1,450 separate government domains, each running its own stack and its own WAF.
Company and professional profiles, directories and firmographic enrichment — long-running pipelines that re-check the same records on a schedule.
Marketplace and retailer product pages, price history, stock and assortment tracking across many sites at once.
Search result pages by keyword and locale, rank monitoring and share-of-voice reporting for agencies and in-house teams.
Airline and OTA fare and availability checks, where the answer depends on the country the request comes from.
Listing portals plus the county assessor, recorder and land-record sources that carry the authoritative ownership and tax data.