Solutions › Property & real-estate data

Scraping property and real-estate data

Property data comes from two very different kinds of source, and a pipeline usually needs both: the listing portals, which are modern sites protected by commercial WAFs, and the county assessor, recorder and land-record systems, which hold the authoritative ownership, valuation and transfer data and are frequently older, slower and behind a challenge page. Rotating datacenter IPs with a browser-like TLS fingerprint and a US exit cover both, because in each case the obstacle is a per-address request budget and a generic WAF rather than consumer-grade bot detection.

Two different sources, one pipeline

• Listing portals and brokerage sites — active inventory, asking prices, days on market, photos, agent details.
• County assessors — parcel records, assessed values, tax rolls, characteristics, exemptions.
• County recorders and land registries — deeds, mortgages, liens, transfer history.
• Permit and code-enforcement portals — construction activity and property condition signals.
• Real-estate licensing commissions — agent and broker verification.

The government half of this list behaves exactly like the rest of the public-records world, and the detail on those targets lives on the public records and government registries page. This page covers the property-specific version.

County records: what actually blocks you

There is no national property database in the United States; there are thousands of county systems. We probed 50 US state and federal public-data URLs directly from a clean US datacenter IP and found only 36% returned usable content. The protections encountered were off-the-shelf products rather than bespoke defences: Cloudflare challenges (FL, CO, NC, IA, GA), a Cloudflare hard block (UT), Imperva/Incapsula (MA, NV), DataDome (AK), reCAPTCHA (LA), a CloudFront 403 (AR), and plain connection timeouts (NE, MO, TN, WI, OK, OH).

Two failure modes are worth designing for specifically. One state real-estate commission answered HTTP 202 with an empty body — a soft block that any status-code-based check records as a success, leaving you with rows that are structurally fine and empty. And timeouts here are often genuine: county systems can be slow rather than hostile, so a short client timeout will manufacture failures that a patient one would not see.

US targets need US exits

Geography is checked before fingerprinting. In our probe, two state sites served a US datacenter IP normally and returned 403 to a European IP for the same request. Counties are, if anything, more parochial than states. If you are collecting US property data from infrastructure in Europe or Asia, exit from the US — and use a state-level location where a portal is regional:

pxu1000-0+US+county-batch-11
pxu1000-0+us-fl+county-batch-11    # Florida exit for a Florida county system

Listing portals

The commercial side of property data behaves like the rest of the modern web: a WAF that fingerprints the TLS handshake, a per-IP request budget, and content rendered client-side. Nothing here needs residential exits; it needs a browser-shaped fingerprint and enough distinct addresses to keep any one of them under the budget. What it does need is care about terms of service — listing data is frequently licensed, and MLS-derived content usually carries explicit redistribution rules. Check what you are permitted to store and republish before scaling a crawl, not after.

How to rotate correctly

Rotation does not happen automatically. Every request through BotProxy runs inside a session; a session holds one exit IP for its lifetime; and with no session id supplied a single default session is used, so every request leaves from the same address. The session id is the last field of the proxy login:

login + location + SESSIONID

Change the session id to change the IP. A session is dropped after 5 minutes idle, and limit_session_age caps its total life — 60 seconds on most credentials, or 0 for no cap.

What that means for property work:

• Hold one session for a parcel lookup sequence — search form, result list, detail page, document image — because these systems carry server-side state and will drop you if the address changes mid-sequence. Raise limit_session_age on the credential you use for them.
• Rotate between parcels, and rotate per county, so no single address accumulates volume against one system.
• Rate is the real constraint. Across 268 targets measured in production traffic, 28 show failure rates rising sharply as the same request volume is concentrated onto fewer exit IPs. Spreading the work is what keeps a county crawl alive.

Anti-Detect Mode

Anti-Detect Mode rewrites your client's TLS and HTTP/2 handshake to match a real browser, and it is what gets past the WAF-based blocking listed above — AWS WAF, Imperva and Cloudflare fingerprint checks. It is enabled by default. Because it works as a man-in-the-middle, your client must accept the proxy's certificate, and the impersonated browser profile supplies the User-Agent, replacing the one your client sends.

Postback forms, map searches and PDFs

County systems are where the web's archaeology lives: ASP.NET postback navigation, frames, session- bound result pagination, and documents delivered only as scanned PDFs. Portals have the opposite problem — map-driven single-page applications with no server-rendered listing at all. Both are handled by the cloud Chrome workers included with every plan: fill the form, page through results, download the document, capture the screenshot. Same account, same IP pool, same bill. See the Browser API reference.

Look for a bulk extract before you write a crawler

A large number of counties publish assessor and parcel data as a downloadable extract, an ArcGIS or Socrata endpoint, or a records request that costs less than a week of engineering. Those files are more complete and more stable than anything scraped from the search UI. Outside the US, data.gov.uk and OpenStreetMap cover a good deal of the same ground. Scrape the counties that publish nothing, the fields the extract omits, and the daily changes a quarterly file cannot give you.

Where this will not work

Portals behind a Cloudflare managed challenge, anything requiring an agent login or MLS credentials, and data whose licence forbids redistribution are not jobs for a rotating proxy. The rest — slow county systems, WAF interstitials, geo-gates and per-IP rate limits — is ordinary work.

---

Try it on the target you actually need

Paste your URL into the live tester and see the real response before you create an account. Every plan includes all locations, Anti-Detect Mode and the Browser API, and is billed on traffic volume alone.

Test your target See plans Read the documentation

Other things people scrape with BotProxy