Solutions

BotProxy is used for sustained, high-volume collection of public data. These pages describe the workloads it is built for, what actually blocks each one, and how to configure rotation, exit country and fingerprinting for it. They are written from measurement rather than marketing: where we have probed the targets ourselves, the numbers are on the page.

Public records & government registries

Licensing boards, Secretary of State business registries, county assessors and recorders. Roughly 1,450 separate government domains, each running its own stack and its own WAF.

B2B data & business directories

Company and professional profiles, directories and firmographic enrichment — long-running pipelines that re-check the same records on a schedule.

E-commerce price & catalog monitoring

Marketplace and retailer product pages, price history, stock and assortment tracking across many sites at once.

SEO & SERP rank tracking

Search result pages by keyword and locale, rank monitoring and share-of-voice reporting for agencies and in-house teams.

Travel & airline fare monitoring

Airline and OTA fare and availability checks, where the answer depends on the country the request comes from.

Property & real-estate data

Listing portals plus the county assessor, recorder and land-record sources that carry the authoritative ownership and tax data.

What these workloads have in common

Different as they look, they fail in the same three ways, and every page below returns to the same three fixes.

  • Geography is checked first. Direct probing of US public-data sites found servers that answered a US datacenter IP normally and returned 403 to a European one. Match the exit country to the target before debugging anything more exotic. All locations are included on every plan.
  • The limit is per IP, not on datacenter IPs as a class. Measured across 268 targets in production traffic, 28 show failure rates rising sharply as the same request volume is concentrated onto fewer exit IPs. Rotation is how you stay inside a budget that is counted per address.
  • The handshake gives you away before your headers do. Anti-Detect Mode rewrites the TLS and HTTP/2 fingerprint to match a real browser, which is what gets past WAF-based blocking — AWS WAF, Imperva and Cloudflare fingerprint checks.

Rotation is not automatic

The single most common mistake is assuming the IP changes by itself. Every request through BotProxy runs inside a session, and a session holds one exit IP for its life. With no session id supplied, a default session is used and all traffic leaves from one address. The session is the proxy login: 

login + location + SESSIONID

Change the session id to change the IP. Sessions are dropped after 5 minutes idle, and limit_session_age caps their total life — 60 seconds on most credentials, 0 to disable the cap. Full detail is in the sessions documentation.

Where an official source exists, use it

SEC EDGAR, UK Companies House, data.gov.uk, OpenStreetMap, many US state registries and a large number of counties publish APIs or bulk downloads. Those are cheaper, more complete and more stable than any crawl. Use a proxy for the sources that publish nothing, for the fields an extract leaves out, and for the daily changes a quarterly dump cannot give you.

Start with your own target

Paste the URL you actually need into the live tester and look at the real response before creating an account.

Test your target See plans Read the documentation