Solutions

BotProxy is built for collecting public data at high volume, day after day. These pages cover the workloads it handles best, what typically blocks each one, and how to set up rotation, location, and fingerprinting. Where we've tested the targets ourselves, the numbers are on the page.

Public records & government registries

Licensing boards, Secretary of State business registries, county assessors and recorders. About 1,450 separate government domains, each with its own stack and its own WAF.

B2B data & business directories

Company and professional profiles, directories, and firmographic enrichment — long-running pipelines that re-check the same records on a schedule.

E-commerce price & catalog monitoring

Marketplace and retailer product pages, price history, and stock tracking across many sites at once.

SEO & SERP rank tracking

Search result pages by keyword and locale, rank monitoring, and share-of-voice reporting for agencies and in-house teams.

Travel & airline fare monitoring

Airline and OTA fare and availability checks, where the price depends on the country the request comes from.

Property & real-estate data

Listing portals plus the county assessor, recorder, and land-record sources that hold the official ownership and tax data.

Brand protection & counterfeit monitoring

Marketplace listings and product images collected on a schedule to find copies of your products and document them for takedowns.

What these workloads have in common

Different as they look, they run into the same three problems — and the same three fixes show up on every page below.

  • Location matters most. When we tested US public-data sites, some returned normal responses to US IPs and 403 errors to European ones. Match your exit country to the target before trying anything else. All locations are included on every plan.
  • Rate limits are per IP, not against datacenter IPs as a whole. Across 268 targets in our production traffic, 28 showed failure rates climbing sharply when too many requests came from too few IPs. Rotation keeps you within a limit that's counted per address.
  • Your connection fingerprint gives you away before your headers do. Anti-Detect Mode rewrites the TLS and HTTP/2 fingerprint to match a real browser — the thing that gets past fingerprint checks from AWS WAF, Imperva, and Cloudflare.

Rotation is not automatic

The most common mistake is assuming the IP rotates on its own. Every request goes through a session, and a session keeps one IP for its lifetime. If you don't set a session ID, a default one is used and all your traffic comes from one address. The session is the proxy login: 

login + location + SESSIONID

Change the session ID to change the IP. Sessions are dropped after 5 minutes of inactivity, and limit_session_age caps their total lifetime — 60 seconds on most credentials, 0 to turn off the cap. Full details in the sessions documentation.

Where an official source exists, use it

SEC EDGAR, UK Companies House, data.gov.uk, OpenStreetMap, many US state registries, and a lot of counties offer APIs or bulk downloads. Those are cheaper, more complete, and more reliable than scraping. Use a proxy for sources that don't publish their data, for fields that official extracts leave out, and for daily changes a quarterly dump can't give you.

Start with your own target

Paste the URL you need into the live tester and see the real response before creating an account.

Test your target See plans Read the documentation