Solutions › B2B data & business directories
B2B data collection — company profiles, directory listings, professional registries and firmographic enrichment — is served well by rotating datacenter IPs because these targets throttle per address rather than block datacenter address space outright. What the workload needs is a large pool, a browser-like TLS fingerprint, and a session strategy that keeps any single exit IP under the target's per-address request budget over a crawl that runs for weeks.
The defining shape of this work is that it does not end. A one-off directory dump is a weekend project; an enrichment product re-checks the same records on a rolling schedule forever, which makes cost per gigabyte and predictable behaviour matter far more than peak throughput.
It varies more within this segment than any other. A plain HTML directory hands over its pages to any client that asks politely. A large aggregator, on the other hand, sits behind a commercial WAF and inspects the TLS handshake before serving a byte.
The official registers at the bottom of many enrichment pipelines are the hardest part, and they fail in the same ways government portals do. In a direct probe of 50 US state and federal public-data URLs from a clean US datacenter IP, only 36% returned usable content; of 22 state business registry portals, three served content directly. The rest answered with Cloudflare challenges, Imperva or Incapsula interstitials, DataDome, a reCAPTCHA gate, a CloudFront 403, or a bare connection timeout. One class deserves singling out: several returned HTTP 202 with an empty body. A pipeline that validates on status codes records those as successes and silently produces empty rows, so make your success check assert on the content you expected, not on the status line.
Across 268 targets measured in production traffic, 28 show failure rates rising sharply as the same request volume is concentrated onto fewer exit IPs. The site is not refusing datacenter traffic; it is refusing this address's eleventh request this minute.
That is good news for a B2B pipeline, because a request budget is something you can plan around. It means throughput is a function of pool size and pacing, not of luck, and it means the fix for a rising error rate is usually to spread the same work over more sessions rather than to buy a more expensive class of IP.
Every request through BotProxy runs inside a session, and a session holds one exit IP. With no session id supplied, everything shares one default session and one IP — the worst possible configuration for a crawl that runs all day. The session is encoded in the proxy login:
login + location + SESSIONID
pxu1000-0+US+enrich-07 # worker 7, US exit
pxu1000-0++enrich-07 # same session, no location constraint
Change the session id to change the IP. Two limits bound a session: it is dropped after
5 minutes idle, and limit_session_age caps its total life —
60 seconds on most credentials, or 0 for no cap.
Practical patterns for enrichment:
The larger directories buy the same WAF products as everyone else, and those products fingerprint the TLS and HTTP/2 handshake. Anti-Detect Mode replaces your client's handshake with a real browser's, which is what gets past AWS WAF, Imperva and Cloudflare fingerprint checks. It is enabled by default. Two things to account for: your client must accept the proxy's certificate, and the impersonated browser profile provides the User-Agent, overriding whatever your client sends — choose the profile rather than hand-writing a UA header.
Geography is checked before fingerprints are. Two of the US state sites in our probe served a US
datacenter IP normally and returned 403 to a European IP for an identical request.
If your enrichment stack runs in an EU region and hits US sources, that alone can look like bot
detection. Pin the country in the login (pxu1000-0+US); every location is included on
every plan, so it costs nothing to match the exit to the source.
A surprising amount of B2B ground truth is published properly, and pulling it from an API is cheaper, more complete and more stable than any crawl:
Use the proxy for the sources with no API, for fields the official extract omits, and for the daily changes a quarterly dump cannot give you.
Directories that require a logged-in account, professional networks that treat scraping as a terms violation, and anything behind a Cloudflare managed challenge are not the job for rotating datacenter IPs. Build on sources that publish to the public web, and check the target's terms before you scale a crawl against it.
Paste your URL into the live tester and see the real response before you create an account. Every plan includes all locations, Anti-Detect Mode and the Browser API, and is billed on traffic volume alone.
Test your target See plans Read the documentation
Licensing boards, Secretary of State business registries, county assessors and recorders. Roughly 1,450 separate government domains, each running its own stack and its own WAF.
Company and professional profiles, directories and firmographic enrichment — long-running pipelines that re-check the same records on a schedule.
Marketplace and retailer product pages, price history, stock and assortment tracking across many sites at once.
Search result pages by keyword and locale, rank monitoring and share-of-voice reporting for agencies and in-house teams.
Airline and OTA fare and availability checks, where the answer depends on the country the request comes from.
Listing portals plus the county assessor, recorder and land-record sources that carry the authoritative ownership and tax data.