Solutions › E-commerce price & catalog monitoring
Price and catalog monitoring is a rate-limit problem before it is a bot-detection problem. Retailer and marketplace pages are cheap to fetch and expensive to fetch often, so the site meters requests per IP address. Rotating datacenter IPs with a browser-shaped TLS fingerprint handle this class well, provided you actually spread the load — which means varying the session id, not just buying a bigger pool.
The workload is wide and repetitive: the same URL list, re-fetched on a schedule, across many domains. Cost per gigabyte and consistency matter more than a single fast request.
The common assumption is that retailers block datacenter IPs outright. Our production traffic says something more specific and more useful. Measured across 268 targets, 28 show failure rates climbing sharply as the same request volume is concentrated onto fewer exit IPs. The most extreme case in the set is a major gaming storefront: 2.4% failures when the load is spread across the pool, and 73.3% when it is concentrated — same target, same requests, thirty times the failure rate purely from how the traffic was distributed.
That is a per-address request budget being enforced, not a judgment about the IP's origin. Two practical consequences: a rising error rate usually means your effective pool has shrunk (or you never rotated in the first place), and adding threads without adding sessions makes things worse, because it concentrates faster.
Every request through BotProxy runs inside a session, and a session holds a single exit IP for its lifetime. If you do not supply a session id, a default session is used and every request leaves from the same address — the exact pattern that produced 73.3% failures above. The session lives in the proxy login:
login + location + SESSIONID
pxu1000-0+US+px-014 # US exit, worker 14
pxu1000-0+DE+px-014 # same worker checking the German storefront
Change the session id to change the IP. A session is dropped after 5 minutes idle,
and limit_session_age caps its total life at 60 seconds on most
credentials (set 0 to disable the cap).
Sensible defaults for a price crawler:
Large retailers run commercial WAFs that fingerprint the TLS and HTTP/2 handshake, so a default Python or Go client is identified as automation before any header is examined. Anti-Detect Mode rewrites the handshake to match a real browser and is what gets past AWS WAF, Imperva and Cloudflare fingerprint checks. It is on by default. Note that it works as a man-in-the-middle, so your client must accept the proxy's certificate, and that the browser profile supplies the User-Agent, replacing the one your client sends.
In e-commerce, exit country is not an access question, it is a correctness question. Currency, tax display, assortment, regional pricing and delivery estimates all follow the IP. A German price collected through a US exit is not a German price. Pin the country in the login and keep one exit country per dataset; all locations are included on every plan, so running the same URL list from several countries costs nothing extra in plan terms.
Plenty of storefronts render the price, the variant matrix or the availability badge in JavaScript after the HTML has loaded, and some hide it behind a postcode or store selector. Every plan includes cloud Chrome workers driven over HTTP for exactly these pages: navigate, set the store, read the rendered value, screenshot the result for an audit trail. Same account, same IP pool, same bill. See the Browser API reference. Use plain HTTP for the bulk of the list and a browser only where it is genuinely required — a rendered page costs far more traffic than a fetch.
Price monitoring is unusually easy to break silently. A soft block frequently arrives as HTTP 200 with a page that has no price on it, and we have seen public portals answer 202 with a zero-byte body. If your pipeline validates on status codes, both are counted as successes and the dataset degrades without an alert. Assert on the extracted field: a page without a parsable price is a failure regardless of what the status line said.
Sites behind a Cloudflare managed challenge, anything requiring a logged-in shopper account, and limited-drop retail — sneakers, ticket resale, console restocks — are not what rotating datacenter IPs are for. Those markets are adversarial by design and want residential exits. For ordinary catalogue and price monitoring at volume, this is the cheaper and steadier tool.
Paste your URL into the live tester and see the real response before you create an account. Every plan includes all locations, Anti-Detect Mode and the Browser API, and is billed on traffic volume alone.
Test your target See plans Read the documentation
Licensing boards, Secretary of State business registries, county assessors and recorders. Roughly 1,450 separate government domains, each running its own stack and its own WAF.
Company and professional profiles, directories and firmographic enrichment — long-running pipelines that re-check the same records on a schedule.
Marketplace and retailer product pages, price history, stock and assortment tracking across many sites at once.
Search result pages by keyword and locale, rank monitoring and share-of-voice reporting for agencies and in-house teams.
Airline and OTA fare and availability checks, where the answer depends on the country the request comes from.
Listing portals plus the county assessor, recorder and land-record sources that carry the authoritative ownership and tax data.