Solutions › E-commerce price & catalog monitoring
Price and catalog monitoring is a rate-limit problem before it's a bot-detection problem. Retailer and marketplace pages are cheap to fetch once and expensive to fetch often, so the site meters requests per IP address. Rotating datacenter IPs with a browser-like TLS fingerprint handle this well — as long as you actually spread the load. That means varying the session ID, not just buying a bigger pool.
The workload is wide and repetitive: the same URL list, re-fetched on a schedule, across many domains. Cost per gigabyte and consistency matter more than raw speed.
People assume retailers block datacenter IPs outright. Our production traffic tells a more specific and more useful story. Across 268 targets, 28 show failure rates climbing sharply as the same request volume is concentrated onto fewer exit IPs. The most extreme case is a major gaming storefront: 2.4% failures when the load is spread across the pool, and 73.3% when it's concentrated — same target, same requests, thirty times the failure rate purely from how the traffic was distributed.
That's a per-address request budget being enforced, not a judgment about the IP's origin. Two practical consequences: a rising error rate usually means your effective pool has shrunk (or you never rotated in the first place), and adding threads without adding sessions makes things worse because it concentrates faster.
Every request through BotProxy runs inside a session, and a session holds a single exit IP for its lifetime. If you don't supply a session ID, a default session is used and every request leaves from the same address — the exact pattern that produced 73.3% failures above. The session lives in the proxy login:
login + location + SESSIONID
pxu1000-0+US+px-014 # US exit, worker 14
pxu1000-0+DE+px-014 # same worker checking the German storefront
Change the session ID to change the IP. A session is dropped after 5 minutes idle,
and limit_session_age caps its total life at 60 seconds on most
credentials (set 0 to turn off the cap).
Sensible defaults for a price crawler:
Large retailers run commercial WAFs that fingerprint the TLS and HTTP/2 handshake, so a default Python or Go client is flagged as automation before any header is examined. Anti-Detect Mode rewrites the handshake to match a real browser and is what gets past AWS WAF, Imperva, and Cloudflare fingerprint checks. It's on by default. It works as a man-in-the-middle, so your client needs to accept the proxy's certificate, and the browser profile supplies the User-Agent, replacing the one your client sends.
In e-commerce, exit country isn't an access question — it's a correctness question. Currency, tax display, assortment, regional pricing, and delivery estimates all follow the IP. A German price collected through a US exit isn't a German price. Pin the country in the login and keep one exit country per dataset. All locations are included on every plan, so running the same URL list from several countries doesn't cost extra.
Many storefronts render the price, variant matrix, or availability badge in JavaScript after the HTML loads, and some hide it behind a postcode or store selector. Every plan includes cloud Chrome browsers you control over HTTP for exactly these pages: navigate, set the store, read the rendered value, screenshot the result for an audit trail. Same account, same IP pool, same bill. See the Browser API reference. Use plain HTTP for the bulk of the list and a browser only where it's actually needed — a rendered page costs far more traffic than a simple fetch.
Price monitoring is unusually easy to break silently. A soft block often shows up as HTTP 200 with a page that has no price on it, and we've seen public portals return 202 with a zero-byte body. If your pipeline validates on status codes, both count as successes and the dataset degrades without an alert. Check the extracted field instead: a page without a parsable price is a failure regardless of what the status line says.
Sites behind a Cloudflare managed challenge, anything requiring a logged-in shopper account, and limited-drop retail — sneakers, ticket resale, console restocks — aren't what rotating datacenter IPs are for. Those markets are adversarial by design and need residential exits. For ordinary catalog and price monitoring at volume, this is the cheaper and steadier tool.
Paste your URL into the live tester and see the real response before creating an account. Every plan includes all locations, Anti-Detect Mode, and the Browser API. You're billed on traffic volume only.
Test your target See plans Read the documentation
From our published proxy tier benchmark: what each of these targets actually required, measured rather than asserted.
Licensing boards, Secretary of State business registries, county assessors and recorders. About 1,450 separate government domains, each with its own stack and its own WAF.
Company and professional profiles, directories, and firmographic enrichment — long-running pipelines that re-check the same records on a schedule.
Marketplace and retailer product pages, price history, and stock tracking across many sites at once.
Search result pages by keyword and locale, rank monitoring, and share-of-voice reporting for agencies and in-house teams.
Airline and OTA fare and availability checks, where the price depends on the country the request comes from.
Listing portals plus the county assessor, recorder, and land-record sources that hold the official ownership and tax data.
Marketplace listings and product images collected on a schedule to find copies of your products and document them for takedowns.