Solutions › SEO & SERP rank tracking

Scraping search results for rank tracking

Rank tracking means fetching the same pages over and over, at high volume, where the results change depending on location and each IP address has a query budget. What you need from a proxy is a large rotating pool, precise control over the exit country, a browser-like TLS fingerprint, and session handling that spreads queries across addresses instead of piling them onto one. Datacenter IPs do this at a cost per query that residential pricing can't touch.

What counts as a SERP target

  • Organic rank tracking — positions by keyword, device, and locale, on a daily or weekly schedule.
  • SERP feature monitoring — featured snippets, people-also-ask, local packs, shopping and news blocks, AI answer panels.
  • Paid-listing observation — which advertisers show up for which terms in which markets.
  • Share of voice and competitor reporting across a keyword portfolio.
  • Marketplace and app-store search, which works the same way and has the same constraints.

What makes this workload different is its shape: tens of thousands of nearly identical requests, all to one host, all needing to be comparable with yesterday's run.

Location decides the result set, not just access

A search result page depends on where the request comes from. The same keyword returns different ordering, different local packs, and sometimes a different index depending on country and region. The exit IP is part of the measurement, not a transport detail. A rank reported from the wrong country isn't a noisy rank — it's a different number entirely.

Two rules follow. Pin the exit explicitly in the proxy login (pxu1000-0+US, or a state like us-ny), and keep it stable across runs — a tracker that silently drifts from a US to a European exit will report a ranking change that never happened. All locations are included on every plan, so tracking several markets is a login change, not an upgrade.

Location is also checked before anything else: when we probed US public-data sites, two servers answered a US datacenter IP normally and returned 403 to a European one. Search engines aren't those sites, but the lesson applies — match the exit to the market before you debug anything more exotic.

Query budgets are counted per IP

Search engines limit how many queries one address can issue in a time window. That's a budget, not a ban, and it's what controls rank-tracking throughput. Our production traffic shows the same pattern across the board: measured on 268 targets, 28 show failure rates rising sharply as the same volume is concentrated onto fewer exit IPs. Throughput scales with how many distinct addresses your queries are spread across and how evenly they're paced — not with how many threads you start.

Rotation for rank trackers

Every request through BotProxy belongs to a session, and a session holds one exit IP. If you don't supply a session ID, a single default session is used, pinning all your queries to one address — which for a SERP workload burns the budget almost immediately. The session is part of the proxy login:

login + location + SESSIONID

pxu1000-0+US+kw-00042      # keyword batch 42, US market
pxu1000-0+DE+kw-00042      # same batch, German market

Change the session ID to change the IP. A session is dropped after 5 minutes idle, and limit_session_age caps its total lifetime — 60 seconds on most credentials, 0 to remove the cap.

Patterns that work for trackers:

  • A fresh session per keyword, or per small batch. Queries are independent, so there's nothing to keep sticky.
  • Hold the session across pagination when you need results beyond the first page, then drop it.
  • Rotate at the first sign of throttling instead of retrying through the same address.
  • Spread a keyword portfolio over time. Bursting the whole list in five minutes concentrates load no matter how many sessions you use.

Anti-Detect Mode

Search engines fingerprint the TLS and HTTP/2 handshake, and a default HTTP client is recognizable as automation before its headers are even read. Anti-Detect Mode replaces the handshake with a real browser's — it's what gets past fingerprint-based checks on WAF-protected targets. It's on by default. Because it works as a man-in-the-middle, your client must accept the proxy's certificate. The impersonated profile also supplies the User-Agent, overriding your client's — worth knowing for a tracker, since mobile-versus-desktop results depend on that header. Pick the browser profile to pick the device you're measuring.

Detecting a degraded SERP

The dangerous failure in rank tracking isn't an error — it's a result page that arrives with HTTP 200 but fewer results than a real one: a consent wall, a "verify you are human" body, or a trimmed set. By status code it's a success; parsed naively it silently reports a ranking drop. Validate the shape of the page: expected number of organic results, presence of the query echo, plausible result count. Anything that fails those checks is a fetch error, not a ranking change.

Where an API is the better answer

If what you need is your own site's performance, Google Search Console and Bing Webmaster Tools give you impressions, clicks, and average position directly — no crawling, no sampling error. Use those for first-party reporting. Scraping is the right tool for the parts they can't answer: competitor positions, SERP feature layout, and the result page as a searcher in a given market actually sees it.

Where this won't work

Anything that requires a signed-in search session, and engines that have moved the results behind an interactive challenge, are outside what rotating datacenter IPs can do. Check the target's terms of service before you scale, and keep your query rate defensible — a tracker that's polite per address is also the one that keeps working.


Try it on your target

Paste your URL into the live tester and see the real response before creating an account. Every plan includes all locations, Anti-Detect Mode, and the Browser API. You're billed on traffic volume only.

Test your target See plans Read the documentation

Other use cases

Public records & government registries

Licensing boards, Secretary of State business registries, county assessors and recorders. About 1,450 separate government domains, each with its own stack and its own WAF.

B2B data & business directories

Company and professional profiles, directories, and firmographic enrichment — long-running pipelines that re-check the same records on a schedule.

E-commerce price & catalog monitoring

Marketplace and retailer product pages, price history, and stock tracking across many sites at once.

SEO & SERP rank tracking

Search result pages by keyword and locale, rank monitoring, and share-of-voice reporting for agencies and in-house teams.

Travel & airline fare monitoring

Airline and OTA fare and availability checks, where the price depends on the country the request comes from.

Property & real-estate data

Listing portals plus the county assessor, recorder, and land-record sources that hold the official ownership and tax data.

Brand protection & counterfeit monitoring

Marketplace listings and product images collected on a schedule to find copies of your products and document them for takedowns.