Solutions › Travel & airline fare monitoring
Fare monitoring is different from most scraping in two ways: the answer depends on the country the request comes from, and getting one price takes a sequence of requests that the site expects from a single visitor. So the proxy requirements are specific — precise exit-country control, and sessions that stay sticky for the length of a search flow but rotate between flows to stay inside each address's request budget.
Airline and OTA pricing is set per point of sale. The same route on the same date can be a different fare, in a different currency, with a different set of available fare families, depending on which country the request appears to come from. The exit IP therefore is not a delivery detail — it is one of the inputs to the measurement, alongside route, date and cabin.
Two rules follow. Pin the exit country explicitly in the proxy login and record it with every price you store, so a fare is always attributable to a market. And keep it stable across runs: a fare series that quietly changes exit country will show a "price change" that is only a change of market. Every location is included on every plan, so covering multiple points of sale is a login change rather than an upgrade.
pxu1000-0+US+route-JFKLHR-0312 # US point of sale
pxu1000-0+GB+route-JFKLHR-0312 # UK point of sale, same queryAccess can also be gated by geography before anything else is considered. In direct probing of US public-data sites we saw servers that answered a US datacenter IP normally and returned 403 to a European one. Matching the exit to the market is the first thing to check when a target starts refusing you.
Unlike a product page, a fare is rarely one request. A search posts an itinerary, the site issues a search token, results poll in over several calls, and the fare detail is fetched against that token. If the exit IP changes mid-flow, the site sees the session move country and typically discards it.
BotProxy's session model maps onto this directly. Every request runs inside a session, a session holds one exit IP for its whole life, and the session id is part of the proxy login:
login + location + SESSIONIDUse one session id for one complete search flow, then a new id for the next itinerary. Two lifetimes matter here more than anywhere else:
limit_session_age caps total session life — 60 seconds on most credentials. A multi-step fare flow usually needs this raised, or set to 0 for no cap, on the proxy user you use for travel work.If you supply no session id at all, a single default session is used and every flow shares one IP — which produces both cross-contaminated searches and rapid throttling.
A fare search is genuinely expensive for the operator: it touches inventory and pricing systems, not a cache. So carriers and OTAs meter searches per address aggressively. This is a request budget rather than a ban on datacenter traffic — the same pattern we measure elsewhere, where across 268 targets, 28 show failure rates rising sharply as the same volume is concentrated onto fewer exit IPs.
Practically: budget searches per IP per hour, not requests per second; spread a route portfolio over the day instead of bursting it; and when a search starts returning empty result sets, rotate to a new session rather than retrying on the address that has already been cut off.
Travel sites are heavy users of commercial WAFs, and those inspect the TLS and HTTP/2 handshake before any application logic runs. Anti-Detect Mode rewrites your client's handshake to match a real browser, which is what gets past AWS WAF, Imperva and Cloudflare fingerprint checks. It is on by default. It operates as a man-in-the-middle, so your client must accept the proxy's certificate, and the impersonated profile provides the User-Agent, replacing your client's own — relevant here because mobile and desktop fare pages can differ.
Many booking engines are single-page applications whose search state never appears in a plain HTML response, and some gate results behind interactions a plain HTTP client cannot perform. Every plan includes cloud Chrome workers driven over HTTP for those: run the search, wait for results to settle, read the fare, screenshot the quote for an audit trail. Same account, same IP pool, same bill — see the Browser API reference. Keep browser use to the flows that require it; a rendered search consumes far more traffic than a fetch.
Where an authorised feed exists, take it. Airline NDC APIs, GDS access and hotel distribution APIs give exact, contractual fare data with no fragile parsing and no rate-limit game. They require a commercial relationship, which is precisely why scraping exists for the rest — competitive monitoring, market surveys, coverage of carriers you have no agreement with. Both approaches coexist in most serious travel pipelines.
Anything that requires completing a booking, a logged-in loyalty account, or passing a Cloudflare managed challenge is outside what rotating datacenter IPs will do. Fare data is also unusually sensitive commercially — check the target's terms and keep the search rate defensible. A monitor that behaves like a heavy user is one that keeps working; a monitor that behaves like a denial-of-service attempt does not.
Paste your URL into the live tester and see the real response before you create an account. Every plan includes all locations, Anti-Detect Mode and the Browser API, and is billed on traffic volume alone.
Test your target See plans Read the documentation
Licensing boards, Secretary of State business registries, county assessors and recorders. Roughly 1,450 separate government domains, each running its own stack and its own WAF.
Company and professional profiles, directories and firmographic enrichment — long-running pipelines that re-check the same records on a schedule.
Marketplace and retailer product pages, price history, stock and assortment tracking across many sites at once.
Search result pages by keyword and locale, rank monitoring and share-of-voice reporting for agencies and in-house teams.
Airline and OTA fare and availability checks, where the answer depends on the country the request comes from.
Listing portals plus the county assessor, recorder and land-record sources that carry the authoritative ownership and tax data.