The 15 targets excluded from the proxy tier benchmark
Fifteen of the 47 candidate targets never reached the benchmark. Four are disallowed by their own robots.txt and eleven have a robots.txt that could not be read at all. They are published here rather than omitted, because which targets a study could not measure is part of its result: the exclusions are disproportionately the most defended sites in the candidate list, so every pass rate in the published run is biased upward.
Florida's Sunbiz corporation search, Georgia's eCorp, Alaska's entity search, Massachusetts CorpWeb, Nursys and Cook County IL are all in this table. They are also, on the reconnaissance probe, six of the hardest targets that were looked at.
The rule, and the one judgement call in it
A robots.txt that cannot be read is not permission. Several candidates
answer HTTP 403 to the benchmark's own User-Agent for /robots.txt itself;
since robots.txt is a published policy document rather than protected content,
the compliance gate re-reads it with an ordinary browser User-Agent and then
obeys whatever it says. Which User-Agent obtained the file is recorded, because
a site that will not serve its crawl policy to a self-identifying crawler is
worth reporting.
In this run the fallback never rescued a single target. Every target that
was benchmarked had its robots.txt read by the benchmark User-Agent on the
first attempt; every target where the benchmark UA failed also failed with a
browser UA, and was dropped. Where a note below reads browser_ua:HTTP 302,
the redirect could not be followed to a policy file either.
The four disallowed by policy
us_al_corpname, us_az_maricopa_prop, osm_way and pinterest_pin publish a
robots.txt that disallows the path. These pages record that fact and describe no
method for collecting from them. OpenStreetMap additionally publishes a full
API, and its bulk data is downloadable, which is the route to use.
All fifteen
| Target | URL | robots.txt | Why it was dropped |
|---|---|---|---|
osm_way |
https://www.openstreetmap.org/way/26659127 |
disallowed | robots.txt 886 bytes via benchmark_ua |
pinterest_pin |
https://www.pinterest.com/pin/1688918733523/ |
disallowed | robots.txt 30977 bytes via benchmark_ua |
us_al_corpname |
https://arc-sos.state.al.us/CGI/CORPNAME.MBR/INPUT |
disallowed | robots.txt 113 bytes via benchmark_ua |
us_az_maricopa_prop |
https://mcassessor.maricopa.gov/ |
disallowed | robots.txt 100 bytes via benchmark_ua |
uk_national_lottery |
https://www.national-lottery.co.uk/ |
unreachable | benchmark_ua:HTTP 403; browser_ua:HTTP 403 |
us_ak_entities |
https://www.commerce.alaska.gov/cbp/main/search/entities |
unreachable | benchmark_ua:HTTP 403; browser_ua:HTTP 403 |
us_fl_sunbiz_search |
https://search.sunbiz.org/Inquiry/CorporationSearch/SearchResults?inquiryType=EntityName&searchTerm=publix |
unreachable | benchmark_ua:HTTP 403; browser_ua:HTTP 403 |
us_ga_ecorp |
https://ecorp.sos.ga.gov/BusinessSearch |
unreachable | benchmark_ua:HTTP 403; browser_ua:HTTP 403 |
us_il_cook_prop |
https://www.cookcountyassessor.com/address-search |
unreachable | benchmark_ua:HTTP 403; browser_ua:HTTP 403 |
us_ma_corpweb |
https://corp.sec.state.ma.us/CorpWeb/CorpSearch/CorpSearch.aspx |
unreachable | benchmark_ua:HTTP 403; browser_ua:HTTP 302 |
us_nursys_lookup |
https://www.nursys.com/LQC/LQCSearch.aspx |
unreachable | benchmark_ua:HTTP 403; browser_ua:HTTP 302 |
us_oh_sos_biz |
https://businesssearch.ohiosos.com/ |
unreachable | benchmark_ua: |
us_ok_sos_corp |
https://www.sos.ok.gov/corp/corpInquiryFind.aspx |
unreachable | benchmark_ua: |
us_tn_bear |
https://tnbear.tn.gov/Ecommerce/FilingSearch.aspx |
unreachable | benchmark_ua: |
us_wi_corpsearch |
https://www.wdfi.org/apps/CorpSearch/Search.aspx |
unreachable | benchmark_ua: |
Two of the eleven are not what the census called them
PROTECTION-CENSUS.md groups Nebraska, Missouri, Tennessee, Wisconsin, Ohio and
Oklahoma together as hosts that "accepted the connection and never answered".
The probe data separates them. Nebraska, Missouri and Tennessee did time out —
ReadTimeout or ConnectTimeout after 20-30 seconds, which is the tarpit
signature. Wisconsin and Ohio failed DNS resolution (Temporary failure in
name resolution) and Oklahoma failed TLS certificate verification
(CERTIFICATE_VERIFY_FAILED). Those are different failures with different
causes, at least one of which may be the benchmark host rather than the target,
and none of the three is evidence of anti-bot behaviour.
Run date: 2026-08-18 · All 32 benchmarked targets · Full report · Exclusion data
Test your own target
The result above describes one URL on one day. Paste your own URL into the live tester and see the real response before creating an account.