If you found this page from a BotProxyBenchmark/1.0 entry in your
server logs: that traffic is a low-volume, published measurement of which class
of proxy a public web target actually requires. It fetches a handful of pages per
target, never faster than your robots.txt asks, obeys
robots.txt at both target selection and request time, and excludes
any target whose robots.txt we cannot read rather than assuming
permission. It touches public, unauthenticated pages only. The harness, the full
target list and every raw request record are published under an MIT licence so the
results can be checked or re-run — and where a site offers an official API or
bulk download, the report recommends that instead of scraping.
Read the methodology and findings in our documentation · Harness and raw results on GitHub
To have your site excluded from future runs, or to ask anything about this traffic, email [email protected] and we will action it.