About, method and logging
What this site is
Vendors document how their crawlers and AI agents identify themselves, which IP addresses they use and whether they follow robots.txt. This site checks those statements against requests that actually arrive at two sites we run: this one, and a separate quiet test site. Each agent page shows the documented facts, the observed facts and any disagreement between them.
Observed figures are small and say only what we saw. They are not rates for the whole web.
How an agent is verified
A user agent is only a claim, because anyone can send any user agent. Each request that claims to be a known agent gets the strongest check it passes.
Signed: the request carries an HTTP message signature (Web Bot Auth) that verifies against the vendor's published key directory.
IP list or Cloudflare verified: the address is in the vendor's published IP list, or Cloudflare labels the request as a verified bot.
Reverse DNS: the address resolves to a hostname under a suffix the vendor documents, and that hostname resolves back to the same address.
Vendor network: the address is in a block registered to the vendor, or in a vendor network that customers cannot rent.
Claimed only: none of the above. These requests are counted separately and are never used to say an agent disagrees with its documentation.
The robots.txt test
robots.txt allows every agent to read every page except one. The page at /trap/ is disallowed for all user agents. It exists only to measure which agents fetch a disallowed page. It is linked here on purpose so that crawlers can find it: /trap/.
What is logged
This site logs every request it receives: the time, path, status, a few request headers such as User-Agent and Accept, the network (ASN and country) and a classification of the visitor. IP addresses are stored as received when the visitor identifies as a bot or connects from networks operated by major bot operators (Google, Microsoft, Amazon, Meta, Apple and ByteDance). All other IP addresses are stored only as a salted one-way hash. Referrers are stored without query strings.
Raw logs are kept for 30 days, or 90 days for verified bot traffic. Hourly counts per agent are kept without limit. The site sets no cookies and serves the same content to every visitor. A small script on each page reports that JavaScript ran. It sends only a random per-page-view identifier.
What is published
Pages show counts per agent, and the IP prefix (a /24 for IPv4, a /48 for IPv6) of verified bot traffic. No full IP address is published, and nothing about visitors who are not known agents is published. User-agent strings sent by visitors are never shown. The only user-agent strings on this site are the examples that vendors document.
Each page carries a short reference code. If an assistant can quote the code, it read the page.
Licence and data
The registry and the observed data are published under CC BY 4.0. You may reuse them with attribution to whoreadsthe.com.
Machine-readable versions: /agents.json, /changes.json, /status.json. Every page also has a Markdown version at the same address with .md added.