CCBot
Reference: H33-3XJ-SBP
Collects pages to train models. Vendor: commoncrawl. robots.txt token: CCBot.
Documented
What the vendor states. Each fact links to its source and the date we checked it.
| Fact | Value | Source |
|---|---|---|
| User agent, as documented | CCBot/2.0 (https://commoncrawl.org/faq/) | source, checked 2026-11-22 |
| robots.txt behaviour | The vendor says it follows robots.txt | source, checked 2026-11-22 |
| Runs JavaScript | No | source, checked 2026-11-22 |
| Published IP list | https://index.commoncrawl.org/ccbot.json | source, checked 2026-11-22 |
| Signs requests (Web Bot Auth key directory) | Not documented | |
| Reverse DNS suffixes | .crawl.commoncrawl.org | source, checked 2026-11-22 |
| Network registered to the vendor | Not documented |
Observed
Not observed yet.
Where documented and observed disagree
None found so far.
Prompted tests
None yet.