ai-crawler-registry: machine-readable registry of AI crawlers with measured block rates

A machine-readable registry of more than 50 AI web crawlers, published as ai-crawlers.json by Crawl Census. Each entry carries robots.txt tokens, user-agent strings, crawler purpose, and compliance status, with block-rate data measured across more than 4,194 domains. The dataset separates training crawlers from answer-engine and retrieval crawlers run by the same operator, including OpenAI, Anthropic, and Google, and records which agents state that they disregard robots.txt. Licensed CC BY 4.0 with a live API.