robots.txt Status Database

Every domain's robots.txt status — has one, doesn't have one, or disallows crawling. Built for SEO and crawl-strategy research.