Robots.txt is a text file placed in a website’s root directory that instructs web crawlers — including AI bots — which pages or sections they are permitted or forbidden to access. Each major AI crawler (GPTBot, ClaudeBot) checks robots.txt before crawling, though some crawlers such as PerplexityBot have been observed to not consistently adhere to these directives.
Robots.txt misconfiguration is one of the most common causes of AI citation invisibility. A brand that blocks all bots, or uses overly broad disallow rules, may inadvertently prevent AI crawlers from accessing key content. Reviewing robots.txt specifically for AI crawler rules — ensuring GPTBot, ClaudeBot, and PerplexityBot are permitted — is a foundational AI SEO audit step.
