AI bot crawl permissions
Checks if AI bots (GPTBot, ClaudeBot, PerplexityBot, etc.) are allowed to crawl your site.
Waarom is dit belangrijk?
AI search engines like ChatGPT, Perplexity and Gemini use their own crawlers. If you block them in robots.txt, your content won't appear in AI search results. Blocking consciously is an option, but you need to know what you're blocking.
Hoe los je het op?
Check your robots.txt for rules about GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot. If you want to be discoverable in AI search results, allow these bots. For control, block selectively.
Deze check scoort van 0 tot 100. Onder de 50 is een fail, 50 tot 89 een waarschuwing, 90 of hoger is goed. Na elke crawl zie je de score per pagina in je rapport, met een korte uitleg wat we vonden.
Veelgestelde vragen
Checks if AI bots (GPTBot, ClaudeBot, PerplexityBot, etc.) are allowed to crawl your site.
AI search engines like ChatGPT, Perplexity and Gemini use their own crawlers. If you block them in robots.txt, your content won't appear in AI search results. Blocking consciously is an option, but you need to know what you're blocking.
Check your robots.txt for rules about GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot. If you want to be discoverable in AI search results, allow these bots. For control, block selectively.
Voorbeeld
Kopieer de code hieronder en pas de domeinnaam, merknaam en inhoud aan naar jouw situatie.
# Allow AI crawlers (to appear in AI search results) User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Amazonbot Allow: / # Regular bots User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Sitemap: https://www.yourdomain.com/sitemap.xml