Pourquoi est-ce important ?
Robots.txt tells crawlers which parts of your site they may and may not visit. A missing or incorrectly configured robots.txt can cause Google not to index pages or crawl private sections.
Comment le corriger ?
Create a robots.txt at /robots.txt. Block /wp-admin/, /private/ and other non-public directories. Reference your sitemap.xml. Test your robots.txt via the Google Search Console robots.txt tester.
Ce check note de 0 à 100. Sous 50 c’est un échec, 50 à 89 un avertissement, 90 ou plus c’est bon. Après chaque crawl vous voyez la note par page dans le rapport, avec une courte explication.
Questions fréquentes
Checks if a valid robots.txt file is present.
Robots.txt tells crawlers which parts of your site they may and may not visit. A missing or incorrectly configured robots.txt can cause Google not to index pages or crawl private sections.
Create a robots.txt at /robots.txt. Block /wp-admin/, /private/ and other non-public directories. Reference your sitemap.xml. Test your robots.txt via the Google Search Console robots.txt tester.
Exemple
Copiez le code ci-dessous et adaptez le nom de domaine, la marque et le contenu à votre situation.
User-agent: * Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /private/ Disallow: /checkout/ Disallow: /*?s= # Allow AI crawlers User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / Sitemap: https://www.yourdomain.com/sitemap.xml