Why does this matter?
Robots.txt tells crawlers which parts of your site they may and may not visit. A missing or incorrectly configured robots.txt can cause Google not to index pages or crawl private sections.
How to fix it?
Create a robots.txt at /robots.txt. Block /wp-admin/, /private/ and other non-public directories. Reference your sitemap.xml. Test your robots.txt via the Google Search Console robots.txt tester.
This check scores from 0 to 100. Below 50 is a fail, 50 to 89 is a warning, 90 or above is good. After each crawl you see the score per page in your report, with a short note on what we found.
Frequently asked questions
Checks if a valid robots.txt file is present.
Robots.txt tells crawlers which parts of your site they may and may not visit. A missing or incorrectly configured robots.txt can cause Google not to index pages or crawl private sections.
Create a robots.txt at /robots.txt. Block /wp-admin/, /private/ and other non-public directories. Reference your sitemap.xml. Test your robots.txt via the Google Search Console robots.txt tester.
Example
Copy the code below and adapt the domain name, brand name and content to your situation.
User-agent: * Allow: / Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /private/ Disallow: /checkout/ Disallow: /*?s= # Allow AI crawlers User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / Sitemap: https://www.yourdomain.com/sitemap.xml