robots.txt mistakes that hide your site

Last month: Disallow: / from staging, shipped to production. Two days invisible to polite crawlers. Search Console said “excluded”. The site “worked” fine in Chrome. That is the trap: you see it, Googlebot does not.

Disallow: /wp-admin is fine. Disallow: /wp- can take posts, media or the sitemap with it. Too wide, noticed too late. Same with /api if HTML lives behind it, or a wildcard that takes /blog because someone meant to block /b.

The file belongs at https://yourdomain.com/robots.txt. No 3xx to another host. A Sitemap: line whose XML returns 200. Do not block the CSS and JS the page needs to render, or Google thinks the site is broken. AI bots (GPTBot, ClaudeBot, PerplexityBot) get a yes or no on purpose, not through a group that happens to be too wide.

This is not a one-off. Every deploy. Five seconds. Open the file, look for Disallow: /, check the sitemap line. Saves you a weekend explaining why traffic vanished. And keep staging rules in a separate file that never ships to live.

I also check whether www or http serves a second robots.txt. One 301 to the canonical is fine. Two different files is how a bot gets the wrong rules. And if a CDN sits in front: it has to pass the file through, not answer 200 with an empty body. I have seen that. The site “worked”. The crawler read nothing.