27 October 2026

crawl-delay in robots.txt that Google ignores

Crawl-delay in robots.txt slows Bingbot and leaves Googlebot untouched. A global rule under a star therefore makes the crawl uneven, even when the HTML is the same on both sides.

Why one robots.txt line splits two crawlers

A Crawl-delay line asks a crawler to pause between requests. Google does not support that field. Its robots.txt documentation lists only user-agent, allow, disallow and sitemap, and names crawl-delay as unsupported. Googlebot can see the line and still ignores it. Google's pace stays tied to host health and to how often the site appears to publish or change URLs.

Bingbot treats the same line as a hard gap. Bing honors crawl-delay and lets it override Crawl Control in Bing Webmaster Tools. The documented window is 1 to 20 seconds, and inside it Bingbot fetches at most one URL. Five seconds works out to about 17280 URLs a day, ten seconds to about 8640, and twenty seconds to about 4320. A catalog with tens of thousands of URLs, or pages that change stock every day, then refreshes much more slowly in Bing than in Google, even though the HTML is identical.

What a global delay actually changes

The usual mistake is a single Crawl-delay under User-agent *. Generators and old plugins drop that line in to calm every bot. Google skips the field anyway, so only crawlers that implement it, Bingbot among them, slow down. New product URLs, price edits and removed pages can already be processed in Google while Bing is still on an older snapshot.

A high delay also does nothing to lighten Googlebot when your host is busy. Status 429, a slow first byte or bursts of 5xx during peaks stay in place, because Google never reads the pause. Capacity, caching and stable responses are what move that rate. The URL stays allowed, and only the gap between requests changes, for crawlers that understand the field.

If you want Bing quieter on purpose, put the rule in its own Bingbot group and pick a low value that fits the size of the site. When the intervals of Googlebot and Bingbot drift far apart in the server logs and robots.txt contains one global crawl-delay, that line explains the uneven refresh. A technical check can read the rule next to the crawl pattern.

Frequently asked questions

Yes. Google supports only user-agent, allow, disallow and sitemap. Crawl-delay sits outside that list and Googlebot skips the line. Its pace stays tied to host health and to new or changed URLs.

About 8640 a day. Inside the window Bingbot fetches at most one URL. Five seconds works out to about 17280, and twenty seconds to about 4320. The rule overrides Crawl Control.

A global rule slows only crawlers that read the field, Bingbot among them. Googlebot does not change. If you want Bing quieter, put the rule in its own Bingbot group.

Want this on your site today?

First report free. Then auto-scan, a PDF in your inbox and alerts when a score drops.

Start a free report
Start free