skip to content →
criticalSEO RULE · R14

X-Robots-Tag header: how it deindexes pages you wanted to rank

The X-Robots-Tag HTTP header gives the same instructions as a robots meta tag — and it overrides what the meta tag says. A misconfigured noindex header can silently deindex pages you absolutely want ranking.

The X-Robots-Tag HTTP header gives the same instructions as a robots meta tag — and it overrides what the meta tag says. A misconfigured noindex header can silently deindex pages you absolutely want ranking. X-Robots-Tag is an HTTP response header (X-Robots-Tag: noindex, nofollow) that tells crawlers how to treat a URL. It commonly gets set globally by reverse proxies, CDN rules, or framework defaults — and applies to PDFs, images, and non-HTML responses too.

Last updated·part of the 50-rule library

What it is

X-Robots-Tag is an HTTP response header (X-Robots-Tag: noindex, nofollow) that tells crawlers how to treat a URL. It commonly gets set globally by reverse proxies, CDN rules, or framework defaults — and applies to PDFs, images, and non-HTML responses too.

Why it matters

Unlike the meta tag (visible in HTML), the X-Robots-Tag only shows in the raw HTTP response. A misconfigured staging-to-prod CDN rule or a single line in a server config can deindex an entire site overnight. The damage is often discovered weeks later.

How to fix it

  1. Inspect the header on every indexable URL. curl -I https://example.com/your-page — look for X-Robots-Tag. Should not contain noindex on pages you want ranking.
  2. Check CDN/edge rules. Cloudflare, Fastly, CloudFront, and Vercel all let you set this at the edge. A staging rule pushed to prod is a common culprit.
  3. Remove the header in your server config. Apache: Header unset X-Robots-Tag. nginx: more_clear_headers X-Robots-Tag. Test with curl after deploy.
  4. Set up an alert. A monitor that scans 50 random URLs daily and alarms when X-Robots-Tag: noindex appears unexpectedly catches regressions before Google does.

Authoritative sources

Related SEO rules