Jerilo

When robots.txt blocks the whole site

robots.txt controls crawling, not indexing. A wide Disallow still hurts you because Googlebot never fetches the pages you care about, so discovery and refreshes stall.

What it means

A rule like Disallow: / under User-agent: * tells crawlers to stay out. That often shows up after someone pastes a staging robots file onto production, turns on a "block all bots" security plugin, or ships a preview template by mistake. Search Console then fills with blocked resources and stale coverage even when the site looks fine in Chrome. See Google's robots.txt introduction and how Google interprets robots.txt.

How to confirm

  • Open https://yoursite.com/robots.txt in a private window and read the live file, not a local copy.
  • In GSC, check the robots.txt tools and Page Indexing for "Blocked by robots.txt."
  • Fetch as Googlebot with curl and make sure you are not hitting a different host or a WAF challenge page.
curl -s https://example.com/robots.txt
curl -sI -A "Googlebot" https://example.com/

How to fix

  • Remove sitewide Disallow: / (and similar catch-alls) from production.
  • Allow the paths you want crawled. Keep Disallow for admin, cart, noisy faceted search, or private areas.
  • Point Sitemap: at your real sitemap once crawl works again.
  • If you need pages out of the index, use noindex on those URLs. Do not lean on robots.txt for that. See Indexed though blocked by robots.txt.

How it comes back

Staging robots files get shipped with the theme. CDN "block scrapers" presets sometimes overreach. WordPress "discourage search engines" plus a plugin can rewrite aggressive rules again. Diff robots.txt on every production deploy, or you will find this the hard way.

We built Jerilo to catch breaks like this early.