Indexed, though blocked by robots.txt
Google Search Console's "Indexed, though blocked by robots.txt" means the URL is still in the index, but robots.txt still blocks crawling it. That is the opposite of what most people wanted when they added Disallow.
What it means
Robots.txt can stop Googlebot from fetching a URL. It does not reliably keep that URL out of Search.
If other sites link to it, or Google already knew about it, the URL can stay indexed with a near-empty snippet ("A description for this result is not available because of this site's robots.txt").
To drop a URL from the index, Google usually needs to crawl a noindex. Blocking crawl blocks that too.
That mismatch is spelled out in Google's
robots.txt introduction
and the
noindex documentation.
How to confirm
- Open the Page Indexing report in GSC and filter for "Indexed, though blocked by robots.txt."
- Inspect a sample URL. You should see crawl blocked by robots, and the URL still indexed.
- Read the live
robots.txtand match the Disallow rule to the URL pattern.
curl -s https://example.com/robots.txtHow to fix
Pick the outcome you actually want.
- You want it indexed and useful: remove the Disallow, let Googlebot crawl, request indexing.
- You want it out of the index: remove the Disallow first, add page-level
noindex(meta orX-Robots-Tag), wait for recrawl, then optionally re-block crawl later if you need to save budget. - Do not add noindex while Disallow is still there. Google cannot see the noindex if it cannot fetch the page.
If the Disallow was accidental and sitewide, start with robots.txt blocking the whole site.
How it comes back
Security plugins, "block scrapers" presets, and staging robots files published to production recreate Disallow rules while nobody watches the GSC report fill up again. Keep an eye on robots.txt, and remember Disallow is not privacy for confidential URLs. Use auth for that.
We built Jerilo to catch breaks like this early.