Robots.txt Unreachable in GSC: What It Means

Search Console troubleshooting

Robots.txt unreachable is alarming because it can make the live URL test fail on every page. But it does not automatically mean every page is broken. It means Google could not fetch the one file it checks before deciding whether crawling is allowed.

Real questions this answers

• Why does URL Inspection say robots.txt unreachable?
• Why does live test fail on every page?
• Should I change my pages when robots.txt is unreachable?
• How do I know if this is a real robots problem or a temporary GSC issue?

Most likely causes

• The robots.txt URL does not return 200 OK consistently.
• DNS or SSL problems affect Googlebot or the Inspection Tool from some locations.
• A CDN, firewall, bot rule, middleware, or hosting edge blocks Googlebot.
• The file was temporarily unavailable and Search Console has not refreshed the robots status yet.
• A redirect chain or invalid response prevents Google from reading the file quickly.

What to fix first

1. Open https://yourdomain.com/robots.txt and confirm it returns 200 OK with plain text.
2. Test with Googlebot-like user agents and from more than one network if possible.
3. Keep robots.txt simple: one User-Agent: * group is often enough for public sites.
4. Exclude robots.txt, sitemap.xml, and llms.txt from auth middleware, redirects, and private route logic.
5. Use Search Console robots.txt report to request a recrawl, then retry Live Test later.

Tools to use

• curl -I https://yourdomain.com/robots.txt for status and headers.
• Search Console robots.txt report for Google cached fetch status.
• VisRank scan to confirm crawlability, robots, sitemap, and noindex signals.

Credible sources

These are the reference docs used to keep the guidance grounded instead of guessing:

• Google URL Inspection Tool
• Google robots.txt introduction
• Google robots.txt report

Next useful steps

• Run a technical SEO audit
• Read the technical SEO checklist
• Diagnose indexing problems