robots.txt Mistakes That Quietly Block Google (and How to Check Yours)
The robots.txt file tells search engine crawlers which parts of a site they may visit. One wrong line can stop Google from reading the whole site. This is common after launch, because staging sites often block everything and the rule is never removed. Here is how to find and fix the usual mistakes.
Where the file lives
robots.txt must sit at the root of the domain, for example example.com/robots.txt. A file in a folder is ignored. Each subdomain needs its own file. If the address returns a 404, crawlers treat the whole site as open to crawling.
Mistake 1: Disallow everything left on from staging
A rule that disallows the whole site is a common setting on staging. If it is copied to the live site, Google is told not to crawl anything. Look for a line that blocks the root path for all crawlers, and remove it or limit it to private areas only.
Mistake 2: Blocking CSS, JavaScript or images
Google renders pages like a browser. If style or script files are blocked, it may see a broken page. Do not disallow the folders that hold the files your pages need to display.
Mistake 3: Using robots.txt to hide pages from search
Blocking a page in robots.txt stops crawling, but the address can still appear in results if other sites link to it. To keep a page out of search, use a noindex tag and leave the page crawlable so Google can see the tag. Never rely on robots.txt for private content.
Mistake 4: No sitemap line
You can list your sitemap address in robots.txt with a Sitemap line. It is not required if you submit the sitemap in Search Console, but it helps crawlers find it.
Mistake 5: Typing errors and case
- Paths are case sensitive, so a rule for /Blog/ does not match /blog/.
- A missing slash can block more than you meant. Test each rule.
- Rules apply to the crawler named above them. Check which user-agent group each rule sits in.
How to check yours
- Open your-domain/robots.txt in a browser and read it.
- Look for any rule that blocks the whole site or key folders.
- In Search Console, use the URL inspection tool on an important page and check whether crawling is allowed.
- Run a Nyrolo audit, which reads robots.txt and warns if it blocks all crawlers or has no sitemap line.
Frequently asked questions
Does robots.txt stop a page being indexed?
No. It stops crawling. A page can still be indexed from links elsewhere. Use a noindex tag to keep a page out of search results.
Is robots.txt required?
No. Without one, crawlers may visit everything. It is useful for keeping crawlers out of areas such as admin pages and for pointing to your sitemap.
How long until Google notices a change?
Google refreshes its copy of robots.txt regularly, often within a day, but it can take longer. You can ask Search Console to re-crawl important pages after fixing it.
Keep reading
Check your own site in about 15 seconds
Nyrolo runs a bounded pre-launch audit and gives you a 0-100 score, a verdict and fix steps for WordPress, Webflow, Shopify and custom code.
Run Free Audit