Is your robots.txt hiding you from search?
Twenty lines of text decide whether search is allowed to see you at all.
The most expensive silent mistake there is
Every other thing that goes wrong in search costs you a position. This one costs you all of them, and it does it without a single symptom you would think to look for.
One line can end your traffic
Disallow: / tells every crawler to stay off every page. It is the normal setting on a staging server, and it ships to production more often than anyone admits.
Nothing warns you
There is no error and no email. Pages stop appearing in search, slowly, and the cause is a file most people have never opened.
It is the cheapest thing to check
The file is public, plain text, and usually under twenty lines. Reading it takes a minute, and it is the first thing to rule out when rankings fall.
A file in, an answer out
Open yoursite.com/robots.txt
It is public, so no sign-in and no tools. If nothing loads, that is an answer too.
Paste it here
Select all, paste. The check runs in your browser, so the file goes nowhere.
Read what it actually does
In plain words: what is blocked, what is reachable, and the line to change.
What it checks
Whether your whole site is blocked
The staging leftover. It also catches a rule aimed only at Google, which hides you from the one crawler that matters most.
Whether you block your own CSS and scripts
Search renders your page to judge it. With the styles blocked it sees a broken layout, and rates the page on that.
Whether your sitemap is declared
This file is where every crawler looks for it. It also catches a relative path, which is ignored and reads the same as having no sitemap.
Rules that belong to nobody
A Disallow above the first User-agent line is skipped entirely. The author thinks a folder is blocked, and it is not.
Crawl-delay, which Google ignores
It looks like it is slowing crawlers down. For Google it does nothing at all, and for Bing it may be slowing your new pages down.
Typos that read as silence
Disalow with one L is not a directive. Crawlers skip it without complaint, so the rule you wrote was never a rule.
Straight talk about what it cannot see
It reads the file you paste, carefully. Everything outside that file is outside what it can tell you.
It cannot confirm the file is being served
A perfect robots.txt on your laptop does nothing. Crawlers read it at yoursite.com/robots.txt, and only opening that address proves it is there.
Blocked is not the same as hidden
Blocking a page stops it being crawled, not indexed. It can still show up if other sites link to it, and only a noindex tag on the page itself keeps it out.
It does not know your paths
It can tell you a rule blocks /assets. Whether /assets is something you needed crawled is a question about your site, not about the file.
Your file never leaves your machine
The whole check runs in your browser. Nothing is uploaded, which is also why there is no limit on it.
Rule it out in a minute
Before you rewrite a page or chase a link, make sure search is allowed to look at any of it.
Check my robots.txt