Missing robots.txt

robots.txt tells crawlers which areas they may access and where your sitemap lives.

What it is

This check flags sites that return no robots.txt at the root.

Why it matters for SEO

Without robots.txt, crawl directives are undefined; while crawling still works, you lose a useful control point and the sitemap hint.

Example

GET /robots.txt   ->  404 Not Found

How to fix

1. Add a robots.txt at the domain root.
2. Allow what should be crawled; disallow only what must not be.
3. Include a Sitemap: line.
4. Re-crawl to confirm robots.txt is served.

← Knowledge base