Robots.txt Generator
Build a correct robots.txt to tell search engines what they can and can't crawl. Start from a preset, fine-tune the rules, then copy or download the file straight to your site's root.
Choose a preset to generate your file…
Quick rules of the road
- Disallow blocks a path from being crawled.
Disallow: /blocks the whole site; an emptyDisallow:blocks nothing. - Allow carves out an exception inside a disallowed folder.
- Don't use robots.txt for secrets. The file is public, and listing a path can advertise it. Protect private pages with authentication.
- Add your sitemap so crawlers can discover every page you do want indexed.
Test it before you ship it
A single stray line can deindex an entire site — the classic disaster is a Disallow: / left over from a staging server going live in production. After you upload the file, open it at yoursite.com/robots.txt in a browser and check it in Google Search Console's robots.txt report.
When rules overlap, Google follows the most specific match rather than file order, so sequence doesn't matter. One thing to avoid: don't block your CSS and JavaScript folders. Google renders pages like a browser, and hiding those files can make your site look broken to the crawler and hurt rankings.
TrafficBud.io
One snippet is a start. TrafficBud audits your whole site, finds the keywords you're missing, and hands you a monthly SEO plan.
Frequently asked questions
What is a robots.txt file?
robots.txt is a plain text file at the root of your site (yoursite.com/robots.txt) that tells search engine crawlers which parts of your site they may or may not request. It's a crawling directive, not a security control.
Does robots.txt keep a page out of Google?
No. Disallow stops well-behaved crawlers from fetching a page, but a blocked URL can still be indexed if other sites link to it. To keep a page out of search results, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
Where do I put robots.txt?
It must live at the root of your domain — for example https://www.yoursite.com/robots.txt. A robots.txt in a subfolder is ignored.