Free Robots.txt Generator Online
Generate a valid robots.txt file for your website, directly in your browser.
Processed in your browser. Nothing is uploaded.
How to generate a robots.txt file
- Enter a user-agent (or leave as * for all crawlers).
- List any paths to disallow or allow.
- Optionally add a crawl-delay and sitemap URL.
- Copy or download the result.
How this works
The generator builds standard robots.txt syntax directly from the fields you fill in — nothing is uploaded, and no server-side processing is involved.
Common use cases
- Create a robots.txt file for a new website
- Block search engine crawlers from specific paths
- Add a sitemap reference to an existing robots.txt
Frequently asked questions
Is this robots.txt generator free?
Yes. It is completely free, with no signup and no limits.
Where do I put the generated file?
Save it as robots.txt and upload it to the root directory of your website (e.g. yoursite.com/robots.txt).
What does "Disallow: /" do?
It blocks the specified user-agent from crawling your entire site — use the "Block all" preset for this.
Do I need a robots.txt file?
It's optional, but recommended for controlling how search engines crawl your site and for pointing crawlers to your sitemap.
Can I have different rules for different crawlers?
This tool generates rules for one user-agent at a time; copy the output and repeat with a different user-agent to add more blocks.