Technical SEO tools
Robots.txt File Generator
Build a robots.txt file with crawler rules, sitemap references, crawl delays, and common SEO presets.
◎ Dynamic Rules ◎ Sitemap Support ◎ Copy & Download
Rules
Build crawler rules
Add allow and disallow directives for all crawlers or specific bots.
Your data stays private
The file is generated in your browser. Nothing is uploaded or stored.
Crawler & indexing basics
What is a robots.txt file?
A robots.txt file tells search engine crawlers which parts of your website they are allowed or not allowed to crawl.
It is commonly used to block low-value pages, prevent unnecessary crawling, and help search engines discover important sections of your site.
Important:
Robots.txt controls crawling - not indexing or security.
How to use this tool
- Add user agents and allow/disallow rules.
- Optionally include sitemap URLs and crawl delays.
- Generate the robots.txt file instantly.
- Copy or download the generated output.
- Upload the file to your domain root at
/robots.txt.
Robots.txt best practices
- Always test rules before deploying them.
- Be specific with blocked paths and wildcards.
- Include your XML sitemap for faster discovery.
- Do not block CSS or JavaScript files unnecessarily.
- Avoid using robots.txt to hide sensitive content.
Example robots.txt file
A simple robots.txt file might look like this:
User-agent: *
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xmlThis allows most pages to be crawled while blocking admin and private sections.