Build a valid robots.txt file in under 30 seconds. Pick your default crawler policy, list paths to disallow, optionally reference your sitemap, and download a ready-to-deploy file.
robots.txt is a plain-text file at the root of your site that tells search engine crawlers which pages they can and can't access. It's not a security mechanism (anyone can read it) but it's the standard way to control crawl budget, prevent indexing of staging or admin URLs, and reference your XML sitemap. Most CMSs generate one automatically; this tool helps you customise it without touching the command line.
Choose your default policy: allow all crawlers (most public sites), block all crawlers (under-construction sites), or custom. List the paths you want to disallow (one per line, e.g. /admin, /private, /cgi-bin). Optionally add a Sitemap: line pointing to your sitemap.xml. Click Generate to preview, then Download to save robots.txt — upload it to your site root.
Generate a fresh robots.txt when launching a new site, after restructuring URLs, when blocking a staging area, when stopping crawl of duplicate content (faceted search, sort orders), or when adding a new sitemap reference. Always validate in Google Search Console's robots.txt tester after deployment.
Don't use robots.txt to hide sensitive information — it's publicly readable. To actually exclude content from search results, use a noindex meta tag instead. Always include a Sitemap: line. Test changes with Search Console's robots.txt Tester before going live — a typo can accidentally deindex your entire site.
robots.txt prevents crawling; noindex (a meta tag) prevents indexing. A page disallowed in robots.txt can still appear in search results without a description if other sites link to it.
No — it tells well-behaved crawlers to skip URLs, but malicious bots ignore it. For real protection use HTTP authentication.
It must live at https://yourdomain.com/robots.txt — the root of your site, not in any subdirectory.
Explore more website management tools on the tool hub — or jump straight to the Website SEO Score Checker, Online Ping Website Tool, Page Speed Test.