Robots.txt Generator
Create crawling instructions for search engines, define allowed and blocked paths and reference your sitemap.
Workspace
Build your crawler rules
Configure crawler access, blocked paths and sitemap references, then review the generated robots.txt file.
Generated robots.txt
Review the file before publishing it.
User-agent: *
Disallow:Important
A Disallow directive is a crawling instruction, not an access-control mechanism. Do not use robots.txt to protect confidential or sensitive URLs.
Publishing
Save the generated content as robots.txt and make it available from the root of the relevant host.
Give crawlers clear instructions
Robots.txt can communicate crawling preferences and sitemap locations to compatible web crawlers.
Crawler rules
Define user-agent instructions for compatible crawlers.
Allow & Disallow
Create path rules for areas you want crawlers to access or avoid.
Sitemap reference
Include a sitemap URL so crawlers can locate your XML sitemap.
Review before publishing
Inspect the generated rules before placing them on your site.
Learn
What is robots.txt?
A robots.txt file provides crawling instructions to web crawlers. It can specify which paths a crawler should or should not crawl and can also reference your XML sitemap.
Where should robots.txt be placed?
The file normally belongs at the root of the website. For example, a site at example.com would typically make its robots.txt file available at example.com/robots.txt.
Robots.txt is not a security feature
Do not use robots.txt to protect passwords, private files, customer information or other sensitive content. Blocking a URL from crawling does not prevent people from accessing that URL. Use proper authentication and access controls for private resources.