Where should I place the robots.txt file?
The robots.txt file must be uploaded to the root directory of your website's domain, making it accessible at https://yourdomain.com/robots.txt.
Robots.txt Generator
Webmaster ToolGenerate clean, error-free robots.txt files to manage search bot crawling behavior. Safeguard private directories and prevent indexing loops.
rules_config.txt
Allow Google (Googlebot)
Allow Bing (Bingbot)
Allow Baidu (Baiduspider)
Allow Yandex (YandexBot)
/admin/cgi-bin/private/api
robots.txt
What is a Robots.txt File?
A robots.txt file is a simple text file placed in your website's root directory. It is part of the Robots Exclusion Protocol (REP) and instructs search engine web crawlers (bots) which directories or files they are allowed or disallowed to crawl and index.
How to Use the Robots.txt Generator
Specify your sitemap URL, set access permissions for default bots (Googlebot, Bingbot, etc.), choose standard crawl delays, and specify directories you want to block (like /admin, /cgi-bin, or private user dashboard paths). The generator outputs a clean robots.txt snippet to save.
FAQ
Can robots.txt hide pages from search engines completely?
A robots.txt disallow rule tells search engines not to crawl the page. However, if other sites link to that page, search engines might still index it without crawling. To guarantee a page is not indexed, use a 'noindex' meta tag instead.
SEO Toolkit ReferenceAudit website health using our collections of sitemap, robots, redirect, and header checkers.
Related Resources
Related Tool
Meta Tag Generator
Generate perfect HTML meta tags for your website including title, description, robots, open graph, and twitter cards to boost your search rankings.
Related Tool
Sitemap Health Checker
Fetch and validate XML sitemaps to score HTTPS coverage, duplicate URLs, lastmod usage, and sitemap size limits.