Robots.txt Generator

Good to know

• robots.txt is advisory — well-behaved bots respect it. Malicious scrapers won't. It is not a security measure.
• Google-Extended — blocks your content from being used to train Gemini AI, while still allowing normal Google search indexing. Recommended for most sites.
• CCBot — Common Crawl is the dataset most open-source LLMs (LLaMA, Mistral…) were trained on. Blocking it is now widely recommended.
• WordPress tip — always pair Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php or front-end AJAX calls will break.
• Sitemap — declaring it in robots.txt helps search engines discover it faster than waiting for manual submission.