SEO Tools
Robots.txt Generator
Generate robots.txt files with custom allow and disallow rules for web crawlers
Trusted by 10k+ developers
Global Parameters
User-Agent Group #1
robots.txt Source
# Generated by FreeTools.io User-agent: * Disallow: /admin/ Disallow: /private/ Allow: / Sitemap: https://example.com/sitemap.xml
Crawler Guide
User-agent: * applies to all bots.
Disallow: / prevents bots from crawling anything.
Continue Exploring
Other SEO Tools you might like...
Meta Tag Generator
Generate complete HTML meta tags for SEO including Open Graph and Twitter Card tags
Try Now
Keyword Density Checker
Analyze keyword frequency and density percentage in your content
Try Now
Word Counter
Count words, characters, sentences, paragraphs, and estimate reading time
Try Now
Slug Generator
Convert any text into a URL-friendly slug with lowercase letters and hyphens
Try Now
XML Sitemap Generator
Generate XML sitemaps from a list of URLs for better search engine indexing
Try Now
Open Graph Preview
Preview how your page looks when shared on Facebook, Twitter, and other social media
Try Now
SERP Preview
Preview how your page appears in Google search results
Try Now
Readability Checker
Calculate Flesch-Kincaid readability score and grade level for your content
Try Now