How to Create a robots.txt File for Your Website
A robots.txt file sits at the root of your site and tells search engine crawlers which areas they may and may not visit. It is a small plain-text file with a big job — used well it guides crawlers efficiently; used carelessly it can accidentally hide your whole site from Google. So it is worth understanding before you edit it.
Where it lives and how it looks
It must be at yourdomain.com/robots.txt — right in your document root. A basic, friendly file looks like this:
User-agent: *
Disallow:
Sitemap: https://yourdomain.com/sitemap.xml
Here User-agent: * means "all crawlers", an empty Disallow: means "nothing is blocked", and the Sitemap: line points crawlers to your XML sitemap.
Step 1: Create the file
- Open File Manager in cPanel and go to your document root.
- Create a new file named exactly
robots.txt. - Edit it and add your rules.
On WordPress, an SEO plugin can manage a virtual robots.txt for you without creating a physical file.
Common rules
- Block a folder:
Disallow: /private/ - Block a specific crawler: name it in
User-agent:thenDisallow: / - Allow everything: leave
Disallow:empty, as above.
The one mistake to avoid
A stray Disallow: / under User-agent: * blocks your entire site from search engines. This is a shockingly common way sites vanish from Google after a redesign. Always double-check this line before saving.
Frequently asked questions
Does robots.txt make a page private?
No. It asks well-behaved crawlers not to index a page, but the URL is still public and bad bots ignore it. For genuine privacy, use password protection instead.
robots.txt vs blocking bad bots — what is the difference?
robots.txt is a polite request that good crawlers respect. Malicious bots ignore it entirely, so to actually stop them you need .htaccess bot blocking.
Do I even need one?
Not strictly — sites work fine without it. But adding one with your sitemap line is good practice and gives you a place to guide crawlers if you ever need to.
Was this article helpful?