How to Block Bad Bots and Scrapers Using .htaccess

Not all bots are welcome. While Google and Bing crawl politely, plenty of others scrape your content, probe for weaknesses, or hammer your server hard enough to slow it down. Adding a few rules to your .htaccess file lets you turn away the worst offenders and protect your bandwidth and resources.

How bot blocking works

Bots identify themselves with a "user agent" string. You cannot trust it completely — bad bots can lie — but many aggressive scrapers and outdated crawlers use recognisable names, and blocking those cuts a real chunk of junk traffic. This helps directly with high resource usage caused by relentless crawling.

Step 1: Open your .htaccess file

Edit the .htaccess in your site's document root (you may need to show hidden files first). Our guide on editing your .htaccess file covers the basics. Back it up before changing it.

Step 2: Add a bot-blocking rule

A common pattern blocks requests whose user agent matches names you specify:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (BadBot|ScraperBot|AnotherBot) [NC]
RewriteRule .* - [F,L]

Replace BadBot|ScraperBot|AnotherBot with the actual agents you want gone (separate each with |). The [F,L] returns a forbidden response and stops processing.

Step 3: Find the bots worth blocking

Do not block blindly. Check your cPanel Visitors and raw logs to see which user agents are hammering you, then add the genuinely abusive ones. Be careful never to block legitimate crawlers like Googlebot or Bingbot.

Frequently asked questions

Will this stop all bad bots?

No — the worst ones forge their user agent or rotate IPs. .htaccess blocking handles the honest-but-unwanted crawlers; for serious abuse and DDoS you want a firewall or a CDN like Cloudflare in front of your site.

Could I accidentally block Google?

Only if you add its user agent to the block list, so never include Googlebot, Bingbot or other legitimate crawlers. Double-check your list, then confirm your pages still appear in Search Console.

Is blocking by IP better?

For a single persistent attacker, yes — see restricting access with the IP Blocker. For patterns of bad crawlers, user-agent blocking scales better. Often you use both.