Google announced recently that it would be rewarding websites converting to https by increased ranking position.

Ok, sure, fine, WHATEVER.

But wait, maybe this is a good thing? Let me spend 6 or 7 hours on this to find out.

Thanks for Nothing, Google

Well, things didn’t work out for us on one of our web properties. We converted to HTTPS on the day after Google’s announcement. Our traffic declined by over 18% compared to the same period in the prior month (comparing 14 days of traffic from a Thursday to a Wednesday, to avoid weekend/weekday fluctuations).

Conclusion: I Heart Hate You, Google

In this tutorial, we’ll learn how to block bad bots and spiders from your website. This is a standard safety measure we implement with our WordPress SEO service. We can save bandwidth and performance for customers, increase security, and prevent scrapers from putting duplicate content around the web.

Quick Start Instructions/Roadmap

For those looking to get started right away (without a lot of chit-chat), here are the steps to blocking bad bots with .htaccess:

• FTP to your website and find your .htaccess file in your root directory
• Create a page in your root directory called 403.html, the content of the page doesn’t matter, our is a text file with just the characters “403”
• Browse to this page on AskApache that has a sample .htaccess snippet complete with bad bots already coded in
• You can add any bots to the sample .htaccess file as long as you follow the .htaccess syntax rules
• Test your .htaccess file with a bot spoofing site like wannabrowser.com

Check Your Server Logs for Bad Bots

If you read your website server logs, you’ll see that bots and crawlers regularly visit your site–these visits can ultimately amount to hundreds of visits a day and plenty of bandwidth. The server log pasted above is from TastyPlacement, and the bot identified in red is discoverybot. This bot was nice enough to identify its website for me, but DiscoveryEngine.com touts itself as the next great search engine, but presently offers nothing except stolen bandwidth. It’s not a bot I want visiting my site. If you check your server logs, you might see bad bots like sitesnagger, reaper, harvest, and others.  Make a note of any suspicious bots you see in your logs.

AskApache’s Bad Bot RewriteRules

AskApache maintains a very brief tutorial but a very comprehensive .htaccess code snippet here. What’ makes that page so great is that the .htaccess snippet already has dozens of bad bots blocked (like reaper, blackwidow, sitesnagger) and you can simply add any new bots you identify.

If we want to block a bot not covered by AskApache’s default text, we just add a line to the “RewriteCond” section, separating each bot with a “|” pipe character. We’ve put “discoverybot” in our file because that’s a visitor we know we don’t want :

# IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(verybadbot|discoverybot) [NC,OR]

If you are on the WordPress platform be careful not to disrupt existing entries in your .htaccess file. As always, keep a backup of your .htaccess file, it’s quite easy to break your site with one coding error. Also, it’s probably better to put these rewrite rules at the beginning of your .htaccess file so no pages are served before the bots read the rewrite directives. Here’s a simplified version of the complete .htaccess file:

ErrorDocument 403 /403.html

RewriteEngine On
RewriteBase /

# IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(black.?hole|blackwidow|discoverybot) [NC,OR]

# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule . - [F,L]

Here’s a translation of the .htaccess file above:

• ErrorDocument sets a webpage titled 403.html to serve as our error document when bad bots are encountered; you want to create a page in your root directory called 403.html, the content of the page doesn’t matter, our is a text file with just the characters “403”
• RewriteEngine and RewriteBase simple mean “ready to enforce rewrite rules, and set the base URL to the website root”
• RewriteCond directs the server “if you encounter any of these bot names, enforce the RewriteRule that follows”
• RewriteRule directs all bad bots identified in the text to our ErrorDocument, 403.html

 Testing Our .htaccess File

Once you upload your .htaccess file, you can test it by browsing to your site and pretending to be a bad bot. You do this by going to wannabrowser.com and spoofing a User Agent, in this case, we spoofed “SiteSnagger”:

If you installed properly, you should be directed to your 403 page, and you have successfully blocked most bad bots.

Some Limitations

Now, why don’t we do this with Robots.txt and simply tell bots not to index? Simple: because bots might simply ignore our directive, or they’ll crawl anyway and just not index the content–that’s not a fix. Even with this .htaccess fix, it’ll only block bots that identify themselves. If a bot is spoofing itself as a legitimate User Agent, then this technique won’t work. We’ll post a tutorial soon about how to block traffic based on IP address. But, that said, you’ll block 90% of bad bot traffic with this technique.

Enjoy!

Want to be an awesome SEO or digital marketer? Seek out Those of Epic Awesomeness and learn from them. If you think like a legend in your work life, your work will be legendary. And, after what must be my 5000th listening of A Night at the Opera, I started thinking about what lessons Freddie Mercury can teach internet marketers.

Surround Yourself With Great People

Freddie Mercury had great co-workers. Guitarist Brian May played for a decade on a guitar he built with his father in the family tool shed. The guitar had an advanced tremolo system that wasn’t available at the time. Brian’s guitar fueled millions in record sales for Queen. Today, Brian owns his own guitar company that produces a commercial replica of his homemade guitar.

Queen’s bassist John Deacon began tinkering with piano in the mid-70s. Almost immediately after beginning to learn the new instrument, he crafted the now-iconic song “You’re My Best Friend.” John has said, casually, “basically that’s the song that came out you know when I was learning to play piano.” The song was a major hit, and helped push Queen’s “A Night at the Opera” album to triple-platinum sales.

Great people produce great things. You can’t do everything alone, so partner up or populate a staff with people of great talent and your work product will be great.

Appreciate Your Clients, They’ll Bring You Fame and Fortune

You are nothing without your clients (or your readers and buyers as the case may be). Freddie honored and loved his fans. “You brought me fame and fortune and everything that goes with it; I thank you all,” he insists in “We Are the Champions.” Your clients are gold. Treat them that way and you’ll enjoy riches and be a champion for years to come.

Build on the Past, but Be Original

Queen regularly employed elements of the past in their music. They drew heavily upon the growing heavy metal movement (at the time pioneered by Led Zeppelin and Black Sabbath) but also included less-expected elements like classical and opera. “It’s unheard of to combine opera with a rock theme, my dear,” Freddie once remarked. He also once said that “the whole point of Queen was to be original.” Queen knew where to ground themselves and where to branch out and be original. The resulting effect was a supernatural but radio-ready rock sound.

You can do the same. Web marketing draws on a foundation of concepts that originate with traditional advertising. Web marketers must honor well-settled advertising concepts like calls to action, customer conversion, branding, etc. After all, the psychology of the buyer hasn’t changed much since the beginning of time. But web marketing presents infinite opportunities to be original. When you blend a solid marketing foundation with creative and innovative ideas, you will excel, and you will succeed. Don’t simply imitate the successes of others, build on the successes of others with your own spin.

Be Fabulous and Think Big

“I always knew I was star, and now, the rest of the world seems to agree with me,” Freddie once mused. No one ever accused Queen or Freddie of thinking small — even before they were famous. In marketing as in rock music, it’s the big ideas that get the most attention. If you are building a website for a client or for one of your own properties, make it the best-in-class for that space. Queen wasn’t done with a song until they had lavished it with operatic 4-part harmonies. Aim high and people will love your product.

If you have an idea for a blog post article, why not go for broke and develop it into a full-scale infographic? We re-learned this lesson recently when our infographic study of social media impact on search signals went viral, earning us thousands of social media mentions and hundreds of links from the SEO and entrepreneurial community. Had we issued that material as a blog post, it might have been lost in the shuffle of thousands of other blog posts. We took the route a champion would take and it paid off.

When web content goes viral, it’s the same as when a band gets famous: when people love something, they tell their friends.