Making WordPress and Robots.txt work for you

[bhtweetthis]

Every website should have a robots.txt file! But, unfortunately in the rush to get a new website launched, this unseen file seens to get forgeooten and most people simply forget all about their unseen robots.txt file .

When using wordpress, and trying to minimize your duplicate content exposure to the search engines, then having a proper robots.txt file is really important. I blogged about this a few months ago, and you can find the post .

So a good wordpress robots.txt file should look something like this, including the link to your sitemap (in XML format) at the bottom.

User-agent: *
Disallow: /wp-content/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /wp-includes/
Disallow: /trackback/
Disallow: /cgi-bin/
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /wp-
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /page/
Disallow: /date/
Disallow: /category/
Disallow: /archive/
Disallow: /rss/

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

sitemap:http://www.yoursite.com/sitemap.xml

But, like the true geek I really am. I got tired of typing this in every time I made a new site. So I created a wordpress plugin that makes this plugin for you.

Yes, there are other wordpress plugin out there, some better, some worse, but this one works exactly like I want it to work. In otherwords, it optimized to the MAX!

So if you want to get the plugin for free, simply .

Go to your plugin menu in the admin area, and activate.

Then click on the BH Robots.txt menu item in the Settings sections and DON'T forget to remove the leading # symbole at the start of the last line. Only remove it if you have a sitemap, otherwise leave it there.

Let me know what you think about this plugin.

[bhtweetthis] Are two or more pronouns joined by a conjunction or a i need help with my math homework preposition that!