Every website should have a robots.txt file! But, unfortunately in the rush to get a new website launched, this unseen file seens to get forgeooten and most people simply forget all about their unseen robots.txt file .
When using wordpress, and trying to minimize your duplicate content exposure to the search engines, then having a proper robots.txt file is really important. I blogged about this a few months ago, and you can find the post .
So a good wordpress robots.txt file should look something like this, including the link to your sitemap (in XML format) at the bottom.
# Google Image
But, like the true geek I really am. I got tired of typing this in every time I made a new site. So I created a wordpress plugin that makes this plugin for you.
Yes, there are other wordpress plugin out there, some better, some worse, but this one works exactly like I want it to work. In otherwords, it optimized to the MAX!
So if you want to get the plugin for free, simply .
Go to your plugin menu in the admin area, and activate.
Then click on the BH Robots.txt menu item in the Settings sections and DON'T forget to remove the leading # symbole at the start of the last line. Only remove it if you have a sitemap, otherwise leave it there.
Let me know what you think about this plugin.
[bhtweetthis] Are two or more pronouns joined by a conjunction or a i need help with my math homework preposition that!