Browsing the archives for the robots.txt tag.

ROBOTS.TXT and Web Public Relations done Properly Part 2 - Creating a New E-Commerce Site

Fundisi History

Well waddayouknow - following on from Friday’s post about how to perform web publicity well, there is now a comment from Eli of Market Theme (the competition of WP e-Commerce) that just missed my post on Friday or I would have mentioned it.

The post is a bit different to Dan’s but also enjoys my pleasure for using it as a demonstration of good Internet public relations:

  • The comment is relevant to the blog post.
  • It offers value to any blog reader who is also looking at e-commerce plugins for Wordpress.
  • It gets his company’s URL onto my blog which the search engine takes notice of and he scores another inbound link.
  • It provides a useful tip.
  • His post directs me to a product demo.

I am not taking Eli’s advice yet; instead I am using an entirely different site as a test site, which is totally blank without a single file in it except just one called ROBOTS.TXT.

This little file prevents any search engine spiders from indexing the site until I am ready for them to. I would hate all my fiddling of pages to get indexed until matters were ready for prime time.

The way to do this is very simple:

Create a pure text file called ROBOTS.TXT

Place the following two lines in it:

User-agent: *
Disallow: /

The first line means:  “Attention all spiders and web-bots”

The second line means: “No indexing of any folder whatsoever is allowed”

For an easy overview of Robots.txt go here.

No Comments