What is a Robots.txt File?
Robots.txt is a plain text file that Search Engines use to see if there are areas within your website which should not be indexed. 
This file must be placed in your 'www' (home directory) in order for a search engine to see it. 

The sample below tells search engines not to index pages in the specified folders. 

---- robots.txt  (contents follow)

User-agent: * 
Disallow: /ares
Disallow: /cgi-bin/
Disallow: /images
Disallow: /image
Disallow: /secret
Disallow: /members
Disallow: /css
Disallow: /invoices
Disallow: /mofcart
Disallow: /orderdata
Disallow: /orderinfo
Disallow: /stats718
Disallow: /Templates
Disallow: /statistics
Disallow: /bkpmofcart