how can i control bot info retrieved
Posted: Wed May 28, 2008 8:56 amas much as i want bots to crawl my site for advertising on search engines, is there anyway we can control what information or pages/forums they can access on the site.
whilst checking up on what info the search engines were picking up, i noticed they were picking up on logins ect and stuff irrelevant to what readers want to see.
any help appreciated
thanks x
whilst checking up on what info the search engines were picking up, i noticed they were picking up on logins ect and stuff irrelevant to what readers want to see.
any help appreciated
thanks x