as much as i want bots to crawl my site for advertising on search engines, is there anyway we can control what information or pages/forums they can access on the site.
whilst checking up on what info the search engines were picking up, i noticed they were picking up on logins ect and stuff irrelevant to what readers want to see.
any help appreciated
thanks x
how can i control bot info retrieved
Moderator: Integra Moderator
3 posts
• Page 1 of 1
how can i control bot info retrieved
Author: woo » Wed May 28, 2008 9:56 am
Last edited by woo on Wed Dec 31, 1969 5:00 pm, edited 1 time in total.
Re: how can i control bot info retrieved
Author: CaNNon » Wed May 28, 2008 10:21 am
google robots.txt <img>
Last edited by CaNNon on Wed Dec 31, 1969 5:00 pm, edited 1 time in total.
3 posts
• Page 1 of 1
Who is online
Registered users: App360MonitorBot, Bing [Bot], Google [Bot], Majestic-12 [Bot], Vendethiel
It is currently Tue Jan 14, 2025 3:42 am
IntegraMOD® is Powered by phpBB® Forum Software © phpBB Group
IntegraMOD® is Powered by phpBB® Forum Software © phpBB Group
