Adding CrawlTrack

[CaNNon]

Your phpBB Version: 2.0.
phpBB Type: Standard phpBB
MODs: No
Your knowledge: Beginner
Board URL: [url]http://[/url]

PHP Version:
MySQL Version:


What was done before the problem appeared?



What was done to try to solve the problem?




De.scription and Message

This package is really something for working with robots. It's pretty much stand alone except for linking it to your install.

The edits are necessary in two files, due to the way the portal works. The site can be accessed from either file. So both should have your code inserted. The install info usually only mentions one though. The code to add is supplied in the install and except for the second location it's what i used.

Please note this needs to be the first code to run so you must place the edits at the start.

Portal.php

Code: Select all

//-- adding CrawlTracker ( location 1 ) ----------------------------------------------------$crawltsite=1;require_once("/your-site-path/your-crawltrack-folder/crawltrack.php");// end add

common.php

Code: Select all

//-- adding CrawlTracker ( location 2 ) ----------------------------------------------------$crawltsite=1;require_once("/your-site-path/your-crawltrack-folder/crawltrack.php");// end add

Link to site at the demo page [url=http]http]

I also have the security functions enabled, I thought there would be some conflicts with the 2 included packages in integramod but have found no problems.

From the way this is working I'm pretty sure it could be run with all integra versions as long as you meet the posted requirements

To use CrawlTrack you need a web hosting with:

-php version 4.3 minimum

-graphique library GD2 to be able to see the graphs

-one MySQL database, it needs its own it won't mess with others so you must be able to make and link to another.

* Edited with a better choice of second include

[CaNNon]

I have submitted the edits, so now you can view the Integramod edits at:
[url=http]http://www.crawltrack.fr/doccms.php[/url]

If anyone tests this on other Integramod versions Please make sure your edits are submitted.

[richiebgood]

sorry, what does it do?

[CaNNon]

It allows you to do things like optimize the the way your found by search words.
It will keep track and allow you to pick the ones you get the most hits from.

Complete graphing of all the bot activity, our bot mod tracks like a dozen types of bots? other than allow or block them you can't work with it much. This tracks 100's and gives 5x more info. you need to know if a bot is moving tru a private area or has bad manners (not obeying your robots.txt) this will tell you. It also allows you to create test robots for your site to help you see what happening.

It also adds a small extra security package, if you would like PM me and I will give you a url/pass/user to look at a working one. you'll see everything except the tool page.

[richiebgood]

pretty cool, i saw the demo there alright.

And are all the bots added automatically i.e. ip ranges etc? or is there some maintenance with it?

[CaNNon]

No, you get updates from the mod site it's trough 2 buttons one does the bot updates and the other is security updates. That stuff is on the tool page and for security you won't get that access (if it's a working demo).

The demo is the same as you can allow for your users and pretty close to what you can show if you use the public viewing pages. this area could be added to your site and as it's using css I'm sure a little work and it could fit right in with any style sheet your using.

[richiebgood]

cool, i will add it to my 'to do' list of things i am doing to improve my site

[richiebgood]

Installed it there now, the crawl tracker works but i got an error when adding tags to portal and index. Perhaps i have it done wrong.

Code: Select all

<?php//-- adding CrawlTracker ( location 2 ) ----------------------------------------------------$crawltsite=1;require_once("/crawltrack/crawltrack.php");// end add

Is it the full url to the crawl tracker folder needed?


*edit, i did not need the first '/', solved

[CaNNon]

your site path is missing in front ---> /your-site-path/

yea the "/" would make it look off root instead off the working folder.

[richiebgood]

do you mean the paintballer.ie bit?

all i have at the moment is 'crawltrack/crawltrack.php'

the url is http://www.paintballer.ie/crawltrack/crawltrack.php.

So is what i have correct?

[CaNNon]

create a test robot, and see you may not need that?

[richiebgood]

ok test robot is working.

In the admin index, is shows a google bot on the forum but that is not showing up on the crawltrack. Does the bot have to hit the portal or index in order to get registered?

[CaNNon]

If the bot was on the site before the install it won't see it.
The bot would not have passed tru your additional code.

[richiebgood]

i see. nice one, cheers for the help. I will report back in a few days.

[richiebgood]

Well i have had crawl track for a few days now and i think it is great. It showing up more bots and differienciating more bots that the acp/bots managment.

Here is my public stats: http://www.paintballer.ie/crawltrack

[CaNNon]

Now that you have access to the tool page check the suppress a bot, (don't do it though as you will stop the bot) if you manage to find anything not on the list let me know. <img>

[richiebgood]

I deleted the crawler code from portal.php and index.php and instead added it to includes/page_tail.php instead.

Now crawtracker tracks the whole site and now just entry pages.

here is my page_tail.php

Code: Select all

<php>sql_query($sql)) )     {         message_die(GENERAL_ERROR, "Couldn't update banners data", "", __LINE__, __FILE__, $sql);     } }  // End add - Complete banner MOD  global $do_gzip_compress;  //// Show the overall footer.//include_once($phpbb_root_path . 'includes/functions_jr_admin.' . $phpEx);$admin_link = jr_admin_make_admin_link();$cookies_link = '<a>' . $lang['cookies_link'] . '</a>';  $template->set_filenames(array(     'overall_footer' => ( empty($gen_simple_header) ) ? 'overall_footer.tpl' ])? $lang['ctracker_ma_on'] : $lang['ctracker_ma_off'];  //Begin Lo-Fi Mod$path_parts = pathinfo($_SERVER['PHP_SELF']);$lofi = '<a>' . (empty($_COOKIE['lofi']) ? ($lang['Lofi']) : ($lang['Full_Version']) ) . '</a><br>';$template->assign_vars(array(     'L_LOFI' => $lang['Lofi'],     'LOFI' => $lofi,     'L_FULL_VERSION' => $lang['Full_Version']));//End Lo-Fi Mod  //// IM Portal//  if(!$layout_forum_wide_flag&&$portal_config['portal_tail']&&(!defined('HAS_DIED'))&&(!defined('IN_LOGIN'))){     $template->set_filenames(array(         'portal_tail'         => 'portal_page_tail.tpl')     );     portal_parse_blocks($portal_config['default_portal'], TRUE, 'tail');     $template->assign_vars(array('PORTAL_TAIL' => portal_assign_var_from_handle($template, 'portal_tail')));}      include($phpbb_root_path . 'includes/pseudocron.'.$phpEx);  //// Close our DB connection.//$db->sql_close();    /* Un-comment the line below to restrict Admins only to view page generation info */  if( ($userdata['session_logged_in']) and ($userdata['user_level'] == ADMIN) ){     $gzip_text = ($board_config['gzip_compress']) ? 'GZIP enabled' : 'GZIP disabled';       $debug_text = (DEBUG == 1) ? 'Debug on' : 'Debug off';       $excuted_queries = $db->num_queries;       $mtime = microtime();     $mtime = explode(" ",$mtime);     $mtime = $mtime[1] + $mtime[0];     $endtime = $mtime;       $gentime = round(($endtime - $starttime), 4);       $sql_time = round($db->sql_time, 4);       $sql_part = round($sql_time / $gentime * 100);     $php_part = 100 - $sql_part;  }//if( defined('DEBUG') ){     $debug_out = '<div>[Page generation time: '. $gentime .'s (PHP: '. $php_part .'% | SQL: '. $sql_part .'%) | SQL queries: '. $excuted_queries .' | '. $gzip_text .' | '. $debug_text .']</div>';}//else{     $debug_out = '<br>';}  $template->assign_vars(array(     'PHPBB_VERSION' => ($userdata['user_level'] == ADMIN && $userdata['user_id'] != ANONYMOUS) ? ' ' . '2' . $board_config['version'] : '',     'INTEGRAMOD_VERSION' => ($userdata['user_level'] == ADMIN && $userdata['user_id'] != ANONYMOUS) ? ' ' . $board_config['integramod_version'] : '',     'BLOCKED'    => str_replace('%T%', '<b>'. number_format($board_config['phpBBSecurity_total_attempts']) .'</b>', $lang['PS_blocked_line']),     'PROTECTED'  => ($userdata['user_level'] == ADMIN && $userdata['user_id'] != ANONYMOUS) ? ' ' . $lang['PS_blocked_line2'].' :: ' : 'Protected by phpBB Security ÂÂ © <a>phpBB-Amod</a> ::',     'TRANSLATION_INFO' => (isset($lang['TRANSLATION_INFO'])) ? $lang['TRANSLATION_INFO'] : ((isset($lang['TRANSLATION'])) ? $lang['TRANSLATION'] : ''),     'COOKIES_LINK' => $cookies_link,     'CRACKER_TRACKER_FOOTER' => create_footer_layout($ctracker_config->settings['footer_layout']),     'L_STATUS_LOGIN' => ($ctracker_config->settings['login_ip_check'])? sprintf($lang['ctracker_ipwarn_info'], $output_login_status) : '',     'ADMIN_LINK' => $admin_link,     //'INTEGRAMODCOPY' => "Powered by <a>IntegraMOD</a> $iversion ÂÂ © 2004, 2005, 2006 The Integramod Group",     'STYLECW' => "Style <a>Integra2 </a> ÂÂ © IntegraMod Team 2007",     ));  $template->pparse('overall_footer');  echo $debug_out;          //-- adding CrawlTracker ( location 1 ) ----------------------------------------------------$crawltsite=1;require_once("/home/irishpai/public_html/paintballer/crawltrack/crawltrack.php");// end add            //// Compress buffered output if required and send to browser//if( $do_gzip_compress && headers_sent() != TRUE ){     $gzip_contents = ob_get_contents();     ob_end_clean();     $gzip_size = strlen($gzip_contents);     $gzip_crc = crc32($gzip_contents);     $gzip_contents = gzcompress($gzip_contents, 9);     $gzip_contents = substr($gzip_contents, 0, strlen($gzip_contents) - 4);     $gzip_contents .= pack("V",$gzip_crc) . pack("V", $gzip_size);     header("Content-Encoding: gzip");     header("Vary: Accept-Encoding");     header("Content-Length: ".strlen($gzip_contents));     header('X-Content-Encoded-By: Integramod '.$board_config['integramod_version']);     echo "x1fx8bx08x00x00x00x00x00";     echo $gzip_contents;}exit;  ?>

[CaNNon]

double check the security link I pm'ed you and check times I would like to know if it picked up that attempt while it was in the tail or the portal and index.

I'm wondering, might it be better to use 3 locations for the extra security?

[richiebgood]

Oh I see what you mean now. The attack happened yesterday, I only changed the code into the page_tail today about 5 hours ago.

[CaNNon]

Yea I want the best of both worlds <img>

maybe even checking for a better spot to add our line. I don't have the traffic you do I'm just using this on a clan site.

[richiebgood]

I don't really know where is the best place to add or if it makes a difference. The only knowledge i have of php is to keep changing stuff untill i get an error, undo, and do something else <img>

I now have crawl tracker security on and cracker on, so i should be save enough. Perhaps some more senior members can assist with the correct / proper placement of the code.

[CaNNon]

Try adding the code to the top of common.php with portal.php richie. I'm testing it now. JidÃÂ ©bÃÂ © offered it as a possible solution in his forum.

It seems to be covering everything but i would like you to test it too.

[richiebgood]

Don't forget, once you have crawltracker installed, add your own url to the trusted sites list via the tool page. If you don't, things like the 'email topic' button in IM will get blocked.

[CaNNon]

Ok common.php is the way to go, I have edited the first post with the change.

After 24 hours of watching it, I get all the forum links like you wanted plus the security I wanted think were good to go. <img>

[salmercker]

has anything changed for this would love to add it to my site and my friend __chandler__ site. I am the one that does 90% of the code edits, upgrades, and mods.

[CaNNon]

Yep everything was keep up to date in my first post. <img>
Everything else just do as the crawltracker install file says.