IM141 and IM3 VPS hosting needs [ cpu / mem]

[viragotech]

IM141 and IM3 VPS hosting needs [ cpu / mem]

Ok Im about to give up and just move the main forum to its own VPS.
They said it was out of date software bla bla bla but Im seein CPU spikes well beyond what i have seen before the 141 forum upgrade. They are gonna shut us down again soon. [ I see spikes well over 50% cpu load, never seen over 40% with the old software.]

I want to know if you think a VPS account of
CPU 1.13 GHZ
RAM 768 MB
Disk Space30 GB
Bandwidth 500 GB

Will be plenty for a active 141 and when it happens a IM3 forum???
[disc space n bandwith is fine]

Its the most reasonable I can get for $40 and still have it semi-managed.

[Helter]

under normal circumstances that should be 5 times what you would need.
I have seen on occasion though where a 132, 140 or 141 site had an issue I could not trace down where the cpu use was out of control.
This is rare though because I have dealt personally with close to 100 member sites and had maybe 5 of them exhibit this behavior.
Since all of them were basically stock, im inclined to think it has more to do with the server setup than the software as all of the out of control sites ran fine on my server. This is not to say that IM140/141 is not a resource hog, because it is a known fact that it uses alot of resources.

[viragotech]

Im using hostgator, which resells ThePlanet
but I really don't expect to make it through the weekend, unless I turn it off every time Im away from the PC, where I can watch the cpu load n kill the forum if it goes above 25% for more then 90sec [which is the rules].

[viragotech]

Though I wish they would just come up with a hard block or wall limit for shared, where I could never cause the server to run slow, my site would just run slower or not respond.
I wouldn't care.

[viragotech]

the site has been around since 03

Number of posts: 196976 Posts per day: 72.28
Number of topics: 20796 Topics per day: 7.63
Number of users: 12720 Users per day: 4.67
Board started: Tue Apr 01, 2003 1:22 pm
Database size: 275.57 MB

300 active members in a 36hr peroid

[Helter]

here is another suggestion
http://www.siteground.com/integramod-hosting.htm
They are a large company and familiar with the requirements of integramod.

[viragotech]

Well my host has sent this suggestion.

not sure how to do that, did the search rebuild but the search word match list is still near 175megs itself.



"The conversion takes lots of SQL time, but at present here are the MySQL stats for your user. The MySQL time for your user is actually 5 times higher than any other user. | xvnetwor_batman | 1 | 14035 | 5306 | 152431 | 84666 | 54168 | 21537 | 228802876 | 2147483647 | I would check into the search functions in each site. This, actually, is where we see most of the SQL usage with phpBB sites. I would make sure to mark common words out and make other changes to reduce the size of this table. "

[viragotech]

starting to think that is it, I disabled the search n hide all the search links and we got 10+ folks online and IM not seeing hugh cpu spikes.

how do I taper off the search problems with out deleting years of post with valuable info???

I could easily set up a google search box have users use them to search the site right?

But id hate to lose the search for new post function.

[viragotech]

Well search and banners are problem.

used to have about 30 in rotation in the top right.

turned about 10 on, didn't take many users or long to spike the cpu.

[Helter]

im testing this on one of my 141 site atm.
if you try this pay close attention to the stopwords edit.

overwrite your current files. It gives much more control over your search tables

[viragotech]

Will do soon as I get back from dinner, just follow the install.txt ehy?

really appreciate all your help.

[viragotech]

where does it output the most common words?
it ran fine but dunno where to find what it found?

## If you want to find your most commonly used words, so you can put some of them into this blacklist,
## you can run the following db query in phpmyadmin, before running this mod :
##
## SELECT swl.word_id, swl.word_text, COUNT(swm.word_id) as total
## FROM `phpbb_search_wordlist` swl, `phpbb_search_wordmatch` swm
## WHERE swl.word_id = swm.word_id
## GROUP BY swm.word_id
## ORDER BY total
## DESC LIMIT 0,50

[Helter]

add the words you found that you dont want included in your searches to language/lang_english/search_stopwords.txt
then go to acp and rebuild the search index. It will remove all the words you added to search_stopwords.txt from the search table

[viragotech]

Yo guy does the search stopword list have to be in alph order abc 123.....

just wonder a user found the stopword list google uses and it has allot of ones not in the default list but will be work to merge them if they gotta be in alph order.

[viragotech]

I guess that the dummest question thus far <img>

ill try n get someoe to combin the two list for me.
the google list is long

I can post it, if you want to see it. the google one.

[Helter]

im not sure if it matters, but the google list will be generic and the list you get from the mods db queries come from your own db

[viragotech]

Yea I messed all that up, I was not getting an output list of words, was running the query in the wrong place. So I totaly missed out on grabbing a whole bunch of words to exclude.

only got to like 10,000 post done when I stopped it as the table grew just as large, then figured it out to run that on the sql tab, not the queary tab.

So I only had 1 page of results to grabs n add new words.

gonna really suck as it takes so long as I must go slow.
150 post every 60 sec so not to blow up the cpu load. it may take only 20-22 sec to do it, but I must have a nice delay between CPU spikes or they wil get pissed.
rules are no more then 25% cpu load for 90 sec.

[viragotech]

brain fart, dummy me, though I was smart enough that when I did the upgrade to 141 I used a copy of the DB, the old DB is still up and is only maybe 100 post smaller then it. I can run that and get a word list from it. DOH

thought id end up running it many times to get more word.

[viragotech]

I ran the ocde but change it to "DESC LIMIT 0,3000"
And got one monster of a list, which I wanted but it took over an hr just to browse through n remove the words I don't want.

SELECT swl.word_id, swl.word_text, COUNT(swm.word_id) as total
FROM `phpbb_search_wordlist` swl, `phpbb_search_wordmatch` swm
WHERE swl.word_id = swm.word_id
GROUP BY swm.word_id
ORDER BY total
DESC LIMIT 0,50

[viragotech]

Should you remove forum usernames from the stop list?

Or will the forum find their post on a user search its own way?

[Helter]

you should probably remove them, though the search post by user function uses user_id instead of username.

[viragotech]

WOW

Our search stop list is near 4000 words long now, and as such I have to lower the limits al lthe way do to only doing 5 threads per process, with a 60sec pause between to stop from blowing up the server.

[Helter]

that is alot of words. If your trying to completely stop searching your forum there are better ways to disable it. I would think 4000 words would make searching impossible.

[viragotech]

that is alot of words. If your trying to completely stop searching your forum there are better ways to disable it. I would think 4000 words would make searching impossible.

Well I used that SQL queary to gather all the words I could form the DB.

Then removed all the words that were important to search our motorcycle forum.

All those left that are not needed, is near 4000 of them.

-------
HelterSkelter

Question, IM starting to think it would be best to use one of the mysql/php programs that will run on my computer to "fake" a server.

Could I then rebuild the search on my computer and run it full tilt so it doen't take a month to do on the real server???

Also will it matter if I have say all the images n files uploaded by users in the copy I would be using on my PC???

just want to make sure, we got allot of user uploads.

IM kinda up creek here right now, users want to search back but can't have it getting us shut down again.

[Helter]

yes, you can do the sql work on your desktop with xamp, winamp or other similar local LAMP. You do not need to import any images or attachments to your desktop install. Just dont use a few of the acp/general admin/db tools functions that check attachments and such.
Make sure the php and mysql versions are the same as the server so you wont have any trouble viewing it.

btw what is your domain?

[viragotech]

yes, you can do the sql work on your desktop with xamp, winamp or other similar local LAMP. You do not need to import any images or attachments to your desktop install. Just dont use a few of the acp/general admin/db tools functions that check attachments and such.
Make sure the php and mysql versions are the same as the server so you wont have any trouble viewing it.

btw what is your domain?

Kinda don't matter now, I guess just disabling the 1 search.php file was not enough. someone did a end run around it or there must be other places to make a search command and blew the server up today.

we are getting the boot.

And that siteground place you suggest who says they do IM hosting will not answer or reply to any of my question. Been trying to call n email all day.
[can't be to great of a site if they can't even be around on a Sat afternoon]

Only gonna cost $20 the min to move there but are they really ready for the load the forum will need for $4 a month.

[Helter]

$4 or $24 a month?
The problem with shared servers is they are strictly for low use recreational sites. It looks like your site outgrew that type of server a long time ago.

I can set you up a temp site here so you can do all your modding. If you choke this server then there is a major snafoo in your db. This server is a monster for a webserver.

[viragotech]

Awful nice of ya, but gotta get the site live, members are startin to get itchy already.

Problem is we are to big for shared, but to small for anything real $$$.


and it $4 a month for IntegraMOD hosting at siteground but it cost a min of $20 to get started.

[viragotech]

Oh Thank GOD, no really.

I had to smooth things over with the host as they went from hey lets help you fix it, to get all your stuff and leave. [ prob my fault, not a people person]

I have been spared and given time.

here is what IM looking at I guess.

$50 a month for fully managed,
could get it for $40 but only semi managed n no CP.
http://www.hostgator.com/vps-hosting/
CPU
1.13 GHZRAM
768 MBDisk Space
30 GBBandwidth
500 GB

http://www.siteground.com/semi_dedicated_hosting.htm
I will still ask siteground what they offer.
Page just says 2x shared cpu lvls and useage allowed.
for $30 a month free, CP

[viragotech]

Oh and if you wanna check up on us its

ViragoTechForum.com

kinda of an easy guess <img>

[viragotech]

Now that its running mostly ok on VPS I was gonna rebuild the search before all the memers hord in, but when I goto the rebuild search page its not displaying any info.

LIke where it left off last time, or the total thread count, and it doens't know the board is disabled.

re-run that sql that builds its table???

[viragotech]

................

[viragotech]

I have re uploaded all the files and checked the file edits.
Only thing left would be to run the sql string to create a table, though the table is there.

Its not automaticly getting the total post count anymore, starting post id doesn't even show, other fields don't auto fills with default post cycle n such like it use too.


[img]http://xvnetwork.com/rsearch.jpg[/img]

[Helter]

did you restore any files after installing the rebuild search mod?

[viragotech]

No but I lost some threads, using those db matiance tools.
In a desperate attempt to fix things.

One where it says it will removed post without any txt, lost like 10 threads.

[viragotech]

ONly thing I haven't done is just try to run it, set it to 1 post with a big delay, then stop it, maybe it would refind itsself.

but dunno if that would harm things.

[viragotech]

Should I go ahead and just tryto run it???

[Helter]

make sure you have a current db backup if you do.

[viragotech]

make sure you have a current db backup if you do.

Ok what do you suggest I do then, your comment has the tone of its not a good idea?


Un-do all the coding and files and use the stock search rebuild?


Now that I can't get booted for to much CPU load, I need to get that run to get the search function useable again.

[Helter]

I only say that because the db tools mod sometimes does not like CH and will put all your subforums in a new forum at the end of your current forums. then you have to manually put them back where they go. On a lard site like yours, that can be time consuming. There was a fix for this applied to 141e but a few users have reported this issue since then.

[viragotech]

I only say that because the db tools mod sometimes does not like CH and will put all your subforums in a new forum at the end of your current forums. then you have to manually put them back where they go. On a lard site like yours, that can be time consuming. There was a fix for this applied to 141e but a few users have reported this issue since then.

Back from the dead, so again the worse that will happen is it will scramble the forum section, and Ill have to move them back around?

The busted search is the worst part of things, took 20 mins and 3 tries to get a new thread posted as it was having problems entering in new search words.

[Helter]

can you zip a copy of your db and pm it to me? Your having alot of difficulties that sound more like a lack of server resources and Id like to check it on my server before I begin bashing The Planet <img>

[viragotech]

can you zip a copy of your db and pm it to me? Your having alot of difficulties that sound more like a lack of server resources and Id like to check it on my server before I begin bashing The Planet <img>

That would be awesome, though I will have to set you up with a user name n password to download it by FTP, its over 150 megs.

I will PM you the info.

[viragotech]

Though I would not be quick to blame any one thing cept that its a real old forum, lots of trial n error hacking n failed upgraded over the years.

Before the 141 upgrade it was a 132d file system with a 140 DB as the 140 upgrade failed.

I was able to see our forum blow up the shared hosting server with allot more power. And it does the same to the VPS and it seems to ramp up.
works great then get slowly worse, you then get a error page when the server dies and then it stats all over. Could only be 3 people on the site or 30 when it goes goofy.


If it was a bike, Id check the fuel system and clean the carbs <img>

[Helter]

one thing i noticed is that your main page has two images linked to external sites. If either of those sites is slow, it will slow you down while your forum waits to load the images. If possible, you should copy the images and host them locally.

btw, ive got 4 quad core cpus on this server. Rebuilding the search table used 94% of on of the 16 cores and 77% of another. It used a total of 7megs of ram. This will put you in the realm of a dedicated server if you plan to reindex the search table again.

another option is since it is mostly cpu usage that is out of control, "might" be godaddy's new Ultimate 4GH shared hosting. It appears to be on a cloud.
[url=http]godaddy Ultimate 4GH[/url]

[viragotech]

one thing i noticed is that your main page has two images linked to external sites. If either of those sites is slow, it will slow you down while your forum waits to load the images. If possible, you should copy the images and host them locally.

only images not ours are the paypal donate buttons, less you see something I don't.

[Helter]

[img]http://xvnetwork.com/nreg.jpg[/img]

[viragotech]

[img]http://xvnetwork.com/nreg.jpg[/img]

Nope that's us too, the letter code for Virago is XV, its on the old shared server, where viragotech.com is still also.

That site isn't ever slow [since we moved for the forum off that account <img> ]

[Helter]

I put your forum up here
http://viragotech.freespace.me/index.php
and it seems to run fine. I did not import all your photos though because you have over 6000 of them!
I hope you have hotlink disabled for your site!

[viragotech]

I put your forum up here
http://viragotech.freespace.me/index.php
and it seems to run fine. I did not import all your photos though because you have over 6000 of them!
I hope you have hotlink disabled for your site!

is the search fixed???

That would be what got us kicked off shared, search was blowing the site up.
And since moving, we never got to rebuild the search as that search mod you had me installed would not work once moved.

90% of the errors or server gone away errors are search ones adding words

[viragotech]

I put your forum up here
http://viragotech.freespace.me/index.php
and it seems to run fine. I did not import all your photos though because you have over 6000 of them!
I hope you have hotlink disabled for your site!

Would the numger of images make a difference also?
Ill get as many people on the site as I can but is it a fair load test?

[viragotech]

ARG you didn't use a modded file system, I hate those pink screen crackertracker warnings ]

[viragotech]

Lots of newer members can't use the new site, my recently made mobilexv would not work also, you get a no user in IM table and just get white error pages.
[DB grew to over 400megs on the freespace.me site once the search index was done]



But lets ignore that, can you help get the search working on the VPS?

[Helter]

search is working fine. The only think that will kill your server is rebuilding the search table, but you should almost never have to do that. The only reason I can think of is if it gets corrupted.

Your other alternative would be to use an external search system like sphinx similar to phpbb.com.
I dont think your going to like phpBB3's search system because it ignores common words by default. What that means for you is if someone searched for a 1983 Virago, the search will ignore Virago since it is a common word in your db

[viragotech]

The search on VPS needs reindex, many many years of post are missing and adding new words causes errors.

And then once on VPS that search mod broke, I was unable to get it or the stock search rebuild to work on VPS.


we made an extensive ignore word list to cut it down. its a virago site, no need to search for it. Its on there with allot of common stuff. cause you seen how big the database grew once reindexed on that test you set up.

[viragotech]

most common error on post with the VPS

Critical Error!
message_die() was called multiple times.


Error #1
Could not insert new word

Line : 235
File : /home/vtfvpsx3/public_html/includes/functions_search.php
SQL : INSERT IGNORE INTO phpbb_search_wordlist (word_text, word_common) VALUES ('cam', 0), ('dirty', 0), ('filtered', 0), ('tube', 0), ('vent', 0)

[Helter]

I still think this is a server error. I got similar errors here at integramod using IM3 and it was due to exceeding the limits set on it in CPanel. It is possible that there is a db table size limit because im not getting any posting errors at freespace

[viragotech]

I still think this is a server error. I got similar errors here at integramod using IM3 and it was due to exceeding the limits set on it in CPanel. It is possible that there is a db table size limit because im not getting any posting errors at freespace

I don't know why it would not unless it cause I was able to rebuild the search on free space and or it was a error in the files, since you used all clean files??? [also why some CT errors]

And it is a VPS, so I got sever side 999 settings I have no clue about.
But don't see anything like that.

[Helter]

my server has 16 cores, each core is the equivalent of one CPU and rebuilding your search maxed two of them and still took 14 hours to complete. I dont think there are any shared, or VPS servers that will withstand that kind of punishment.
Your options are imo a dedicated server, or a true cloud server.
The problem with a cloud server is you pay for what you use. Your daily use would probably not be that expensive, but rebuilding the db or a phpBB3 update could cost you.

[viragotech]

my server has 16 cores, each core is the equivalent of one CPU and rebuilding your search maxed two of them and still took 14 hours to complete. I dont think there are any shared, or VPS servers that will withstand that kind of punishment.
Your options are imo a dedicated server, or a true cloud server.
The problem with a cloud server is you pay for what you use. Your daily use would probably not be that expensive, but rebuilding the db or a phpBB3 update could cost you.

were still talking 141 though right?

Cause our test of IM3 used no power, it was back on the shared account we got kicked off of. And the search rebuild took like 30sec and it found everything.

[viragotech]

you can go ahead and shut down the freespace site, don't wany my users getting spoiled.

I thought you tossed it up on something like we got as a compare test, of course it runs great on a server 16x more powerful <img>

[Helter]

i put it there to see if your search or forum was broken or just hitting your servers limits. Im doing an IM3 conversion now.

[viragotech]

i put it there to see if your search or forum was broken or just hitting your servers limits. Im doing an IM3 conversion now.

ok cool, another stress test then ehy to see how much power we really need?

[viragotech]

i put it there to see if your search or forum was broken or just hitting your servers limits. Im doing an IM3 conversion now.

I set you up with a tip jar on the main site for your efforts.
more then sure folks will stuff it as they can over the next few weeks.
[dind't make sense to give paypal a cut twice by having them give it to me then to you]

email is XXXX out

Code: Select all

<CENTER><FORM><b><INPUT><INPUT><INPUT><INPUT><INPUT><INPUT><INPUT><INPUT></b></FORM> <br><b> Tips from ViragoTech Members.</b>
</CENTER>

[Helter]

thank you <img>

I did the update and to convert your post table to phpBB3 took just over 4hours. To build the search index took about 9 hours, but it should never have to be done again unless it gets corrupted somehow

http://viragotech.freespace.me/forum/

[viragotech]

[quote=""HelterSkelter""]thank you

Funny is every so often someone make comment about that very old image I made many years ago still being up, and folks would try and make something new but I never like it.

Ok so what do we do, have everyone bombard the site, and evlauate the power use needed to see if we can go back to cheap shared or get a 2nd job to get our own server???


[ Got the banner up on the site now <img> ]

[viragotech]

Grrrrrrrrrrrrrrr

Got the image up on the real site, but many are having problems seeing the .png file.

Seems IE from 1 to the latest some times have problems seeing them.

But old as opera 6 has no problem.

Even with the IE on my machine it won't display, try to load the image itself alone and f'in quick time starts......

[Helter]

do a google for pngfix.js
there are many versions of it that give png support to IE5x and up.
If anyone is still running IE4, they should really think about upgrading because Windows95 is simply not a viable solution anymore <img>

[viragotech]

do a google for pngfix.js
there are many versions of it that give png support to IE5x and up.
If anyone is still running IE4, they should really think about upgrading because Windows95 is simply not a viable solution anymore <img>

Well I even made a screen cap to get a low quality jpg but still get the hover effect but it still wouldn't show up in IE.

so back to the real nice .png

told em to go grab old Opera 6.01 which it displays fine in if they got an older PC.

http://www.oldversion.com/download_Opera_6.01.html
[really nice if you got 2000 or older]

[Helter]

OPEN
forum root/templates/YOUR TEMPLATE/overall_header.tpl
FIND

Code: Select all

<Prillian>

AFTER, ADD

Code: Select all

<script></script>
<script></script>

UPLOAD the attached jquery.pngFix.js to your template folder. Your banner should now show to IE 5, 6 and 7 users.