Recent server issues

Invision Power Board is the other "big" commercial forum software out there, and I don't think it's much better or worse than vB with respect to efficiency and scalability.

I assume the low hanging fruit has been handled already, stuff like making sure XCache is installed and configured to run properly, analyzing the mysql instance and tuning the cache values so that most of the common queries are cached, making sure there's no memory bottleneck (RAM is comparatively cheap). You can use Yslow for Firebug to check the pages and make sure that the site is using the browser cache effectively, we made some Apache changes from that to improve the caching performance, reduced the raw # of hits to the server.

You could also look at using lighttpd (or there's another newer one that's supposed to be even better but the name eludes me) instead of Apache, much lighter and more efficient. We haven't done this one yet.

There also seems to be quite a few add-ons to vBulletin here, some research into what each one does and how much load it might put on the server might be worthwhile, in case there's one that's causing an inordinate amount of load.

One other thing we've done to reduce the load on our server is use a content delivery network for all the static resources. All the forum images, avatars, CSS and JS files can be fairly easily relocated to a content delivery network and the settings changed in the control panel. This reduced the # of hits to our server by 50% or more, it was a pretty huge difference. Won't help directly if the problem is MySQL, but will help indirectly by reducing the amount of resources the http server is using. SimpleCDN is the one we used, very easy to use if you use their mirroring functionality.
 
Hey!! Speak for yourself. We the inmates have taken over the asylum. :D
 
Invision Power Board is the other "big" commercial forum software out there, and I don't think it's much better or worse than vB with respect to efficiency and scalability.

I assume the low hanging fruit has been handled already, stuff like making sure XCache is installed and configured to run properly, analyzing the mysql instance and tuning the cache values so that most of the common queries are cached, making sure there's no memory bottleneck (RAM is comparatively cheap). You can use Yslow for Firebug to check the pages and make sure that the site is using the browser cache effectively, we made some Apache changes from that to improve the caching performance, reduced the raw # of hits to the server.

You could also look at using lighttpd (or there's another newer one that's supposed to be even better but the name eludes me) instead of Apache, much lighter and more efficient. We haven't done this one yet.

There also seems to be quite a few add-ons to vBulletin here, some research into what each one does and how much load it might put on the server might be worthwhile, in case there's one that's causing an inordinate amount of load.

One other thing we've done to reduce the load on our server is use a content delivery network for all the static resources. All the forum images, avatars, CSS and JS files can be fairly easily relocated to a content delivery network and the settings changed in the control panel. This reduced the # of hits to our server by 50% or more, it was a pretty huge difference. Won't help directly if the problem is MySQL, but will help indirectly by reducing the amount of resources the http server is using. SimpleCDN is the one we used, very easy to use if you use their mirroring functionality.
I think we've just had a volunteer to provide technical assistance. :D Someone give this guy admin powers, quick!
 
I don't understand why the post count of an individual poster being high would affect anything, but if dropping mine to zero and keeping it there would help, go for it.
If that actually means deleting every post I ever made, let me save the valuable ones........ok, done.. Pull the control rods.
 
Lol, I'm just regurgitating a lot of what our server admin guy and I researched out and decided was good, so I had help. I'm not a linux guru by any means (which I hate, but never seem to have time to remedy).

Though I did compile and install XCache from source, so I was impressed with myself.
 
I don't understand why the post count of an individual poster being high would affect anything, but if dropping mine to zero and keeping it there would help, go for it.
If that actually means deleting every post I ever made, let me save the valuable ones........ok, done.. Pull the control rods.

The actual post count # that's displayed is just a field in the database and won't affect anything.

The number of records can have an impact though, I haven't researched it but I assume there's ways to archive older posts out to improve things; my forum has 1/3 the # of posts here so I haven't had to worry about that yet.

One other thing I forgot to mention is gzip compression; make sure it's enabled somewhere (either in the forum itself, or through apache with mod_deflate), but make sure it isn't enabled in BOTH places! :D I had that so every page the forum generated was being compressed twice, which of course uses up resources needlessly. :)
 
The only thing worse than being a paid IT with a knowledgeable boss must be being an volunteer IT with about 60 knowledgeable members.

Why don't y'all quit jostling his elbows? If he had to read all your stuff he'd never be able to find the time to type his own SQL.

<duck and cover>
 
Lol very true, I was just offering up the things we've done that have worked for us, I'm sure Terry's looked at 90% of what we went through already, thought he content delivery network thing is not very common for vBulletin forums yet so it's not a well known option.
 
I don't understand why the post count of an individual poster being high would affect anything,

It doesn't affect anything until someone hits "See more posts by Soapy_Sam", at which point a very long-running query is generated which examines every post you've ever made, sorts them by date, and shows the most recent 400.

but if dropping mine to zero and keeping it there would help, go for it.
If that actually means deleting every post I ever made, let me save the valuable ones........ok, done.. Pull the control rods.

I'd rather not start throwing away data. If that does happen, it'll be because JREF has decided it is the appropriate course, and most likely that will be against my recommendation.
 
One other thing I forgot to mention is gzip compression; make sure it's enabled somewhere (either in the forum itself, or through apache with mod_deflate), but make sure it isn't enabled in BOTH places! :D I had that so every page the forum generated was being compressed twice, which of course uses up resources needlessly. :)

I suspect that for now it should be turned off completely,unless bandwifth is really restricted.

And additionaly ( ;) ) I suppose that sooner or latery it is going to be either hardware or rewrite bad SQL queries.(But with proper documentation of changes it should not be difficult to upgrade later)
 
I'm just shooting ideas, but if there's a problem with users having many posts, would it be effective to split up really old posts to an archive sock puppet? So if I have 5000 posts older than a year, what about some task makes a sock puppet as close as possible to me and cuts&pastes those posts to the sock puppet (which would be Mitchell314_ or something)? And if I get 5000 more old posts, push them on the old sock puppet? Probably not too practical but I'm just throwing stuff out there. Beats banning the old members. Then again...
 
I suspect that for now it should be turned off completely,unless bandwifth is really restricted.

I guess it depends on what is limited, it's not just bandwidth, it takes apache less time to transfer the gzipped resource, quite a bit less time for raw HTML pages. Typically it's worth the extra CPU to compress them.
 
It doesn't affect anything until someone hits "See more posts by Soapy_Sam", at which point a very long-running query is generated which examines every post you've ever made, sorts them by date, and shows the most recent 400.
Would it help to simply disable that feature?
 
Would it help to simply disable that feature?
I find this feature useful for researching particular posts that I remember seeing, but not which thread they are in. I'm in two minds - in one sense I would not want to see it disabled, rather we should all self-regulate and not use it to find posts by people with a large number of posts. But I'm not naive enough to beleive that this could possibly work.
 
vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with huge threads or members with huge numbers of posts. The other thing is the server needs more RAM (and ideally to be a 64-bit kernel so that a single process can grab a larger amount of RAM too). Almost invariably when the "no DB connections left" messages start, it's because we're thrashing swap. I've tried various things to tune the db to use VM better, but once one of those giant "runs for several hundred seconds" queries kicks in, we usually end up dropping people.

ETA: if we didn't drop people, it probably wouldn't recover on its own, and it would be hung until rebooted (or until someone restarted Apache and/or the database, at least).
Yeah, it's what I call the LAMP death spiral.

Your server is trundling along more or less happily, then you get a query that takes longer than usual for whatever reason. Other queries back up behind it, which means that the Apache/PHP processes involved are sitting around taking up memory while they wait. As more requests come in, Apache starts up more processes to handle them, which takes up memory that would otherwise be used to cache the data from the database.

That means that MySQL has to hit the disk to fetch its data instead of just grabbing it from memory, which means the queries run even slower, and in a matter of minutes you can go from everything running fine to the server being so overloaded that you can't even log in to fix things.

The solution is exactly what you are proposing to do: Throw hardware at it.

64-bit is nice but not critical; the Linux filesystem cache can grow to 64GB (I think) even on a 32-bit system, and while it's not quite as efficient as MySQL's own buffers (assuming you're using InnoDB), it gets the job done.

There's one other silver bullet you can use, and that's a Fusion-io ioDrive. They're not cheap (start at around $3000) but they're about 1000 times faster than a regular disk drive. I'm working on a system that handles about 1.5 million posts a day, and we do all sorts of crazy indexing (full text indexes, geospatial indexing) and one ioDrive handles everything we throw at it without blinking.
 
I googled these last time you mentioned them. Very nice bit of kit. Presumably the price of these will fall in the months to come?
 
Actually, the price has gone up. They're selling them as fast as they can churn them out, and the closest competition is a year away from shipping, so for the moment they have the market all to themselves.

Worth every penny, though.
 

ISF - Join now!

Every member here is approved by hand. No bots, no spam, just people who care about evidence and honest debate.

Membership is free!

Create your free account

Back
Top Bottom