• Due to ongoing issues caused by Search, it has been temporarily disabled
  • Please excuse the mess, we're moving the furniture and restructuring the forum categories

Recent server issues

The Bad Astronomer

Muse
The Bad Astronomer
Joined
Feb 19, 2002
Messages
828
Folks-

I know that some of you are unhappy with the problems we've been having with the forum/website server. Jeff has been working closely with our webhost to isolate and solve the problem... which is, simply put, too many people accessing the site! Yes, we're a victim of our own success. We've been trying different incremental fixes to relieve the pressure, and we still have some ideas up our sleeves.

Let me assure you, we've been taking this issue very seriously. I live my life online, and I am personally aware of how important this forum is to the JREF. I read the forum as often as I can, though of course my time is limited (helping plan TAM 7 and TAM London turns out to be somewhat time-intensive). Jeff, of course, is also very busy trying to get this thing solved, so feel free to ask questions, but I'm asking that you please be patient with us.

I assure you that getting these problems solved is a top priority. At some point eventually we may need help from you, our community, but we'll work that out when the time comes.

For now, please be aware -- and this is coming straight from me, a guy who owes his career to the internet and the people who use it -- this forum and this community are highly prized by the JREF, and the problems will be fixed as quickly as we can make it happen.
 
Last edited:
Can I humbly suggest significantly over-engineering your next solution point. Plan for, say, twice or three times as much traffic per unit time and concurrent logins and visitors, etc.

Otherwise you will be right back here again very soon, doing this all again.

I hear Deep Blue is currently inactive. Make a bid! :)
 
Phil -- sorry I didn't see this thread before I responded on the other thread!!

Apologies, Miss Kitt
 
Just added the slow forum tag to this thread and another thread. If you look at the threads you will see
- Every few months a new thread has been created to discuss the issue.
- The problem started about the start of 2008 (13 months ago)
- If you read the threads Darat and Terry have spent a lot of time on the issue.

Other notes.
The forum has not grown at all over this period. This is partly due to the CT sub forum shrinking (maybe due to its success?). Shrinking means the number of threads created per month has gone down since the start of 2008. I suggest the lack of growth of this forum is also due to the topic at hand.

How much and how fast this forum will grow when the performance issues have been fixed can only be guessed at. I am eagerly looking forward to that date.
 
Thanks for the update Phil. It's good to know the organisation values the forums highly. What (else) can we do to help?
 
The Bad Astronomer said:
Overengineering -- what web hosts call "room to grow" -- is in the plan.
Click to expand...
From an old computer scientist: ...why stop there?! :)

Seriously, I could arrange for a relatively monster server with Kenworth powered CPUs, mountains of RAM and heaps-o-disk to be made available solely for JREF, for possibly little or no money at all. But if the software ain't efficient and rugged, and the ISP network connection flows slower than treacle at the South Pole, there's no point. The forum will still seem like it is on a 486DX16 with 16MB RAM, Windows ME and terminal asthma.

As has been asked previously, Terry, what's the real problem? And is there anything we can possibly assist with?
 
Zep said:
From an old computer scientist: ...why stop there?! :)

Seriously, I could arrange for a relatively monster server with Kenworth powered CPUs, mountains of RAM and heaps-o-disk to be made available solely for JREF, for possibly little or no money at all. But if the software ain't efficient and rugged, and the ISP network connection flows slower than treacle at the South Pole, there's no point. The forum will still seem like it is on a 486DX16 with 16MB RAM, Windows ME and terminal asthma.

As has been asked previously, Terry, what's the real problem? And is there anything we can possibly assist with?
Click to expand...
...please?
 
As chillzero pointed out in the other thread, if you want Terry to see a question it might be best to PM him to point him here!

Re: the server size issue, it's clear to see what the requirements are if you look at the size of this forum relative to the rest of the internet.

From a forum ranking website (by metric), this forum is the 362nd largest forum on the internet. 362 out of hundreds of thousands. That's a big forum.

Compare this to the Bad Astronomy forum at number 1264, the Richard Dawkins or Skeptic Society forums which don't even rank (ETA the latter two may be because they haven't been submitted or crawled but if you visit them you can see the member and post counts for yourselves), and you can see just how huge it actually is.

Heck, this forum is bigger than the Penny Arcade forum. It's bigger than Mozillazine. It's just below the Doctor Who forum. It's massive.

http://rankings.big-boards.com/?p=all

Some of the biggies carry proper banner and skyscraper advertising, but not all do. I wonder, therefore, if it would be worth contacting the admins of some of those forums to ask then what their server spec is or what their costs are. They're meeting those costs somehow, so their knowledge might be useful.
 
Last edited:
arthwollipot said:
How the heck do you find out statistics like that???
Click to expand...

I gave a link in my post :D

(the answer is, I believe they pull data from each forum's own measuring tools. There's an FAQ but it's not hugely detailed).
 
Last edited:
I do not think that the total number of posts ever made is the best measure of how big a board is. For example we would be much bigger if we had not lost heaps of posts in the past. Forums grow or shrink over time.

A better measure is how many posts were made in a week. This is the link from the same site. Here is the link for that from the same site http://rankings.big-boards.com/?sort=week&filter=all,all&p=12. It shows we are 235 out of 1176 listed with 20,883 posts last week (subject to change over time). Compare that with the 14th biggest with 211,332 posts last week. 10 times bigger. In short we got potential to grow. On the other hand there are not too many other forums with a large general science sub forum or related subjects that are bigger than us.

I have been keeping stats of this over time.
 
rjh01 said:
I do not think that the total number of posts ever made is the best measure of how big a board is.
Click to expand...

Well, it depends. It isn't an unreasonable thing to look at if you're concerned about database performance.
 
Zep said:
As has been asked previously, Terry, what's the real problem? And is there anything we can possibly assist with?
Click to expand...

vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with huge threads or members with huge numbers of posts. The other thing is the server needs more RAM (and ideally to be a 64-bit kernel so that a single process can grab a larger amount of RAM too). Almost invariably when the "no DB connections left" messages start, it's because we're thrashing swap. I've tried various things to tune the db to use VM better, but once one of those giant "runs for several hundred seconds" queries kicks in, we usually end up dropping people.

ETA: if we didn't drop people, it probably wouldn't recover on its own, and it would be hung until rebooted (or until someone restarted Apache and/or the database, at least).
 
You could try shoving a weiner in the warp drive. That sometimes helps...
 
362. biggest forum on the internet, sounds like it needs good hardware and good connections.
Might explain why it is not always avaible when I want to log on.

The forum does a lot of good against woo, so it would be worth expending the hardware on. Probably best value for the money.
 
Terry said:
vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with huge threads or members with huge numbers of posts. The other thing is the server needs more RAM (and ideally to be a 64-bit kernel so that a single process can grab a larger amount of RAM too). Almost invariably when the "no DB connections left" messages start, it's because we're thrashing swap. I've tried various things to tune the db to use VM better, but once one of those giant "runs for several hundred seconds" queries kicks in, we usually end up dropping people.

ETA: if we didn't drop people, it probably wouldn't recover on its own, and it would be hung until rebooted (or until someone restarted Apache and/or the database, at least).
Click to expand...

Does vBulletin have support for sever-side caching?Or at least Apache?This could decrease number of connection,but not sure how efficient with vB or Joomla it is/would be.

Or just new hardware,where new server would be dedicated for database and old one for website(PHP).

P.S:And I am ,I suspect, Mr.Obvious (Klimax)...
 
Terry said:
ETA: if we didn't drop people, it probably wouldn't recover on its own, and it would be hung until rebooted (or until someone restarted Apache and/or the database, at least).
Click to expand...
Speaking of dropping people, why not cull old unused user accounts from the database?

Is there a query that you can use to determine the last time an account was used? I see by looking through the members list that there are a large number of accounts with 0 posts. If a script could be formulated such that {IF post count = 0 AND time last logged on > 365 days ago THEN delete the account} that would free up a modicum of space, wouldn't it?
 
arthwollipot said:
Speaking of dropping people, why not cull old unused user accounts from the database?

Is there a query that you can use to determine the last time an account was used? I see by looking through the members list that there are a large number of accounts with 0 posts. If a script could be formulated such that {IF post count = 0 AND time last logged on > 365 days ago THEN delete the account} that would free up a modicum of space, wouldn't it?
Click to expand...

I don't see how that would make a significant difference to the issues we're seeing.
 
From a forum ranking website (by metric), this forum is the 362nd largest forum on the internet. 362 out of hundreds of thousands. That's a big forum.
Click to expand...

Before we start the self-congratulations, we are well below the Delta Goodrem forum.:(
 
Terry said:
vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with huge threads or members with huge numbers of posts. The other thing is the server needs more RAM (and ideally to be a 64-bit kernel so that a single process can grab a larger amount of RAM too). Almost invariably when the "no DB connections left" messages start, it's because we're thrashing swap. I've tried various things to tune the db to use VM better, but once one of those giant "runs for several hundred seconds" queries kicks in, we usually end up dropping people.

ETA: if we didn't drop people, it probably wouldn't recover on its own, and it would be hung until rebooted (or until someone restarted Apache and/or the database, at least).
Click to expand...
Thanks, Terry. That does explain a fair bit. I appreciate your situation, and you have my full support!

Clearly the root cause is the vBulletin software. So just throwing more hardware at it (i.e. the Microsoft option) is unlikely to make a huge dent in it. That could be money and effort wasted just now.

The grizzled old coder here says that you will get far more bang for your buck by tracking down and "efficientising" the noticeable software roadblocks you have already identified. We have been this path before, hence the limited Search results, etc.

It seems clear that long topics and high post-count users are RAM hogs. That suggests vBulletin is making routine calls on MySQL that return giant tables in memory, possibly containing non-normalised data. (This would also show as high disk IO counts - disk thrashing.) My first "programmer analyst" question would be: Why are all these records back to the year dot needed every time in routine queries? Surely they would be expected to consume huge amounts of memory per instance, and yet it is highly unlikely all the records will ever be required for subsequent processing. Sounds like the vBulletin designers never allowed that these numbers could really get so high.

So I would look at reducing that memory demand in software before trying to accommodate it in hardware. If that can be done, you reduce RAM consumption per user and lower overall resource demand (i.e. RAM and disk IO) on your existing unmodified platform, allowing you to defer heavy-duty upgrades until things are calmer and planned. And paid for. :)

For example, would the most recent 1000 records be sufficient under most common circumstances? Or could a much smaller recordsize be requested, to be used as an index into one or two selected bigger records as required? (SQL programmers can stop laughing now! You suggest better code solutions, OK?) More to the point, can you re-code the guts of vBulletin at all anyway?

Good luck!
 
Terry said:
vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with ... members with huge numbers of posts. <snip>
Click to expand...

Ever thought about banning members with too many posts? But give them permission to create a sock puppet.

Or if our forum has too many posts then remove all old threads. Give a link to the archive on the main menu. The archive would have these deleted threads.
 
Well,what about to start locking threads at 5000 range?There are some active.
Or even lower loke from 2000 or so...
 
I clicked on this thread and got a database error. How Ironic. =/
 
Zep said:
Thanks, Terry. That does explain a fair bit. I appreciate your situation, and you have my full support!

Clearly the root cause is the vBulletin software. So just throwing more hardware at it (i.e. the Microsoft option) is unlikely to make a huge dent in it. That could be money and effort wasted just now.
Click to expand...

Not sure I agree that throwing more hardware at the situation is unwarranted.

Zep said:
The grizzled old coder here says that you will get far more bang for your buck by tracking down and "efficientising" the noticeable software roadblocks you have already identified. We have been this path before, hence the limited Search results, etc.
Click to expand...

Of course, that's optimization 101. But this is effectively shrinkwrap software. We are in a world of hurt if we start changing vBulletin, and I for one am not up for supporting a fork of a licensed commercial product just because we happen to be able to read the PHP.

Zep said:
It seems clear that long topics and high post-count users are RAM hogs. That suggests vBulletin is making routine calls on MySQL that return giant tables in memory, possibly containing non-normalised data. (This would also show as high disk IO counts - disk thrashing.) My first "programmer analyst" question would be: Why are all these records back to the year dot needed every time in routine queries?
Click to expand...

They aren't, but you have to look at all the posts in a thread to generate the thread display. Or at least look at an index containing all the posts.
 
Klimax said:
Well,what about to start locking threads at 5000 range?There are some active.
Or even lower loke from 2000 or so...
Click to expand...

There are 8 threads that are open with more than 5k posts. I'll ask for them to be... TERMINATED!
 
I help run a vB forum that isn't quite as big as this one (not as many posts), but has about the same number of active users and will range from 350 to over 700 active users at any given time.

If you want you can send me a PM and I can share a few things we've done to help tune things up and get the max performance out of our server.. we're running on a single box with 2 Xeon processors, 2GB of RAM and mirrored SCSI drives and a throttled connection, and we rarely get slowdowns so I think we've done a good job of maximizing our resources without going to extremes.

Though eventually you get to the point that you need more than one physical server, but without knowing more details it's hard to be more helpful. Have you determined what the bottleneck is, sounds like it's mysql?

Anyway, if you are interested Terry shoot me a PM, or I can share in-thread if you don't mind the technical stuff in here.

In response to Zep, it's not quite as bad as all that, throwing more hardware if done properly can make a huge difference, there are some truly massive forums, I am a moderator on one (though I'm not really active anymore) that has 4-5000 active members normally, and lots more during really active times, so the software does scale, it's just a harsh reality of having hundreds of thousands of people hitting refresh every 30 seconds on a server where the pages are by necessity dynamically generated.

You are right though about identifying the bottlenecks, in my optimization quests I talked to a few vB users who had done just that; they'd basically gone through and rewritten some of the SQL queries to be more efficient and a few of the pieces of the software. That's tough though, not everyone has that kind of time (let alone skill, writing great queries isn't easy) to dedicate to their forum.
 
El_Spectre said:
Thats crazy talk! Tachyons! we need more tachyons!
Click to expand...

Captain! I've giving you every ounce of power this tub can muster. The engines will blow at any minute! We need more Berilium crystals captain Kirk!
 
We could just get rid of all the crazy people...that would free up a lot of bandwidth. Although I think there would only be like..5 people left. :)
 
You must log in or register to reply here.
Back
Top Bottom