Future of the Forum

[This is The End]

While in principle I agree, the reality is that the forum is broken. There's severe performance issues that need addressing.

You may know something I do not, but the forum seems to work great when it is not having lag issues. So if those "severe performance issues" really aren't just improper hardware or software setup, then I would agree with you.

As for mobile users, that is what we have tapatalk for. As for security updates, are you sure those are even being added to this forum now, let alone when it stops in three years?

If you do take on the task eventually I wish you the best of luck.

Also, thanks for all of the server statistics in post #144!!

 

[temporalillusion]

Fortunately...

In addition to this, there is a custom script for importing big boards onto Xenforo. The built in script works (I've run it a few times as tests), but it by no means fast.

There is a custom import script that might be worth considering: https://marketplace.digitalpoint.com/vbulletin-big-board-importer-vbulletin-3-vbulletin-4.1535/item

Or there are even people who specialize in such things: http://xenforo.com/community/thread...tin-big-board-1m-posts-import-services.55328/

If donations are part of the equation allocating some to something like that might be worth doing, when that time comes anyway.

 

[temporalillusion]

Here's the current basic server configuration. It's a dedicated box installed 5.5 years ago -

Bare Metal Server Installed 2/19/2009 in Seattle @ Softlayer
CentOS 2.6.18-92.e15
SuperMicro X7QCE Intel Xeon HexCore QuadProc Sata [4Proc]
4x2GB Generic RAM
4x2.13GHz Intel Xeon-Tigerton (7320-Quadcore)
SuperMicro AOC-SIMSO-plus Remote Management Card
2xSuperMicro PWS-1K01-1R Power Supply
SuperMicro BPN-SAS-828TQ Backplane
Adaptec 3405 Drive Controller
Western Digital Raptor 10,000 RPM WD1500ADFD (sdb) for Database
Seagate Cheetah ST373455SS [73GB] (sda) for system

etc, etc

Huh, that's not nearly as bad as I'd thought, it's not a VPS anyway. RAM isn't too bad, though how it's allocated might be an issue (i.e. if MySQL is only getting 2GB). How much is given to Apache might be part of it as well. CPU could be part of it, and disk might be limiting the database as well. My forum runs off an NFS share for the database drives but they are pretty high I/O. An SSD would of course be huge.

 

[temporalillusion]

Any preference on a crowd-funding platform to support this?

I mentioned this already to you but I thought I'd bring it up here as well as a more general discussion.

There's two ways to do it, one is to purchase servers and colocate them, the other is to just rent servers. Renting costs more in the long run in general, but you have the not insignificant benefits of being supported by someone else, hardware repairs being someone else's problem, warranty being someone else's problem, and what to do if moving to updated hardware being someone else's problem.. well you get the idea Paying monthly for servers is just so much simpler IMO.

Purchasing servers costs more up front, then has lower cost for a while (assuming the colocation is cheaper than renting the server), then increases again when things start to break or require purchasing more warranty or paying hands and eyes at the colo to do repairs.

I prefer renting, but that kind of goes against the donation style funding.

Not saying donations shouldn't be done, or part of it, or be the whole funding method, just raising the point for consideration.

 

[temporalillusion]

Ouch. That takes it out of my range of experience.

I also saw mention of this free backup tool:
http://www.percona.com/software/percona-xtrabackup
.. which looks pretty good.

Just a comment on this, while I LOVE Percona and have used their products in the past, this won't work for the current forum without alterations, as vBulletin 3.x uses MyISAM table types (and that's for InnoDB or XtraDB).

That said I have seen some vBulletin 3.x forums alter their table types to InnoDB, so that might be an option if that was done.

 

[dasmiller]

Huh, that's not nearly as bad as I'd thought, it's not a VPS anyway. RAM isn't too bad, though how it's allocated might be an issue (i.e. if MySQL is only getting 2GB). How much is given to Apache might be part of it as well. CPU could be part of it, and disk might be limiting the database as well. My forum runs off an NFS share for the database drives but they are pretty high I/O. An SSD would of course be huge.

Do we know what else is running on the hardware? I've seen antivirus programs that would use >90% of the system resources for half an hour or more, at random intervals (and that was considered a feature!), rendering an otherwise fast machine completely useless for even things like word processing.

Could we have some rogue mundane process occasionally using all the CPU, or allocating so much RAM that the Forum can't get any and has to resort to disk thrashing?

 

[temporalillusion]

One last post, I missed a bunch of posts so I apologize for all the replies in order.

So some general my thoughts about the various areas:

Ownership: Be it buying servers via donations, or holding donations to pay for monthly services, the question of who owns what and how they're accountable and such is something to consider. No one wants a situation where someone gets mad and takes their servers and goes home. Not a pleasant thing to think about something that warrants some thought.

Hardware Source: As mentioned you can rent servers or buy and colocate them, so which path will need to be decided.

Reliability: In general (and users will of course disagree) I don't think the forum is a mission critical service requiring five nines of availability. So spending a lot of money on redundancy probably isn't necessary; depending on the service provider's or vendor's warranty and reasonable levels of backups is probably fine. A day of downtime while a server's power supply is replaced or the database drive is replaced and restored from backup is acceptable for the forum IMO.

Hardware: I'd suggest two servers, one for the database and one for the web server. An SSD for the database is a good idea, spindle drive for OS and database logs and SSD for database files. The web server doesn't require as much in the way of disk I/O, so two mirrored drives is sufficient IMO.

OS: CentOS is fine to use, lots of support and info out there. I think CentOS 7 is even available now.

Web Server: Apache works well and I'm not sure nginx is any faster for PHP performance. Static file performance is higher, but I think we should be offloading static resources to a CDN anyway, so Apache is probably fine, but nginx might be fun to use anyway (haven't played with it personally).

Database Server: As mentioned previously, vBulletin 3.x uses MyISAM table types, in which case I'd suggest MariaDB as the database server. It's a drop-in replacement for MySQL but has other things that MySQL either doesn't have, is coming, or is exclusive to the enterprise version. If we converted the table types to InnoDB then we could use Percona which is also a drop in replacement for MySQL. I prefer Percona but maybe that's just because I've been reading the mysqlperformance blog for a long time.

CDN: As mentioned using a CDN like MaxCDN can reduce the load on the HTTP server a lot (most of the requests in a vBulletin page are for static resources). Then the HTTP server can focus on serving up PHP pages. It's relatively cheap and there are a few ways to set it up, someone really skilled with rewrite rules can do it completely via rewrite rules. Personally I just changed the file locations in the vBulletin software itself; seemed easier to me.

Images: Related to that is images in general. These forums allow for images to be uploaded, and icerat mentioned that the forum DB is 32GB which seems quite high to me (my forum is 4.2GB for 5 million posts). If images are stored in the DB then yes that's a significant performance issue, and a storage issue too. In my opinion there are a lot of different free image hosts out there and this is a forum, not an image server. On my forum we simply don't allow image uploads other than avatars, if someone wants to show an image they can upload it to imgur.com themselves. But at bare minimum images should be on the filesystem not in the database, and the CDN could also be used to offload the request for images from the server.

Backups: Technical issues on how to backup the database aside, something like Crashplan ($10/mo business) is inexpensive IMO and gets you offsite backups as well (though Crashplan for Linux is a more complex setup). Some level of backups might also be included with whatever server or colocation plan is obtained.

All I can think of at the moment.

 

[temporalillusion]

Do we know what else is running on the hardware? I've seen antivirus programs that would use >90% of the system resources for half an hour or more, at random intervals (and that was considered a feature!), rendering an otherwise fast machine completely useless for even things like word processing.

Could we have some rogue mundane process occasionally using all the CPU, or allocating so much RAM that the Forum can't get any and has to resort to disk thrashing?

I don't think anyone has any access to the server as of yet, the stuff icerat posted was forwarded to him if I understand correctly.

So yeah could be something else on the server, heck could be a hacker running a botnet or mining bitcoins!

I doubt the server's been compromised but until one looks one never knows.

 

[Klimax]

I don't think anyone has any access to the server as of yet, the stuff icerat posted was forwarded to him if I understand correctly.

So yeah could be something else on the server, heck could be a hacker running a botnet or mining bitcoins!

I doubt the server's been compromised but until one looks one never knows.

And it wouldn't be first time... (A tech was caught running bitcoin mining on servers, when they ran too slow.)

 

[arthwollipot]

I'd just like to say something. A lot of people are posting in this thread, contributing ideas and thoughts. Sharon has asked for people who might be able to help, and that's all very well, but we don't want a huge number of people on the committee unless we want to recreate the forum in the form of a camel.

It seems that everyone feels entitled to input on the discussion. A project like this cannot work that way. A small lean team will be able to get the project completed faster and with fewer complications. This does of course mean that some people are going to feel ignored. I think we as a community need to get over that and provide only the help that is asked for, when it is asked for.

Let's let those who are working work, and not expect them to take every possible opinion into account.

 

[icerat]

I'd just like to say something. A lot of people are posting in this thread, contributing ideas and thoughts. Sharon has asked for people who might be able to help, and that's all very well, but we don't want a huge number of people on the committee unless we want to recreate the forum in the form of a camel.

It seems that everyone feels entitled to input on the discussion. A project like this cannot work that way. A small lean team will be able to get the project completed faster and with fewer complications. This does of course mean that some people are going to feel ignored. I think we as a community need to get over that and provide only the help that is asked for, when it is asked for.

Let's let those who are working work, and not expect them to take every possible opinion into account.

Personally I'm quite happy for people to give whatever their thoughts or ideas are and discuss them here.

Techinical issues are but one area though. From what I understand the general feeling withing JREF is that they would like the forum to be wholly independent of JREF. I don't specifically know why but given the number of lawsuits and threats that have been flying around the skeptic community on various issues of late, I can understand why there might be motivation. Options for doing this while still remaining affiliated (and keeping the name) are being considered.

The question arises as to the organisational form of any future body operating the forum. A new non-profit? Part of an existing non-profit or for-profit entity? Couple of dudes in their mother's basement? None of the above?

Just FYI, the current advertising income does not come close to covering the costs of the forum. While an upgrade to better equipment will, paradoxically, probably result in lower costs, it will likely still not be enough so ongoing funding solutions will need to be found.

Throw in your 2 cents or relevant currency.

 

[zooterkin]

I'd rather wait and see what the JREF's position regarding the forum is, before making plans to go solo. As I understand it, TAM would not have happened without the forum. Certainly I would not have heard of TAM otherwise, nor attended half a dozen. The forum provides the volunteers and TAMbassadors that help run TAM, as well as providing scholarships for those unable to afford the cost. Of course, some of could still happen without the forum, but I think participation would be lower.

 

[Spindrift]

I'd just like to say something. A lot of people are posting in this thread, contributing ideas and thoughts. Sharon has asked for people who might be able to help, and that's all very well, but we don't want a huge number of people on the committee unless we want to recreate the forum in the form of a camel.

It seems that everyone feels entitled to input on the discussion. A project like this cannot work that way. A small lean team will be able to get the project completed faster and with fewer complications. This does of course mean that some people are going to feel ignored. I think we as a community need to get over that and provide only the help that is asked for, when it is asked for.

Let's let those who are working work, and not expect them to take every possible opinion into account.

Should we ask everyone first if they think everyone should be involved?

Why do you hate camels? We are traversing the desert of woo on the camel of skepticism.

 

[icerat]

Mr. Adams, on the JREF board, is able to get in and do some technical tweaks to the hardware and software.

That would be this Mr Adams for those interested. So as you can imagine he has the technical nous but perhaps has other interests he prioritises over this forum

One insight he has given me is that right now 40% of the queries are from search engine bots, a good proportion of the traffic is from China. Legitimate traffic?

Quite probably hack attempts. Our servers are regularly attacked by chinese IPs attempting to gain access through various means.

40% of queries from search engine bots sounds a bit high (around 30% is more typical) but certainly within normal range.

 

[Klimax]

Right now, Sucuri proxy is causing problems.
http://www.internationalskeptics.com/forums/showpost.php?p=10149231&postcount=11

 

[BenBurch]

If systems help is needed, consider me a volunteer.

 

[Soapy Sam]

Personally I'm quite happy for people to give whatever their thoughts or ideas are and discuss them here.

SNIP

Just FYI, the current advertising income does not come close to covering the costs of the forum. While an upgrade to better equipment will, paradoxically, probably result in lower costs, it will likely still not be enough so ongoing funding solutions will need to be found.

Throw in your 2 cents or relevant currency.

icerat- can you put some numbers on this? (Approximate).
Current
Cost of hardware:-
Cost of software:-
Cost of hosting:-
Cost to maintain:-

 

[icerat]

icerat- can you put some numbers on this? (Approximate).
Current
Cost of hardware:-
Cost of software:-
Cost of hosting:-
Cost to maintain:-

From my current understanding of the system (and I don't have all the facts) I think we could get a way more than good enough dedicated server or VPS for the $250 to $300/mth range.

We'd then want to add some firewalling and backup, maybe another $200.

So we're talking max $500/mth for a better set up than now for hardware/hosting at less cost (which I believe is $500/mth+ not counting firewalling/backup?)

The current vbulletin licence is privately owned, no ongoing costs, but I tend toward upgrading to Xenforo ($250 incl enhanced search) plus a yearly fee for ongoing upgrades/support ($65).

While there's quite a lot of expertise available here I think it's worth paying for some migration software and an experienced tech to handle it. Maybe $500. (ironically some of that is licencing fees to Brian Dunning's partner in crime Shawn Hogan, but that's another story)

So I'd say we're looking at say $1500 for setup and first month, then max $5-600/mth ongoing.

Maintenance is currently all volunteer. Experience to date suggests that might need to change. Personally I'm in favor of pro-moderators/admins on a site this well trafficked. I don't have enough info on the workload to judge costs at this stage.

Right now there is very limited advertising on the site that brings in little revenue. A mix of appropriate advertising plus member donations/subscriptions should be enough to support the site.

My preference - and others I've spoken to seem to feel the same - is that a new non-profit spun off from JREF is probably the way to go. (any legal/accounting folk with time/expertise in setting up and maintaining a non-profit, put your hand up)

I'm also getting the distinct impression there's a lot of backroom history we're dealing with here which has lead to the current somewhat critical situation. That may also make this entire discussion moot and something entirely different happens.

Left hand, meet right hand, pleased to meet you.

 

[Spindrift]

From my current understanding of the system (and I don't have all the facts) I think we could get a way more than good enough dedicated server or VPS for the $250 to $300/mth range.

We'd then want to add some firewalling and backup, maybe another $200.

So we're talking max $500/mth for a better set up than now for hardware/hosting at less cost (which I believe is $500/mth+ not counting firewalling/backup?)

The current vbulletin licence is privately owned, no ongoing costs, but I tend toward upgrading to Xenforo ($250 incl enhanced search) plus a yearly fee for ongoing upgrades/support ($65).

While there's quite a lot of expertise available here I think it's worth paying for some migration software and an experienced tech to handle it. Maybe $500. (ironically some of that is licencing fees to Brian Dunning's partner in crime Shawn Hogan, but that's another story)

So I'd say we're looking at say $1500 for setup and first month, then max $5-600/mth ongoing.

Maintenance is currently all volunteer. Experience to date suggests that might need to change. Personally I'm in favor of pro-moderators/admins on a site this well trafficked. I don't have enough info on the workload to judge costs at this stage.

Right now there is very limited advertising on the site that brings in little revenue. A mix of appropriate advertising plus member donations/subscriptions should be enough to support the site.

My preference - and others I've spoken to seem to feel the same - is that a new non-profit spun off from JREF is probably the way to go. (any legal/accounting folk with time/expertise in setting up and maintaining a non-profit, put your hand up)

I'm also getting the distinct impression there's a lot of backroom history we're dealing with here which has lead to the current somewhat critical situation. That may also make this entire discussion moot and something entirely different happens.

Left hand, meet right hand, pleased to meet you.

Somewhat rambling thoughts:

I've worked with a lot of non-profit organizations and have been president of several. While $1500 startup and $6000-$7000 a year does not sound like a lot, it can be to a huge amount to non-profit.

I agree with your point about a pro-admin/mod being needed. Volunteers are great, but their priorities can shift so needed activities don't happen when needed, they get done when they can be done. So you're talking more money. I don't know how much time would be needed for a pro-admin and therefore how much money. I'll throw out a number of $20K per year. I don't know if that's too much or more likely too little.

Will the community here support about $2000+ a month? Advertising can bring in revenue, but how much? I believe we have about 1500 frequent users. Could we get 150 to subscribe at $10-$15 a month? 10% participation in something like that is probably optimistic.

In non-profits it comes down to a matter of whether or not something can be done financially, not whether or not it should be done.

If you're spinning it off to a new non-profit, that can be done for $1000 or less, but would JREF want that?

 

[rjh01]

Making the forum independent of JREF would have huge consequences. Like loss of control of what happens on the forum. Maybe rule 8 could go out the door. Want to sue JREF while still an active member here and boast about it on the forum? Fine. Go ahead.