Future of the Forum

[George 152]

The place is running like a dream. No real delays.
Some-one needs a lot of thanks for their work behind the scenes

 

[Lisa Simpson]

Yes, it's running quicker, but many people still can't log in and I can't access the registration queue, which is growing and growing and growing:

 

[SezMe]

My preference - and others I've spoken to seem to feel the same - is that a new non-profit spun off from JREF is probably the way to go. (any legal/accounting folk with time/expertise in setting up and maintaining a non-profit, put your hand up)

I'm also getting the distinct impression there's a lot of backroom history we're dealing with here which has lead to the current somewhat critical situation. That may also make this entire discussion moot and something entirely different happens.

I would very much like not to see this happen. I know nothing of the backroom but I still am of the opinion that there is a mutually beneficial relationship between JREF and this forum. TAM, as noted, is just one example. I suspect the forum acts as a feed for new JREF members and that that is a two-way street. The spin-off of Skeptical Community is an example to avoid.

 

[Spindrift]

I would very much like not to see this happen. I know nothing of the backroom but I still am of the opinion that there is a mutually beneficial relationship between JREF and this forum. TAM, as noted, is just one example. I suspect the forum acts as a feed for new JREF members and that that is a two-way street. The spin-off of Skeptical Community is an example to avoid.

I thought the forum provides very few JREF members.

 

[icerat]

Somewhat rambling thoughts:

I've worked with a lot of non-profit organizations and have been president of several. While $1500 startup and $6000-$7000 a year does not sound like a lot, it can be to a huge amount to non-profit.

Yes, been involved in several myself. As I mentioned earlier I'm a fan of developing business systems that allow community organizations to be self-funding and even profitable. The challenge is finding ways to do that. In the US there's actually a formal structure for this now - B-Corporations.

I agree with your point about a pro-admin/mod being needed. Volunteers are great, but their priorities can shift so needed activities don't happen when needed, they get done when they can be done. So you're talking more money. I don't know how much time would be needed for a pro-admin and therefore how much money. I'll throw out a number of $20K per year. I don't know if that's too much or more likely too little.

Without having enough info to say for sure, I'm probably willing to have one of my community managers do this (it's part of their role with other communities). As such if the money is there then we willing accept it, if not so be it.

Will the community here support about $2000+ a month? Advertising can bring in revenue, but how much? I believe we have about 1500 frequent users. Could we get 150 to subscribe at $10-$15 a month? 10% participation in something like that is probably optimistic.

I agree, however I think there should be an acceptable advertising model that can generate income. What we are doing with other communities is finding community members who have particular products/services that are directly related to that community, and as part of a "pro" membership we co-market to the community.

It has to be done in a way that is acceptable and useful to the community in question. It's also more work, but generates more revenue than things like googleads.

As an example, if this was an independent community it may take a small commission on TAM ticket sales that occur via the site. I'm sure there are other skeptic events around the world that could do with co-marketing through this site.

In non-profits it comes down to a matter of whether or not something can be done financially, not whether or not it should be done.

If you're spinning it off to a new non-profit, that can be done for $1000 or less, but would JREF want that?

I think JREF as an organization hasn't yet clearly solidified an answer to that question.

 

[Childlike Empress]

Like I suspected and icerat confirmed:

Likely [the problem is current hardware], and not unlikely the contract is a couple of years old given that the forum doesn't seem to have been a JREF priority for the last few years.

Meaning that today you would get a lot more power for the bucks you signed in for, likely enough for the current needs, and I suggest that you lay all of this out to the amazing Randi and let him make the phone call for a new deal.

Just FYI, the current advertising income does not come close to covering the costs of the forum. While an upgrade to better equipment will, paradoxically, probably result in lower costs, it will likely still not be enough so ongoing funding solutions will need to be found.

I'd advise to focus on first fixing the immediate problems and pressuring the host a bit (that specs are way above what I expected, there could be simply some defect stuff involved as it was the last time with that router/switch that wasn't fixed for ages) before getting into some pipedreams. I don't doubt that at this stage some serious money could be raised from frustrated forum members, but in the long run it will be difficult to sustain such high running costs including however enthusiastic professional helpers.

Oh, and it seems to be much faster already indeed.

 

[Startz]

This is a very constructive discussion. I can't contribute anything to the technical side, but a brief comment about money might be helpful.

We have a detailed estimate from icerat that things could be made to run quite well for $500~600 a month. To be way generous, call it $10,000 a year. According to JREF's 2012 990 form, annual JREF revenue is around $1.3 million. Now there are lots of calls on that revenue, but it seems the forum could be kept running for about 1/10th of 1 percent of the JREF budget. This suggests to me the issue is JREF's level of interest, rather than any real money problem.

It would be good to have a frank answer from JREF before worrying too much about funding models.

 

[wombatwal]

Oh, and it seems to be much faster already indeed.

That's the power of wanting it to be faster. The Universe responded.

 

[jhunter1163]

This is a very constructive discussion. I can't contribute anything to the technical side, but a brief comment about money might be helpful.

We have a detailed estimate from icerat that things could be made to run quite well for $500~600 a month. To be way generous, call it $10,000 a year. According to JREF's 2012 990 form, annual JREF revenue is around $1.3 million. Now there are lots of calls on that revenue, but it seems the forum could be kept running for about 1/10th of 1 percent of the JREF budget. This suggests to me the issue is JREF's level of interest, rather than any real money problem.

It would be good to have a frank answer from JREF before worrying too much about funding models.

I think you're off one decimal place; it would be about 1% of JREF's revenue. Still and all, it's not a huge amount of money, even for a comparatively small non-profit such as JREF.

Could the forum be self-sustaining? Possibly; if the membership put up, say, the hardware cost plus the first year's ongoing expenses, there would be time to develop some sort of model to generate revenue (I hate banner ads like the plague, but I would suffer them to preserve the forum, and I think most here would agree).

 

[AdMan]

I think you're off one decimal place; it would be about 1% of JREF's revenue. Still and all, it's not a huge amount of money, even for a comparatively small non-profit such as JREF.

Could the forum be self-sustaining? Possibly; if the membership put up, say, the hardware cost plus the first year's ongoing expenses, there would be time to develop some sort of model to generate revenue (I hate banner ads like the plague, but I would suffer them to preserve the forum, and I think most here would agree).

With the number of forum grants to TAM given out each year (20+ this year, for example), I'm quite sure that the forum can probably be self-sustaining just from the income that brings the JREF. I'm hoping they are taking that into account--or maybe they think they can raise more money elsewhere with equal or less effort. I'd be skeptical of that.

That's another reason I agree with those that think it's likely more than just a money issue, but in any case the money issue will need to be dealt with eventually if we do want the forum to continue.

 

[Startz]

I think you're off one decimal place; it would be about 1% of JREF's revenue. Still and all, it's not a huge amount of money, even for a comparatively small non-profit such as JREF.

Hey, what's an order of magnitude between friends?

 

[temporalillusion]

My forum went from a single server to two servers at around the 6-700 concurrent users level (I suspect my forum usage profile has heavier peaks than this one though), so while yeah I was surprised the hardware was more robust than I had thought it was, I still wouldn't be shocked if the slowness was just server scaling.

And even if it is something specific that's eating up resources away from what should get the resources, I still think it's worth moving since you can likely get better servers for cheaper.

But that's still a good point, taking a look at the server and making sure there isn't something else going on should be one of the first things done. vBulletin itself has some cron type jobs it runs on various intervals.

I had an issue once where someone's browser half crashed in a way that was re-submitting the same request to the server many times a second, basically DoSing the server. I figured the user from the IP and emailed them and they rebooted their computer and it stopped. Strange stuff can happen.

But granting volunteers root access to the server is something I can see them being reluctant to do, that's a pretty big amount of trust.

 

[This is The End]

but I tend toward upgrading to Xenforo ($250 incl enhanced search) plus a yearly fee for ongoing upgrades/support ($65).

If the hardware fixes the problems please do not do this. The forum works great (see below post). Do not fix what is not broken.


ETA: And do not overfix what is broken. Like how some computer repair shops actually fix problems, even if it takes some time. Other computer repair shops just instantly reformat and reinstall.

 

[This is The End]

Been a developer for client/server database apps and/or web services for the last 25+ years. From the limited info I have seen, I don't think new hardware is required. I think there's a specific root cause, which has not been identified, that is causing it to appear to be a system that has outgrown its hardware.

I have to agree with Dan here.

The forum works absolutely fantastic. Lightning fast page loads. For hours on end. And then randomly it will work horribly slow for hours.

On and off, just like that, for at least 1 year now.

Does not compute.

My first guess would be the large volume of traffic from China and search engines, but it could be some random glitch not yet troubleshot.

 

[blobru]

Yes - the suggestion of a rogue process, sitting in memory and butting in on everything else, that many have made makes sense of the vmstats (the over 200 X as many interrupts as normal, only half as many processes running, free memory shrinking, even before things start to back up midway through the snapshot); certainly worth looking into.

 

[SezMe]

Could the forum be self-sustaining? Possibly; if the membership put up, say, the hardware cost plus the first year's ongoing expenses, there would be time to develop some sort of model to generate revenue (I hate banner ads like the plague, but I would suffer them to preserve the forum, and I think most here would agree).

I'd be a bit leery of such an approach. What happens after year one if a revenue model is not found? Is there some way to estimate revenue streams from various business models before starting down one road? Surely, yes, because we can't be the first non-profit to fact this situation.

 

[Wolfman]

I've got a problem...I can view the forum no problem, but every time I try to make a post, I'm taken to the Sucuri Website Firewall, with the following information:

Block details

Your IP: 123.151.32.96
URL: forums.randi.org/newthread.php?do=postthread&f=26
Your Browser: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0
Block ID: BLACK02
Block reason: Your IP address is listed in our blacklist and blocked from completing this request.
Time: Tue, 05 Aug 2014 02:35:34 -0400
Server ID: cp413

I've had issues with the Sucuri thing before, but never this error message. It seems to arise only when I'm making posts, not when I'm just viewing the forum. Given the previous information that a lot of traffic is coming from China, I'm concerned that perhaps someone changed the security to block Chinese traffic?

I'm currently accessing the forums through a VPN, but that's far from an ideal solution.

 

[Michel H]

Mr. Adams, on the JREF board, is able to get in and do some technical tweaks to the hardware and software.

One insight he has given me is that right now 40% of the queries are from search engine bots, a good proportion of the traffic is from China. Legitimate traffic?

...
That would be this Mr Adams for those interested. So as you can imagine he has the technical nous but perhaps has other interests he prioritises over this forum
...

Yes, it's running quicker, but many people still can't log in and I can't access the registration queue, which is growing and growing and growing:

[qimg]http://www.internationalskeptics.com/forums/imagehosting/230853dff92cabfa9.png[/qimg]

I've got a problem...I can view the forum no problem, but every time I try to make a post, I'm taken to the Sucuri Website Firewall, with the following information:

Block details

Your IP: 123.151.32.96
URL: forums.randi.org/newthread.php?do=postthread&f=26
Your Browser: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0
Block ID: BLACK02
Block reason: Your IP address is listed in our blacklist and blocked from completing this request.
Time: Tue, 05 Aug 2014 02:35:34 -0400
Server ID: cp413

I've had issues with the Sucuri thing before, but never this error message. It seems to arise only when I'm making posts, not when I'm just viewing the forum. Given the previous information that a lot of traffic is coming from China, I'm concerned that perhaps someone changed the security to block Chinese traffic?

I'm currently accessing the forums through a VPN, but that's far from an ideal solution.

Sharon (idoubtit), I hope Mr. Adams, who is perhaps the person who's been installing the Sucuri website firewall (see here for an explanation) for the forum in recent days, is fully aware of the problems mentioned by Lisa Simpson and Wolfman above. The number of active users is currently rather low (about 325 now, usually about 700 [?]), this may explain why the forum is fast. It would be nice if you could convince this person (or whoever is currently in charge of the hardware) to post in the Forum Help & Member Support sub-forum. In this way, he could explain what he's doing, interact with, and receive advice from the experts of this forum.

 

[Klimax]

Sharon (idoubtit), I hope Mr. Adams, who is perhaps the person who's been installing the Sucuri website firewall (see here for an explanation) for the forum in recent days, is fully aware of the problems mentioned by Lisa Simpson and Wolfman above. The number of active users is currently rather low (about 325 now, usually about 700 [?]), this may explain why the forum is fast. It would be nice if you could convince this person (or whoever is currently in charge of the hardware) to post in the Forum Help & Member Support sub-forum. In this way, he could explain what he's doing, interact with, and receive advice from the experts of this forum.

You missed at least one other problem caused by Sucuri: Missing avatars.

 

[lauwersw]

System Performance

%vmstat 3

%vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 7  0    252  86180  55660 7092544    0    0   219    71    7    7 26  3 62  9  0
 4  0    252  80972  55740 7094984    0    0   912  1376 1625 2253 20  1 76  3  0
 3  0    252  57136  55820 7100208    0    0  1621   692 1396 3459 21  1 74  4  0
 3  1    252  57728  55824 7094740    0    0  1211   616 1423 2835 19  1 76  3  0
 4  5    252  47628  55536 7052016    0    0  3113  1072 1399 11270 21  3 65 11  0
 2  5    252  47948  55572 7050052    0    0  3379   479 1529 2090 10  1 79 10  0

I'm not an expert on interpreting vmstat but this sample doesn't appear too bad (ie it's not a time when things are lagging badly) but a cpu upgrade would provide some immediate benefits, as I suspect would moving images out of the db, if they're not already.

The last few columns of CPU stats suggest there's plenty of CPU time left. The "wa" column is relatively high, indicating that the OS has to wait for I/O (probably disk, could be network too), but it's nothing problematic. System cpu usage is really low and most work is done in user space, indicating that the application software isn't eating unnecessary kernel resources. To finish there's plenty of CPU time left, so CPU is not the bottleneck for this snapshot. Context switches are fairly high but nothing extraordinary. I have systems that run way higher on all accounts and are still responsive.

It smells indeed like a software scaling problem, which is best solved in the application, but you could improve a lot with better hardware of course. I would concentrate on the disks first, no the CPU. Any chance you can take some "top" snapshots at peak load, or more vmstats, maybe that will show a different picture?

If money allows, go indeed for SSD. Alternatively for disks don't use RAID5: RAID10 or RAID6 are much better for performance and resilience but require more disks. On the other hand RAM is cheap, you could cache most of the DB in memory, only leaving writes loading the disks.

Upgrading from CentOS 5 to 6 might give between 10 to 30% more performance on the same hardware, contributed from different parts of the OS. CentOS 6 supports the ext4 filesystem, CentOS 7 (no personal experience yet) uses XFS by default. Both will give some improvement for MySQL file system performance. The kernel and networking stack will do the rest. Even Upgrading to the latest CentOS version can already help, I've seen about 10% difference in CPU load on a heavily loaded server when upgrading from 5.4 to 5.10.

You should also checkout ktune and tuned, especially on CentOS 6 this might give a free additional benefit.