Nutritional Suppliments question

[icerat]

Right, it is merely much more likely to be false than true.

http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124

Interesting looking article, but it will take a while to digest properly. I have an elderly friend, now an adjunct professor at a globally known medicial university. He used to be head of research for a major multi-national pharmaceutical company. He has told me the multi-national would as part of it's standard operating procedure have a study on the efficacy of a drug be done at least 20 times.

One study would be published.

I'm sure you understand the implications of that.

Yes, that is the heart of evidence-based medicine. And that is why use of these supplements is not generally recommended.

absence of evidence is not evidence of absence. When something makes logical sense, it still makes logical sense.

No it doesn't - immortality would have to be conferred by taking the anti-oxidants and by not taking the anti-oxidants. There were 405 studies excluded because there were no deaths in both study groups, not that there were no deaths in the anti-oxidants group but some deaths in the control group.

This has to be considered in the context of the researchers publicising the finding that antioxidants may *increase* risk of death - which come to think of it was a classic example of the problem of sub-group analysis as highlighted earlier.

And even if they manage to not to obscure the results, you still obtain all your information about the results from the studies that were not excluded.

This may be the case when determining whether or not anti-oxidants decrease mortality, but not I believe with the claim they increase it.

The studies included in the Cochrane analysis were tested for homogeneity and were found to be homogeneous. I agree that it is reasonable to be cautious about meta-analysis. But a careful reading of the study shows that it has the characteristics that allow one to form reliable conclusions - that is, it fulfills the criteria one uses to judge whether it is valid.

Might fulfill yours and the Cochrane researchers criteria, it doesn't fulfill mine! And I'm not the only statistician (though well out of practice) to feel that way about meta-analyses.

 

[littlehulkster]

That is probably the only group that supplements make any sense for at all.

There are proven disorders that are remedied by taking the particular supplement. Carnitine comes to mind, if you have the particular mitochondrial genetic disease. There are others too, in which the body lacks some enzyme to make a certain protein out of the amino acids in the diet.

Whether their testing will actually show up a particular disorder or is just a scam needs to be shown. Those diseases are each in the range of one per 5,000 to one per hundreds of thousands. I doubt that Amways business plan is to make a sale to one in 5,000 subjects.

I am truly a believer in the coming of pharmacogenetics. That will be when we get a real handle on our individual health. But gene testing is still waaaay too expensive to test subjects randomly.

I don't know, supplements is a pretty broad category and many do actually do something (protein powders and creatine come to mind), however even the effective ones usually have wildly exaggerated claims.

 

[jon]

supplementation should not be done in isolation, however I'd also argue that in many cases an ongoing assessment procedure might cost more than the price of supplementation and with similar outcomes. Still, in general you are correct, there are plenty of non-supplement interventions that should have higher priority in most peoples lives.

As to assessment costing more than supplementation - this isn't just a financial issue. Insofar as supplements have effects, there is also the risk of side effects (not to mention contamination, etc.) It does not seem good practice to recommend an intervention which carries some (even if small or unknown) risk, without knowing if the problem you're trying to deal with even exists. Why take supplements for high cholesterol without knowing whether you have it (or without good evidence that these supplements are safe and useful when taken by those who don't have high cholesterol).

Pill=lower plasma homocysteine, therefore take pill is a "magic bullet" type approach I don't prescribe to. Amway's "premium pills" aren't targeted to that approach. They actually incorporate 23 different plant concentrates as well as various other vitamins and minerals. Nutrilite does do studies comparing with the generic brands as well, and it does do a better job, as you would expect if you believe that it's better to get nutrients from food than from a chemistry lab.

'Better' in what sense. I'm not particularly interested in raising/lowering my plasma homocysteine levels, unless there's good evidence that doing so will lead to improved health outcomes.

Rather than messing round with plant concentrates etc., I'd much rather eat plants. If there's good evidence that taking supplements is necessary or useful, fair enough (I appreciate that this is the case with some supplements, for some groups). Otherwise, I'd rather eat food.

 

[littlehulkster]

Well, supplements such as whey protein are basically food. It's just that whey, although nutritious, is not very palatable, so it needs to be mixed in with something else, be it in a protein shake or in cooking.

Whey and it's nutritional properties are very well understood and can be objectivley tested in the same way any other food product would be. I am, of course, opposed to many supplements, but I don't think we can tar all of them with such a broad brush.

 

[fls]

Interesting looking article, but it will take a while to digest properly. I have an elderly friend, now an adjunct professor at a globally known medicial university. He used to be head of research for a major multi-national pharmaceutical company. He has told me the multi-national would as part of it's standard operating procedure have a study on the efficacy of a drug be done at least 20 times.

One study would be published.

I'm sure you understand the implications of that.

Which is why the high-impact peer-reviewed journals require trial registration before the trial is undertaken or they will not publish the results, in order to track what has and has not been published.

absence of evidence is not evidence of absence. When something makes logical sense, it still makes logical sense.

The point is that recommendations made in the absence of evidence have been wrong far more often than they have been right. And that 'logical sense' usually does not translate into 'effective'.

This has to be considered in the context of the researchers publicising the finding that antioxidants may *increase* risk of death - which come to think of it was a classic example of the problem of sub-group analysis as highlighted earlier.

You cannot decrease the risk of death in the anti-oxidant group relative to the control group more by including studies without any deaths in either group unless you manage to make the 'no deaths' in the anti-oxidant groups count for more than the 'no deaths' in the control group. But how are you going to do that without admitting that the groups aren't comparable in the first place? You are screwed either way.

This is also not a sub-group analysis, since it was a statistically necessary/methodologically valid a priori group formation. The point of sub-group analysis is that the groups are formed a posteriori.

This may be the case when determining whether or not anti-oxidants decrease mortality, but not I believe with the claim they increase it.

It applies to both situations.

Might fulfill yours and the Cochrane researchers criteria, it doesn't fulfill mine! And I'm not the only statistician (though well out of practice) to feel that way about meta-analyses.

What are your criteria then?

Linda

 

[icerat]

As to assessment costing more than supplementation - this isn't just a financial issue. Insofar as supplements have effects, there is also the risk of side effects (not to mention contamination, etc.) It does not seem good practice to recommend an intervention which carries some (even if small or unknown) risk, without knowing if the problem you're trying to deal with even exists.

No disagreement there, however decent companies very strongly monitor for contamination. Unfortunately there is indeed a lot of crap out there. Consumerlab.com tests and reports often on this.

Why take supplements for high cholesterol without knowing whether you have it (or without good evidence that these supplements are safe and useful when taken by those who don't have high cholesterol).

No disagreement there either, supplements should be targetted to specific needs. As mentioned the company I work with has for example been developing genetic testing to help determine the best/most useful supplements for different people. There is also of course lifestyle assessments that should be done. In most cases the latter can actually be done by the individual.

'Better' in what sense. I'm not particularly interested in raising/lowering my plasma homocysteine levels, unless there's good evidence that doing so will lead to improved health outcomes.

There's good evidence lower plasma homocysteine levels decrease risk of CHD.

Rather than messing round with plant concentrates etc., I'd much rather eat plants. If there's good evidence that taking supplements is necessary or useful, fair enough (I appreciate that this is the case with some supplements, for some groups). Otherwise, I'd rather eat food.

Absolutely, but most people don't eat properly, and in today's world it's increasingly difficult to do so. Indeed, with an inactive lifestyle it may even be the case that getting optimal nutrition through food may lead to a level of caloric intake that cause obesity and all the problems that entails. Yes, ideally people should therefore lead more active lives. Again though, reality is they don't.

This is all of course before we get into the theorised "enhancement" benefits of various supplements, such as the anti-oxidant anti-aging camp etc etc. In most cases that type of thing still is highly theoretical and mostly unsupported by clinical research.

 

[fls]

This has to be considered in the context of the researchers publicising the finding that antioxidants may *increase* risk of death - which come to think of it was a classic example of the problem of sub-group analysis as highlighted earlier.

This may be the case when determining whether or not anti-oxidants decrease mortality, but not I believe with the claim they increase it.

Also, if you are a statistician, can you explain how one goes about combining studies with no observed events?

Linda

 

[icerat]

Which is why the high-impact peer-reviewed journals require trial registration before the trial is undertaken or they will not publish the results, in order to track what has and has not been published.

That's a relatively new innovation and a much needed one. Alas it's not that widespread, and there's an awful lot of research that went before that is still heavily cited.

The point is that recommendations made in the absence of evidence have been wrong far more often than they have been right. And that 'logical sense' usually does not translate into 'effective'.

I disagree with both assertions.

You cannot decrease the risk of death in the anti-oxidant group relative to the control group more by including studies without any deaths in either group unless you manage to make the 'no deaths' in the anti-oxidant groups count for more than the 'no deaths' in the control group.

But how are you going to do that without admitting that the groups aren't comparable in the first place? You are screwed either way.

It depends on the study design, which brings you back to the problems of meta-analysis.

This is also not a sub-group analysis, since it was a statistically necessary/methodologically valid a priori group formation. The point of sub-group analysis is that the groups are formed a posteriori.

I haven't read the full study since it came out, you may be correct here, only entered my mind after reading the earlier link on subgroup analysis.

What are your criteria then?

For a start the included studies should be trying to study the same thing. "anti-oxidants" are not even remotely a monolithic subject area.

 

[icerat]

Also, if you are a statistician, can you explain how one goes about combining studies with no observed events?

lol! well retired from that field I'm afraid, and more than a little rusty, so you're shaking things out a bit! IMO it's flaw in the meta-analysis methodology in the first place, you can't really include them in this type of study - but that doesn't mean they're not important. If I recall the Cochrane guys did actually try to adjust for their removal, which supports my position that it's a problem. I just think it's a worse problem than they do

In a simplified form, take an extreme example.

1000 seperate, well designed studies with 1000 subjects all taking anti-oxidants.

Nobody dies.

10 studies with 100 subjects all taking anti-oxidants.

A significant number die.

The first group of studies is excluded because nobody died.

The second group is analysed and the researchers conclude anti-oxidants increase risk of mortality.

It's simply not a valid conclusion.

I suspect perhaps we're debating at cross-purposes and you're more concerned with whether anti-oxidants have a positive effect, whereas I'm more concerned with their conclusion they may have a negative effect?

 

[jon]

In a simplified form, take an extreme example.

1000 seperate, well designed studies with 1000 subjects all taking anti-oxidants.

Nobody dies.

10 studies with 100 subjects all taking anti-oxidants.

A significant number die.

The first group of studies is excluded because nobody died.

The second group is analysed and the researchers conclude anti-oxidants increase risk of mortality.

It's simply not a valid conclusion.

Example doesn't work. You're talking about a Cochrane meta-analysis of RCTs. The hypothetical '10 studies' you mention don't have a control group.

 

[icerat]

hmmm ... I actually immediately edited my post adding control groups, must have forgot to press post or something.

The groups of course need matched controls.

 

[fls]

I disagree with both assertions.

I realize that you do, however, that is what the evidence shows.

It depends on the study design, which brings you back to the problems of meta-analysis.

Are you under the impression that meta-analysis can not address this issue?

For a start the included studies should be trying to study the same thing. "anti-oxidants" are not even remotely a monolithic subject area.

Then the separate meta-analyses that they did for vitamin C, vitamin A, vitamin E, beta carotene and selenium are all valid?

Linda

 

[jon]

supplements should be targetted to specific needs. As mentioned the company I work with has for example been developing genetic testing to help determine the best/most useful supplements for different people. There is also of course lifestyle assessments that should be done. In most cases the latter can actually be done by the individual.

OK - but I'd want to see good evidence that the testing provided clinically useful results, and that the lab doing it was doing it correctly, before bothering to get it done. Likewise, I'd want to see good evidence that the supplements sold as 'most useful' for those with certain genes actually improved health outcomes in people with those genes.

I appreciate that I am likely not your target market

There's good evidence lower plasma homocysteine levels decrease risk of CHD.

Lower plasma Hcy is associated with decreased risk of CHD. Correlation is not the same as causation, though - and results of using e.g. B vitamin supplements to (successfully) lower plasma Hcy have been less than stunning, in terms of levels of CHD after lowering Hcy. I'm not aware of anything near to the standard of evidence that would convince me to take supplements to lower my plasma Hcy.

Again - I'd want to see robust evidence of health outcomes. In terms of deciding whether to take a supplement or drug to lower my homocysteine levels, I'm interested in whether it will make me healthier - no point in lowering Hcy just for the sake of it.

Absolutely, but most people don't eat properly, and in today's world it's increasingly difficult to do so.

I just don't buy that. It's now relatively easy to buy a nutritionally adequate diet (in Western countries like the US) even if people often don't eat well.

 

[icerat]

Are you under the impression that meta-analysis can not address this issue?

This takes us back to the original issue, the more hoops you jump through, the less real-world reliable your result. I fully admit this is more a philosophical position than a statistical one, though there's certainly statistical support for it.

Then the separate meta-analyses that they did for vitamin C, vitamin A, vitamin E, beta carotene and selenium are all valid?

You're going to make me go read the whole damn study again, aren't you

See point 1!

 

[icerat]

OK - but I'd want to see good evidence that the testing provided clinically useful results, and that the lab doing it was doing it correctly, before bothering to get it done. Likewise, I'd want to see good evidence that the supplements sold as 'most useful' for those with certain genes actually improved health outcomes in people with those genes.

And if I was trying to sell it to you, I'd endeavour to present that to you. As it happens these products and tests aren't even available in my market.

I appreciate that I am likely not your target market

Which is actually a highly important and oft ignored appreciation!

Too many folk go from "I'm not interested in this" to "therefore there's no market for it".

In many fields the most successful brands and products have marketshare of around 5% - so 95% of folk decided they didn't want it!

I'm not aware of anything near to the standard of evidence that would convince me to take supplements to lower my plasma Hcy.

Well, this is "magic bullet" stuff again, and in my opinion nutrition doesn't work that way. Too many interactions etc. Having said that, I work in a different manner - absent any indication of increased risk of negative outcomes, I see no reason not to use something that may have positive outcomes, pending further study.

Again - I'd want to see robust evidence of health outcomes. In terms of deciding whether to take a supplement or drug to lower my homocysteine levels, I'm interested in whether it will make me healthier - no point in lowering Hcy just for the sake of it.

This study[/url] (combined with others) suggests to me for example that it has a positive effect, but in healthy patients it may be too small to statistically observe. My view is that there there thus is a point in lowering Hcy just for the sake of it - it may be beneficial and doesn't do any harm. Having said that, I'm not taking any supplements for the express purpose of lowering my homocysteine levels.

I just don't buy that. It's now relatively easy to buy a nutritionally adequate diet (in Western countries like the US) even if people often don't eat well.

Well, as you say "relatively". As I outlined elsewhere, the overall nutritional content of commercially produced food does appear to be decreasing, so it is "harder". On the other hand increased variety and availability makes it "easier". On the third hand, we're also not active enough so may consume too many calories to get that nutrition.

It's relatively easy *if* you have the time and money to source, purchase, and prepare nutritionally rich food. With the advent of more organic farming and whole food stores the sourcing part may indeed be easier for many, but the purchasing and preparing are still a challenge, as is the lack of activity.

Folk buy car insurance too.

 

[jon]

Well, this is "magic bullet" stuff again, and in my opinion nutrition doesn't work that way. Too many interactions etc. Having said that, I work in a different manner - absent any indication of increased risk of negative outcomes, I see no reason not to use something that may have positive outcomes, pending further study.

Many supplements aren't particularly well-tested in terms of adverse effects, though. And in the UK, the system for reporting such effects from supplement is (even) worse than that for drugs.

I imagine risks are low with most common supplements. However, why expose oneself to the risk, cost and hassle, unless there's good reason to think there will be benefits.

This study[/url] (combined with others) suggests to me for example that it has a positive effect, but in healthy patients it may be too small to statistically observe. My view is that there there thus is a point in lowering Hcy just for the sake of it - it may be beneficial and doesn't do any harm. Having said that, I'm not taking any supplements for the express purpose of lowering my homocysteine levels.

I'm not denying that homocysteine levels are interesting. That's a study on a very narrow group of patients, though, and authors are explicit that more research is needed. A number of other trials found no significant benefit - e.g. this was in women with elevated risk of CHD http://www.ncbi.nlm.nih.gov/pubmed/...ez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum

If benefits are small enough that they don't show up in this kind of trial, it looks unlikely that we'll be talking about anything that dramatic. If future research shows significant benefit in the general population, great - I'd be delighted if B vitamin supplementation provided a cheap, easy way to lower CHD risk. There's nothing that would convince me to date, though.

Well, as you say "relatively". As I outlined elsewhere, the overall nutritional content of commercially produced food does appear to be decreasing, so it is "harder". On the other hand increased variety and availability makes it "easier". On the third hand, we're also not active enough so may consume too many calories to get that nutrition.

It's relatively easy *if* you have the time and money to source, purchase, and prepare nutritionally rich food. With the advent of more organic farming and whole food stores the sourcing part may indeed be easier for many, but the purchasing and preparing are still a challenge, as is the lack of activity.

Buying fresh (or good quality frozen) fruit and veg is not all that expensive - often cheaper than 'junk' food. Likewise, things like tinned sardines are cheap sources of 'good' fats etc. Food can generally be simply prepared, unless you enjoy cooking more complex meals. They needn't be calorie dense, if people have low calorie needs. No need to go to whole food stores IMO - perfectly OK fresh food is available from markets, supermarkets, etc.

Compared to times when fresh food was simply unavailable for long periods - or when stores were sometimes so depleted that people faced starvation - we're in a wonderful situation nowadays. If someone's diet is so poor that they need vitamin pills in order to avoid a numerous deficiencies, the pills are a poor substitute for eating better.

 

[fls]

it's flaw in the meta-analysis methodology in the first place, you can't really include them in this type of study - but that doesn't mean they're not important. If I recall the Cochrane guys did actually try to adjust for their removal, which supports my position that it's a problem. I just think it's a worse problem than they do

They didn't adjust for their removal. They did perform an exploratory analysis by pretending that the excluded studies were one large study and that there was one death in each group (the answer to my previous question was "you can't"). This allows them to short circuit any naive arguments over whether or not these studies should be excluded by pointing out that it doesn't matter anyway.

In a simplified form, take an extreme example.

1000 seperate, well designed studies with 1000 subjects all taking anti-oxidants.

Nobody dies.

10 studies with 100 subjects all taking anti-oxidants.

A significant number die.

The first group of studies is excluded because nobody died.

The second group is analysed and the researchers conclude anti-oxidants increase risk of mortality.

It's simply not a valid conclusion.

That there was a significant increase in deaths can't simply be ignored. What is important to know is why some studies showed events and some didn't. In the case of the 405 excluded studies, they were short-term studies measuring surrogate end-points. This means that they had a low probability of any events which gives them a low power to detect a large effect on mortality. That is, one cannot draw valid conclusions about mortality from the excluded studies which means that one cannot overturn conclusions from other studies.

There were more than 10 times as many participants in the studies that were included than in the studies that were excluded (as opposed to your example).

But here's the kicker. You are assuming that if your 1000 no death studies were combined with your 10 death studies, that it would change the results from significant to not significant. However, it does not. If you combine the studies as one large study, the relative risk of the excess mortality remains unchanged and the significance level also remains essentially unchanged. That is, it doesn't matter whether or not your criticism is valid, it doesn't change the conclusion that there is a significant increase in excess mortality with anti-oxidant use.

I suspect perhaps we're debating at cross-purposes and you're more concerned with whether anti-oxidants have a positive effect, whereas I'm more concerned with their conclusion they may have a negative effect?

No, I am concerned with both possibilities. The problem is that your criticism is not valid, and even if it were, it would not change the conclusion.

Linda

 

[fls]

This takes us back to the original issue, the more hoops you jump through, the less real-world reliable your result. I fully admit this is more a philosophical position than a statistical one, though there's certainly statistical support for it.

External validity is relevant to whether or not your results are real-world reliable. However, you have yet to bring up threats to external validity for discussion, despite your handwaving reference to hoops.

Linda

 

[fls]

Absolutely, but most people don't eat properly, and in today's world it's increasingly difficult to do so. Indeed, with an inactive lifestyle it may even be the case that getting optimal nutrition through food may lead to a level of caloric intake that cause obesity and all the problems that entails.

This is utterly wrong. It is not difficult to eat properly. The resources for doing so are readily available, as is the ability to lead an active lifestyle.

Linda

 

[icerat]

I imagine risks are low with most common supplements. However, why expose oneself to the risk, cost and hassle, unless there's good reason to think there will be benefits.

I believe there's good reason to think there will be benefits. You're free to disagree

I'm not denying that homocysteine levels are interesting.

Actually there's some interesting research that indicates they may be a symptom rather than a cause, so it may indeed be a waste of time. Or it may provide other benefits. Or it could have a negative effect - but there's no evidence at all I'm aware of to indicate that.

Buying fresh (or good quality frozen) fruit and veg is not all that expensive - often cheaper than 'junk' food. Likewise, things like tinned sardines are cheap sources of 'good' fats etc. Food can generally be simply prepared, unless you enjoy cooking more complex meals. They needn't be calorie dense, if people have low calorie needs. No need to go to whole food stores IMO - perfectly OK fresh food is available from markets, supermarkets, etc.

You're in the UK? Very interesting study by a few years ago analysing 50 years of government food composition data. There were clear decreases in nutrient content in a wide range of fruits and vegetables. I have similar studies for North America, Scandinavia, and Australia.

Again though, you're right that it's possible. The reality is however that most people don't. I know I don't.

Compared to times when fresh food was simply unavailable for long periods - or when stores were sometimes so depleted that people faced starvation - we're in a wonderful situation nowadays. If someone's diet is so poor that they need vitamin pills in order to avoid a numerous deficiencies, the pills are a poor substitute for eating better.

100 years ago problems caused by deficiencies or diseases probably far outweighed any problems caused by nutritional depletion, ie low levels that cause negative effects but not explicit disease conditions. Compared to 50 years ago we have a different situation. Take one area I've researched, Vitamin D levels in Scandinavia. About 5 years back I noticed that many of the signs of possible Vitamin D depletion or deficiency seemed to be increasingly prevalent, ranging from high levels of osteoporosis to SAD to alzheimers to suppressed immune functions in late winter etc etc. It seemed likely to me that Swedes were low in D, something which made sense given the low-levels of UVB for much of the year.

However local authorities and researchers said there was no evidence of this. I looked into the research and discovered they were correct. Studies of blood plasma levels seemed to indicate no significant problems. So what was going on? I later realised I'd been foolish to focus just on UVB. D can also be sourced from fatty fish. I also found was that most of the research into blood plasma levels was actually two or three decades old. When I looked into diet studies it became very clear that Swedish fish consumption had dropped quite dramatically in recent decades. Partly this has happened due to increased variety in the food supply, so fish was being replaced by things like chicken that wasn't such a common meal 20 years ago, partly due to increased warnings about heavy metal contamination in many fish.

So, it was quite clear to me that dietary intake of Vitamin D had likely plummeted in a very short period of time and many Swedes most likely were D deficient. In the last couple of years research has now confirmed this.

I've seen very similar issues with Omega-3 (n-3), where dietary intake has almost certainly dropped dramatically, combined with significant increases in Omega-6 (n-6), also essential but a metabolic competitor. We're also seeing significant increases in the kind of problems you would predict if there was indeed a lack of biologically available omega-3. As an aside, the increased n-6 issue is IMO one of the reasons why many n-3 supplementation studies don't show much benefit - flooded with n-6 the body simply can't effectively use n-3.