Media Coverage of Antioxidant Study - Lies, Damned Lies and Statistics!

Response from Quest Vitamins

Media Coverage of Antioxidant Study - Lies, Damned Lies and Statistics!

A rigorous assessment of the methodology of the original meta-analysis of vitamins by Bjelakovic et al; was carried out by Dr. Steve Hickey, Dr. Len Noriegai and Dr. Hilary Roberts (academics at Staffordshire University and Manchester Metropolitan University). This assessment uncovered the meta-analysis as being nothing more than basically lies, dammed lies and statistics!

Some of the main points made by Drs. Hickey, Noriegai and Roberts are listed below.
The statistics were inappropriately applied to poorly selected data, thus the conclusions are invalid.
Experimenter bias was compounded by a basic misuse of statistical testing.
Of the initial studies, 14,910 (93%) were discarded, with only a brief explanation of the exclusion criteria.
747 (92%) of the 815 were rejected, for example, because no subject died during the experiment. The remaining 68 studies were included in the analysis.
The large number of studies by Bjelakovic himself raises concerns in respect of objectivity, as the probability of trials being selected for inclusion in a meta-analysis can be influenced by knowledge of their results, leading to inclusion bias.
critical failing of the Bjelakovic paper is the absence of detail on the number of statistical tests performed on the data.

The fact that this many tests were carried out on just one of the supplements (vitamin A) investigated suggests the results of the study are unreliable. Conventionally, a single statistical test has a 1 in 20 probability of being significant by chance alone. With 100 such tests, we would therefore expect five 'significant' results, just by chance.

Media reports gave the impression that scientific evidence suggests vitamins may be harmful. In fact, no evidence has been provided to this effect. The statistics provided were insufficient to support a claim that vitamin supplements will increase mortality. Moreover, the results cannot validly be generalised to a relatively healthy general population

Bjelakovic's meta-analysis has little biological meaning, because of the large number of ill-defined substances that have been grouped together. The meta-analysis includes a diverse range of doses of the individual supplements, with no concern for the expected physiological effects.
A The authors, by not controlling for experimenter bias, have produced a paper that might simply reflect their own personal bias.