Chalmers’ meta-analysis (1975)

by Harri Hemilä



This text is based on pages 36-38 of Hemilä (2006).
This document has up to date links to documents that are available via the net.
Harri Hemilä
Department of Public Health
University of Helsinki,  Helsinki, Finland
harri.hemila@helsinki.fi
Home:  http://www.mv.helsinki.fi/home/hemila

This file:  http://www.mv.helsinki.fi/home/hemila/reviews/chalmers

Version May 29, 2012




Thomas Chalmers was an eminent pioneer in controlled trials, and one of the early promoters of meta-analysis (Dickersin 1996; Huth 1996; Liberati 1996; Maclure 1996a 1996b; NLM 2005d). Greenhouse (1996) commented that Chalmers was the most influential non-statistician to affect the acceptance and direct the course of the randomized, controlled clinical trial among clinical investigators, at least in the USA. Chalmers was a founding member of the Society for Clinical Trials, and president of that society in 1984 (Knatterud 1996). "In leadership roles at the US Veterans Administration and National Institutes of Health, Chalmers’ intellectual and administrative support of randomized trials helped lead to their establishment as a prime source of therapeutic evidence" (Feinstein 1996). "Tom Chalmers’ name is synonymous with randomized, controlled trial and metaanalysis… Many of the medical advances discovered in the past several decades through the use of randomized, controlled trials can in large measure be attributed to his tireless advocacy" (Lau 1996). It was even commented that Thomas Chalmers "contributed more than anyone else to the general field of clinical science" (Ian Chalmers 1996).

A volume of the Annals of the New York Academy of Sciences was dedicated "To Thomas C. Chalmers - A Meta-Analyst for All Seasons" (Warren & Mosteller 1993), and a book entitled Systematic Reviews was dedicated to "Thomas C Chalmers in appreciation of his many pioneering contributions to the science of reviewing health research, and in particular, for the first clear demonstration of the dangers of relying on traditional reviews of research to guide clinical practice" (Ian Chalmers & Altman 1995). At each annual Cochrane Colloquium "The Thomas C Chalmers MD Award" is given for the best oral or poster presentation at the Colloquium (Cochrane 2005b).

Because of this personal background, a meta-analysis on vitamin C and the common cold by Thomas Chalmers obviously became highly influential, and found its place in major medical textbooks as soon as the ink on the paper had dried.

For his meta-analysis in the American Journal of Medicine, Chalmers (1975   CH ) selected 8 placebo-controlled trials and presented their results in a table. Chalmers calculated that the common cold episodes were on average 0.11 ± 0.24 (SE) days shorter in the vitamin C groups than the placebo groups. Even if real, a 0.11 day (= 2.6 hours) decrease in the duration of colds would have no clinical relevance, and the great variation in the results indicates that the slight difference is not real. Apparently, because of the presentation of the published results in a table, and the clear-cut negative findings, Chalmers’ meta-analysis met ready approval.

Chalmers’ meta-analysis has been frequently cited as evidence that vitamin C has been shown to be worthless for colds. For example, it was cited in the US nutritional recommendations (FNB 1980 p 77, 1989a p 120 ), major textbooks of infectious diseases (Gwaltney 1979, 1985, 1990, 1995; Liu 1989; Dick et al. 1992, 1998), and numerous textbooks on nutrition (Halsted 1993; Thurnham et al. 2000; Hamilton & Whitney 1982, 1994; Whitney & Rolfes 1993; Shils et al. 1994).

The American Medical Association based its official statement that "One of the most widely misused vitamins is ascorbic acid. There is no reliable evidence that large doses of ascorbic acid prevent colds or shorten their duration" wholly on Chalmers’ review (AMA 1987 p 1934). Lancet also referred to Chalmers’ meta-analysis in stating that vitamin C has no worthwhile effect on the common cold (Editorial 1979 [see Pauling 1979] ).

Chalmers’ meta-analysis was cited in several major reviews of common cold therapy (Hirsch & Swartz 1980; Anderson et al. 1983; Sperber & Hayden 1988; Steele 1988; Lorber 1996).

Chalmers’ meta-analysis was recently shown to contain a large number of serious errors (Hemilä & Herman 1995). In some cases the data was inconsistent with the original published data, there were errors in calculation, the selection of trials was not consistent, and in some trials a clinically less meaningful outcome was selected. Table 14 shows the data for common cold duration that Chalmers presented in his main table (one trial did not provide data for common cold duration and is not listed in Table 14: Franz et al. 1956; see Table 3 in Background). The figures which are erroneous or misleading in Chalmers’ table are underlined in Table 14 (for details see Hemilä & Herman 1995).

Chalmers did not consider the dose of vitamin C used in the trials. At the extreme, Karlowski et al. (1975) administered up to 6 g/day of vitamin C to their subjects, whereas Cowan et al. (1942) administered 25 mg/day as their lowest supplementary dose, meaning an up to 240-fold difference between the lowest and highest vitamin C doses used; however, these two trials were presented side-by-side by Chalmers (1975) in his table without noting this difference (Table 14). Furthermore, Chalmers justified his metaanalysis by referring to Pauling’s 1970 book, but Chalmers included in his table a trial in which only 0.025 - 0.05 g/day of the vitamin was used (Cowan et al. 1942). This latter trial obviously does not serve as a meaningful test of Pauling’s hypothesis that ≥1 g/day might be beneficial against colds.

In the case of the Anderson et al. trial (1972) Chalmers did not justify his outcome selection. He showed the effect of vitamin C on days ‘symptoms present’ per episode which was reduced by 5% (P = 0.15) in his table, but did not mention the effect on days ‘confined to house’ per common cold episode which was reduced by 21% (P = 0.007), even though the latter outcome would seem more relevant for public health considerations since it is a measure of how much these infections cause actual functional limitations, whereas a runny nose is simply a minor nuisance.

Furthermore, there are severe arithmetical problems in Chalmers’ meta-analysis. In his table, the shortest cold duration in a placebo group is 1.0 days and the longest 6.3 days (Table 14). Chalmers calculated the mean for the absolute difference in duration without paying any attention to the great variation in the control group duration. For example, if a 6-day cold is shortened by 1 day, it is not equivalent to a 1-day cold being shortened by the same amount, although both differences are equal in absolute units. Furthermore, a 6-day cold can be shortened by 2 days, but a 1-day cold cannot. Thus the outcome scales of such trials are incompatible in absolute units. Consequently, it seems much more reasonable to calculate the relative effect of vitamin C, so that a 6-day cold shortened by 2 days and a 1-day cold shortened by 0.33 days both correspond to an equivalent 33% reduction. Calculating the relative effect corresponds to normalization of all control groups to an episode duration of one unit or 100%.

Another shortcoming in Chalmers analysis was not considering the size of the trials. For example, Anderson et al. (1972) recorded 1,170 episodes, whereas Karlowski et al. (1975) recorded only 249 (Table 10 in Karlowski's problems ). In calculating a pooled estimate, the former trial should obviously be given much greater weight than the latter trial. Lack of weighting by study size leads to an inappropriate significance of small inaccurate trials in the calculation of the summary estimate.

A further arithmetical problem in Chalmers’ meta-analysis relates to the presentation of papers that contain more than one trial. In the case of the Cowan et al. paper (1942), Chalmers correctly presents the two reported trials separately (Table 14). However, in the case of the Coulehan et al. paper (1974), Chalmers combines the two reported trials together, one with 1 g/day and another with 2 g/day of vitamin C. This combination of Coulehan trials gives more emphasis in his table to the two trials by Cowan et al. which administered only 0.2 g/day and 0.025-0.05 g/day. Furthermore, when the unweighted mean effect is calculated, the two small-dose trials of Cowan et al. are accorded a weighting factor of two, while the two large-dose trials of Coulehan et al. get a weighting factor of only one.

The trials known to Chalmers which had used ≥1 g/day of vitamin C, i.e., that were testing Pauling’s proposal, were reanalyzed and the common cold episodes were calculated as 0.93 ± 0.22 (SE) (unweighted mean; P = 0.005) days shorter in the vitamin C groups (Hemilä & Herman 1995). Thus, an estimate of more than 8 times Chalmers’ estimate was obtained by employing correct values and keeping to trials which used doses as high as Pauling had suggested. Furthermore, a more useful measure of effect from the same trials is a 21.2% ± 3.0% (SE) reduction of symptom duration. As expected, the relative effect leads to a more accurate estimate of effect as indicated by the smaller P-value (P = 0.0005).

Chalmers review (1975) is no longer cited in the recent editions of major textbooks on infectious diseases (Liu 1994; Gwaltney 2000, 2005), but it is still cited in the current US nutritional recommendations (FNB 2000 p 127). Also, the Chalmers review (1975) was cited in a recent review on echinacea and the common cold when proposing that the effects of echinacea in placebo-controlled trials are caused by the placebo effect (Caruso & Gwaltney 2005 [see Hemilä 2005a]).

Table 14. Ascorbic acid and the common cold: "reasonably well-controlled studies" according to Chalmers (1975)


Trial Ascorbic acid Placebo

No. of subjects Mean duration (days) No. of subjects Mean duration (days) Difference in duration (days)
Anderson-1972 407 3.96 411 4.18 0.22
Anderson-1974 583 3.28 578 3.18 -0.10
Coulehan-1974 321 4.71 320 5.92 1.21
Wilson-1973 158 2.65 144 2.79 0.14
Karlowski-1974 101 6.80 89 6.30 0.50
Cowan-1942 233 1.10 194 1.60 0.50
Cowan-1942 227 1.70 120 1.00 -0.70




Mean for the above: 0.11  ±  0.24 (SE)
Ritzel-1961 139 1.35 140 1.95 0.60

Erroneous and misleading numerical values are indicated by underlining and bold.
Chalmers also listed the incidence of colds, but these are left out to save space.
This table was originally published in Hemilä and Herman (1995).

References

AMA [Council of Scientific Affairs, American Medical Association] (1987) Vitamin preparations as dietary supplements and as therapeutic agents. JAMA 257:1929-36   Extract

Anderson LJ, Patriarca PA, Hierholzer JC, Noble GR (1983) Viral respiratory illnesses. Med Clin North Am 67:1009-30

Anderson TW, Reid DBW, Beaton GH (1972) Vitamin C and the common cold: a double-blind trial. Can Med Assoc J 107:503-8

Anderson TW, Suranyi G, Beaton GH (1974) The effect on winter illness of large doses of vitamin C. Can Med Assoc J 111:31-6

Caruso TJ, Gwaltney JM Jr (2005) Treatment of the common cold with echinacea: a structured review. Clin Infect Dis 40:807-10 * comments in: Hemilä 2005a

Chalmers I, Altman DG, eds (1995) Systematic Reviews. London: BMJ Publishing Group

Chalmers TC (1975) Effects of ascorbic acid on the common cold: an evaluation of the evidence. Am J Med 58:532-6 PubMed CH BM

Cochrane (2005b)

Coulehan JL, Reisinger KS, Rogers KD, et al. (1974) Vitamin C prophylaxis in a boarding school. N Engl J Med 290:6-10

Cowan DW, Diehl HS, Baker AB (1942) Vitamins for the prevention of JAMA 120:1268-71 CH BM

Dick EC, Inhorn SL (1992) Rhinoviruses. In: Textbook of Pediatric Infectious Diseases, 3rd edn [Feigin RD, Cherry JD, eds]. Philadelphia, PA: Saunders. pp 1507-32
Dick EC, Inhorn SL, Glezen WP (1998) Rhinoviruses. In: Textbook of Pediatric Infectious Diseases, 4th edn [Feigin RD, Cherry JD, eds]. Philadelphia, PA: Saunders. pp 1839-65

Dickersin K (1996) Thomas Clark Chalmers. JAMA 276:656-7 *  see also: jameslindlibrary   pubmedcentral

Editorial (1979) Ascorbic acid: immunological effects and hazards. Lancet 313:308 * comments in: Pauling (1979) Ascorbic acid [letter]. Lancet 313:615

Feinstein AR (1996) Editor’s note. J Clin Epidemiol 49:1079

FNB [Food and Nutrition Board, National Research Council] (1980) Recommended Dietary Allowances, 9th edn. Washington DC: National Academy of Sciences * Vitamin C pp 72-82

FNB [Food and Nutrition Board, National Research Council] (1989a) Recommended Dietary Allowances, 10th edn. Washington DC: National Academy Press * Vitamin C pp 115-25 p 120

Franz WL, Heyl HL (1956) Blood ascorbic acid level in bioflavonoid and ascorbic acid therapy of common cold. JAMA 162:1224-6 CH

Greenhouse SW (1996) Tributes to Dr. Thomas C. Chalmers. Cont Clin Trials 17:474-5  * see also: (1996);17:471-2

Gwaltney JM (1979) The common cold. In: Principles and Practice of Infectious Diseases, 1st edn [Mandell GL, Douglas RG, Bennett JE, eds]. NY: Churchill Livingstone. pp 429-35

Gwaltney JM (1985) The common cold. In: Principles and Practice of Infectious Diseases, 2nd edn [Mandell GL, Douglas RG, Bennett JE, eds]. NY: Churchill Livingstone. pp 351-5

Gwaltney JM (1990) The common cold. In: Principles and Practice of Infectious Diseases, 3rd edn [Mandell GL, Douglas RG, Bennett JE, eds]. NY: Churchill Livingstone. pp 489-93

Gwaltney JM (1995) The common cold. In: Principles and Practice of Infectious Diseases, 4th edn [Mandell GL, Bennett JE, Dolin R, eds]. NY: Churchill Livingstone. pp 561-6

Gwaltney JM (2000) The common cold. In: Mandell, Douglas, and Bennett’s Principles and Practice of Infectious Diseases, 5th edn [Mandell GL, Bennett JE, Dolin R, eds]. Philadelphia: Churchill Livingstone. pp 651-6

Gwaltney JM (2005) The common cold. In: Mandell, Douglas, and Bennett’s Principles and Practice of Infectious Diseases, 6th edn [Mandell GL, Bennett JE, Dolin R, eds]. Philadelphia: Elsevier Churchill Livingstone. pp 747-52

Halsted CH (1993) Water-soluble vitamins. In: Human Nutrition and Dietetics, 9th edn [Garrow JS, James WPT, eds]. London: Churchill Livingstone. pp 239-63

Hamilton EMN, Whitney EN (1982) Nutrition, Concepts and Controversies, 2nd edn. NY: West Publ. pp 277-93

Hamilton EMN, Whitney EN (1994) Nutrition, Concepts and Controversies, 6th edn. NY: West Publ. pp 403-29

Hemilä H (2005a) Echinacea, vitamin C, the common cold, and blinding [letter]. Clin Infect Dis 41:762-3 * comments on: Caruso & Gwaltney (2005)

Hemilä H (2006) Do vitamins C and E affect respiratory infections? [Dissertation]. University of Helsinki, Finland   Hemilä 2006

Hemilä H, Herman ZS (1995) Vitamin C and the common cold: a retrospective analysis of Chalmers’ review. J Am Coll Nutr 14:116-23 Abstract 

Hirsch MS, Swartz MN (1980) Antiviral agents. Part II. N Engl J Med 302:949-53

Huth EJ (1996) Tributes to Thomas Chalmers. Ann Intern Med 124:696

Karlowski TR, Chalmers TC, Frenkel LD, Kapikian AZ, Lewis TL, Lynch JM (1975) Ascorbic acid for the common cold: a prophylactic and therapeutic trial. JAMA 231:1038-42   CH * see problems

Knatterud G (1996) Tributes to Dr. Thomas C. Chalmers. Cont Clin Trials 17:473-4 * see also: (1996);17:471-2

Lau J (1996) Tributes to Thomas Chalmers. Ann Intern Med 124:696

Liu C (1989) The common cold. In: Infectious Diseases: A Modern Treatise of Infectious Processes, 4th edn [Hoeprich PD, Jordan MC, eds]. Philadelphia, PA: Lippincott. pp 288-93

Liu C (1994) The common cold. In: Infectious Diseases: A Treatise of Infectious Processes, 5th edn [Hoeprich PD, Jordan MC, Ronald AR, eds]. Philadelphia, PA: Lippincott. pp 336-41

Lorber B (1996) The common cold. J Gen Intern Med 11:229-36

Maclure M (1996) Dr. Tom Chalmers, 1917-1995. Can Med Assoc J 155:757-60  and 155: 986-8

NLM [National Library of Medicine] (2005d) Finding Aid to the Thomas C. Chalmers Papers, 1927-1995.

Ritzel G (1961) Kritische Beurteilung des Vitamins C als Prophylacticum und Therapeuticum der Erkältungskrankheiten [in German; Critical analysis of the role of vitamin C in the treatment of the common cold]. Helv Med Acta 28:63-8 TRANSLATION    Ritzel 1961 in German     Pauling note 30    Pauling note 33   Ritzel 1976 JAMA   Ritzel 1976 JAMA  Ritzel biography ch (129 KB)   Ritzel biography bm (86 KB)

Shils ME, Olson JA, Shike M, eds (1994) Modern Nutrition in Health and Disease, 8th edn. Malvern, PA: Lea & Febiger. pp 652-61

Sperber SJ, Hayden FG (1988) Chemotherapy of rhinovirus colds. Antimicrob Agents Chemother 32:409-19

Steele RW (1988) Antiviral agents for respiratory infections. Pediatr Infect Dis J 7:457-61

Thurnham DI, Bender DA, Scott J (2000) Water-soluble vitamins. In: Human Nutrition and Dietetics, 10th edn [Garrow JS, James WPT, Ralph A, eds]. London: Churchill Livingstone. pp 249-87

Warren, KS, Mosteller F, eds (1993) Doing More Good than Harm: The Evaluation of Health Care Interventions. Ann NY Acad Sci, vol. 703

Whitney EN, Rolfes SR (1993) Understanding Nutrition. St Paul, MN: West Publ. pp 332-5


Copyright:
© 2006-2009 Harri Hemilä. This text is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.  

Creative Commons License
Chalmers’ meta-analysis (1975) by Harri Hemilä is licensed under a Creative Commons Attribution 1.0 Finland License.
Based on a work at www.mv.helsinki.fi/home/hemila/reviews/chalmers.



Valid HTML 4.01 Transitional