Evidence-based medicine: Yet more concerns | Ear, Nose & Throat Journal Skip to content Skip to navigation

Evidence-based medicine: Yet more concerns

| Reprints
July 18, 2017
by Robert T. Sataloff, MD, DMA, FACS
Editorial

This journal has expressed reservations previously about the rush to evidence-based medicine (EBM).1 I have continued to follow writings on this topic with interest over the years, and my concerns about the strong emphasis on evidence-based research and the implication that it results in superior validity have not diminished. I am comforted to know I am not alone. Many authors have expressed skepticism, some in articles well worth reviewing. One example was a commentary published in 2016, “Evidence-based medicine has been hijacked: A report to David Sackett” by Dr. John Ioannidis.2 This article might have been missed by those of our readers who do not follow the Journal of Clinical Epidemiology. It was sent to me by my colleague Dr. Brian McKinnon. The article is both entertaining and insightful.

Dr. Ioannidis is an epidemiologist and professor of medicine with appointments in three departments and a Meta-research Innovation Center at Stanford University, who has studied EBM for more than a decade, and whose insights might be different (and potentially more sophisticated) than those of us in otolaryngology (certainly including this author). I will not recount his observations in detail in this editorial (as tempting as that is), because I would prefer to encourage our readers to review them for themselves. However, this editorial will summarize some of the highlights that I find credible, predictable, and somewhat disturbing.

My previous editorial noted that EBM has some intrinsic drawbacks and limitations, and that prospective, double-blind research is not the only literature worth publishing. Moreover, the fact that a study appears to meet the criteria for high-quality, evidence-based research does not necessarily mean that its conclusions are valid. Nevertheless, EBM may have a greater likelihood of producing valid (or at least defensible) results than other, less rigorous, research designs, and it has developed a certain cache. Hence, we should not be surprised with the argument put forth by Ioannidis that EBM has been “hijacked.”

Ioannidis is correct in noting that as EBM has become more influential, the temptation to exploit it for commercial purposes has grown. A large number of randomized trials currently are sponsored by, or actually done by, industries affected commercially by the results of the research. We used to joke (sort of) that a good statistician could prove anything. Good statisticians are intrinsic to EBM, and results can be biased by subtleties of study design and data analysis that might not be obvious to readers.

In my opinion, this might be particularly dangerous because there is a growing tendency for readers to believe that if an article meets the highest standards of evidence-based research, then it has been vetted already, and its conclusions can be believed. Therefore, readers might be tempted to read highly rated EBM articles less critically and less incisively than articles that do not have the imprimatur of EBM (I have no evidence to support this suspicion, of course, and it is not part of the wisdom put forth by Ioannidis).

Along with other EBM, Ioannidis attacks meta-analyses and guidelines as having “become a factory,” and also as largely serving “vested interests.” I agree with his observations that a great deal of funds (private and federal) are expended on projects, including some meta-analyses, that have little relevance to clinical medicine, especially to improving outcomes.

I would add that all of us should remember that meta-analyses combine data from a number of published studies. While this statistical technique can be valuable in detecting trends and extracting information missed because of smaller numbers in each component study, it does not correct fundamental design flaws and other errors in the studies that it includes. In fact, the technique can compound them. Worse, the fact that a publication is a “meta-analysis” might give it credibility similar to that of other EBM in the eyes of many readers who may believe conclusions that are supported statistically by the meta-analyses, but that are not correct.

All of our journals strive for quality, and EBM has become synonymous with quality in the eyes of many academicians, readers, and even some editors. All of us need to remain vigilant and to recognize that there is no approach to research that ensures truth in and of itself and that, like the results of all other investigations, products of EBM are subject to unintentional, or even intentional, bias. As readers, we still need to read with a critical eye.



References

  1. Sataloff RT. Evidence-based medicine Ear Nose Throat J 2006; 85 (10): 624-5.
  2. Ioannidis JPA. Evidence-based medicine has been hijacked: A report to David Sackett. J Clin Epidemiol 2016; 73:82-86.
Editor-in-Chief, Ear, Nose & Throat Journal
Ear Nose Throat J. 2017 July;96(7):234