‘Scientific evidence’ part 2 – misplaced faith in meta analysis

“He uses statistics like a drunken man uses a lamppost, for support rather than illumination”

Mark Twain

img92In the absence of the context provided by natural laws, and the credibility conferred by replication, modern science has turned to the meta analysis approach to judge what ought to be taken as ‘best available’ evidence. Meta analysis is a statistical procedure that combines individual studies on a topic. The studies entered into the meta analysis often have a wide range of outcomes as each study contains different individuals and sometimes different ways of measuring the outcome of interest. The goal of meta analysis is to derive a new ‘overall’ result that supposedly represents one huge study with a much larger sample. It is the top of the evidence-based pyramid and is believed to provide the most trustworthy facts / evidence. However, like all statistical procedures, the output is only as good as the input. Given that there is heavy publication bias in journals towards positive effects, with replications and zero-effect studies not being represented (Rosenthal, 1979) meta analyses outcomes are also highly likely to be biased (Oakes, 1986). In the world of evidence-based practice, this is known as bias in = bias out, or the BIBO monster. This is using statistics for support not illumination.

So what is credible, trustworthy scientific evidence?

Trustworthy scientific evidence is:

  1. That which agrees with natural laws and
  2. After satisfying 1, that which has been replicated many times in different samples but with the same or a similar result.

Applying these simple filters can help anyone to avoid confusion in the face of equivocal research findings, and to avoid being duped by the latest craze / fad / fashion. The BTR model is based solely on scientific evidence fitting this definition i.e. trustworthy, credible and undisputed.

References.

Oakes, M. (1986). Statistical Inference: A commentary for the social and behavioural sciences. New York: John Wiley & Sons.

Rosenthal, R. (1979). The “File Drawer” problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641.