Ever looked at an old photo and winced at your clothes? For me – hyper-colour pants in primary school. Only later can we see how awkward our choices are. Ideologies and traditions can be fashions which encourage us to look for fashions in conservative places like clinical research. The limitations of our most trusted evidence-generating apparatus – the randomised control trial (RCT) – are being exposed by recent studies
Clinicians face the challenge of trying to apply the take-aways from an RCT manuscript to the patient who, not infrequently, neither looks, nor behaves, like the carefully selected trial participants. And, what if the trial, itself, is flawed?
Diana Herrera-Perez et al released a study in 2019 examining the stability of the findings of more than 3000 RCTs published across three major medical journals. Her team found 396 cases of “medical reversal” – that is, where a newer and methodologically superior clinical trial produced a result contradicting existing clinical practice and the older trials on which it was based; 13% of published studies were eventually reversed.
Evolving statistical models play a role. A 2018 study exposed the lack of concordance in modern statistical analysis. The research team gave the same data to 29 teams of statisticians; 69% produced analysis that was for the null hypothesis but 21% of teams produced data supporting the opposite result. Remarkably, all teams presented internally validated and rational statistical approaches. Deciding whether to include data outliers is a good example of where differences can emerge (are they aberrations or ‘real’ phenomena worth including?). The study demonstrated how subjective, yet defensible, analytical choices could influence research results.
In response to these tensions, major regulators such as the FDA and EMA are increasingly interested in real world evidence (RWE) as a complement to, or in lieu of, traditional clinical trial evidence.
RWE is clinical evidence derived from the careful analysis of real-world data (RWD) such as insurance claims, electronic health records, disease registries and patient-generated data. By definition, RWD is collected outside of clinical trials.
It is an exciting trend. Firstly, it’s becoming easier to incorporate digital health technology into our own practices and thereby contribute to the body of RWD. Secondly, as the analysis of this data improves, we can more effectively evaluate the efficacy and safety of treatments directly in the line-of-care and provide patients access to new treatments sooner.
As the limitations of RCTs become better understood, more pragmatic evidence-generating methods that absorb the real-world variances of patients and care delivery are welcome and, with luck, this kind of evidence-generating care can help expose our fashions a little earlier.
Key messages
- Medical research has its own fashions
- RCTs are not the only form of evidence
- The digitisation of health data provides new sources of evidence
- Real world evidence opens new doors
References available on request.
Questions? Contact the editor.
Author competing interests: nil
Disclaimer: Please note, this website is not a substitute for independent professional advice. Nothing contained in this website is intended to be used as medical advice and it is not intended to be used to diagnose, treat, cure or prevent any disease, nor should it be used for therapeutic purposes or as a substitute for your own health professional’s advice. Opinions expressed at this website do not necessarily reflect those of Medical Forum magazine. Medical Forum makes no warranties about any of the content of this website, nor any representations or undertakings about any content of any other website referred to, or accessible, through this website.