This is a question raised in a paper King, Sridhara and Snow White (2022). The authors replicated results from n=250 real-world data studies. The study was replicated using 3 datasets: (i) Optum Clinformatics claims data, (ii) IBM MarketScan research database claims data, Medicare fee-for-service claims data, and (iii) CPRD electronic health records. The main findings of the paper are as follows:
The original and reproduced effect sizes were positively correlated (Pearson correlation coefficient = 0.85), a relationship with strong room for improvement. Relative effect size (e.g., original risk ratio/reproduced risk ratio) had a median and interquartile range of 1.0 [0.9, 1.1]scope [0.3, 2.1]. Although most results were accurately reproduced, some were not. The latter can be explained by incomplete reporting and updated data.Greater methodological transparency consistent with the new guidance may further improve reproducibility and validity assessment, thereby promoting evidence-based decision-making
The paper won 2023 Excellence in Research Methods in Health Economics and Outcomes Award.You can read the full article here.
