Dear ISPE colleague,

The most recent retractions of two high-profile COVID-19 studies have reemphasized the volatility of the public and scientific acceptance of evidence from real-world data and the use of observational designs to make causal inferences about drug effects. The two papers, published in The Lancet and NEJM, used a data source, Surgisphere, which – quite surprisingly – most of us are not familiar with. Commentaries pointing to inconsistencies in the descriptive data that was presented, unclear data provenance and other concerns with measurement and analysis motivated the lead author to request an independent audit and re-analysis of the data, which was not granted by the data vendor. In the meantime, results were heavily politicized and those of us who had not read the articles may have considered these studies as excellent examples where RWE can provide rapid guidance in moments of crisis.

The retractions and their negative implications for the reputation of observational research require careful reflection upon what we as a Society, and each of us as pharmacoepidemiologists, can do to prevent conduct and publication of poor quality pharmacoepidemiologic research. Regarding conduct, we may want to point to the fact that no trained pharmacoepidemiologist was involved in either of these studies, but the wide availability of RWD and the rapidly expanding RWE field will render the proportion of such studies that has involvement of ISPE members small. In our new strategic plan, ISPE outlines increased outreach to other scientific disciplines, and ISPE's Real-World Evidence Task Force has several subgroups that are specifically focusing on such outreach. But what else can we do?

On behalf of the society, the RWE Task Force is working on a position statement with particular focus on integration of best pharmacoepidemiologic practices in the peer review process. The task force will also reach out to key scientific journals, such as The Lancet and NEJM, with the aim to reinforce this message and offer the ISPE membership as reviewers for high-profile RWE studies.

As individuals, we can ensure in our peer review activities that best practices were followed, and we should not be afraid to call out shortcomings if they could represent fatal flaws. Importantly, we should call for a peer reviewer with demonstrated experience with the RWE data source if we don’t have this experience ourselves. Assessment of data provenance and fit for purpose is difficult within the confined information provided in a manuscript. I am sure none of us would have pursued a study of such high impact without understanding the source(s), quality and validation efforts in Surgisphere and our message to collaborators and authors who consider such adventures should be a word of caution. Finally, we should strive to share our experience with data sources with the broader research community to maximize appreciation of strengths and weaknesses and enhance everyone's ability to assess what a data source can and absolutely cannot do.

The RWE Task Force welcomes other suggestions for how we can ensure use of rigorous methods and balanced assessments of RWE studies.

Warm regards,

Almut Winterstein