Reproducibility — the ability to take the same data and analysis strategy and arrive at the same results — is one form of repeatability used to assess trust in research outputs, and a critical starting point in evaluating the credibility of published findings. Without reproducibility, it becomes difficult to assess the claims presented in a paper. But what can recent large-scale studies actually tell us about whether published results can be reproduced?

Here, we report on 3 complementary efforts that examine this question across different research contexts, offering some of the most comprehensive evidence to date.

META-REP

The META-REP program, a metascientific program funded by the German Research Foundation, has been systematically examining reproducibility across a range of behavioral, social, and cognitive science disciplines. A subproject focusing on research involving large-N observational data published its findings in a recent paper. Rather than sampling from a particular journal or discipline, the META-REP team used as their sampling frame publications drawing on the European Social Survey (ESS) — a publicly available, repeated cross-sectional survey covering 39 countries and more than half a million respondents. This approach allowed the researchers to examine reproducibility across disciplines including sociology, political science, and management science, many of which weren’t well-represented in prior reproducibility audits.

Among more than 1,100 articles identified, only 35% of authors shared code when contacted. Of those who didn’t share, the most common reason was that the code had been lost. Some authors indicated they hadn’t used code at all, on the assumption that standard analyses didn’t require it.