Reproducibility of Research: Issues and Proposed Remedies
March 8-10, 2017; Washington, D.C.
Organized by David B. Allison, Richard Shiffrin, and Victoria Stodden
Overview
This colloquium was held in Washington, D.C. on March 8-10, 2017.
This colloquium brought together scientists and researchers from multiple disciplines to lay out the scope of the problem of reproducibility in a more tactical way that permits each problematic aspect to be measured, assessed for baseline levels, targeted with proposed interventions to reduce the occurrence, and monitored for improvement.
Videos of the talks are available on the Sackler YouTube Channel. Some speakers have delayed making their talks public due to publication embargos.
Agenda
March 8, 2017
Colloquium Call to Order, David Allison, University of Alabama at Birmingham
Welcoming Remarks, Marcia McNutt, President, National Academy of Sciences
Remembering Stephen Fienberg, Robert Groves, Provost, Georgetown University
Framing the Issues, Victoria Stodden, University of Illinois
Part 1 – TAXONOMY OF ISSUES AND CHALLENGES TO IMPROVING RESEARCH REPRODUCIBILITY
Issues of Design & Measurement, Kay Dickersin, Johns Hopkins University
Issues of Analysis, David B. Allison, University of Alabama at Birmingham
Taking Stock/Making Change: Surveying Practices, Assessing Perspectives, and Re-training Personnel as methods of supporting reproducibility in a learning health system, Madhu Mazumdar, Icahn School of Medicine at Mount Sinai Medical College
Incomplete Reporting in Primary Scientific Literature, Lehana Thabane, McMaster University
Honest Learning for the Healthcare System: Large-Scale Evidence from Real-World Data, David Madigan, Columbia University
Distortion of Research Results in Primary Scientific Literature and Beyond, Isabelle Boutron, Centre d'Epidémiologie Clinique, Hôpital Hôtel Dieu
PANEL DISCUSSION – Moderator: Andrew W. Brown, UAB
Panelists: Inder Verma (PNAS), Phil Campbell (Nature), Kelvin Droegemeier (Oklahoma University), Veronique Kiermer (PloS), Ginny Barbour (COPE)
Annual Sackler Lecture: The Challenge of Reproducibility in the Biomedical Sciences, Randy Schekman, University of California, Berkeley.
March 9, 2017—REMEDIES
Raw Data Depositing and Sharing, Victoria Stodden, University of Illinois
The Role of Registries and Repositories, Roberta W. Scherer, Johns Hopkins University
Statistical Power and Evidence in the Psychological Literature, Joachim Vandekerckhove, University of California, Irvine
Rigorous Assessment of Whether A Result can be Said to Have Been Replicated, Yoav Benjamini, Tel Aviv University
Training the Current and the Next Generations in Statistics, Emery Brown, Massachusetts Institute of Technology
Biomedical replication studies, Elizabeth Iorns, Science Exchange
Agents of Cultural Behavior Change, Susan T. Fiske, Princeton University
Post-Publication Peer-Review and Certificate Systems, Hilda Bastian, National Center for Biotechnology Information, NIH
Science Journalism and Public Dialog, Trevor Butterworth, Sense About Science USA
Government Perspectives, Catherine Woteki, Former Undersecretary for Research, Education and Economics, USDA
PANEL DISCUSSION – Moderator: Kathryn A. Kaiser, University of Alabama at Birmingham
Panelists: Daniele Fanelli, Stanford University; Jeffrey Flier, Harvard University
March 10, 2017—RESEARCH GOALS
The Problem with Remedies: The Need to Study Unintended Consequences, Richard M. Shiffrin, Indiana University, Bloomington
Minimizing Inadvertent Distortion of Science by Scientists and Media, Kathleen Hall Jamieson, University of Pennsylvania
Systems Matter: Research Environments and Institutional Integrity, C. K. Gunsalus, University of Illinois
Testing Interventions at the Levels of the Journal, David Moher, University of Ottawa
Funder’s Perspective, Richard Nakamura, NIH Center for Scientific Review
Improving Reproducibility, Brian Nosek, University of Virginia, Center for Open Science (COS)
New statistical approaches to reproducibility, Giovanni Parmigiani, Dana Farber Cancer Institute, Harvard University
Sharing scientific data and replicability, Keith Baggerly, MD Anderson Cancer Center
Closing Remarks, Richard Shiffrin, Indiana University, Bloomington
