Back to top

Readings

These are some initial readings about reproducibility and open science, the relationship between rigor and replicability, tools to improve research practices, challenges, and solutions.

Overview

  • Baker, Monya, and Dan Penny 2016. “Is There a Reproducibility Crisis?” Nature 533: 452–54.

Is Research Reproducible?

Theoretical

  • Ulrich, Rolf, and Jeff Miller 2020. “Questionable Research Practices May Have Little Effect on Replicability.” eLife 9: e58237. DOI: 10.7554/eLife.58237.

Empirical

  • Open Science Collaboration 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251). DOI: 10.1126/science.aac4716.
  • Stodden, Victoria, Jennifer Seiler, and Zhaokun Ma 2018. “An Empirical Analysis of Journal Policy Effectiveness for Computational Reproducibility.” Proceedings of the National Academy of Sciences 115 (11): 2584–89. DOI: 10.1073/pnas.1708290115.
  • Archmiller, Althea A., Andrew D. Johnson, Jane Nolan, Margaret Edwards, Lisa H. Elliott, Jake M. Ferguson, Fabiola Iannarilli, et al. 2020. “Computational Reproducibility in the Wildlife Societys Flagship Journals.” The Journal of Wildlife Management 84 (5): 1012–17. DOI: 10.1002/jwmg.21855.

Questionable Research Practices & Researcher Degrees of Freedom

  • John, Leslie K., George Loewenstein, and Drazen Prelec 2012. “Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling.” Psychological Science 23 (5): 524–32. DOI: 10.1177/0956797611430953.
  • Silberzahn, R., E. L. Uhlmann, D. P. Martin, P. Anselmi, F. Aust, E. Awtrey, Š. Bahnı́k, et al. 2018. “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results.” Advances in Methods and Practices in Psychological Science 1 (3): 337–56. DOI: 10.1177/2515245917747646.
  • Steegen, Sara, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel 2016. “Increasing Transparency Through a Multiverse Analysis.” Perspectives on Psychological Science 11 (5): 702–12. DOI: 10.1177/1745691616658637.

Data Sharing

  • Miyakawa, Tsuyoshi 2020. “No Raw Data, No Science: Another Possible Source of the Reproducibility Crisis.” Molecular Brain 13 (24). DOI: 10.1186/s13041-020-0552-2.

Statistics and Reporting

  • Weissgerber, Tracey L, Oscar Garcia-Valencia, Vesna D Garovic, Natasa M Milic, and Stacey J Winham 2018. “Why We Need to Report More Than ’Data Were Analyzed by t-Tests or ANOVA’.” eLife 7: e36163.

Credit

  • Lamprecht, Anna-Lena, Leyla Garcia, Mateusz Kuzak, Carlos Martinez, Ricardo Arcila, Eva Martin Del Pico, Victoria Dominguez Del Angel, et al. 2020. “Towards FAIR Principles for Research Software” 3: 37–59. DOI: 10.3233/DS-190026.
  • Moher, David, Lex Bouter, Sabine Kleinert, Paul Glasziou, Mai Har Sham, Virginia Barbour, Anne-Marie Coriat, Nicole Foeger, and Ulrich Dirnagl 2020. “The Hong Kong Principles for Assessing Researchers: Fostering Research Integrity.” PLoS Biology 18 (7): e3000737. DOI: 10.1371/journal.pbio.3000737.

Pipelines, Automation, & Tools

  • Beaulieu-Jones, Brett K, and Casey S Greene 2017. “Reproducibility of Computational Workflows Is Automated Using Continuous Analysis.” Nature Biotechnology 35 (4): 342. DOI: 10.1038/nbt.3780.
  • White, Ethan P., Glenda M. Yenni, Shawn D. Taylor, Erica M. Christensen, Ellen K. Bledsoe, Juniper L. Simonis, and S. K. Morgan Ernest 2019. “Developing an Automated Iterative Near-Term Forecasting System for an Ecological Study.” Methods in Ecology and Evolution 10: 332–44. DOI: 10.1111/2041-210X.13104.

Diversity, Equity, and Inclusion

  • Bahlai, Christie, Lewis Bartlett, Kevin Burgio, Auriel Fournier, Carl Keiser, Timothée Poisot, and Kaitlin Whitney 2019. “Open Science Isn’t Always Open to All Scientists.” American Scientist 107 (2): 78. DOI: 10.1511/2019.107.2.78.
  • Chan, Leslie, Angela Okune, Rebecca Hillyer, Denisse Albornoz, and Alejandro Posada, eds. 2019. Contextualizing Openness: Situating Open Science. University of Ottawa Press. https://www.idrc.ca/en/book/contextualizing-openness-situating-open-science.
  • Murphy, Mary C., Amanda F. Mejia, Jorge Mejia, Xiaoran Yan, Sapna Cheryan, Nilanjana Dasgupta, Mesmin Destin, et al. 2020. “Open Science, Communal Culture, and Women’s Participation in the Movement to Improve Science.” Proceedings of the National Academy of Sciences, 201921320. DOI: 10.1073/pnas.1921320117.

Solutions

  • Markowetz, Florian 2015. “Five Selfish Reasons to Work Reproducibly.” Genome Biology 16 (1). DOI: 10.1186/s13059-015-0850-7.
  • Nosek, B. A., G. Alter, G. C. Banks, D. Borsboom, S. D. Bowman, S. J. Breckler, S. Buck, et al. 2015. “Promoting an Open Research Culture.” Science 348 (6242): 1422–25. DOI: 10.1126/science.aab2374.
  • Lowndes, Julia S. Stewart, Benjamin D. Best, Courtney Scarborough, Jamie C. Afflerbach, Melanie R. Frazier, Casey C. O’Hara, Ning Jiang, and Benjamin S. Halpern 2017. “Our Path to Better Science in Less Time Using Open Data Science Tools.” Nature Ecology & Evolution 1 (6): 0160. DOI: 10.1038/s41559-017-0160.