Teaching Open Science and Reproducibility

Hanne Oberman

Junior assistant professor at Utrecht University

TODOs:

  • discuss how Open Science (OS) was adopted as core principle for education at Utrecht University (UU)
  • showcase teaching materials from the Methodology & Statistics (M&S) department on reproducibility in statistics and data science
  • engage with the audience to exchange ideas about teaching reproducibility

Open Science at UU

Open Science Programme

uu.nl/en/research/open-science

COSiE

openscience-utrecht.com/cosie

OER on OS

openscience-utrecht.com/cosie

Teaching reproducibility
in M&S courses

OS as mandatory element


Showcase: Markup Languages and Reproducible Programming in Statistics

mlarpis.github.io/markup

Markup Languages and Reproducible Programming in Statistics

Course aims include the development of a publication-ready reproducible research compendium that contains:

  • a typeset manuscript following a markup language,
  • data and code,
  • everything that allows for successful reproduction and reuse of the materials (e.g. a license).

Full course aims

  1. Students develop fundamental knowledge and understanding in the state of the art in statistical markup languages and reproducible programming and development

  2. They can determine the most effective markup strategies to address a typesetting problem

  3. They can efficiently organize a reproducible programming and development process

  4. They can produce repositories up to the standards of international programming and coding conventions and initiatives

  5. They can produce publications up to the typesetting standards of international peer-reviewed journals

Markup languages

Version control

Licensing

Sharing R code

Research compendiums

Research compendiums (counterfactual)

Discussion

Which resources to rely on?

forrt.org

How to convince administrators?

The Turing way

🗣️

Outro

Take aways

  • reproducibility is important
  • we should all learn reproducible workflows
  • we should teach reproducible workflows

Thank you!

References

COSiE: Community for Open Science in Education (n.d.). Open Science Community Utrecht. URL: https://openscience-utrecht.com/cosie/

Eglen, S., & Nüst, D. (2024). CODECHECK. URL: https://codecheck.org.uk

FORRT - Framework for Open and Reproducible Research Training (n.d.). URL: https://forrt.org/

Mosteiro P., Oberman H.I. (2026). Markup Languages and Reproducible Programming in Statistics (2024). Course materials. URL: https://www.mlarpis.github.io/markup

The Carpentries (n.d.). The Carpentires. URL: https://carpentries.org/

The Turing Way Community (2025). The Turing Way: A handbook for reproducible, ethical and collaborative research. URL: https://doi.org/10.5281/zenodo.15213042

References (cont.)

Utrecht University (2023). Best Practices for Writing Reproducible Code. URL: https://utrechtuniversity.github.io/workshop-computational-reproducibility

Utrecht University (2023). Writing Reproducible Manuscripts in R & Python. URL: https://utrechtuniversity.github.io/workshop-reproducible-manuscripts

Utrecht University (2024). The educational model. URL: https://www.uu.nl/en/education/educational-vision/the-educational-model

Utrecht University (2026). Markup languages and reproducible programming in statistics (202000010). Osiris course catalogue. URL: https://osiris-student.uu.nl/#/onderwijscatalogus/extern/cursus?cursuscode=202000010&taal=en&collegejaar=huidig

Poster

Mini lecture: Reproducibility

Why bother?

We would like our results to be as fully reproducible as possible:

A. Reproducibility is one of the pillars of science

  • It is the standard by which to judge scientific claims
  • It helps the cumulative growth of knowledge - no duplication of effort

B. Reproducibility may greatly benefit you

  • You’ll develop better work habits
  • Better teamwork - especially with new team members
  • Changing or amending your work is much easier
  • Higher research impact - more likely to be picked up and cited

Crisis?

Baker (2016)

Definition

A result is reproducible when the same analysis steps performed on the same dataset consistently produces the same answer.

The Turing way

Definition

Research results are replicable if there is sufficient information available for independent researchers to make the same findings using the same procedures.

King (1995)

True or false?

In computational sciences - such as statistics - simply having the data and code means that the results are not only replicable, but fully reproducible.

Reproducibility of R scripts

Reproducible research is not the norm:

74% of R files failed to complete without error

Trisovic, Lau, Pasquier, et al., 2022

Reproducibility spectrum

Peng (2011)

Making research reproducible

  • reproducible documents
  • research compendiums

Reproducible but not replicable

set.seed(1)
run_simulation()
set.seed(2)
run_simulation()
set.seed(3)

Mini lecture: Research Compendiums

Research compendium

DOI

Definition

A research compendium is a collection of all digital parts of a research project including data, code, texts…

The collection is created in such a way that reproducing all results is straightforward1


The compendium serves as a means for distributing, managing, and updating the collection2

Basic compendium

A basic research compendium is just a folder…

compendium/
├── data
│   └── my_data.csv
├── analysis
│   └── my_script.R
├── requirements.txt
└── README.md

(Not so) basic compendium

… but it can become extensive…

|
├── paper/
│   ├── paper.qmd       
│   └── references.bib  
| 
├── figures/            
|
├── data/
│   ├── raw_data/       
│   └── clean_data/   
|
└── templates
    └── journal_template.csl

(Not so) basic compendium

…or even executable!

|
├── _targets.R
├── R/
│   ├── functions_data.R
│   ├── functions_analysis.R
│   ├── functions_visualization.R
└── data/
    └── input_data.csv

(Not so) basic compendium

Peng (2011)

Guidelines

  • Completeness
  • Organization
  • Economy
  • Transparency
  • Documentation
  • Access
  • Provenance
  • Metadata
  • Automation
  • Review

In practice

Research Data Management Support workshop:

Writing Reproducible Manuscripts in R and Python

Compendium step-by-step

  • Think about a good folder structure
    • Split up ‘read-only’, ‘human-generated’, and ‘project-generated’ files
  • Create folder structure (main directory and sub directories)
    • Add a landing page in the form of a README document
    • Make the compendium executable (to automatically generate the results; optional)
    • Make the compendium into a git repository (optional)
  • Add all files needed for reproducing the results of the project
    • Avoid ‘hard coded’ parameters or human intervention in the execution
  • Make the compendium as clean and easy to use as possible
    • Include a citation file and a LICENSE file with info on how it can be used
  • Publish your compendium
    • E.g. on Zenodo (optional, more on this in the last course week)

Adapted from The Turing Way