diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index c1194c1..119f7cf 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -18,6 +18,13 @@ jobs: - name: Checkout code uses: actions/checkout@v4 + - name: Setup python env + uses: actions/setup-python@v6 + with: + python-version: '3.13' + cache: `pip` + run: pip install -r requirements.txt + - name: Set up Quarto uses: quarto-dev/quarto-actions/setup@v2 diff --git a/exercises/dependencies_exercises.qmd b/exercises/dependencies_exercises.qmd index 82ebe69..443debc 100644 --- a/exercises/dependencies_exercises.qmd +++ b/exercises/dependencies_exercises.qmd @@ -2,13 +2,61 @@ Returning to the Python scripts, try to answer the following questions: -1. What are the python dependencies? -2. What are the system dependencies? -3. What are the data dependencies? +::: {.panel-tabset} + +### What are the python dependencies? + +:::: {.columns} + +::::: {.incremental .column width=40%} + - `xarray` + - `matplotlib` + - `numpy` + - `netcdf4` / `h5netcdf`! +::::: + +::::: {.fragment .column width=60%} +The last dependency is tricky, because it is not required by `xarray`, unless you +want to read/write `netcdf` files! +If we include the dependecies of the dependencies, the list is even longer! +``` +ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see: +https://docs.xarray.dev/en/stable/user-guide/io.html +https://docs.xarray.dev/en/stable/getting-started-guide/installing.html +``` +::::: + +:::: + +:::: {.fragment} + +:::: + +### What are the system dependencies? + +- python3 +- netcdf +- lapack library +- blas library +- sqlite +- curl +- glibc +- gcc/gfortran +- mpdecimal +- libffi +- ... + +### What are the data dependencies? + +- HadCRUT 5.0.0, specifically: + - `HadCRUT.5.0.0.0_analysis_summary-series_global_annual.nc` + - `HadCRUT.5.0.0.0_analysis_summary-series_northern-hemisphere_annual.nc` + - `HadCRUT.5.0.0.0_analysis_summary-series_southern-hemisphere_annual.nc` + - `HadCRUT.5.0.0.0_analysis_summary-series_global_monthly.nc` ### Extension -Using your favourite dependency management tool (e.g. Conda, Nix), create a +1. Using your favourite dependency management tool (e.g. Conda, Nix), create a reproducible environment configuration to handle the python and system dependencies. - -Create a way to automatically fetch the data dependencies when using the scripts. +2. Create a way to automatically fetch the data dependencies when using the scripts. +::: diff --git a/exercises/version_control_exercises.qmd b/exercises/version_control_exercises.qmd index 4adbf48..aca9432 100644 --- a/exercises/version_control_exercises.qmd +++ b/exercises/version_control_exercises.qmd @@ -12,5 +12,5 @@ Specifically in the directory `exercises/problem`. Tasks: 1. Download the Python scripts from the GitHub repository -2. Create a new git repository on your local machine -3. Add the Python scripts to git for version control! +2. Create a new `git` repository on your local machine +3. Add the Python scripts to `git` for version control! diff --git a/flake.nix b/flake.nix new file mode 100644 index 0000000..13d53ec --- /dev/null +++ b/flake.nix @@ -0,0 +1,14 @@ +{ + description = "Reproducibility in Scientific Software environment"; + + inputs.flake-utils.url = "github:numtide/flake-utils"; + + outputs = { self, nixpkgs, flake-utils }: + flake-utils.lib.eachDefaultSystem + (system: + let pkgs = nixpkgs.legacyPackages.${system}; in + { + devShells.default = import ./shell.nix { inherit pkgs; }; + } + ); +} diff --git a/shell.nix b/shell.nix new file mode 100644 index 0000000..6fd05cb --- /dev/null +++ b/shell.nix @@ -0,0 +1,14 @@ +{ pkgs ? import { } }: +pkgs.mkShell { + packages = [ + pkgs.quarto + (pkgs.python3.withPackages (python-pkgs: with python-pkgs; [ + # select Python packages here + pandas + numpy + matplotlib + jupyter + jupyter-cache + ])) + ]; +} diff --git a/src/dependencies.qmd b/src/dependencies.qmd index b70961c..48950a5 100644 --- a/src/dependencies.qmd +++ b/src/dependencies.qmd @@ -2,16 +2,20 @@ ## Dependencies +\ + - All software has dependencies - Some are more obvious than others: - Data/input - - Packages/libraries e.g. numpy, Eigen + - Packages/libraries e.g. `numpy`, `Eigen` - System libraries - Compiler/Interpreter - If your code can't run without it, it's a dependency! ## How to discover dependencies +\ + - Some dependencies may be "implicit" - For example, you may have a library installed on your system - Since the code "just works", you may not be aware of the dependency @@ -19,32 +23,39 @@ ## How to declare dependencies +\ + - List them in a tracked file in the repository - e.g. add a "Dependencies" section to your README.md - Specify: - - Versions of each dependency e.g. numpy 2.3.9 + - Versions of each dependency e.g. `numpy` 2.3.9 - Where/how to aquire the dependency ## How to declare data dependencies +\ + :::{.columns} -:::{.column width="30%"} +:::{.column width="60%"} - Links to publicly accessible data dependencies - Zenodo or some other long term solution - Reference specific versions of the data + - Use DOI from hosting platform - Ideally automate the fetching of the data ::: -:::{.column width="70%"} -![Example of a zenodo listing. Note how each version of the dataset has a different reference](img/zenodo.jpg) +:::{.column width="40%"} +![Example of a zenodo listing. Note how each version of the dataset has a different reference](img/zenodo.png) ::: ::: ## Dependency metadata +\ + - There are automated ways of resolving dependencies - Usually language/tool specific - Some tools automatically update dependency metadata - - e.g. Rust's cargo, Julia's Pkg, uv for Python + - e.g. Rust's `cargo`, Julia's `Pkg`, `uv` for Python - Project file: Depencies and compatible versions - Lock file: Write exact version (plus other metadata e.g. source) of *every* dependency you are using @@ -52,6 +63,8 @@ ## System dependencies - Conda +\ + - Python centric package manager - Can be used for installing python packages - Also able to manage environments @@ -60,9 +73,12 @@ - even different python versions! - Available on most HPC systems (usually as Miniconda) - Can export environments to text-based files +- There are other alternatives, e.g. `pixi`, but `conda` is most available ## System dependencies - Containers +\ + :::{.columns} :::{.column width="60%"} - Containers package up code and all dependencies @@ -83,13 +99,15 @@ ## System dependencies - Nix/Guix +\ + :::{.columns} :::{.column width="60%"} -What they have in common: - Declarative package managers - Use pure functional languages to create environments - same inputs = same outputs + - Useful for caching artifacts - Packages are hashed - can lookup if package already exists locally - Publicly hosted and *versioned* package repository diff --git a/src/documentation.qmd b/src/documentation.qmd index c3bcfcf..9481a5e 100644 --- a/src/documentation.qmd +++ b/src/documentation.qmd @@ -9,11 +9,23 @@ ## README +:::{.columns} + +::::{.column width=50%} + +  + - Markdown file at the project root - Should contain: - Description of project - Dependencies - Instructions on building/running +:::: + +::::{.column width=50%} +![](img/ftorch-readme.png) +:::: +::: ## Comments @@ -28,3 +40,12 @@ - Single source of truth - Comments/Docstrings embedded in code - Reduce separation between code and docs + +## Licences + +- Licences are necessary for reproducibility! +- They define how code can be used/reused +- Enforce openness of code and data + - FOSS licenses eg. GPL, MIT, BSD, etc +- Many explanations of which licence to use exist (and I'm not a lawyer!) +- [choosealicense.com](https://choosealicense.com/) diff --git a/src/figs/perception.csv b/src/figs/perception.csv new file mode 100644 index 0000000..d738301 --- /dev/null +++ b/src/figs/perception.csv @@ -0,0 +1,5 @@ +question,strongly disagree,disagree,rather disagree,don't know,rather agree,agree,strongly agree +Most published science in my field is reproducible,8,29,24,2,24,11,3 +Reproducibility is a major problem in my field,4,15,20,2,16,29,15 +A lack of reproducibility is jepoardizing the trust in our results,6,13,20,2,24,27,10 +My own scientific work is reproducible,1,6,11,2,27,38,15 diff --git a/src/figs/reasons.csv b/src/figs/reasons.csv new file mode 100644 index 0000000..2423b06 --- /dev/null +++ b/src/figs/reasons.csv @@ -0,0 +1,7 @@ +question,strongly disagree,disagree,rather disagree,don't know,rather agree,agree,strongly agree +Workflow is poorly documented, 2, 6, 12, 3, 26, 34, 17 +Code is poorly documented, 2, 6, 11, 5, 24, 33, 19 +Code is too complex, 4, 11, 20, 6, 25, 22, 11 +Input data is not available, 3, 7, 13, 2, 16, 32, 27 +Code is not available, 2, 8, 12, 3, 20, 27, 28 +Code is written in a language I don't know, 10, 17, 21, 7, 17, 18, 10 diff --git a/src/img/ftorch-readme.png b/src/img/ftorch-readme.png new file mode 100644 index 0000000..b385863 Binary files /dev/null and b/src/img/ftorch-readme.png differ diff --git a/src/img/reprohack-hub.png b/src/img/reprohack-hub.png new file mode 100644 index 0000000..856daa7 Binary files /dev/null and b/src/img/reprohack-hub.png differ diff --git a/src/img/zenodo.jpg b/src/img/zenodo.jpg deleted file mode 100644 index b3fee9e..0000000 Binary files a/src/img/zenodo.jpg and /dev/null differ diff --git a/src/img/zenodo.png b/src/img/zenodo.png new file mode 100644 index 0000000..fb188c6 Binary files /dev/null and b/src/img/zenodo.png differ diff --git a/src/introduction.qmd b/src/introduction.qmd index e73c249..51f76ed 100644 --- a/src/introduction.qmd +++ b/src/introduction.qmd @@ -12,6 +12,8 @@ We will use the definitions used in [The Turing Way](https://book.the-turing-way ## What is reproducibility? +\ + For a result to be reproducible, it should be possible for someone else to: - Access your code @@ -33,11 +35,107 @@ By making our work reproducible, we ensure that both these things are not just possible, but straightforward! ## Reproducibility in Earth sciences -\ -Poll of researchers in Earth sciences: -![](/img/es_reproducibility_perception.jpg) -![](/img/es_reproducibility_reasons_only.jpg) +Perception of reproducibility: +``` {python} +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt + +positive_colours = ["#607c40", "#a4c484", "#e2ecd9"] +negative_colours = ["#ab5445", "#f99a89", "#ffe0da"] +neutral_colour = "#949482" + +positive_labels = ["rather agree", "agree", "strongly agree"] +negative_labels = ["rather disagree", "disagree", "strongly disagree"] + +df = pd.read_csv("figs/perception.csv") + +fig, axs = plt.subplots(4,1, sharex=True) + +for i, row in df.iterrows(): + question = row["question"] + + positive = [row["rather agree"], row["agree"], row["strongly agree"]] + negative = [row["rather disagree"], row["disagree"], row["strongly disagree"]] + neutral = row["don't know"] + + # Cumulative sum, but reverse so taller bars are behind smaller bars + positive_bars = np.flip(np.cumsum(positive)) + negative_bars = np.flip(np.cumsum(negative)) + # Don't need to do this for neutral as it is just one number + + + axs[i].barh("Positive", positive_bars, color=positive_colours, label=positive_labels[::-1]) + axs[i].barh("Negative", negative_bars, color=negative_colours, label=negative_labels[::-1]) + axs[i].barh("Neutral", neutral, color=neutral_colour, label="don't know") + + axs[i].yaxis.set_inverted(True) + axs[i].yaxis.set_visible(False) + axs[i].set_title(question) + +fig.subplots_adjust(hspace=0.5) + +axs[-1].set_xlabel("Responses, %") +axs[0].legend(ncols=7, bbox_to_anchor=(0.5, 2), + loc='upper center', fontsize='small') + +plt.show() +``` + +::: footer +[R Reinecke et al (2022)](https://iopscience.iop.org/article/10.1088/1748-9326/ac5cf8) +::: + +## Reproducibility in Earth sciences + +Reasons for non-reproducibility: +``` {python} +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt + +positive_colours = ["#607c40", "#a4c484", "#e2ecd9"] +negative_colours = ["#ab5445", "#f99a89", "#ffe0da"] +neutral_colour = "#949482" + +positive_labels = ["rather agree", "agree", "strongly agree"] +negative_labels = ["rather disagree", "disagree", "strongly disagree"] + +df = pd.read_csv("figs/reasons.csv") + +fig, axs = plt.subplots(6,1, sharex=True) + +for i, row in df.iterrows(): + question = row["question"] + + positive = [row["rather agree"], row["agree"], row["strongly agree"]] + negative = [row["rather disagree"], row["disagree"], row["strongly disagree"]] + neutral = row["don't know"] + + # Cumulative sum, but reverse so taller bars are behind smaller bars + positive_bars = np.flip(np.cumsum(positive)) + negative_bars = np.flip(np.cumsum(negative)) + # Don't need to do this for neutral as it is just one number + + + axs[i].barh("Positive", positive_bars, color=positive_colours, label=positive_labels[::-1]) + axs[i].barh("Negative", negative_bars, color=negative_colours, label=negative_labels[::-1]) + axs[i].barh("Neutral", neutral, color=neutral_colour, label="don't know") + + axs[i].yaxis.set_inverted(True) + axs[i].yaxis.set_visible(False) + axs[i].set_title(question) + +fig.subplots_adjust(hspace=1.0) + + +axs[-1].set_xlabel("Responses, %") +axs[0].legend(ncols=7, bbox_to_anchor=(0.5, 2.5), + loc='upper center', fontsize='small') + +plt.show() +``` ::: footer [R Reinecke et al (2022)](https://iopscience.iop.org/article/10.1088/1748-9326/ac5cf8) @@ -63,12 +161,12 @@ Other findings: - By making our software more reproducible, we also get to reap the benefits! :::{.columns} -:::{.column width="50%"} +:::{.column .fragment width="50%"} ### Safely implement changes - Automation and testing help catch mistakes before they become an issue ::: -:::{.column width=50%} +:::{.column .fragment width=50%} ### Rerun analyses - Create reusable pipelines @@ -77,13 +175,13 @@ Other findings: ::: :::{.columns} -:::{.column width=50%} +:::{.column .fragment width=50%} ### Faster onboarding - Better documentation on how to use the software - Relevant metadata present - no hunting down dependencies ::: -:::{.column width=50%} +:::{.column .fragment width=50%} ### Better collaboration - Version control for keeping track of code history @@ -93,7 +191,7 @@ Other findings: ## Where do we go from here... - +\ Throughout the rest of this session, we will walk through the steps that we can take to go from an ad hoc collection of scripts into a reproducible scientific workflow! diff --git a/src/reproducibility_initiatives.qmd b/src/reproducibility_initiatives.qmd index 75af3b5..14cf6b7 100644 --- a/src/reproducibility_initiatives.qmd +++ b/src/reproducibility_initiatives.qmd @@ -30,12 +30,23 @@ thing (internationally). --- -## ![](img/riots.png){width=1.5cm} RIOT Science +## ![](img/riots.png){style="width: 5%"} RIOT Science + +:::: {.columns} + +::: {.column width="50%"} +- **R**eproducible **I**nterpretable **O**pen **T**ransparent - Groups at universities (mostly Psychology, mostly UK) - Conferences, events, seminars open to everyone -![](img/RIOT_principles.png){style="width:80%; fig-align:center;"} +::: + +::: {.column style="width: 50%; fig-align: center; "} +![](img/RIOT_principles.png){style="width: 80%; background-color: white;"} +::: + +:::: --- @@ -56,26 +67,42 @@ thing (internationally). ::: --- -## ![](img/acm.png){width=10%} ACM Reproducibility Badges +## ![](img/acm.png){style="width:20%; vertical-align:text-bottom;"} ACM Reproducibility Badges   -![](img/acm_badges.jpg){width=90%, fig-align="center"} +:::: {.columns} + +::: {.column width="50%"} +- Applied to research artifacts e.g. papers +- Used across ACM journals and conferences +- Rewards researchers for reproducibility efforts +::: + +::: {.column width="50%"} +![](img/acm_badges.jpg){style="width:90%; fig-align:center"} +::: + +:::: --- ## ![](img/sc26.png){width=10%} SC Reproducibility Initiative +  + - SC (formerly Supercomputing), The International Conference for High Performance Computing Networking, Storage, and Analysis -- Initiative started in 2015, Artifact Descriptions (ADs) optional for the -first years -> used in Student Cluster Competition +- Initiative started in 2015 + - Artifact Descriptions (ADs) optional for the first few years + - used in Student Cluster Competition - Then gradually made mandatory for more categories/prizes - Computational Results Analysis (CRA) -> Artifact Evaluation (AE) appendix still optional + - Computational Results Analysis (CRA) + - Artifact Evaluation (AE) appendix still optional -- AD/AE committee evaluates appendices and recommends ACM badge awards (IEEE badges seem to have vanished) +- AD/AE committee evaluates appendices and recommends ACM badge awards - Reproducibility challenge introduced 2021 @@ -83,7 +110,7 @@ first years -> used in Student Cluster Competition ## ![](img/reprohack_logo.jpg){width=5%} ReproHack -- Challenge: Reproduce the results of a paper in one day! +- __Challenge__: Reproduce the results of a paper in one day! - Started in 2016 and 2017 as satellite events of OpenCon (inspired by a course by Owen Petchey) @@ -93,12 +120,17 @@ course by Owen Petchey) - More events, a team formed, remote ReproHacks became a thing.... - ReproHack Hub launched in 2021 - - Material and checklists for organisers - - Paper database - - Evaluation forms - - Events listing - - Support through ReproHack Slack - + +### ReproHack Hub + +- Material and checklists for organisers +- Paper database +- Evaluation forms +- Events listing +- Support through ReproHack Slack + +![](img/reprohack-hub.png) + --- ## And more! @@ -114,8 +146,3 @@ course by Owen Petchey) - Climate Informatics Reproducibility Challenge - ... - - - - - diff --git a/src/slides.qmd b/src/slides.qmd index c8565c4..f742a10 100644 --- a/src/slides.qmd +++ b/src/slides.qmd @@ -10,6 +10,11 @@ format: authors: - name: Jack Franklin - name: Marion Weinzierl + +jupyter: python3 + +execute: + cache: true --- {{< include introduction.qmd >}} @@ -37,21 +42,29 @@ authors: ## Reproducibility is important Primary benefits: + - Confidence in scientific results - Peer review/cross analysis Additional benefits: -- Allows for code resuse + +- Allows for code reuse - Better collaboration ## Ingredients for reproducibility: +  + - Version Control - Dependency Metadata - Public Accessibility + - Licence +- Documentation ## Even better if +  + - Testing for: - Verification - Regression checks diff --git a/src/testing.qmd b/src/testing.qmd index 55e792c..0221e71 100644 --- a/src/testing.qmd +++ b/src/testing.qmd @@ -24,18 +24,14 @@ ## A very short intro to testing ---- - -## Unit tests +### Unit tests - Test the smallest logical unit of the code - Ensure each component works as intended - Test functions for known results - Compare to previously produced results ---- - -## Integration tests +### Integration tests - Test that components work together - Try to have a range of complexity of tests diff --git a/src/version_control.qmd b/src/version_control.qmd index f8ac915..3904194 100644 --- a/src/version_control.qmd +++ b/src/version_control.qmd @@ -2,6 +2,8 @@ ## Version Control +\ + - The first thing we should do is move our project into version control (VC) - This way we never lose the original state of the project - We can then try things without worrying about breaking anything! @@ -9,10 +11,12 @@ ## What to add to VC +\ + - DON'T do this: -``` bash -git add . -``` + ```console + git add . + ``` - Our repository should only contain: - Code/scripts @@ -25,14 +29,18 @@ it will be true. ## What to add to VC +\ + - Large datafiles should be hosted separately (e.g. on Zenodo) - External dependencies should be declared - e.g. link to Zenodo dataset in docs and code -- Use .gitignore to automatically ignore any unwanted files +- Use `.gitignore` to automatically ignore any unwanted files - e.g. build outputs ## Aside - testing with worktrees +\ + - git worktrees are like "local clones" of a repository - Create a worktree: ``` bash @@ -44,8 +52,8 @@ git worktree add -b ## What to do next? +\ + - The repository can then also be hosted a remote service (e.g. GitHub, GitLab, Codeberg, Bitbucket) - This will make collaboration with other people a lot easier! - It will also mean that any work done can be accessed by collaborators - -