Reproducibility

Could someone else get my numbers?

That question is the one I find most useful to hold myself to, and it is harder than it sounds when your analysis depends on a public API whose contents change underneath you. This page sets out what I do, and where the honest limits are.

01 Data provenance

Every analysis states where its data came from and whether it is real, public, illustrative or simulated. That distinction appears in the project's status label, in its findings, and in its limitations — not in a footnote.

KindWhat it meansWhere it appears
Live public dataFetched from a named API at run time, on a stated genome assemblyEnsembl, gnomAD, PubChem, STRING
IllustrativeValues defined in the code to demonstrate a methodExpression workflow, drug–gene categories
SimulatedGenerated from a statistical model, with no biological claimAll GWAS projects, statistics studio
The rule I follow. If a reader could mistake simulated data for a real finding, the project has failed regardless of how good the code is. So the word "simulated" appears in the findings themselves, not only in the limitations.

02 Environments and dependencies

The published analyses are notebooks with their dependencies stated. For the pipeline work I propose, the standard is higher: pinned versions in a lockfile, a container image, and a workflow engine that records which step produced which file.

The gap between "it ran on my machine" and "it will run on yours next year" is mostly about pinning. An unpinned dependency is a slow-acting bug.

03 Randomness

Anything stochastic gets a fixed seed, and the seed is a visible parameter rather than a hidden constant. In the statistics studio the seed is a slider: you can move it and watch the "significant" hits change, which is a more honest demonstration of sampling variability than any paragraph I could write about it.

# Deterministic by construction
rng = numpy.random.default_rng(seed=7)

04 The live API problem

Several analyses query Ensembl at run time. This is good for freshness and bad for reproducibility: Ensembl adds transcripts between releases, so the same notebook can return a different count of transcript annotations months apart without anything being wrong.

This is not hypothetical. The CYP2C19 analysis recorded 66 transcript annotations when the notebook was run. The live Lab tool queries the same variant today and the consequence-term counts no longer match, because the annotation set has moved.

How I handle it. Results are reported with the assembly stated and a note that the counts track the Ensembl release. For work where exact reproducibility matters more than freshness, the right answer is to pin the release and cache the response alongside the analysis — which is what I would do for anything intended as a formal result rather than a demonstration.

05 Status honesty

Every project carries one of four labels, and I apply them strictly. Work in development is not described as published. Proposals do not have a results section, because they have no results.

PublishedComplete, documented
LiveIn production
In developmentPartly built
ProposalNot undertaken

06 What I don't claim

Some things that would be easy to imply on a portfolio site, and which are not true of this work:

  • No peer-reviewed publications. The research outputs here are independent work, not reviewed by anyone.
  • No clinical validation of anything. The pharmacogenomics work is educational and explicitly not a prescribing tool.
  • No performance metric appears anywhere unless a model was actually trained and evaluated. There are currently no such metrics on this site.
  • No claim that simulated results demonstrate biological findings.
  • Several published projects use small illustrative datasets, which is stated in their findings rather than only in the repository.
Why list this at all. A portfolio that only advertises strengths is hard to calibrate. If you can see exactly where I have drawn the line, the claims on the other side of it are worth more.
See the projects Ask me about any of it