EC-17.3 Reproducibility and Peer Review

What this is and why it exists

Reproducibility is presented here as an engineering property rather than a scientific virtue, because that is what makes it actionable.

A result nobody can repeat is a result you cannot build on, including your own result from six months ago. That is the argument, and it applies to a laboratory measurement, a simulation and a piece of software equally.

Reviewing is included because giving and receiving review well is a learnable skill and is almost never taught.

The vocabulary

• Reproducible — able to be repeated by someone else, from what you published.
• Provenance — the record of where a result came from and how it was produced.
• Version pinning — recording the exact versions of tools and libraries used.
• Raw data — the measurements as recorded, before any processing.
• Selection effect — a result shaped by unconscious choices about what to include.
• Constructive review — feedback that ranks findings and says what would change the assessment.

The mental model

A result is reproducible when a competent stranger could repeat it from what you wrote. Five things are needed.

The method, in enough detail to follow. The exact conditions. The data. The tools, with their versions. And any choices you made along the way.

Anything a reader would have to guess at is a place where their attempt can differ from yours without either being wrong. That is the useful test, and applying it to your own write-up usually finds two or three gaps.

Packaging your own work is the practical half. Scripts rather than remembered steps. Recorded versions rather than latest. Raw data kept alongside the processed results.

The first person this helps is you. Six months later you will not remember which setting produced the good figure. A script that regenerates it from raw data answers in a minute what memory cannot answer at all.

Results fail to reproduce for four reasons, and none of them involves anybody lying.

An undocumented setting, which was obvious at the time and is not now. A difference in a tool version, where a library changed a default. A selection made unconsciously, such as discarding a run because something felt wrong with the setup. And a result that was always partly chance, and happened to come out well the day it was recorded.

All four are common, and recognising them is more useful than assuming dishonesty, which is rare and would not explain most cases.

Reviewing someone else's work usefully has a shape. Say what the work does well, say what would change your assessment, and say what is unclear.

Note what that excludes. A review that lists faults in order of discovery is less useful than one that ranks them, because the author cannot tell what matters. And a review about your taste in presentation is not a review of the work.

Receiving a review is the other half and it is harder. Separate the reviewer's conclusion from their reasoning, and answer the reasoning.

A reviewer who misread something has usually found a place where the writing was unclear, which is still your finding and still worth fixing. Treating a misreading as the reviewer's fault discards the most useful signal in the review.

Finally, the limits, stated honestly. Review catches unclear reasoning, missing controls and overstated conclusions.

It rarely catches fabricated data, and it almost never catches a subtle equipment fault. A reviewer sees what you wrote rather than what you did. Treating peer review as a guarantee of correctness is a mistake, and it is one that both authors and readers make.

What you should now be able to explain or do

• List the five things a reproducible result must provide.
• Package your own work so that you could repeat it in six months.
• Name the four reasons results fail to reproduce without anyone being dishonest.
• Write a review that ranks findings and says what would change your assessment.
• Receive a review by answering the reasoning rather than the conclusion.
• State what review does and does not catch.

Check yourself