Make an experiment reproducible¶
Reproduction needs the program, configuration, data and analysis method. A seed is only one part of the record; it cannot recover a changed algorithm version, parameter ordering or dependency environment.
| Keep | Question it helps answer |
|---|---|
| SDK version and commit, Python and numerical dependency versions | Was the same implementation used? |
| Original program, bindings, units and target specifications | Were inputs and physical assumptions the same? |
| Requested and applied execution settings | Which method, precision, concurrency and budgets took effect? |
| Root seed, parameter order and repeat identifiers | How were random inputs assigned? |
| Raw results, failed points and execution costs | Were all observations retained? |
| Reference method, predetermined metrics and analysis code | How was the conclusion obtained? |
Scan child seeds depend on point indices, so changing point order can change counts. Even with the same seed, dependency or execution changes need not preserve every random integer or floating-point bit. Compare input meanings first, then use appropriate numerical tolerances and statistical checks.
Prepare a record someone else can rerun¶
Keep raw JSON and the script that creates the report. The report supports reading; raw results support inspection. Distinguish analytic values, numerical references, sample estimates and conclusions. If using local job history, retain the database and its result directory; see job recovery.
Before comparing methods, specify whether the budget fixes total shots, objective evaluations, time or another cost. These are not interchangeable. Choose the number of repeats in advance and retain unfavorable outcomes and failures instead of reporting only the best run.
The experimental-design lesson gives a small repeated experiment, and learning routes includes a recording template. Reports and raw data in the MIS project provide an example of a reproducible submission.