Design a small reproducible experiment¶
Use the ideas from noise and precision to answer a specific question: how does a rotation angle affect the probability of 1? Choose three angles before running. Keep the circuit structure and shot budget fixed, and use four separately seeded runs per angle.
Each row should retain its parameter value and all repeat results. A single best-looking outcome is not a summary of the experiment.
"""Repeat a small parameter scan and keep each seed and observed frequency."""
from __future__ import annotations
import json
from math import sin, sqrt
from statistics import mean, stdev
import cascaqit
from cascaqit import Circuit
def experiment() -> dict[str, object]:
shots, repeats = 256, 4
rows = []
for index, theta in enumerate((0.4, 0.8, 1.2)):
seeds = [101 + 10 * index + repeat for repeat in range(repeats)]
frequencies = []
for seed in seeds:
result = Circuit(1).ry(theta, 0).measure_all().run(shots=shots, seed=seed)
frequencies.append(result.counts.get("1", 0) / shots)
rows.append(
{
"theta_rad": theta,
"seeds": seeds,
"frequencies": frequencies,
"mean_frequency": mean(frequencies),
"standard_error_across_repeats": stdev(frequencies) / sqrt(repeats),
"ideal_probability": sin(theta / 2) ** 2,
}
)
return {
"sdk_version": cascaqit.__version__,
"shots_per_run": shots,
"repeats": repeats,
"total_shots": len(rows) * shots * repeats,
"points": rows,
}
if __name__ == "__main__":
print(json.dumps(experiment(), sort_keys=True))
python examples/learning/foundations/experimental_design.py
{
"points": [
{
"frequencies": [
0.0390625,
0.04296875,
0.0390625,
0.02734375
],
"ideal_probability": 0.039469502998557456,
"mean_frequency": 0.037109375,
"seeds": [
101,
102,
103,
104
],
"standard_error_across_repeats": 0.0033829117335329633,
"theta_rad": 0.4
},
{
"frequencies": [
0.16015625,
0.15625,
0.16796875,
0.1875
],
"ideal_probability": 0.1516466453264173,
"mean_frequency": 0.16796875,
"seeds": [
111,
112,
113,
114
],
"standard_error_across_repeats": 0.006951222820332885,
"theta_rad": 0.8
},
{
"frequencies": [
0.32421875,
0.33203125,
0.28515625,
0.30078125
],
"ideal_probability": 0.31882112276166324,
"mean_frequency": 0.310546875,
"seeds": [
121,
122,
123,
124
],
"standard_error_across_repeats": 0.010756973725168168,
"theta_rad": 1.2
}
],
"repeats": 4,
"sdk_version": "1.0.8a",
"shots_per_run": 256,
"total_shots": 3072
}
The mean frequency estimates sin²(θ/2). The script estimates the standard error of that mean as the sample standard deviation across repeats divided by sqrt(4). Four repeats give a noisy estimate of uncertainty; the printed value is not automatically a 95% confidence interval. All shots are still accounted for: three points × four runs × 256 shots = 3,072 shots.
The ideal probability is an independent analytical comparison for this simple circuit. It is not an experimental measurement. Record both the baseline and its assumptions when interpreting a difference.
Keep the comparison fair¶
Before trying another method, decide what to hold fixed: shots, total objective evaluations, elapsed time or another resource. These are different budgets. Equal iteration counts need not mean equal backend cost, especially for gradient or repeated-sampling algorithms.
Exercise: add θ = 1.6, with a new set of seeds. How many additional shots are needed? If one angle appears better than another after one run, what would you report before calling the difference reliable?
Answer
Another four runs require 1,024 shots. Report the raw runs, means, uncertainty, comparison budget and the meaning of “better.” Repeat a predeclared comparison rather than stopping only when a favorable result appears. If you later use paired random inputs, analyze the differences as paired data; do not pretend the runs are independent.
Save your own experiment record, then choose a Digital, Analog or optimization route.
Concept reference: reproducibility.