# Prompt experiment run checklist

## Before the offline experiment

- [ ] Write one change hypothesis and one primary outcome.
- [ ] Record guardrails separately from the primary outcome.
- [ ] Freeze the dataset or record the exact dataset revision.
- [ ] Keep model, tool, retrieval, and generation settings fixed across prompt
      variants.
- [ ] Define how failures, missing outputs, and retries enter the analysis.
- [ ] Choose the analysis method before reading variant results.

## Before the production canary

- [ ] Confirm the candidate passed the offline dataset and manual failure
      review.
- [ ] Give the two prompt versions distinct Langfuse labels.
- [ ] Assign by a stable subject identifier, not independently on every request.
- [ ] Do not place raw user identifiers in traces or exported evidence.
- [ ] Record the intended candidate allocation and inspect the observed sample
      ratio.
- [ ] Define a stop condition for every safety or business guardrail.
- [ ] Confirm that the previous production label can be restored.

## Before a decision

- [ ] Check duplicate subjects, missing outcomes, retries, and allocation drift.
- [ ] Compare the primary outcome and every guardrail separately.
- [ ] Review latency, cost, and failure cases rather than only averages.
- [ ] Report uncertainty and the exact observation window.
- [ ] Do not call a small or underpowered difference “no effect.”
- [ ] Keep the candidate out of production when a critical guardrail worsens.
- [ ] Save the prompt versions, dataset revision, code revision, and analysis
      output.

Licensed under CC BY 4.0. Attribution details are in `LICENSE-CC-BY-4.0.md`.