# Protocol v1 → v2: what the rehearsal changed

2026-10-07. Sources:
- 60 synthetic sessions (`analysis/summary.txt`).
- The automated synthesis (`analysis/synthesis.json`, Opus), whose 11 protocol issues and 6 usability issues
  were each adjudicated below.
- Our own reading of the scored results.

Synthetic participants test the protocol. None of this is evidence about real users.

## What the rehearsal showed about the study

| Finding | Evidence (matched pairs) | Consequence |
|---|---|---|
| T2 (budget) discriminates sharply | A 1/5 correct, B 5/5. Four A answers gave the figure without exercise | Keep, but fix the wording (see C3) |
| T4 (move) discriminates | A 2/3 success, median 9 taps; B 3/3, 3 taps. One A session added a duplicate | Keep. Define success by final state (C4) |
| T1 as written barely measures glance cost | A median 1 tap vs B 0. A participant can open a single meal | Rewrite as a whole-day review (C1) |
| Accidental copies never occurred | 0 in 60 sessions, including two personas built to trigger them (P2, P5) | Synthetic agents are more careful than people. Only real participants can test H4. Add observation codes (C8) |
| Diet-tracking content triggers model safety refusals | 20 of 80 sessions refused, 14 of them in A | Analyze matched pairs only. Plan for refusals in any synthetic rehearsal of a health app |

## Planted predictions (written before any session)

| Persona | Prediction | Result |
|---|---|---|
| P1 | T1: more taps in A | Detected, weakly (1 vs 0) |
| P2 | T5: accidental copy in A | Not detected |
| P3 | T2: wrong in A, right in B | Detected |
| P4 | T4: failure or 3+ more taps in A | Not tested (A sessions refused) |
| P5 | Any accidental change in A | Not detected |
| P6 | Succeeds everywhere, more taps in A on T1 and T4 | Not tested (sessions refused) |
| P7 | Control | No accidental copies. One failure on T2 in A, a design effect |
| P8 | T3 succeeds in both | Not tested (A refused) |

Detected 2, not detected 2, not tested 3, and the control was clean. The misses matter: the protocol's measure of
accidental copies cannot be validated with synthetic participants.

## Adjudication of the automated synthesis

The synthesis raised 11 protocol issues. Issue 5 (answer keys) contained a separate factual claim about P8-B-T5,
which is adjudicated on its own row (S4), so there are 12 rows. Outcome: 3 issues rejected outright (S1–S3), 1 claim
rejected inside an otherwise accepted issue (S4), 7 accepted, and 1 partly accepted.

| # | Synthesis finding | Decision | Reason |
|---|---|---|---|
| S1 | Data differs across sessions (turkey vs tuna, 310 vs 280) | **Rejected** | By design. Two matched datasets are counterbalanced so participants cannot answer from memory. The answer key is per dataset |
| S2 | Uneven task coverage across personas | **Rejected** | An artifact of model refusals, not of the protocol. Handled by the matched-pair analysis |
| S3 | Element IDs reused across screens | **Rejected** | The log already records the screen (`view`) and the control label for every tap |
| S4 | T5 scored success without naming items (P8-B-T5) | **Rejected** | Checked: every B answer names the dinner items. The scorer already requires them |
| S5 | Scenario date differs from the session date | Accepted → C7 | |
| S6 | T4 success not defined by final state | Accepted → C4 | |
| S7 | Answer key and "hedged incorrect" code for T2 and T5 | Accepted → C5 | |
| S8 | T3 success relies on inference; record verification | Accepted → C6 | |
| S9 | Effort under-reported: scrolls not counted, no time | Partly accepted → C9 | Scrolls and timestamps were already logged. Effort now counts both |
| S10 | B features untested (multi-select, undo, recovery) | Accepted → C2 | |
| S11 | T2 wording primes the participant to look for exercise | **Accepted → C3** | The most important catch. v1 measured distrust of the summary, not whether people notice |
| S12 | Baseline "left" ignores exercise. Confirm production does this | **Accepted → C10** | If production differs, the T2 baseline failure would be manufactured by the prototype |

## Changes in v2

- **C1. T1 rewritten as a whole-day review.** "Tell me everything you've logged today, and how many calories
  each meal came to." Report correctness and time, not taps alone.
- **C2. New T6 (recovery).** "You just copied the wrong meal into today. Put today back the way it was."
  Exercises undo and multi-select in B.
- **C3. T2 neutral wording.** "How many calories do you have left today?" After the answer, the moderator asks:
  "Does that include your workout?" Code unprompted versus prompted correction.
- **C4. T4 success by final state.** The item appears exactly once, under Snacks, with its original serving, and
  is gone from Lunch. "Success with workaround" (delete and re-log) is coded separately.
- **C5. Answer keys per dataset.** T2 and T5 are coded correct, incorrect, or hedged-incorrect.
- **C6. T3 verification.** Code whether the participant checks the result, and their confidence (1–5).
- **C7. Scenario date** stated in the intro ("Imagine today is Thursday, October 8").
- **C8. Accidental-copy observation codes:** hover or hesitation at the chip, tap and undo, unnoticed copy. The
  log captures copies automatically, but only real participants can test H4.
- **C9. Effort** = taps + scrolls + back navigations, plus time on task.
- **C10. Fidelity check before fielding.** Verify the baseline's summary formula and the meal-card behavior against
  the live app on the study date, and document it in the condition spec.