Protocol revisions from synthetic rehearsal

Ten protocol corrections identified before human recruitment

Synthetic participants ran the real prototype. The protocol changed in ten places, including a task that pointed people to the answer.

MyFitnessPal compact day view · usability study design and AI-assisted rehearsal · follows the review study

Basis in the diary-redesign review analysis

Public reviews of the diary redesign pointed to design costs in repeated logging. The next step was a usability test of a compact day view with established loggers. This case designs that study and tests it before it runs.

Condition A prototype — A · current structure

Condition B prototype — B · compact view

Comparative usability study objectives

Evidence required to evaluate an optional compact day view for established loggers

Research question	Hypothesis	From the review analysis
Daily review time and interaction effort	Less effort and time in B	Day not visible at a glance
Accuracy of remaining-calorie interpretation	Higher accuracy in B	Budget harder to inspect
Copying and moving task performance	Fewer taps, higher success in B	Removed or narrowed functions
Frequency of unintended copying	Fewer unintended copies in B	Accidental "same as yesterday"
Comprehension under increased information density	Guardrail: any drop blocks the change	Risk of the fix itself

Study design

Proposed counterbalanced comparison with twelve established loggers

Participants12 people, 2+ years of use, logging 5+ days a week, mostly on the phone app. iOS and Android mix. UX, product, and competitor employees excluded.

SessionModerated, remote, 60 minutes, on the participant's own phone.

DesignEveryone tries both designs. Half see A first, half see B first. Two matched food logs, so no one answers from memory.

Condition AAn unverified reconstruction of the Today interface, with a calorie card that excludes exercise, content above the diary, collapsed meals, and a same-as-yesterday shortcut on each meal.

Condition BThe compact view: meals expanded, the budget in one line, multi-select, copy in the meal menu with undo.

Condition A, scrolled to the diary — A · diary below other cards

Condition B with an item selected — B · item selected, bulk actions

Tasks and measures

Five prototype tasks with instrumented interactions and outcome scoring

#	Task as rehearsed (v1)	Tests
T1	Say what you logged for lunch and its calories	Glance, navigation
T2	"You exercised and logged it." How many calories are left?	Budget
T3	Copy yesterday's breakfast into today	Copy
T4	The item at lunch was a snack. "Put it under Snacks."	Move
T5	Check yesterday's dinner without changing anything	Accidental copies

Logged on every tap

Taps, scrolls, and back navigations
Every change to the food log
Answer and final state

Derived automatically

Success from the final state
Effort and time on task
Unintended changes
Accidental copies

The moderator watches the person instead of counting taps.

Rehearsal before recruiting

Eight agent profiles tested the instrumented prototype

Eric, AI-assisted

8 synthetic participants

Planted traits, with predictions written before any session. One control.

›

Claude agents

80 task runs planned, 60 completed

Agents operated the screen representation, selected controls, and supplied responses. Interactions were logged.

›

Automatic

Scoring

Success, effort, unintended changes, predictions tested.

›

Separate model

Synthesis

Usability issues and protocol problems proposed.

›

Eric

Adjudication

Which problems are real, and what changes.

Trusts the first number · Condition A · budget task

Same participant · Condition B

Protocol defects identified in agent rehearsal

Rehearsal evidence · tests the study, not users

The agent runs identified task outcomes that the prototype could measure and errors the rehearsal failed to elicit

Tasks that can separate the designs

of synthetic participants found calories left in A, against of in B (matched sessions)

median taps to move an item in A, against in B

Limits of the rehearsal

No accidental copies occurred in completed agent task runs, including those assigned profiles intended to elicit the error. This does not establish that actual users would avoid unintended copying.
of 80 planned runs were refused by model safety screening (diet content). Condition comparisons use matched completed runs.

Problems in the study design

Budget task pointed to the answer. "You exercised and logged it" sends people looking for exercise.

Move task let a wrong action succeed. The literal reader added a second copy to Snacks and declared success:

Lunch task barely measured the glance cost. Opening one meal answers it.

Verification of automated review findings

Eleven proposed issues evaluated against the protocol and recorded interactions

Rejected findings and supporting evidence

"Data differs across sessions"By design: two matched, counterbalanced food logs.

"Uneven task coverage"Caused by safety refusals, not the protocol.

"Control IDs reused"The log already records screen and label.

"An answer scored without the food named"Checked the transcript: it was named.

Protocol v2: ten changes. The four that matter most:

Budget question made neutral, then the probe "Does that include your workout?"
Move task judged from the final log, not from what the participant says
Lunch task becomes a whole-day review, so it measures the glance cost
Before recruiting: check condition A against the live app, or the budget result could be manufactured by the prototype

Plus six scoring, recovery, and observation changes (appendix and protocol/CHANGES.md).

Shortest established interaction routes by prototype

A search of both prototypes compares the shortest routes it can establish, and where one tap changes the log

Task (rehearsal success rule)	Today	Compact
Review today's lunch and its total	1 tap	0
Calories left (exercise included)	1 tap*	0
Copy yesterday's breakfast	1 tap	2 taps
Move an item from Lunch to Snacks	5 taps	4 taps
Check yesterday's dinner	2 taps	1 tap
Undo a wrong copy	7 taps†	4 taps

In a task that should change nothing, 4 of 19 possible first taps change the food log on Today, 0 of 13 on the compact view. The same one-tap copy chip makes copying faster on Today.

* In the reconstructed Today test case, the displayed figure at zero taps is 1,346, compared with the exercise-adjusted value of 1,656. The agent quotations elsewhere use the second matched food log, which has different values. † A separate scripted route established seven taps as an upper bound because the exhaustive search reached its exploration limit.

Breadth-first search over the two implemented prototypes identifies the shortest routes established under the task rules. Rendered content counts as available without additional interaction. These results are not measures of human task time or effort. The reconstructed Today baseline has not yet been verified against the live application.

Baseline validation and proposed human study

Check the baseline against the live app, then run the revised study with twelve people

Fielding plan

Verify condition A's summary and meal behavior against the live app
Recruit 12 established loggers from the screener
Run protocol v2: six tasks in each condition, counterbalanced
Score automatically from the prototype log, code observations, synthesize

Decision rule, set in advance

Support the compact view if the study finds lower effort and improved accuracy on the relevant tasks without a consequential decline in comprehension. Evaluate differences in copying and moving separately rather than requiring improvement on every task. If accidental copies are too rare to estimate in twelve sessions, assess that risk through additional targeted testing before making a decision based on it.

The rehearsal tested the study, not the users. It caught a task that pointed people to the answer, a wording that let someone succeed by doing the wrong thing, and an error hypothesis that synthetic participants could not test, all before anyone was recruited.

Appendix

Rehearsal outcomes and protocol revisions

Open the study materials: Try the prototype (moderator mode) · Protocol v2: screener, tasks, moderator guide · What the rehearsal changed

Matched sessions by task (synthetic, v1 tasks)

Same persona and task in both conditions. Synthetic participants: evidence about the protocol, not about users. Total model cost $.

Planted predictions, written before any session

The other six protocol changes

New recovery task: undo a wrong copy
Answer keys per food log, "hedged incorrect" code
Code whether people verify a copy, and confidence
Effort = taps + scrolls + back navigations, plus time
Observation codes for accidental copies
Scenario date stated in the intro

Refused sessions ():