Trial design simulator: verification support for AI coding tools

This tool evaluates a proposed experiment rather than the feature itself. Each run generates a randomized trial with effects you specify, analyzes it with the prespecified intent-to-treat comparison, applies the ship rule, and checks the decision against the one the specified effects call for. Change the assumptions to see where the trial reaches the right decision, where it stays inconclusive, and where it makes a costly mistake.

Trial Participants per arm Simulated trials per scenario Use the feature if assigned Missing outcomes (at random)

Population (assumptions) Baseline trust in output accuracy Verification time per task, minutes Spread of time (log scale) Error rate in accepted code Time–error latent correlation

Baseline trust is the survey's daily-user figure (49%). The rest are planning assumptions to replace with product data.

Decision rule Smallest worthwhile time reduction Largest acceptable error increase Interval must clear the threshold Allow targeted rollout to low-trust users

Ship when time falls by at least the worthwhile amount and the error interval stays under the margin. Do not ship when errors rise beyond the margin or the best case is below the worthwhile amount. Otherwise the trial is inconclusive.

Scenario (effects of using the feature)	Time, low / high trust	Errors, low / high	Decisions across simulated trials

correct decisioninconclusiveshipped a change that should not shiprejected a change worth shippingshipped to everyone when only low-trust users benefittargeted when everyone benefits

The simulation shows what this design and decision rule can establish under the effects and assumptions entered here. It does not estimate the feature's actual effects, which only the trial can measure. Reported trust is generated and analyzed but is deliberately not part of the ship rule: higher trust with worse code would not be a success.

Source: sim.js (no dependencies). The saved default results are in results.json.