Verification costs and reported trust
So the first test should be a feature that cuts the cost of checking AI code, focused on frequent users who favor the tools but seldom trust their output.
Quantitative UX research, AI-enabled · Stack Overflow Developer Survey 2023–2025 · public data, about 204,000 responses
The product question
A team building an AI coding assistant has to choose which problem to fix first to raise trust and keep people using the tool.
Reported trust in AI output, 2023–2025
Trust in the accuracy of AI output, AI users only
Somewhat or highly trust vs somewhat or highly distrust. Bars: 95% intervals. Dashed: earlier years reweighted to 2025's mix of experience and professional status.
Respondents choose to take the Stack Overflow survey, so levels describe them, not all developers. The direction holds after reweighting.
Analytical workflow and measurement checks
Eric
Product question
Prioritization of reported problems and identification of candidate user segments
Eric, AI-assisted
Measures and models
Chosen for the decisions they support
Automated (Claude)
One rerunnable script
From raw survey files to every figure here
Eric
Inference
What the evidence supports, what to test
Comparability of the trust measure
Three items, one scale
Overall stance on AI tools · trust in accuracy · how well tools handle complex tasks
Item functioning for senior versus junior developers
Odds ratio, senior vs junior, holding the other two items constant. Shaded band: the 0.67–1.5 range set in advance as "no meaningful difference".
Correlates of reported trust
Complaint prevalence and adjusted associations with trust
Adjusted: logistic model holding usage frequency, agent use, experience, role, org size, and manager status constant. Points with 95% intervals. Association, not cause.
Predictor contributions to model fit
Independent problem-solving confidence: respondents reporting lower confidence in their own problem-solving also reported greater trust in AI output. The association motivates a separate outcome measure in the proposed experiment.
Descriptive segment for verification-workflow research
K-means on usage frequency, stance, trust, complex-task rating, agent use, and number of frustrations (n = ). Stable under resampling (adjusted Rand ), but the segments overlap (silhouette about 0.2), so they describe tendencies, not hard types.
Profile-based targeting
Held-out discrimination (ROC AUC)
Target: daily AI user who trusts the output (base rate ). Trained on 75% of users, tested on the other 25%. Profile: experience, role, company size, age, education, work arrangement, manager status.
The model provides weak discrimination among surveyed respondents. Its classifications of current non-users cannot establish which of them would become trusting daily users.
Target by behavior, not demographics: for example, time spent fixing AI output.
Proposed test and sample size
Example feature
Run the project's tests on a suggestion and show what changed before the developer accepts it.
Primary benefit: verification and debugging time.
Quality guardrail: errors in accepted code.
Secondary: reported trust in accuracy, task completion, total task time, suggestion acceptance, confidence in independent problem-solving.
Design: randomize daily AI-tool users to the current workflow or the feature, and prespecify the initially low-trust subgroup.
Sample-size illustration: participants per arm, using reported trust as the outcome
Baseline trust among daily users:
These sample sizes concern the trust outcome. They do not establish adequate precision for the proposed benefit, guardrail, or subgroup decisions, which the next slide simulates.
The survey identifies additional debugging work as a candidate problem for intervention. The proposed experiment would test whether integrated verification reduces that work without increasing errors in accepted code.
Decision classification in simulated trials
| Specified effect of the feature | Right decision | 1,600 per arm correct / inconclusive / wrong | 4,400 per arm |
|---|---|---|---|
| Less debugging, no quality loss | ship | 38% / 58% / 4% | 77% / 21% / 2% |
| Higher trust, worse code quality | no ship | 21% / 79% / 0% | 45% / 55% / 0% |
| Benefit only for low-trust users | targeted | 15% / 63% / 22% | 31% / 37% / 31% |
| Small benefit, high uncertainty | no ship | 14% / 72% / 14% | 33% / 55% / 12% |
| No effect | no ship | 77% / 22% / 1% | 99% / 1% / 0% |
1,000 simulated trials per cell, intent-to-treat analysis, ship rule on verification time and code errors (trust measured, not decisive). Planning assumptions: 85% use the feature, 10% missing outcomes, errors in 12% of tasks, 5% smallest worthwhile time reduction, 2-point error margin.
At 1,600 per arm, sized for trust, a useful feature ships in only about 38% of trials. The error guardrail limits the decision.
Requiring the whole interval to clear the worthwhile reduction removes most over-broad and costly shipments, with little loss of approvals for the useful feature.
Model estimates and inferential boundaries
Data: Stack Overflow Developer Survey 2023–2025, ODbL 1.0. Analysis: tools/analyze.py, one command.