Human–Technology Interaction

Org Signal is a browser-based social-network analysis application I built. Its calculations passed independent numerical checks, but AI agents using the interface drew conclusions the results did not support. They identified a departed employee as a current organizational broker, read a count that included two-way ties twice as the number of distinct contacts, and overlooked how the definition of a relationship changed the network being analyzed.

These errors concern the interpretation of network measures rather than their calculation. Whether someone occupies a central position depends on which people and relationships are included. When reciprocal ties are counted separately, a directed-tie count can exceed the number of distinct contacts. And different definitions of a relationship produce different networks, so a conclusion that ignores the definition may describe a network other than the one analyzed. The agents made these errors while using an interface that did not consistently surface the qualifications needed to interpret its results.

Two rounds of agent evaluation, using synthetic profiles that included a consultant, an instructor, and students, produced 253 issue reports. I consolidated them into sixteen design decisions and implemented them across the application. The revisions included explanations attached to network measures, clearer distinctions between observed and inferred groups, and account-status warnings. A subsequent known-answer evaluation, cut short by model refusals, found that the prominent "Who stands out" summary still listed the departed employee first without her deactivated status. A deterministic regression check located two further references missing the warning and provides a repeatable test for the correction.

The evaluations identified specific ways the agents misread the application and the software states in which relevant qualifications were missing. Whether the revisions help people interpret network results more accurately remains to be tested.

In April 2026, MyFitnessPal replaced its long-standing diary interface with a redesigned Today screen. I examined whether the complaints that followed reflected difficulty learning a new interface or changes that increased the work of routine food logging.

I collected approximately 9,200 public app-store reviews and coded a sample of 950. Part of that sample had been selected for redesign complaints, so the principal comparison over time uses only randomly sampled Google Play reviews. Among complaints identifying a specific source of difficulty, repeated-use design costs predominated both immediately after the redesign and in later periods. Reviewers described opening more screens to inspect meals, losing a compact view of the day, and having difficulty copying or editing entries. These problems differ from difficulty locating an unfamiliar control. A user can learn where an action has moved without reducing the number of steps it requires. The persistence of such complaints is consistent with continuing workflow costs. The reviews come from people who chose to post them and are not repeated observations of the same users, so they cannot establish how common these difficulties were among all users or whether individual users adapted.

The analysis informed a proposed compact day view and a comparative usability study with twelve established food loggers. Before recruiting, I rehearsed the protocol with AI agents on an instrumented prototype. Sixty of eighty planned runs were completed before model refusals stopped the rest. The rehearsal exposed a task that suggested its expected answer, a success criterion that could accept an incorrect action, and an error measure that could not test its intended claim. Together with a separate automated review, it led to ten revisions of the study.

A path search over the two prototypes found that the compact view showed the full day and the correct calories-left figure without a tap and needed fewer actions for several editing and navigation tasks. The reconstructed Today screen offered a shorter path for copying yesterday's breakfast. These comparisons concern the prototypes. The reconstruction still needs to be verified against the current app, and the revised study run with people.

Members of a fantasy-football league I play in have repeatedly dropped players they meant to keep while making waiver claims in Sleeper. A waiver claim can include a conditional drop, which releases the player only if the claim succeeds. In the recorded interaction, tapping a player's name opens a separate page whose prominent Drop action releases the player immediately, independently of the pending claim. The manager can then submit the claim with an open roster spot, and nothing indicates that the earlier drop was unnecessary.

I reconstructed the interaction as a working prototype with three kinds of change. Selection controls keep the user within the claim, the player page names the pending task, and confirmations distinguish conditional drops from immediate releases. Five behavioral policies run against five interface versions produced 25,000 simulated transactions. Under the stipulated name-tapping and row-tapping policies, approximately 81 percent of baseline runs produced immediate drops, and the selection and context changes eliminated them for the claim-based policies. A separate drop-first policy exposed a remaining problem: changes confined to the claim workflow cannot prevent errors that begin before the claim.

I also specified an event-log sequence for finding candidate errors, a procedure for establishing whether recorded drops were contrary to users' intentions, and an experiment to test the selection change. The simulations compare outcomes under specified behavioral policies. The proposed event-log analysis and intention-validation procedure would be needed to determine whether comparable errors occur in actual transactions.

Nextdoor's Ask feature can help someone formulate a local question. In my walkthrough it produced a request for bicycle-mechanic recommendations, but the request could not be carried into the neighborhood posting workflow. Preparing a useful question and deciding what to disclose, to whom, and whether to publish require different judgments.

I designed an Ask-to-Post handoff that carries the prepared question into an editable draft and keeps a separate review stage covering audience, identifying details, named people and businesses, and wording. Claude implemented the working prototype from my specification. Opening the draft publishes nothing, and the review controls prompt reconsideration without certifying that a post is appropriate. I also specified a randomized evaluation whose outcomes include whether users receive useful information and whether the change produces unwanted disclosures, moderation problems, or low-value content.

To test the review stage, I built 160 synthetic drafts containing 164 planted concerns and 149 harmless lookalikes. The prototype's three detection patterns caught 49 items in the categories they support, with 37 false alarms. A contextual AI reviewer produced findings overlapping all 164 planted items, with six false alarms. That comparison favored the reviewer, whose instructions named the categories used to build the cases, so after seeing those results I built a harder set, on which it detected 23 of 24 planted items. These results concern constructed examples. They do not show how the reviewer would perform on actual neighborhood posts or how people would respond to its warnings.

Among Stack Overflow survey respondents using AI coding tools, the share reporting trust in output accuracy declined from 48 percent in 2023 to 39 percent in 2025. Using approximately 204,000 responses from the three survey years, I examined which reported difficulties distinguished trusting from distrusting users.

The most frequent frustration, suggestions that are almost correct, distinguished trusting from distrusting respondents poorly. Reporting extra time debugging AI-generated code was associated with approximately fifteen percentage points lower trust after adjustment for other measured characteristics. Demographic and professional characteristics barely predicted trusting daily use. Reported experience with AI tools predicted it much better but is unavailable before someone has used the product.

The contrast between almost-correct suggestions and debugging effort suggests that the work of verifying and correcting generated code may matter more for trust than encountering imperfect suggestions. The survey cannot establish that debugging effort lowers trust. Task difficulty, prior expectations, and other differences among users could produce the same association.

I proposed a randomized experiment testing integrated verification support, with outcomes for debugging effort, task completion, code quality, and reported trust, and built a Monte Carlo simulator to examine how the trial's decision rules would perform. Under the initial assumptions, a sample sized to detect a five-percentage-point difference in trust approved a broadly useful verification feature in approximately 38 percent of simulated trials. A feature that raised trust while worsening code quality was rarely approved, but the trial often ended inconclusive rather than rejecting it. A stricter benefit criterion substantially reduced mistaken general rollouts when the gains were confined to initially distrustful developers. The simulation examines how the proposed trial would classify treatment effects under specified assumptions. The experiment itself would test whether verification support reduces debugging effort and improves trust without compromising code accuracy.