Allocating human attention in agent-led software development
When AI agents carry software development from planning through verification, where to spend limited human attention becomes a central design question. This paper proposes AFHL (Agent-first, Human-last), a method that concentrates human involvement in a final evaluation of the user experience, and describes its structure: two working modes and a rule for reporting verification status. It also discusses the risks this allocation carries and the limits of its current application.
- Engineering
- Agents
Abstract
AI agents are increasingly able to plan, implement, verify, and document software. As agents come to generate most of the work, the human role shifts from checking every change to choosing where limited attention should go. This paper proposes AFHL (Agent-first, Human-last), a development method that concentrates human involvement in a final evaluation of the user experience. AFHL separates a working mode that prioritizes refining the experience from one that improves internal quality, and requires agents to state the verification status of their work explicitly. The paper also discusses the risks this allocation carries and the limits of its application.
1. Background
In conventional software development, code review, in which a person reads the diff of every change, has been a primary means of maintaining quality. As agents increase the volume of changes, however, this practice consumes human attention in proportion to that volume. In a small organization, the more attention goes to reading diffs, the less time remains for evaluating the user experience.
Meanwhile, qualities of the experience, such as how an interaction feels, how quickly it responds, and how naturally the screen moves, are hard to judge without actually operating the software. There is an asymmetry between what agents can verify on their own and what only a person can judge.
2. Question
This paper asks: when agents carry most of the development work, at which stage should a small organization allocate human attention to maintain the quality of the user experience?
3. Related ideas
- Code review. The practice of a person examining the diff of each change. AFHL does not assume routine code review, and limits technical review to the points at which a person asks for it.
- Human-in-the-loop. A design that builds human judgment into key points of an automated process. AFHL can be seen as one such arrangement: it places human judgment at a final evaluation of the experience and limits involvement along the way to confirming direction.
- Technical debt. The view that deferring design or verification for short-term progress increases the cost of later change (Cunningham, 1992). AFHL’s default working mode deliberately defers tests and cleanup, and so incurs debt. AFHL brings that debt under management by recording what was deferred and repaying it in a separate working mode.
4. Method
4.1 Division of roles
Agents handle planning, implementation, the verification the work requires, self-review, and documentation. A person comes in at the final stage, operates the software on a real device, evaluates the UI and UX, and directs revisions. Because human involvement is placed last, the method is called Agent-first, Human-last (AFHL).
4.2 Two working modes
- Iteration (the default). Priority goes to refining the UI, the feel of interaction, responsiveness, and the continuity of motion. Tests are limited to those directly tied to a change. Data loss, changes to the storage format, and major regressions are still checked before distribution. Broad cleanup and exhaustive testing are deferred, and what was deferred is recorded along with the reason (a refinement backlog).
- Refinement (only on explicit instruction). The design, code, and tests are improved without changing the agreed experience. When this is complete, work returns to Iteration.
The two modes are kept separate because demanding both refinement of the experience and improvement of internal quality at once tends to leave both incomplete. Each mode has its own completion criteria: Iteration is complete when the experience has reached a level that can be evaluated; Refinement is complete when the acceptance criteria for the agreed scope and the verification results have been presented.
4.3 Interim checks and reporting verification status
Human-last does not mean that a person stays out of development until the end. Before a new screen or a major change, the agent presents a preview and a person confirms the direction.
A person’s trial use is also never treated as a substitute for the agent’s own verification. In its reports, the agent distinguishes between a build passing and a person having confirmed the behavior on a real device, and states explicitly what remains unverified.
4.4 Codifying the rules
These rules are written into a guide in each repository (such as AGENTS.md). Because agents read the guide at the start of their work, work can continue under the same rules even when a different agent or model takes over.
5. Discussion
AFHL shifts human attention from checking diffs to evaluating the experience. This allocation rests on the asymmetry between what agents can verify and what only a person can judge. When human attention is the scarcest resource, there is a case for concentrating it on the judgments agents cannot make in a person’s place.
This allocation carries two risks, however. First, the tests and cleanup deferred in the default mode accumulate as technical debt. Recording them and switching to Refinement bring this debt under management, but the timing of that switch is left to human judgment. Second, the evaluation of the experience depends on the evaluator’s tacit standards. With a single evaluator those standards stay consistent, but they are neither validated nor shared.
Moreover, placing human involvement only at the end can lead to costly rework at the final stage. Inserting preview-based checks of direction along the way is a compromise that keeps human involvement small while limiting rework.
6. Current application and limits
Our World has applied this method to its own application development since August 2026. The content of this paper is based on rules established in the course of that work.
- The sole evaluator is the author, and the period of use is short. Whether the method holds for organizations of other sizes or for other kinds of software has not been tested.
- It has not been compared, under equal conditions, with conventional methods centered on code review.
- No evaluation based on the quality of outcomes, such as defects after release or user ratings, is yet available.
7. Future work
- Methods for evaluating the quality of outcomes produced with this approach, based on defects after release and user response.
- Methods for articulating and sharing standards for evaluating the experience once there is more than one evaluator.
- Indicators, beyond human judgment, for deciding when to switch to Refinement.
References
- Cunningham, W. (1992). The WyCash portfolio management system. In Addendum to the Proceedings of OOPSLA ’92 (pp. 29–30). ACM.
Related
Version history
- v0.4Oct 5, 2026Rewritten as a paper under a new title. Added related ideas, discussion, and limits.
- v0.3Sep 30, 2026Added experience from use.
- v0.2Sep 9, 2026Split the process into two modes, Iteration and Refinement.
- v0.1Aug 18, 2026First version: how work is divided between agents and a person.
This document’s URL will not change. When citing it, please include the version number.