Development experiments
Agent Development Lab
I use AI agents to help develop software. I also want to know whether the way we arrange that work is helping.
Adding specialists, reviewers, or more instructions can look like progress. The lab gives us a way to examine that assumption through a specific task, a simpler comparison, and rules for judging the result.
One experiment asked whether a group of specialist agents could review software more effectively than a single reviewing agent. The answer depended on what we counted.
Make the comparison fair
An early comparison had a problem: the two arrangements had received different surrounding context. Differences in their results could have come from that extra information. We set the comparison aside and corrected the mismatch.
That changed the question we had to answer first. Before asking which arrangement performed better, we had to establish that they were being compared on fair terms.
Decide what would count
In the corrected comparison, the specialist group finished faster and found an important problem the single reviewer missed. It also used about 32% more processed tokens, the units of text handled by the models, for the same number of verified findings.
Speed and the distinct finding were useful. They did not satisfy the cost rule set before the comparison. The recorded decision was to hold the proposal rather than expand the coordinator's autonomy.
Keep the conclusion as small as the evidence
This was one comparison on one target. Processed tokens were a proxy for cost, not a measured bill. The result did not settle whether specialist groups are generally worthwhile.
It did show why I want the development system to remain open to criticism. A faster run can be useful and still fall short of the reason we proposed it. A comparison can expose a flaw in our method as readily as a flaw in the software.
The work is to notice those differences and let them change what we do next.
Based on recorded experiments from August and September 2026. The result described here is a bounded comparison, not a general productivity claim.