A New Agent Benchmark Makes the Case for Testing the Team, Not the Demo

Adding AI agents to a workflow can multiply the bill faster than it improves the result. A new Vals AI benchmark gives marketing teams a useful reason to measure the full job, not the polished demo: on one web-app building test, teams cost 1.8 to 5.1 times as much as single agents, and only one of four score comparisons was statistically significant.

Vals ran GPT-6 Sol and Claude Opus 5.5 on 50 full-stack app tasks, testing each model as a single agent and as a team at medium and maximum reasoning effort. Its October 9 report found that Sol’s medium-effort team scored 84.9%, compared with 77.6% for its single agent, at average costs of $3.07 and $1.22 per app. That 7.3-point gain was statistically significant in the test. The other three team-versus-single score differences were not.

The report does not prove that teams are a poor choice for marketing. It tested coding tasks, where work can be divided into a database, an API, an interface, and deployment. It also ran each of the eight setups once, and its authors caution that the results may not transfer to other task types. That limitation is the point for marketers: a benchmark can guide an experiment, but it cannot replace one on your own work.

The study also illustrates why the comparison matters. At maximum effort, the Sol single agent cost $4.82 per app, while the medium-effort team cost $3.07 and completed work in a similar time. For Opus, the report found that a team at maximum effort produced the highest score in the test, but its score advantage over the medium-effort single agent was not statistically significant and it cost roughly 30 times as much. The purchasing decision changes depending on whether the goal is quality, speed, cost, or a combination. That question has surfaced in other tests of agent token costs and in comparisons where the harness can matter more than the model.

John Aspinall, an independent AI practitioner, made a related case in a public LinkedIn post: “Multi-agent only gets useful when conversation is not punished by price.” His point is about routing work to models with different costs and strengths. Vals’ data adds a measurement question: when agents coordinate, does the extra interaction reduce total effort or simply create more tokens, review, and integration work?

Other research points to the same need for cost-aware evaluation, though with different tasks and methods. A June paper on arXiv comparing multi-agent structures at equal inference cost found no statistically significant advantage for its full agent team over a single-agent alternative on its tested benchmarks. It used a frozen 7-billion-parameter model and optimized prompts, so its results do not settle what larger commercial models can do in marketing workflows. Together, the studies suggest that role count alone is a poor proxy for value.

Marketing leaders can apply the lesson with a small, fair test. Choose a recurring task, such as campaign QA, customer-research synthesis, or localization review. Define a quality bar before running it. Then compare a single agent at ordinary effort, the same agent at higher effort, and a team configuration. Use the same input, source material, and human acceptance criteria. Record total model and tool cost, elapsed time, correction hours, factual errors, and how often a person must resolve disagreement among agents.

The correction burden matters because a multi-agent setup can shift work instead of removing it. Vals observed that team leads handled coordination and testing differently: one lead split work quickly and tested the integrated application, while another wrote a shared contract before assigning sequential tasks. For marketing, that can translate into one agent drafting, another checking claims, and a third reconciling style. If the human has to inspect every handoff, “more agents” may mean more process to supervise.

Vals researcher Katrina Drozdov framed the open question in the report: “A team should gain the most when the work splits into parts that can proceed side by side, and the lead can combine them without spending the time it saved on coordination.” The benchmark’s contribution is a set of dimensions to test: cost, runtime, output quality, and where a task can be split cleanly. A coordinated team is more likely to help when subtasks can run in parallel and the final result can be integrated without lengthy reconciliation. That is a hypothesis to test against a live marketing workflow, not a promise from the demo.

Vendors may report token costs or model response quality, while your business pays for the whole workflow. Include storage, orchestration, retries, human review, and the cost of correcting an error. Track the result per accepted deliverable, not per model call. For a campaign QA workflow, that means the cost of a clean, approved review, not the number of checks the agents claim to have completed.

The verdict: Before buying an agent team, run it against the best single-agent alternative on a real task and the same acceptance bar. Pay for the team only when the quality or time saved outweighs coordination, inference, and correction costs. The benchmark says the burden of proof belongs to the workflow, not the headcount of the agent roster.