Evaluation
An API agent can complete the work and still fail the task
APIFlow-Bench tests long, dependent REST workflows and decomposes success into
seven engineering capabilities. Its synthetic API worlds pass a self-test with
no LLM and an oracle solvability check; grading follows a canary through the
actual call path and verifies a typed answer card.
For 19 models running in the same scaffold, success fell from 93% on individual
subtasks to 74% on clean 20-step chains, and to 61% after adding the 8% of trials
that no model completed. Best-of-five improved the result by 7 points, while
all-five reliability differed by 44. The most revealing number is that 77% of
failed clean runs reached the correct final state but failed at delivery.
All results come from the preprint's authors and need replication. Still, the
benchmark identifies an important boundary. “Made the changes” and “returned a
verifiable result to the party waiting for it” are different capabilities. In
production, the latter is often part of the task, not presentation around it.
Source: APIFlow-Bench.
AgentLogs moves research from benchmark tasks to real trajectories
AgentLogs collects public runs of GitHub Copilot's cloud agent. The authors
scanned 1,812,362 popular public repositories and found agent activity in
35,810. The dataset contains 307,416 tasks, 549,239 sessions, and more than 64.2
million log entries: prompts, intermediate reasoning, tool calls, Git
operations, and token usage.
That is far closer to real development than a collection of final patches. But
the public footprint does not represent private enterprise use, and the
availability of reasoning and logs depends on GitHub's interfaces and privacy
filtering. Any analysis first needs the dataset card, license review, and a
separate check for sensitive data.
The dataset's strength is that it makes the process available for study: where
an agent spends tokens, how a person frames a task, which tool sequences lead
to failure, and what happens after review. Its scale matters only because the
dataset preserves intermediate causes rather than merely the successful commit.
Source: AgentLogs.