Evaluating an artificial intelligence agent operating in a scientific domain based solely on the final result can be misleading. Code can run, a hypothesis can seem plausible, or an answer can pass a test even when the path that produced it is fragile, inefficient, or based on lucky guesses. It is this gap that gave rise to OpenDiscoveryTrace, a dataset presented on arXiv by Aayam Bansal and Keertan Balaji to shift the focus from answers to the working processes of agents.
The project makes available 558 complete trajectories of AI agents engaged in 124 scientific tasks. The tests cover drug discovery, materials science, genomics, and scientific literature analysis. The goal is not merely to determine whether a model reaches the desired conclusion, but to allow for reconstructing what happens along the way: which tools it uses, what it observes, when it encounters errors, what signals prompt it to course-correct, and how confidently it claims to proceed.
For the autonomous agents sector, the difference is significant. The most common benchmarks tend to condense complex performance into a single success metric: task completed or not, output correct or not. It is a useful metric, but insufficient when the system is put forward for contexts where the procedure matters as much as the outcome, from biomedical research to evidence synthesis in academic literature. A correct answer obtained by ignoring constraints, misinterpreting an experimental result, or misusing a tool does not offer the same guarantees as a result achieved through a verifiable method.
A structured trace for every step
Each step in the OpenDiscoveryTrace trajectories is described across nine fields. The dataset includes the model's reported thoughts, tool calls, received observations, errors, factors that trigger a revision, and the self-reported confidence level. The availability of these elements makes it possible to study the agent as a sequence of decisions, rather than as a black box evaluated at the end of a task.
This setup does not automatically turn the traces into a faithful snapshot of a model's internal reasoning. Indeed, the recorded thoughts and confidences are content generated or communicated by the system within the experimental framework, not direct evidence of every computational mechanism leading to the answer. Yet they remain useful operational signals for auditing: they show the agent's explicit planning, its interaction with tools and sources, stated revisions, and anomalies that emerge during execution.
The dataset includes three frontier models — GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro — with 124 trajectories each, balanced by domain and difficulty according to the authors. Joining them are Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B, along with 60 trajectories from a variant featuring real-time information retrieval. Bringing proprietary families and open-weight models together in the same archive can help compare not just end accuracy, but execution styles and the areas where each architecture tends to fail.
Similar successes, very different behaviors
The first analysis presented in the paper covers 363 trajectories evaluated by an LLM. According to the researchers, the three frontier models achieve fairly close success rates, ranging between 84% and 89%. Looking at this figure alone, their behavior would appear largely comparable.
The traces, however, tell a more nuanced story. Claude Opus 4.6 records an average of 2.5 errors per trajectory, compared to 0.08 for GPT-5.4: a difference the paper quantifies as roughly thirtyfold and considers statistically significant. The nature of the detected issues also shifts. For Claude, 66.7% of observed errors are tied to incorrect tool use; for GPT-5.4, 83.6% fall instead into the category of reasoning errors.
This is not a definitive ranking of models. The analyzed sample is limited, the classification relies on judgments from another LLM, and a preprint does not equal a completed independent validation. Rather, the value of the comparison lies in demonstrating the limitations of an exclusively outcome-based evaluation: systems with similar completion rates may require very different interventions and controls. An agent that fails primarily when using a database, a search engine, or a specialized tool poses a different challenge than one that uses tools correctly but draws flawed inferences from the retrieved data.
From benchmark to laboratory audit
OpenDiscoveryTrace also defines five benchmark tasks and provides baselines including logistic regression, random forest, LSTM, and Transformer models. Released under a CC BY 4.0 license, the work makes available the dataset, the trace schema, the agent harness used to collect them, and the task definitions. The choice of openness is significant because it enables researchers to replicate analyses, propose alternative metrics, and train tools capable of identifying risk patterns in agent workflows.
Potential applications point in several directions. Teams developing research agents can use these sequences to understand whether an improved output conceals an increase in improper actions, late corrections, or overconfidence. Those integrating such systems within an organization can establish supervision checkpoints based on the type of activity performed: for instance, requiring human review following a call to a critical tool or when the agent modifies a hypothesis after an error. Finally, for governance research, a standardized trace offers a more concrete foundation for discussing accountability, traceability, and oversight requirements.
Naturally, a dataset alone does not solve the problem of scientific verifiability. The trajectories represent a specific set of models, tasks, and experimental setups; behavior can change with different tools, different instructions, new models, or access to live sources. The use of automated evaluators also requires caution: an LLM tasked with identifying errors may introduce questionable classifications or miss flaws that a domain expert would recognize. The next step will therefore be to verify whether process metrics remain informative in environments closer to real-world scientific work and alongside specialist human evaluations.
The preprint, submitted on September 5, 2026, and awarded Best Dataset at the ICML 2026 Workshop on AI for Science dedicated to AI scientists, comes as the industry pushes toward systems capable of planning, seeking evidence, executing tools, and iterating on a problem. In this scenario, accounting for the path taken is not a methodological detail. It is a prerequisite for distinguishing an agent that occasionally produces a good answer from one whose work can be inspected, corrected, and, with the appropriate caveats, reused.
