AI agents capable of autonomously updating instructions, memory, or other persistent elements are often presented as systems that improve with experience. But there is a substantial difference between correcting behavior after an error and truly learning a general rule. A new paper published on arXiv proposes a benchmark designed specifically to separate the two cases: determining whether the agent has identified the criterion connecting certain elements, or whether it has merely added yet another exception to its memory.
The project is called CLR-Bench, short for Curriculum of Latent Rules, and was developed by Chenjie Qiu, Boyan Xu, Shaobin Shi, Bingfeng Chen, Ruichu Cai, and Dongsheng Luo. The starting point is a limitation that is largely invisible in traditional tests: an agent may appear more competent after receiving feedback, despite having extracted no transferable knowledge. If every correction is stored as a new individual case, performance can improve on known examples without the system being able to handle unseen ones.
The research therefore does not claim to have created a better agent, but instead offers an evaluation tool for a capability that will become central if autonomous assistants are to operate over long periods, receive user corrections, and modify their operational state without requiring complete retraining.
The test uses real-world categories, but conceals the criterion
CLR-Bench constructs exercises around namable categories that exist on the web—for example, groups of people or organizations united by a specific fact. In the task, however, the agent does not directly receive the name of the category. It is presented with an opaque label, an initial example value, and then a sequence of signals obtained during practice: one element that belongs to the group and one that does not.
Verification is carried out on members not yet shown and on similar but incorrect cases. To succeed reliably, the agent must infer the underlying category and apply it to new elements. Simply memorizing the received examples is not enough: the list may grow, but it remains incapable of correctly discriminating what has never appeared in the history.
Each chain in the benchmark progressively increases the difficulty. At each level, a new latent rule is added, overlapping with those already introduced. The authors released 20 chains across ten domains and describe an automated pipeline to generate more from new datasets. This curricular structure matters because a truly adaptive agent must not merely solve an isolated puzzle: it must maintain and update an orderly representation of multiple constraints over time.
A key methodological aspect involves the preliminary checks. The model chosen to certify the exercises must already know the members of the categories used, ensuring the score does not simply measure a gap in factual knowledge. At the same time, it must not be able to guess the category without receiving the intended feedback. According to the authors, the tests also verify that the provided clues actually make it possible to identify the group once the labels are interpreted as references to known categories.
Persistent memory does not mean understanding
The study evaluated six recent self-evolution methods—approaches where the agent revises a persistent state in light of previous outcomes. With the model used to certify the exercises, the results are modest. The maximum improvement over an agent that remains unchanged caps at five percentage points.
The most compelling finding, however, is the comparison with an elementary strategy: retaining the full, raw history of practice interactions. In that simple configuration, the system manages to exclude 20% of unobserved members from the count, whereas the analyzed self-evolution methods range between 6% and 12%. This comparison does not prove that structured memory is useless, but it does show that the tested mechanisms do not automatically extract a rule that is more useful than the original feedback log.
Analysis of the internal states updated by the agents reinforces this interpretation. The member revealed during practice appears in memory 75% to 98% of the time, while the category name is barely recorded. And when new feedback introduces a member that was not on the list, the frequent behavior is simply to add that specific item. In other words, the correction is often treated as a data point to be stored, rather than an opportunity to reformulate the principle that triggered the correction.
This is a practical distinction, not merely a theoretical one. An assistant tasked with handling corporate procedures, customer inquiries, or compliance rules may be corrected multiple times. If it turns each correction into a separate entry, its memory becomes increasingly bloated and harder to manage, without necessarily acquiring the ability to handle similar cases. If, on the other hand, it identifies a general rule, the same feedback can improve numerous future decisions while keeping memory more compact.
An explicit lesson significantly alters results
The authors then introduced a different condition: a fixed lesson that explicitly explains the convention used—namely, that labels should be interpreted as groups with a name and a definition. All else being equal across methods, accuracy increased by between 14.5 and 16.5 points. The result suggests that a significant portion of the difficulty does not involve factual retrieval, but rather the conceptual leap required to transform scattered examples into a hypothesis about a category.
This is not to say that agents cannot learn from feedback without additional instructions. In tests with GPT-6-Sol, identified in the study as the strongest model, two methods—ACE and Dynamic Cheatsheet—demonstrate autonomous learning capabilities instead. They achieve 78% accuracy and solve 59–60% of the individual benchmark cells. With the certifier model, those same metrics stood at 53% and 8%, respectively. Furthermore, in this configuration, these techniques outperform simply maintaining a raw history.
The most cautious takeaway is that capability does not depend solely on the agent's updating architecture. It also depends on the language model that this architecture utilizes. Speaking of “agents that learn” as if it were a uniform property of any system with persistent memory therefore risks obscuring very pronounced differences.
A useful benchmark, but not a real-world certification
CLR-Bench measures a well-defined behavior: inferring hidden categories from controlled feedback. It is a useful choice to isolate the problem, but it does not replicate the full complexity of real-world tasks. In daily work, rules can be ambiguous, conflicting, subject to legitimate exceptions, or based on incomplete sources. Furthermore, knowing a category published on the web is not the same as mastering an internal procedure, professional judgment, or a regulatory constraint.
It should also be noted that this is a preprint published on October 29, 2026, on arXiv: while the paper makes its methodology and results available, it should not be confused with an independent evaluation already established by peer review. The figures presented describe the configurations tested by the authors, not a definitive ranking of agents available on the market.
Despite these limitations, the benchmark introduces a concrete question into agent evaluation: when a system improves after feedback, what representation has it truly built? With a view toward agents designed to operate continuously, final accuracy alone risks being an insufficient metric. It will also be necessary to observe how memory grows, whether new information is generalized, and whether behavior remains robust when the specific case changes. CLR-Bench attempts to make this very difference measurable.
