Explanations generated by artificial intelligence systems are often presented as a bridge between complex models and humans: they indicate which variables influenced a decision, help identify anomalies, and, at least in theory, make it possible to assess whether an algorithm is reasoning in an acceptable manner. However, a new position paper published on arXiv calls for separating two issues that are frequently conflated: having an explanation, and having an explanation that genuinely reflects the model's behavior.
The study, authored by Marco Zullich, Emily Schiller, Ivan Gentile, David Dembinsky, Adriano Lucieri, Kilian Göller, and Steffen Seitz, focuses on faithfulness metrics applied to feature importance techniques. These are the tools that assign varying degrees of relevance to a model's inputs: in an image classifier, they might highlight portions of an image; in a tabular system, they might rank the features deemed decisive; while in other contexts, they might flag the elements that reportedly guided a prediction.
The authors' argument is not that Explainable AI (XAI) is useless. Rather, it warns that its evaluation relies on criteria whose robustness is often taken for granted. If a faithfulness metric fails to rigorously establish how closely an explanation actually corresponds to the model, a positive result may be interpreted with far more confidence than it deserves. For those developing or adopting automated systems, the distinction is substantial: a convincing visualization does not prove that the system actually relied on the highlighted information to reach that answer.
The problem of the gap between model and explanation
XAI techniques typically attempt to make observable a behavior that, in more complex models, is not immediately interpretable. To do so, they rely on simplifications: producing a map, a ranking, or a description that is more manageable than the model's internal mechanisms. However, this simplification can introduce a gap between the account provided to the user and the actual logic of the prediction. In research terminology, this gap is described as unfaithfulness.
A fidelity metric should measure precisely that correspondence: to what extent the importance assigned to features by the explicator approximates what happens within the model. It is a relevant property, but it does not tell the whole story when evaluating an explanation. An output can be faithful without being understandable to those who must use it; it can be understandable without being sufficiently faithful; furthermore, it may be of little use for the practical task at hand. The paper treats fidelity as one component of the broader reliability of explanations, rather than a definitive seal of approval on AI quality.
The context highlighted by the authors is just as important. Over the past decade, the idea that making a model “explainable” automatically boosts trust has been challenged by empirical findings. According to the paper, explanations can encourage overreliance even when the prediction is wrong. When that reliance is subsequently undermined, it can turn into the opposite reaction: distrust toward the algorithm. Apparent transparency, therefore, does not necessarily foster a more balanced relationship between humans and automated systems.
Why evaluation is not just a methodological detail
Research on explanations is often showcased through intuitive demos: a heatmap on an X-ray, a list of the most influential words in a text, or the weight assigned to income, age, or payment history in an automated decision. But behind these outputs lies a validation problem. Without a direct, verifiable benchmark of the model’s internal reasoning, measuring the quality of an explanation becomes difficult; if that verification requires an excessive amount of computation, it can become impractical in real-world scenarios.
The authors argue that current metrics suffer from five major limitations, making the quantification of fidelity either not truly viable or computationally very demanding. The abstract does not detail each of the five points, so it would be improper to attribute specific flaws to them not reported in the available document. The central takeaway, however, is clear: simply assigning a score to an explanation is not enough to turn it into an objective measure of alignment with the model.
It is a warning that concerns both researchers and organizations. In a lab, a weak metric can lead to the conclusion that one feature importance method is superior to another when the observed difference depends on the chosen test. In corporate settings or public administration, the same misunderstanding can find its way into audit processes, technical documentation, and risk assessments. The explanation then becomes a reassuring artifact, but not necessarily a tool capable of surfacing shortcuts, spurious correlations, or undesirable criteria learned by the model.
The position paper also proposes a more concrete take on XAI: as a debugging tool. In this role, explanations can offer specific insights into the features the model appears to have learned, helping developers formulate hypotheses to test. It is a valuable use case, but distinct from the promise of automatically making a decision reliable or enabling a non-technical user to validate it. An insight emerging from an explanation should therefore be treated as an investigative signal—to be weighed against data, tests, and domain expertise—rather than self-sufficient proof.
Four research directions, without shortcuts
The contribution goes beyond critique. The authors outline four potential research directions to make fidelity evaluations more reliable. Here again, the publicly released summary does not detail the operational specifics: what is certain is that the proposal aims to address the very foundations of how explanations are measured, rather than presenting a ready-to-use new technique.
This approach aligns with the nature of a position paper. It does not announce a product, a definitive benchmark, or a solution capable of settling the debate; it calls on the community to re-examine established methodological assumptions. The work was presented at the XAI workshop at IJCAI-ECAI 2026 and is available as a preprint on arXiv. It should therefore be read as a contribution to scientific debate, not as a certification or a conclusive systematic review of the field.
For end users, the most immediate takeaway is a call for caution when interpreting statements like “the model decided this because…”. That phrase may refer to a post-hoc explanation—meaning one constructed after the prediction—and how closely it reflects the decision-making process depends on the method adopted and how it was evaluated. Asking which metric was used, what limitations it discloses, and whether independent verifications exist is more useful than merely looking at a chart ordered by importance.
The discussion comes as AI enters an increasing number of procedures where explainability, control, and accountability matter as much as accuracy. The paper does not argue that these techniques should be abandoned. However, it contends that an explanation cannot be considered a reliable anchor if the tools designed to verify it fail to measure its connection to the model with sufficient rigor. For XAI, the next challenge is not producing more persuasive visualizations, but developing verification criteria that help distinguish a plausible narrative from solid technical evidence.
