Decision Transformation Testing
AI agent testing usually evaluates outputs. Decision Transformation Testing evaluates decisions under changes in state — motivation, principles, terminology, procedure, transformation classes, coverage and limitations.
“A consequential AI decision should change when decision-relevant conditions change, and remain stable when they do not.”
The same state, two different histories
The same endpoint, a different history
Part I
Motivation and principle
1. Motivation
Many conventional evaluation methods assess a system at individual observation points.
A system receives an input or state S and produces a decision D(S). That decision may then be evaluated for correctness, accuracy, policy compliance, groundedness, safety, calibration or another task-specific criterion.
This establishes something about the system at S. It does not necessarily establish how the system behaves around S — and for consequential enterprise decisions, that distinction matters.
Consider a system that recommends a payment in a supplier-payment workflow. The decision may be correct in the observed state. Assurance also requires understanding what happens when relevant conditions change.
For example:
- What if the supplier's bank account changes?
- What if an approval expires?
- What if the transaction amount crosses an authorization threshold?
- What if irrelevant metadata changes?
- What if two individually acceptable conditions occur together?
- What if the same apparent state is reached through a different sequence of events?
DTT treats these changes as part of the evaluation problem.
2. Methodological principle
Let S represent the state in which a decision is made, and D(S) the decision produced in that state.
DTT applies a valid transformation T from S to a transformed state S′:
The decision system is then evaluated again against the transformed state, producing D(S′). DTT examines the relationship
against the expected behavioral property associated with the transformation.
3. Transformation Contracts
Every meaningful transformation requires an expected behavioral relationship. DTT represents this concept as a Transformation Contract:
The contract does not require every transformation to change the decision.
Depending on the transformation, the expected relationship may be:
- invariance — the decision should not change: D(T(S)) = D(S);
- material sensitivity — the decision should respond: D(T(S)) ≠ D(S);
- a specific transition — for example PAY → HOLD, or ALLOW → ESCALATE.
The transformation therefore carries semantic meaning, not simply input perturbation.
Part II
Transformation classes
4. Transformation classes
DTT organizes decision transformations according to the type of behavior being evaluated. Nine classes are recognized, and each carries a different expected relationship.
4.1 Invariance
An invariance transformation changes something that should not materially affect the decision. The expected property is:
Examples can include changes to non-material metadata, equivalent representations, or other decision-irrelevant conditions.
A failure indicates that the decision may be sensitive to something the governing requirements consider irrelevant. We refer to this generally as spurious sensitivity.
Methodological question
Does the decision remain stable when the world changes in a way that should not matter?
4.2 Material sensitivity
A material transformation changes a condition that should affect the decision. For these transformations the expected behavior is not invariance — the transformed decision must satisfy the behavior required by the applicable decision requirement.
For example, PAY → HOLD when a required approval expires.
Failure to respond correctly to a material change represents material insensitivity.
Methodological question
Does the decision respond when something important changes?
4.3 Boundary behavior
Many consequential decisions contain explicit or implicit boundaries — authorization limits, expiration windows, risk thresholds, policy cutoffs, eligibility conditions, confidence thresholds and approval levels.
DTT evaluates decision behavior around these boundaries. Two states may differ only slightly while lying on opposite sides of a boundary:
The expected decisions may therefore differ:
This matters because aggregate performance metrics can hide localized failures around consequential decision boundaries.
Methodological question
Does the decision change where the governing requirement says it should change?
4.4 Interaction and composition
Enterprise decisions frequently depend on combinations of conditions, and testing those conditions independently may be insufficient. Suppose transformations T_A and T_B each produce acceptable behavior:
It does not necessarily follow that their composition will also pass:
A system may therefore pass both individual tests while failing when the conditions occur together. DTT explicitly recognizes this as an interaction problem.
4.5 Temporal behavior
Enterprise decisions occur in time. State can become stale, approvals expire, evidence changes, risk accumulates and relationships evolve.
DTT therefore includes transformations in which the material change is temporal. A state at time t
may legitimately support one decision, while a related state at a later time
may require another. Temporal transformations evaluate whether decision behavior reflects these changes appropriately.
Methodological question
Does the decision remain valid as the temporal context changes?
4.6 Sequence and trajectory
Some decisions depend not only on the current state but on the path by which that state was reached. Two trajectories
may reach endpoints that appear equivalent:
while their histories are materially different. DTT therefore does not assume that equivalent endpoints carry equivalent meaning.
This matters for systems in which event ordering, prior actions, timing, accumulated risk or historical relationships affect the meaning of current state — snapshots can hide path-dependent meaning.
Methodological question
Would the same apparent state still justify the same decision if it were reached through a materially different history?
4.7 Evidence sensitivity
Enterprise decisions are often supported by multiple forms of evidence. DTT can examine how assurance behavior changes when the evidence environment changes.
Examples include:
- evidence becomes unavailable;
- evidence becomes stale;
- new contradictory evidence appears;
- an evidence source changes;
- supporting evidence is withdrawn.
The purpose is not to manufacture alternative truth. It is to evaluate whether the decision or assurance outcome is appropriately sensitive to changes in its evidentiary basis.
Methodological question
Does the system respond appropriately when the evidence supporting the decision changes?
4.8 Authority sensitivity
Evidence and authority are different concepts. Two sources can disagree while the enterprise has an explicit rule determining which authority governs the decision.
Authority itself can also vary by jurisdiction, business unit, decision type, policy version or effective period. DTT can therefore examine how an assurance outcome behaves when the applicable authority changes.
AnantVerify does not use this mechanism to invent authority. Authority remains defined by the organization; DTT evaluates behavior relative to that declared authority.
Methodological question
Would the decision still be permitted under a different governing policy or authority?
4.9 State quality
The quality of the available state can itself be transformed. A decision may initially be evaluated using sufficient state; DTT can then examine conditions such as:
- a required fact is removed;
- a fact becomes stale;
- two authoritative sources conflict;
- provenance becomes unavailable;
- uncertainty increases.
A decision system should not necessarily produce the same operational behavior when the state supporting the decision is no longer sufficient. For example, a transition from sufficient to incomplete state may require ALLOW → HOLD rather than another confident decision.
This evaluates whether the system recognizes the limits of what it knows.
Part III
The procedure
5. The DTT procedure
At a methodological level, DTT follows a repeatable sequence. The procedure below describes what each stage establishes; it does not describe how transformations are discovered, generated, compared or searched.
Establish the decision context
Identify the consequential decision being evaluated and define the legal or permitted decision vocabulary. For a payment-assurance task this might be {PAY, HOLD, ESCALATE}; for a general enterprise action it is {ALLOW, HOLD, ESCALATE, BLOCK}.
Establish the required state
Identify the state required to evaluate the decision: the relevant entities, facts, relationships, events, temporal conditions and provenance requirements.
Establish expected decision properties
Determine which behavioral properties should hold — invariance, material sensitivity, boundary behavior, interaction behavior, temporal behavior, sequence behavior, evidence sensitivity or authority sensitivity. These expectations must come from qualified requirements or evidence. DTT itself does not define enterprise truth.
Establish a baseline
Evaluate the decision in the original state, D(S). The baseline provides the reference point for subsequent transformations.
Apply a valid transformation
Construct a transformed state S′ = T(S). The transformation should represent a meaningful change in the decision context.
Re-evaluate the decision
Evaluate the same decision process against the transformed state, producing D(S′).
Evaluate the required property
Compare D(S) and D(S′) against the relevant Transformation Contract. The result is not simply whether the two decisions are equal — the question is whether their relationship is correct for the transformation that occurred.
Record the assurance evidence
The transformation, original state, transformed state, original decision, transformed decision, expected property and result become part of the assurance evidence. This makes DTT repeatable and suitable for regression testing.
Part IV
DTT in context
6. DTT and probabilistic systems
DTT does not require the underlying decision engine to be deterministic. The decision may come from a language model, an AI agent, a machine-learning classifier, a temporal model, a rules engine, a multi-agent system or a human-assisted process.
The underlying system may be probabilistic. The assurance property can still be explicit. An AI system may generate a decision probabilistically while the enterprise requirement remains unambiguous — for example, that an expired approval means a payment is not permitted.
7. DTT and AI agents
Agentic systems introduce additional state transitions. An agent may observe, reason, call tools, update state, delegate to other agents and then act — and each of those stages can change the state in which a consequential decision is made.
DTT can be applied to the consequential decisions emerging from these workflows, and the methodology does not depend on a particular agent framework. Agents may be built with LangChain, LangGraph, Google ADK, CrewAI, the OpenAI Agents SDK, Microsoft Agent Framework, AWS Strands Agents, or a proprietary orchestration environment — DTT is applied at the decision boundary regardless. AnantVerify is model- and framework-agnostic, and integrations with these ecosystems are described on the AI Agent Assurance page.
Methodological question
When the enterprise state changes, does the agent's consequential decision change appropriately?
8. DTT and conventional AI evaluation
DTT is not intended to replace conventional AI evaluation. It addresses a different dimension, and both forms of evidence can contribute to a broader assurance determination.
| Conventional evaluation | Decision Transformation Testing |
|---|---|
| Accuracy, calibration, groundedness, safety, tool correctness, task completion, latency, cost, agent behavior. | Invariance, material sensitivity, boundary behavior, interactions, temporal behavior, trajectory behavior, evidence sensitivity, authority sensitivity. |
| How well did the AI perform? | How did the decision behave when meaningful aspects of the world changed? |
9. DTT and counterfactual analysis
DTT has conceptual similarities to counterfactual testing but addresses a different objective. A typical counterfactual question asks what change would cause the model to produce a different output.
DTT asks a different question: given a meaningful transformation, what relationship should exist between the original and transformed decisions?
A different output is not inherently desirable. For an invariance transformation, changing the decision may indicate failure. For a material transformation, failing to change may indicate failure. For a boundary transformation, a specific transition may be required.
DTT therefore evaluates behavioral relationships, not merely output differences.
10. DTT and red teaming
Red teaming and DTT are complementary. Red teaming typically attempts to discover ways in which a system can be induced to fail. DTT evaluates whether specified decision properties hold across meaningful transformations.
Together they form a useful lifecycle: a discovery leads to a property, the property defines a transformation, and the transformation becomes a regression test. An incident, a red-team finding or a production failure can reveal a new property that should subsequently become part of the DTT assurance suite.
In this way, previously discovered failures become persistent assurance requirements rather than one-off fixes.
Part V
Coverage, interpretation and limitations
11. Assurance coverage
A DTT result should not be interpreted independently of coverage. Passing all executed transformations establishes behavior only for the transformations that were evaluated. It does not establish correctness across the entire decision space.
DTT therefore treats assurance coverage as a first-class concept. Coverage can consider whether relevant categories have been exercised, including invariant transformations, material transformations, boundaries, interactions, temporal conditions, trajectories, evidence conditions, authority conditions and state-quality conditions.
Exhaustive testing is not always feasible. As transformation count and interaction depth grow, DTT uses explicit search policies and budgets — and the search policy used is itself part of the assurance evidence, so a result can always be read alongside the search that produced it.
The purpose is to distinguish three different things:
- what passed;
- what was actually tested;
- what remains unknown.
That distinction is essential for meaningful assurance.
12. Interpretation of results
DTT results should be interpreted as evidence about decision behavior.
A passing result means that the observed decision behavior satisfied the specified property for the evaluated transformation.
It does not mean that the AI system has been proven universally correct or safe. Similarly, a high number of passing tests does not automatically imply a corresponding probability of safety.
DTT is designed to make assurance evidence more structured, reproducible and explicit — not to convert incomplete evidence into certainty.
13. Methodological limitations
DTT intentionally separates behavioral testing from the question of who defines correct behavior. This produces several important limitations, which are not incidental — they are part of the methodology's assurance model.
- DTT does not create ground truth. Expected behavior must be supported by qualified requirements, policies, evidence or designated authority.
- DTT does not prove universal correctness. The method evaluates defined properties over evaluated transformations.
- DTT does not eliminate unknowns. Unexplored regions of the decision space remain unexplored.
- DTT does not replace domain expertise. Domain experts remain necessary for establishing appropriate decision requirements and interpreting consequential findings.
- DTT does not replace conventional evaluation. Accuracy, calibration, safety, agent evaluation and other methods remain relevant.
14. Five methodological questions
DTT can be summarized through five questions. They are methodological questions about decision behavior — distinct from the five questions a consequential decision itself has to answer before it becomes an action.
| Dimension | Question |
|---|---|
| Invariance | What should not change the decision? |
| Sensitivity | What should change the decision? |
| Boundaries | Where should the decision change? |
| Composition | What happens when conditions interact or unfold over time? |
| Coverage | Which of these behaviors have actually been tested — and which remain unknown? |
Traditional evaluation provides observations at individual points. DTT is concerned with the relationships between those points: from point evaluation, to behavior under change, to the decision surface, to assurance coverage.
Instead of asking only whether an answer is correct, DTT also asks what must remain true about the decision as the world around it changes. For enterprise AI systems that increasingly move from generating information to making consequential decisions, that distinction becomes important.
Part VI
DTT within AnantVerify
15. DTT within AnantVerify
Within AnantVerify, DTT is one component of a broader Enterprise Decision Assurance model. Enterprise state, an AI proposal, evidence, declared authority, conventional evaluation, DTT and coverage together produce an assurance determination, which can then inform whether an enterprise action should be allowed, held, escalated or blocked.
DTT therefore does not attempt to answer every question about AI assurance. It addresses one specific and important question.
Methodological question
When the world around an AI decision changes, does the decision change in the right way?
That is the core methodology behind Decision Transformation Testing.
Read the method against the product
This page describes the methodology. The product pages show how DTT is applied inside AnantVerify, alongside state assurance, evidence, declared authority, coverage and runtime control.