Passing every test you ran does not tell you how much you tested
AnantVerify makes Assurance Coverage and Decision Robustness first-class parts of Enterprise Decision Assurance — so you know where the decision is assured, and where it isn't.
At a glance
Passing your tests tells you nothing about coverage
Coverage is a record of what was exercised, not a probability that the system is safe.
The decision surface is the object
Regions, boundaries and interactions, and what remains unknown about each of them.
Robustness is behavior under change
Stable to what should not matter, responsive to what should.
Coverage sets the autonomy
Thin coverage with high consequence is not a candidate for autonomous execution.
“An AI system passes 10,000 tests. Is it safe to automate? That number alone cannot answer the question.”
Coverage tells you what has been examined. Robustness tells you how easily the decision changes.
Coverage, and what has not been tested yet
The denominator problem
Test counts are operational metrics. They do not describe the structure of what was tested — and the structure is what assurance actually depends on.
The denominator matters
Which system has stronger assurance?
Dots are schematic — not one dot per test. Both pass rates look reassuring. Only one of them tells you what was examined.
Those tests may cover the same kinds of conditions repeatedly while leaving important decision boundaries, interactions, evidence conflicts and temporal conditions unexplored.
System A might contain thousands of variations around already well-understood conditions. System B might systematically cover the most consequential decision boundaries, interactions and failure conditions. The pass rates look almost identical.
The important question is not how many tests we ran. It is how much of the meaningful assurance space those tests covered.
That is the problem AnantVerify addresses with Assurance Coverage.
From test counts to assurance coverage
What behaviors and conditions have actually been assured?
Traditional testing emphasizes tests executed, tests passed, tests failed and pass rate. These remain useful operational metrics.
But they do not describe the structure of what has been tested.
AnantVerify adds a second dimension: the behaviors and conditions that have actually been assured. For a consequential decision, that may include coverage across:
- Invariant conditions
- Material changes
- Decision boundaries
- Interactions
- Higher-order combinations
- Temporal changes
- Event sequences and trajectories
- State-quality conditions
- Evidence conditions
- Authority conditions
- Known failure patterns
- Production incidents
What Assurance Coverage means
Coverage is not a number to maximize. It is a description of which parts of a decision's behavior have been evaluated.
What does Assurance Coverage mean?
Coverage describes the behavior that has been looked at, not the behavior that works
Assurance Coverage describes the extent to which the relevant behaviors surrounding a decision have been evaluated.
Take a system that recommends whether a supplier payment should be PAY, HOLD or ESCALATE. It may perform correctly on historical examples. Assurance may also require understanding whether it behaves correctly when:
- the payment amount changes;
- an approval expires;
- the supplier's bank account changes;
- required evidence becomes stale;
- two risk conditions occur together;
- the transaction crosses an authorization boundary;
- an irrelevant field changes;
- the governing policy changes.
Each represents a different region of the assurance space. AnantVerify helps make those regions visible.
The Assurance Envelope
Understand where confidence in the decision is supported
Think of a consequential decision as existing inside a larger state space. Around the current decision are other possible states. Some are effectively equivalent. Others:
- Contain meaningful changes
- Cross important boundaries
- Introduce conflicts
- Represent conditions the organization has never evaluated
Together, the evaluated conditions form an Assurance Envelope around the decision.
Inside that envelope, the organization has evidence about how the decision behaves. Outside it, uncertainty increases.
Known. Tested. Unknown.
Three concepts that are frequently collapsed
The distinction matters. A policy requirement may be known but never tested against the AI. A transformation may have been tested but lack sufficient authority to establish the expected behavior. A production condition may appear that has never been encountered before.
“We don't know yet.”
A mature assurance system should be able to say that — and it is often safer, and more useful, than manufacturing confidence.
Coverage across decision behavior
Decision Transformation Testing gives AnantVerify a structured way to reason about coverage. Eight classes, each answering a different question.
Eight ways coverage can be incomplete
Each class asks a different question about the decision
Why aggregate scores hide boundary failures
Strong on average, wrong where it matters
A model can have strong aggregate accuracy and still be wrong either side of a boundary like this one — which is exactly where the business consequence sits.
A payment model can score well across a million historical payments and still be unreliable either side of a single authorization limit. Aggregate accuracy is an average over a distribution; the business consequence sits at the boundary.
Boundary coverage asks whether behavior was tested precisely where the decision is supposed to change.
Not all coverage is equal
Risk-weighted assurance coverage
| Decision task | Transformation | Boundary | Interaction | State | Evidence | Authority | Trajectory |
|---|---|---|---|---|---|---|---|
| Payment authorization | |||||||
| Credit limit change | |||||||
| Fraud hold release | |||||||
| Claims escalation |
Illustrative example. Coverage is measured against registered assurance requirements for your decision tasks—not probability of safety.
Testing an inconsequential metadata change is not equivalent to testing a condition capable of causing a multimillion-dollar loss. AnantVerify therefore treats coverage as more than a simple count.
Organizations can prioritize assurance based on
- Business impact
- Decision criticality
- Known failure history
- Policy significance
- Regulatory relevance
- Operational exposure
- Uncertainty
- Potential consequence
Instead of asking whether we tested everything equally, the enterprise can ask whether we have sufficiently tested the conditions that matter most.
Coverage is not a probability of safety
Incomplete knowledge cannot be converted into a safety percentage
Suppose AnantVerify reports that a decision class has strong coverage across the assurance requirements currently defined by the organization. That should not be interpreted as “this AI is 98% safe”.
Assurance coverage measures the extent of evaluated requirements and conditions. It does not convert incomplete knowledge into a universal probability of safety. Equally, 1,000 of 1,000 tests passed is not 99.999% safe.
Decision Robustness
Coverage tells you where you have tested. Robustness asks how stable the decision is as meaningful conditions change.
What Decision Robustness characterizes
How the decision behaves around its current state
Decision Robustness characterizes how a decision behaves around its current state. The goal is to understand questions such as:
- How close is this decision to an important boundary?
- Which state changes could alter the decision?
- Which evidence changes could alter the assurance result?
- Which interactions create fragility?
- Which conditions leave the decision stable?
This moves assurance beyond PASS / FAIL toward understanding the structure surrounding the decision.
The Decision Robustness Radius
Robustness is a distance, not a verdict
Decision Robustness Radius
- Current state
- ALLOW
- Small meaningful change
- ALLOW
- Additional meaningful change
- ESCALATE
How much meaningful change is required before this decision should no longer be treated the same way?
One useful way to think about robustness is the distance between the current decision and a meaningful assurance boundary. The amount and significance of change required before the assurance status changes tells us something about the robustness of the current decision.
It does not mean every possible state can be reduced to a simple physical distance. It provides a framework for asking a sharper question.
Test the decision surface
From isolated points to meaningful regions
- Stable region
- Decision boundary
- Review region
- Escalation region
- Blocked region
Decision behavior read as regions rather than isolated points.
Traditional evaluation gives us points: state A → decision A, state B → decision B, state C → decision C. But enterprise decisions exist within a much larger state space, and DTT lets AnantVerify examine the relationships between those points.
The objective is not merely to know what the AI decided at isolated examples. It is to understand how decision behavior changes across meaningful regions of enterprise state.
Find the fragile decisions
Two decisions can both say ALLOW and not be equivalent
Both decisions say ALLOW. They should not necessarily be treated as operationally equivalent.
Two decisions with the same output can carry very different assurance characteriztics. One may sit far from any boundary, with consistent evidence and well-tested conditions. The other may sit just inside an authorization boundary, with limited interaction coverage and a recent evidence change.
Knowing which one you are holding is operationally useful, because the second will not stay ALLOW under much pressure.
Assurance Gaps
Coverage becomes most valuable when it reveals what is missing.
What an Assurance Gap can represent
The objective is not to hide gaps. It is to make them actionable.
- An untested decision boundary
- A missing transformation class
- Limited interaction coverage
- An unexplored temporal condition
- An unresolved evidence conflict
- An authority configuration that has not been evaluated
- A known production scenario without a regression test
- A new agent behavior outside the existing assurance envelope
Known gaps are better than hidden gaps
Optimize for truth, not for green
Traditional dashboards often optimize for green. An assurance platform should optimize for truth. Sometimes the most valuable finding is:
“This decision passed everything we tested — but this important condition has never been tested.”
That does not necessarily mean the system should be blocked. It means the organization can make a conscious risk decision rather than unknowingly operating outside its assurance evidence.
Prioritize what to test next
Enterprise decision spaces can become extremely large
Testing every conceivable combination may be impractical. The more useful question becomes: what should we test next? The current assurance picture can identify the areas that deserve additional evaluation.
Potential priorities
- High-impact decisions with low coverage
- Conditions near known boundaries
- Frequent evidence conflicts
- New policy changes
- New agent behaviors
- Historically fragile decisions
- Production incidents
- Frequent human overrides
- Newly observed state combinations
- Areas with significant uncertainty
This turns coverage from a reporting metric into a continuous assurance process.
Keeping assurance current
Assurance coverage is not permanent. A system can have strong coverage today and weaker effective coverage tomorrow — because the environment changes.
Coverage changes as the enterprise changes
Coverage is a living property, not a certificate
What moves the picture
- A new model is deployed
- A policy changes
- An agent gains access to a new tool
- A new data source is introduced
- A new customer segment appears
- A regulation changes
- A new decision boundary is discovered
- A production incident reveals an unknown interaction
AnantVerify therefore treats coverage as a living property of the operational decision system.
Know what a change affects
The assurance blast radius
An assurance blast radius, not a global re-test. Test what the change affects; preserve what remains valid.
When an important component changes, the enterprise should not have to rerun every test blindly. It should understand the potential assurance blast radius: which decision types depend on the change, which contracts are affected, which transformations rest on those contracts, and which evaluations should be rerun.
Test what the change affects. Preserve what remains valid. Reassess where assurance may have changed.
Continuous assurance regression
Yesterday's assurance can break after today's change
Continuous assurance regression
- Change
- Impact analysis
- Affected decision assurance
- Targeted evaluation + DTT
- Updated coverage
- Deployment decision
Software teams use regression testing because yesterday's working system can break after today's change. AI decision systems need the same principle — but across a broader environment.
Changes can occur in
- Models
- Agents
- Prompts
- Tools
- Enterprise state
- Policies
- Evidence sources
- Decision contracts
- Authority
- Workflows
This turns assurance into a continuous engineering discipline rather than a one-time certification exercise.
Production expands the assurance space
Pre-deployment testing can never anticipate every real-world condition
Production as assurance intelligence
- Production
- Discover
- Test
- Add to coverage
- Regression protect
Production therefore becomes another source of assurance intelligence. New observations can reveal:
- Previously unseen state combinations
- New boundaries
- Unexpected interactions
- Recurring human overrides
- Evidence conflicts
- Policy inconsistencies
- Decision incidents
- Unexpected outcomes
These discoveries can become new assurance requirements — the assurance envelope becomes stronger through operational experience.
From incident to permanent coverage
Don't discover the same AI failure twice
Incident to permanent coverage
- Incident
- Root cause
- Failure pattern
- Decision transformation
- Assurance requirement
- Regression test
- Coverage
Suppose an AI system produces a problematic decision because of a combination no one previously anticipated. The organization investigates, and the root cause is identified. AnantVerify can help turn that finding into a permanent assurance asset rather than a one-off fix.
The goal is straightforward: the organization should not have to discover the same AI failure twice.
Autonomy and honesty
Coverage is also a governance instrument. It describes where autonomy has been earned — and where it has not.
Coverage for AI agents
“Can the agent complete the task?” is not the whole question
An agent may
- Reason
- Use tools
- Delegate
- Retrieve information
- Modify state
- Call another agent
- Propose an action
A conventional evaluation may show that the agent completes the workflow successfully. AnantVerify adds another question:
“Across which enterprise conditions have we actually established that the resulting decision is appropriate?”
Coverage can therefore extend across
- Individual agent decisions
- Tool interactions
- Agent handoffs
- Multi-agent workflows
- State transitions
- Decision boundaries
- Execution actions
This moves agent evaluation from “can the agent complete the task?” toward “under what conditions are we prepared to let it act autonomously?”
Coverage can define the boundary of autonomy
Not maximum automation — maximum justified automation
Not every AI decision needs the same level of human involvement. An enterprise may have strong assurance for one region of the decision space and limited assurance elsewhere. That creates an opportunity for graduated autonomy: the covered region can act, the partly-covered region is held or escalated, and the unassured region is escalated or blocked.
The objective is not maximum automation. It is maximum justified automation.
From model confidence to enterprise confidence
Four different questions that should not be collapsed
A more honest assurance dashboard
The picture is usually more complicated than one aggregate number
Imagine two systems. Traditional reporting may make System A appear stronger.
| System A | System B | |
|---|---|---|
| Model accuracy | 96% | 94% |
| Tests passed | 9,842 / 10,000 | 1,650 / 1,700 |
Now add boundary coverage, interaction coverage, temporal coverage, evidence coverage, authority coverage, known critical gaps and decision robustness. The enterprise may discover that the picture is much more complicated than the headline numbers suggested.
AnantVerify is designed to expose that complexity rather than hide it behind one aggregate number.
Don't ask only “Did it pass?”
The difference between test execution and assurance engineering
Ask instead
- What did we test?
- What didn't we test?
- Where are the boundaries?
- Which interactions matter?
- How robust is this decision?
- What has changed since the last evaluation?
- Which assurance evidence is still valid?
- Where should we test next?
The principle
You cannot meaningfully claim assurance without knowing the limits of your evidence
Passing tests matters. But so does knowing what those tests represent — and knowing what they do not.
AnantVerify makes those limits visible.
Where coverage and robustness meet
- ALLOW region
- HOLD region
- ESCALATE region
- BLOCK region
- Tested condition
- Known, not yet tested
- Gap outside the envelope
- Current decision
- Assurance envelope
- Unexplored
Regions radiate from the current decision as meaningful state, evidence and authority conditions change.
Illustrative. Regions, points and coverage figures are constructed to show the method — not measured from a system.
Know where your AI is assured — and where it isn't
Enterprise AI does not need artificial certainty. It needs explicit evidence about where automation is justified.
- Decision Transformation Testing
- Assurance Coverage
- Decision Robustness
- Evidence & Authority
- State Assurance
- Continuous Regression
Not “the model passed”. But
“We know what was tested, how the decision behaves, where the gaps are, and where we are prepared to let it act.”
The method behind the coverage classes is the DTT methodology. The states and verdicts themselves are Runtime Control.
Illustrative. Regions, points and coverage figures are constructed to show the method — not measured from a system. Coverage is measured against registered assurance requirements; it is not a probability of safety.