AI governance products increasingly claim runtime visibility. Buyers should separate observation from intervention and identify the system in which a control can actually change the outcome.
Four Levels of Runtime Capability
- Observe: capture prompts, outputs, identities, tool calls and context.
- Alert and advise: identify a policy condition and notify an operator without changing the action.
- Require approval: pause the action until a person or trusted service provides step-up verification.
- Constrain or stop: redact, limit, block, isolate, pause or terminate through an enforcement point.
A supplier may be strong at one level and depend on integrations for another. That is not inherently a weakness, but the boundary must be explicit.
Put Controls Where Authority Is Exercised
Prompt filters cannot govern every consequence. If an agent can change a record, send a payment or deploy code, control may also be needed in identity, transaction, application or workflow systems. The right architecture combines AI-specific context with established enforcement points.
Define the Policy Decision and Enforcement Point
For each material action, specify the signal, applicable policy, decision options and system that can apply the result. A policy engine that returns “deny” has not enforced the decision if the tool or target application still completes the action.
Choose the Response Mode by Consequence
| Agent action | Signal / context | Policy decision | Runtime response | Human role | Evidence |
|---|---|---|---|---|---|
| Read approved knowledge | Identity, source and data classification | Allow within scope | Log and continue | Review exceptions | Access and source trace |
| Send a customer message | Destination, content and customer type | Approve or redact | Pause before transmission | Confirm high-impact cases | Draft, decision and sent version |
| Issue a refund above limit | Value, account, frequency and delegated authority | Deny or step up | Block and escalate | Authorised approver decides | Request, policy, approver and outcome |
Design for Failure
Define what happens if the policy engine, model, approver or integration is unavailable. High-impact actions may need fail-closed behaviour; lower-risk work may pause or continue in a non-action mode. Operators need controlled override, expiry and retrospective review.
Preserve Performance Without Bypassing Control
Measure policy latency and timeout behaviour under realistic load. Do not allow a slow or unavailable control to create an undocumented bypass. Where cached decisions are used, define their scope, duration and invalidation after policy or identity change.
Verify the Outcome
Runtime governance is not complete when an alert is generated. Evidence should show the triggering context, policy version, decision, enforcement result and final state. Connect this record to the wider governance evidence model.
Test Runtime Control Claims
Ask the supplier to show one permitted action, one action requiring approval and one blocked action. Then remove the approver, make the policy service unavailable and repeat the test after a permission change.
An agent attempts a transaction beyond its permitted amount. Verify the policy receives the value and identity, pauses or blocks the request, routes it to the correct approver and records whether the external system changed.
Use the agentic AI governance playbook to place runtime tests inside the wider production gate.