Governing agents at work

What must oversight cover before agents go live, and after?

Direct answer. Testing before deployment shows how the system behaved under the conditions tested. Oversight during operation examines the work performed since, including whether the outcome was achieved and whether the controls worked.

Management defines the intended outcomes, the risk limits that put the board's risk appetite into practice, and accountability for each AI-enabled process. The people responsible need evidence that the work meets those expectations and authority to respond when it does not.

Testing before deployment shows how the system behaved under the conditions tested. Oversight during operation examines the work performed since, including whether the outcome was achieved and whether the controls worked.

The plan-do-check-act cycle in Figure 2 connects these responsibilities before and after agents go live.

Checks run during operation as well as at scheduled reviews. Findings may require immediate intervention or changes to the wider plan. Appendix 1.1 explains how this management cycle connects to the detailed workflow and the frameworks used in this paper.

Which control domains should you extend for agents?

Direct answer. For each AI-enabled process, establish who owns the outcome, what agents may do without approval, and which decisions remain with people.

Start with the controls the organization already uses. For each AI-enabled process, establish who owns the outcome, what agents may do without approval and which decisions remain with people.

The twelve control domains show where those controls may need extending, from planning and change management to oversight of live work. Use them to identify gaps in responsibility, control and evidence. The detailed reference is in Appendix 1.2.

Why do people still have to be in the loop?

Direct answer. As routine cases are automated, reviewers may receive a higher proportion of ambiguous, disputed, or consequential cases. Plan capacity for that workload.

Agents change the work people must do. As routine cases are automated, reviewers may receive a higher proportion of ambiguous, disputed or consequential cases. Plan capacity for that workload, rather than assuming the old mix of cases still applies.

Plan and budget for the people who review, correct and escalate the work as part of the business case. Give reviewers the authority to refuse, the evidence and expertise to judge, and enough time to act. Training, breaks and rotation should reflect the demands of the work.

Set limits on how much work can await review and for how long. Agree what happens when those limits are reached: add capacity, narrow what agents may do, slow the flow or escalate to the business process owner.

Track review delays, overrides, rework and recurring problems alongside speed and cost. To assess whether oversight works in practice, ask reviewers to show a recent case they refused, changed or escalated, and why.

Appendix 1.3 sets out roles, workload, skills and tools, review limits and measures in detail.

How do you monitor outcomes and act when work goes wrong?

Direct answer. Define the intended outcome, operating limits, and the response to problems before the work begins. When outcomes deteriorate, authorized people decide whether the work should continue, be restricted, or stop.

Define the intended outcome, operating limits and response to problems before the work begins. Monitor the business results as well as the tasks completed.

When outcomes deteriorate, controls fail or material conditions change, use the agreed response. Systems enforce required holds; authorized people resolve exceptions and decide whether the work should continue, be restricted or stop.

Match the response to the potential harm, how quickly it could spread, whether it can be reversed, applicable obligations and the consequences of interrupting the service. A problem in one part of a process may justify restricting that part rather than stopping everything.

If a required pre-action check fails, is inconclusive or returns no result, hold the action for review. If the checking service is unavailable, pause the work or use an approved fallback where the risk permits. Record the evidence gap; an outage is not a passed check.

Problems found after an action require correction and a decision about subsequent work. After a stop, reconcile unfinished work, correct affected records and address the cause before the authorized person approves restart. Preserve a route for affected people to challenge the outcome.

Appendix 1.5 lists the impact factors and response options, and explains fallback and restart.

Which controls have to hold in live operation?

Direct answer. A written instruction alone does not prevent an action that the available tools still permit. Authorization does not establish that the work was accurate, complete, or achieved its intended outcome.

Enforce limits through the systems agents use. Give each agent identity a named owner and only the access it needs, and test whether the access paths preserve the separation of duties the process requires. A written instruction alone does not prevent an action that the available tools still permit.

Approve and record material changes to models, instructions, tools, data and permissions. In the proposed design, where a consequential action requires approval, the approval identifies that action and the recorded scope it permits. Reassess it when relevant conditions change.

Then check the business result. Authorization does not establish that the work was accurate, complete or achieved its intended outcome. Appendix 1.4 shows how these controls apply to a refund requiring human approval.

What should assurance evidence show a leader?

Direct answer. Leaders need a supported assessment of whether agent work and its controls met agreed criteria within a defined scope and period, so they can decide how much authority agents should hold.

Why it matters for agents

Assurance gives leaders a supported assessment of whether agent work and its controls met agreed criteria within a defined scope and period, so they can decide with confidence how much authority agents should hold. It requires competent, objective reviewers, an agreed assessment process and systems that capture reliable evidence as the work happens.

Three features of agent-enabled work make this important:

• They change often. Models are updated, instructions and tools change and workflows are recomposed, sometimes by a supplier. Testing before an agent goes live shows how it behaved then, not how it behaves now.
• They can be wrong in convincing ways. Agents built on large language models produce likely answers rather than guaranteed ones. They can carry bias, which NIST's AI Risk Management Framework treats as a risk to manage, and in one study, five widely used AI assistants tended to favor a user's stated views over the correct answer.
• They cannot be relied on to check themselves. In one study, language models struggled to correct their own reasoning without outside feedback. Check the work against rules, source records or other evidence instead. A separate checking component helps, but does not by itself make the review independent.

Software delivery relies on testing and quality assurance before release. Agents need that discipline carried into live operation: checks on the work itself, as it runs and periodically, matched to the risk (Figure 5). Figure 6 gives examples.

Sycophancy: Sharma et al., Towards Understanding Sycophancy in Language Models, arXiv:2310.13548 (2023). Self-correction: Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet, ICLR 2024, arXiv:2310.01798.

Appendix 1 sets out the controls for each control domain.

Before relying on an assurance conclusion, ask what work and period it covers, what it was measured against, what was checked and what remains unresolved. Management's own checks support that assessment; independent assurance needs a reviewer who is independent of the work. Appendix 1.7 describes assurance more fully, including who provides it and how independence is judged.

What the evidence should show

Keep enough evidence to establish what work was done, what requirements applied, what was checked and what each result meant. Show the population in scope, the coverage achieved and any gaps. Preserve the history needed to understand earlier decisions under the organization's retention requirements.

Keep failed, inconclusive and unassessed work separate: they require different follow-up. Give unresolved issues an owner and a next action.

The checks also need scrutiny. Review their test results, false alarms, missed errors, known limitations and failure modes before relying on their findings.

For the records to keep and how to count coverage, see Appendix 1.6.

Which requirements and guidance apply?

Direct answer. First determine, with risk, compliance, and legal teams, which requirements apply to the process. Recurring expectations are accountability, defined authority and limits, controlled changes, human oversight where required, monitoring, evidence, and a response when things go wrong.

In the frameworks reviewed here, the recurring operating expectations are similar: clear accountability, defined authority and limits, controlled changes, human oversight where required, monitoring of live operation, evidence of what happened, and a response when things go wrong.

First determine, with risk, compliance and legal teams, which requirements apply to the AI-enabled process, given the organization's role and jurisdiction, the system's intended use and classification, and application dates.

They come in four kinds. Binding legal and policy requirements, such as the EU AI Act and, for Canadian federal institutions, the Directive on Automated Decision-Making, apply by role, system and date. Supervisory guidance, such as OSFI's E-23, PRA SS1/23 and the US model-risk guidance, applies to regulated firms within its scope; SR 26-2 expressly leaves generative and agentic AI outside its scope. OSFI's July 2026 technology risk bulletin sets out sound practices institutions can consider. Voluntary frameworks, such as NIST AI RMF and ISO/IEC 42001, name functions and management-system requirements. Professional guidance, such as the IIA's, sets expectations for those who review the work.

The detailed table is in Appendix 2. It is a scoping aid, not a conformance assessment: each entry states what the regulation, standard or guidance contributes and what it leaves to you.

What should executives ask management to show?

Direct answer. Use these questions before agents are given more authority and at periodic reviews. Ask management to answer for a specific process, using recent work and supporting evidence.

Use these questions before agents are given more authority and at periodic reviews. Ask management to answer for a specific process, using recent work and supporting evidence.

Appendix 1. Control and operating reference

Contents: 1.1 Oversight and the workflow · 1.2 Control domains · 1.3 Human review capacity · 1.4 Controls in live operation · 1.5 Responding when problems arise · 1.6 Evidence records and coverage · 1.7 Review arrangements and independence.

1.1 Oversight and the workflow

The main paper summarizes management oversight in four steps.

The oversight loop is the organization's management system: plan, do, check and act. It establishes policy and ownership, reviews risk and performance, and directs improvement.

The workflow loop follows one AI-enabled process from planning and design through build, verification and deployment, then operation, monitoring, assurance and response. Evidence from live operation informs decisions about continuing it and about changes to wider governance.

We infer the plan-do-check-act cycle from ISO/IEC 42001's clause structure. The workflow loop adapts the model lifecycle in OSFI's Guideline E-23 and extends it through operation, assurance and response. NIST's Govern, Map, Measure and Manage functions sit across both loops.

1.2 Control domains

For each control domain, the table names the established control in generally accepted terms, the additional controls we propose for agent work, and the regulations and standards that map to it; Appendix 2 sets out their scope. Use it to check whether the controls you already run cover the work agents now do, and where they need extending.

Items 2 to 6 adapt the AI lifecycle stages in NIST AI RMF 1.0; the other domains group established control disciplines.

1.3 Human review capacity

The work that reaches people changes. A team used to handle a mix of cases, many of them simple. Once agents take the routine cases, what reaches people is mostly what the agents escalate: the unusual, the ambiguous and the disputed, often with a customer already waiting. The remaining cases may take longer and require more judgment, and a full day of them is more tiring than a mixed workload. Plan staffing and support for that changed workload. The risk is that reviewers under that load, and under queue pressure, approve what is in front of them: the tendency the EU AI Act calls automation bias.

Plan the people in the loop like any other capacity:

• Roles. Who directs the agents, who reviews their work, who approves consequential actions and who corrects errors. Each needs the authority to say no, enough evidence, the knowledge to judge it and the time to do it.
• Load. Size teams for the harder cases that now reach them. Build in breaks, and rotation between demanding and lighter work, so that a reviewer at the end of a shift can still refuse.
• Skills and tools. Train people to handle escalated cases and to question an agent's output. Give them tools that help: case summaries that show they were AI-generated, what they drew on and their limits; grouping of similar exceptions; and quick routes to specialists.
• Limits. How much work can wait for review, and for how long, before the agents' work is slowed, narrowed or escalated to the business process owner.
• Measures. A few that show whether oversight is working: the review queue against its limit, the age of the oldest case, how often people override the agent, and recurring problems.

1.4 Controls in live operation

The five controls below change most when agents act in live operation. They build on controls most organizations already test, such as accountability, access and change control, and checks on each transaction, rather than adding a separate framework.

1. Limits on acting alone (Governance and risk appetite). Define what an agent may do without further approval, including the consequences of its actions, and enforce those limits through the system.

2. Access and segregation of duties (Security, identity and observability). Give each agent identity a named owner and only the access it needs. Make sure an agent cannot combine access or actions that would be kept apart for a person, and test this through the credentials and tools the agent actually uses.

3. Change control (Deployment and change management). Approve and record changes to models, prompts, tools, data and permissions. Where an action must be prevented, restrict the capability that enables it; a written instruction alone does not enforce that restriction.

4. Approval for one action (Deployment and change management). In our design, approval identifies a specific action and a recorded version of its permitted scope. Reassess it when the relevant conditions change; a standing approval may no longer reflect the circumstances of the action.

5. Checks on completed work (Assurance). Check whether the completed work was accurate, complete, valid and within the approved scope. Authorization is part of that assessment; it does not establish that the intended business outcome was achieved.

1.5 Responding when problems arise

Problems come to light in different ways: outcomes fall short, a check fails, a customer complains, a member of staff notices something wrong, or a supplier changes a model the agents rely on. Sometimes the checking service itself stops working, through an outage, a failed change or a supplier problem. When outcomes fall short or something material changes, the person authorized decides whether the work continues, is restricted or stops. The evidence informs that decision; a person makes it.

Base the response on the harm the problem could do to customers and the business. Weigh:

• how much damage one wrong action can do, and whether it can be reversed;
• how many customers or transactions it could reach, and how quickly;
• whether the problem is still happening or was a single event;
• what regulators, customers and contracts require, since some rules specify the response and its timing (see the box below);
• what it costs customers and the business to slow or stop the work.

The options run from light to severe:

• Continue and watch more closely. Check more cases, tighten alerts and follow the trend. Suits a contained, low-impact problem.
• Hold the action for a person. The system holds the action automatically when a check required before an action finds a problem, cannot reach a conclusion or returns nothing; a person decides how to resolve it.
• Restrict. Lower limits, remove tools or send certain cases to people until the cause is fixed.
• Use an approved manual or fallback path where the risk permits and the work cannot wait. Record the evidence gap and reconcile the work when checking resumes.
• Pause or stop the agents. Use this when harm could spread quickly or cannot be undone.

A problem found in completed work calls for correction and a fresh decision on whether the work should continue, since the action has already happened. Distinguish between a check that found a problem and a checking service that was unavailable.

In August 2026 a crashed check let a test payment through in our build; we changed it so that a check that crashes holds the action for a person.

In our design, an outage can send an existing process down its approved fallback when the risk allows. That design remains open to review. It should not be read as permission for an agent to continue unchecked.

After a stop, the recovery plan must identify who reconciles work already started, corrects affected records and authorizes restart, accounting for unfinished actions and the people they affect.

An authorized person decides how to resolve each exception; neither the agent nor the checker makes that disposition. The permitted decisions are reject; approve once, with conditions and an expiry; require remediation; narrow scope; escalate; suspend; or another disposition the organization's mandate allows.

Confirm closure through a later check, preferably by someone other than the person who performed the remediation. Where a reviewer concludes that an exception was raised in error, retain the original finding and record it as “disputed as raised”, with the reason.

Record when the failure occurred separately from when it was discovered. Preserve the required route for review or recourse where a decision continues to affect a person.

1.6 Evidence records and coverage

Evidence is what lets someone check, later, whether something is or was true. Figure 11 shows what a reviewer asks about any check, and what to keep to answer each.

The five answers help a reviewer judge whether evidence is relevant, reliable and sufficient, the tests the IIA's Global Internal Audit Standards set for the information internal auditors gather (Standard 14.1).

1. What did the check look at? Keep a record made when the check ran, showing the time, the exact material it examined and where that came from, and which version of the check and of the requirement applied.

2. How much was checked? For a stated period, keep counts of the cases in scope, how many were checked and which were not. Take the total from the system of record, so it also counts work the check missed. Say when the total is unknown, whether the check covered every case or a sample, and how any sample was chosen. Figure 7 shows one week of refund counts.

3. What did each result mean? Keep “failed”, “inconclusive” and “not assessed” apart, as Figure 7 does. Evidence of a breach makes a result a failure; missing evidence makes it inconclusive. For a check before an action, all three results hold the action for review. For a check on completed work, each goes to an owner for correction or further checking.

4. What did an earlier decision rely on? Keep the original record and add any correction beside it, with the date, the reason and who made it, so a reviewer can see both. Check that restoring or restarting a system leaves earlier records intact.

5. Can a reviewer check it for themselves? Keep enough for a competent reviewer to follow what each check did and reach the same conclusion. A check that uses a model may answer differently when run again, so keep the answer it gave. Give the reviewer the records as an evidence package, with an integrity check they can run independently of the team that produced it, to confirm the records are unchanged. Integrity checks do not establish that the underlying work was correct or the evidence complete.

Name the owner who applies the organization's retention and deletion requirements to these records, rather than leaving either to the tool's default. Keep the material a record identifies for as long as the record itself, within data protection limits, so a reviewer can still follow the check.

1.7 Review arrangements and independence

The IIA's Global Internal Audit Standards (2024) glossary defines assurance: “Statement intended to increase the level of stakeholders’ confidence about an organization’s governance, risk management, and control processes … when compared to established criteria.” It defines assurance services as “Services through which internal auditors perform objective assessments to provide assurance.”

What assurance involves

Assurance means two things. It is a conclusion: a statement of how far leaders can rely on what was done, judged against suitable criteria. It is also the work that produces that conclusion: examining and testing what actually happened against those criteria.

Delivering assurance takes four things together:

• People. Competent, objective reviewers: management checks the work, risk and compliance oversee it, and internal audit or an external provider reviews it independently.
• Process. Criteria suited to the risks and objectives, a defined scope and period, and an agreed method for testing the work and recording the conclusion.
• Technology. Systems that capture evidence as the work happens, keep it intact and let a reviewer reach it.
• Evidence. Relevant, reliable and sufficient records, each showing where it came from and what it covers.

Management's own checks support an assurance assessment. Independent assurance needs a reviewer who is independent of the work and reaches their own conclusion from the evidence. Management, risk and compliance, and the independent reviewer each test the evidence before relying on it.

Embedded controls and independent assurance are distinct disciplines. An embedded checker can be useful and still not be independent. An operational checking service used by management remains part of management's controls, even when an external supplier operates it. An external assurance engagement is a separate arrangement, assessed against its mandate and independence requirements. Internal audit can provide independent assurance where it is positioned independently, with the chief audit executive reporting directly to the board, as can an external assurance provider within its mandate and independence requirements.

The reviewer weighs all the evidence for relevance, reliability and sufficiency: the operator's own records, model-provider tests and platform controls, each with its origin and scope visible. The reviewer also needs evidence about the checker itself: test results, false alarms, known failure modes and the classes of problem it has not seen. In our own testing, all findings in one overnight run traced to one checker bug.

Internal audit can rely on management's checks and on other assurance providers once it has evaluated them: who did the work, how independent and competent they are, how they did it, and whether the evidence supports their results. It records the basis for that reliance. The reviewer must assess the evidence and reach their own conclusion about what it supports. Each finding then needs an owner and a date for correction.

Appendix 2. Regulations and standards in detail

The Section column shows where Appendix 1 discusses each one: Oversight and the workflow (1.1), Control domains (1.2), Human review capacity (1.3), Controls in live operation (1.4), Responding when problems arise (1.5), Evidence records and coverage (1.6), and Review arrangements and independence (1.7).

Appendix 3. Glossary

Agent

Software that can choose steps or tools and take business actions.

AI-enabled work

Business work in which an AI system decides, recommends or acts.

Assurance

Both a conclusion and the work that produces it: a statement, measured against agreed criteria, of how far leaders can rely on a piece of work, and the examining and testing behind it (after the IIA Global Internal Audit Standards). Independent assurance needs reviewers independent of the work and of the checks built into it.

Continuous assurance, as used in this paper

Checking AI-enabled work against its intended business outcome and agreed requirements, using the evidence to inform decisions on whether the work continues and within what limits.

Coverage

The work assessed over a stated period, with the in-scope population and selection method made clear. State when the total population is unknown.

Failed / inconclusive / not assessed

Failed: examined, and the evidence shows the requirement was not met. Inconclusive: examined but unresolved, for example because evidence was missing. Not assessed: not examined.

Exception and disposition

An issue raised by a check, and the decision an authorized person makes about it.

Evidence package

A bundle of evidence and verification material prepared for review.

Reperformance

Running a check again. It is distinct from an independent re-execution by an outside party.

Non-human identity

The credential or workload identity an agent uses, with a named owner.

Appendix 4. Sources

Regulation and supervisory guidance

• European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). OJ L, 2024/1689, 12 July 2024. eur-lex.europa.eu
• European Parliament and Council of the European Union. Regulation (EU) 2026/1744 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 (Digital Omnibus on AI). OJ L, 2026/1744, 24 July 2026; entered into force 27 July 2026. eur-lex.europa.eu
• Office of the Superintendent of Financial Institutions (Canada). Guideline E-23 – Model Risk Management (2027). Published 11 September 2025; effective 1 May 2027. osfi-bsif.gc.ca
• Office of the Superintendent of Financial Institutions (Canada). Guideline B-13 – Technology and Cyber Risk Management. Effective 1 January 2024.
• Office of the Superintendent of Financial Institutions (Canada). Generative and Agentic Artificial Intelligence: Implications for Technology, Cyber Security, and Operational Resilience. Technology Risk Bulletin, July 2026. osfi-bsif.gc.ca
• Board of Governors of the Federal Reserve System. SR 26-2: Revised Guidance on Model Risk Management. 17 April 2026. federalreserve.gov
• Board of Governors of the Federal Reserve System, Federal Deposit Insurance Corporation and Office of the Comptroller of the Currency. Supervisory Guidance on Model Risk Management. SR 26-2 attachment, 17 April 2026.
• Prudential Regulation Authority. SS1/23 – Model risk management principles for banks. Supervisory statement, current version 16 April 2026 (updating 17 May 2023). bankofengland.co.uk
• Treasury Board of Canada Secretariat. Directive on Automated Decision-Making. Published 29 March 2021; page modified 24 June 2025. tbs-sct.canada.ca
• US Office of Management and Budget. M-25-21 – Accelerating Federal Use of AI through Innovation, Governance, and Public Trust. 3 April 2025. whitehouse.gov
• US Office of Management and Budget. M-25-22 – Driving Efficient Acquisition of Artificial Intelligence in Government. 3 April 2025. whitehouse.gov
• National Association of Insurance Commissioners. NAIC Model Bulletin: Use of Artificial Intelligence Systems by Insurers. Adopted 4 December 2023. content.naic.org

Standards and frameworks

• National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, January 2023. nist.gov
• National Institute of Standards and Technology. Secure Software Development Framework (SSDF) Version 1.1: Recommendations for Mitigating the Risk of Software Vulnerabilities. NIST SP 800-218, February 2022.
• National Institute of Standards and Technology. Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile. NIST SP 800-218A, July 2024.
• The Institute of Internal Auditors. Global Internal Audit Standards. Issued 9 January 2024; effective 9 January 2025. Glossary; Standard 14.1. theiia.org
• The Institute of Internal Auditors. The IIA's Artificial Intelligence Auditing Framework. 2024.
• The Institute of Internal Auditors. Global Technology Audit Guide (GTAG) 3: Coordinating Continuous Auditing and Monitoring to Provide Continuous Assurance. 2nd edition, March 2015.
• The Institute of Internal Auditors. The IIA's Three Lines Model: An Update of the Three Lines of Defense. Position paper, 2020; updated September 2024. theiia.org
• US Government Accountability Office. Standards for Internal Control in the Federal Government (Green Book). GAO-25-107721, May 2025.
• HM Treasury. The Orange Book – Management of Risk – Principles and Concepts. 2023 edition.
• International Organization for Standardization and International Electrotechnical Commission. Information technology – Artificial intelligence – Management system. ISO/IEC 42001:2023, first edition, December 2023. iso.org
• Committee of Sponsoring Organizations of the Treadway Commission (COSO). Internal Control – Integrated Framework. May 2013. coso.org
• Committee of Sponsoring Organizations of the Treadway Commission (COSO). Enterprise Risk Management – Integrating with Strategy and Performance. June 2017. coso.org
• CEN-CENELEC Joint Technical Committee 21. Artificial intelligence – Quality management system for EU AI Act regulatory purposes. EN 18286:2026, available 22 July 2026. cencenelec.eu
• OWASP Gen AI Security Project. OWASP Top 10 for Agentic Applications for 2026. Published 9 December 2025.
• FINOS (Fintech Open Source Foundation). FINOS AI Governance Framework. Version 2, 20 October 2025.
• Cloud Security Alliance, AI Safety Initiative. The AI Agent Governance Gap: What CISOs Need Now. 3 April 2026.

Research on language models

• Sharma et al. Towards Understanding Sycophancy in Language Models. arXiv:2310.13548, 2023. arxiv.org
• Huang et al. Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024; arXiv:2310.01798. arxiv.org

Cases

• Civil Resolution Tribunal of British Columbia. Moffatt v. Air Canada. 2024 BCCRT 149, February 2024. civilresolutionbc.ca