Governing agents at work

A1. Oversight before and after agents go live

Management defines the intended outcomes, the risk limits that put the board's risk appetite into practice, and accountability for each AI-enabled process. The people responsible need evidence that the work meets those expectations and authority to respond when it does not.

Testing before deployment shows how the system behaved under the conditions tested. Oversight during operation examines the work performed since, including whether the outcome was achieved and whether the controls worked.

The plan-do-check-act cycle in Figure 2 connects these responsibilities before and after agents go live.

Checks run during operation as well as at scheduled reviews. Findings may require immediate intervention or changes to the wider plan. Appendix 1.1 explains how this management cycle connects to the detailed workflow and the frameworks used in this paper.

A2. Control domains for governing agents

Start with the controls the organisation already uses. For each AI-enabled process, establish who owns the outcome, what agents may do without approval and which decisions remain with people.

The twelve control domains show where those controls may need extending, from planning and change management to oversight of live work. Use them to identify gaps in responsibility, control and evidence. The detailed reference is in Appendix 1.2.

B1. People in the loop are essential

Agents change the work people must do. As routine cases are automated, reviewers may receive a higher proportion of ambiguous, disputed or consequential cases. Plan capacity for that workload, rather than assuming the old mix of cases still applies.

Industrial automation showed the same pattern: Bainbridge, Ironies of automation, Automatica 19(6), 1983, described how automation leaves operators the tasks designers could not automate, and how skills fade when they go unused.

Plan and budget for the people who review, correct and escalate the work as part of the business case. Give reviewers the authority to refuse, the evidence and expertise to judge, and enough time to act. Training, breaks and rotation should reflect the demands of the work.

Set limits on how much work can await review and for how long. Agree what happens when those limits are reached: add capacity, narrow what agents may do, slow the flow or escalate to the business process owner.

Track review delays, overrides, rework and recurring problems alongside speed and cost. To assess whether oversight works in practice, ask reviewers to show a recent case they refused, changed or escalated, and why.

Appendix 1.3 sets out roles, workload, skills and tools, review limits and measures in detail.

B2. Monitor outcomes and act when things go wrong

Agents may repeat an error across many transactions before anyone intervenes.

Before it runs. Define the intended outcome, operating limits and response to problems.

While it runs. Monitor business results as well as tasks completed. When conditions change significantly, reassess whether the work should continue.

When it goes wrong. Use the agreed response when outcomes deteriorate, controls fail or material conditions change. Systems enforce required holds; authorised people resolve exceptions and decide whether to continue, restrict or stop. Weigh potential harm, how quickly it could spread, whether it can be reversed, applicable obligations and the cost of interrupting service. Restricting one part may be better than stopping everything.

If the checking service is down, hold the work, or use a fallback that management has approved, provided the risk stays within the limits it has set. An outage is not a passed check.

Correct problems in completed work, decide whether subsequent work should continue and preserve a route for affected people to challenge the outcome.

See Responding when problems arise (Appendix 1.5) for required holds, fallback, correction and recovery.

B3. Controls in live operation

A written rule cannot prevent an action that an agent's tools still allow, so enforce limits through the systems it uses.

Record who owns each agent identity and give it only the access it needs. Test whether its access paths preserve the separation of duties the process requires.

Approve and record material changes to models, instructions, tools, data and permissions.

Where an action needs approval, the approval identifies that action and the recorded scope it permits. Reassess the approval when relevant conditions change.

Then check the business result. Authorisation does not establish that the work was accurate, complete or achieved its intended outcome. Controls in live operation (Appendix 1.4) shows how these controls apply to a refund requiring human approval.

B4. The role of assurance

Why it matters for agents

Assurance provides a supported assessment of whether agent work and its controls met agreed criteria within a defined scope and period. The assessment helps leaders decide how much authority agents should hold. It requires competent, objective reviewers, an agreed assessment process and systems that capture reliable evidence as the work happens.

Three features of agent-enabled work make this important:

• They change often. Models are updated, instructions and tools change and workflows are recomposed, sometimes by a supplier. Testing before an agent goes live shows how it behaved then, not how it behaves now.
• They can be wrong in convincing ways. Agents built on large language models produce likely answers rather than guaranteed ones. They can carry bias, which NIST's AI Risk Management Framework treats as a risk to manage, and in one study, five widely used AI assistants tended to favour a user's stated views over the correct answer.
• They cannot be relied on to check themselves. In one study, language models struggled to correct their own reasoning without outside feedback. Check the work against rules, source records or other evidence instead. A separate checking component helps, but does not by itself make the review independent.

Software delivery relies on testing and quality assurance before release. Agents need that discipline carried into live operation: checks on the work itself, as it runs and periodically, matched to the risk (Figure 7). Figure 8 gives examples.

Sycophancy: Sharma et al., Towards Understanding Sycophancy in Language Models, arXiv:2310.13548 (2023). Self-correction: Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet, ICLR 2024, arXiv:2310.01798.

Appendix 1 sets out the controls for each control domain.

Before relying on an assurance conclusion, ask what work and period it covers, what it was measured against, what was checked and what remains unresolved. Management's own checks support that assessment; independent assurance needs a reviewer who is independent of the work. Appendix 1.7 describes assurance more fully, including who provides it and how independence is judged.

What the evidence should show

Keep enough evidence to establish what work was done, what requirements applied, what was checked and what each result meant. Show the population in scope, the coverage achieved and any gaps. Preserve the history needed to understand earlier decisions under the organisation's retention requirements.

Keep failed, inconclusive and unassessed work separate: they require different follow-up. Give unresolved issues an owner and a next action.

The checks also need scrutiny. Review their test results, false alarms, missed errors, known limitations and failure modes before relying on their findings. Look beyond a single accuracy or confidence figure. Count errors by their consequences: unauthorised actions that went ahead, work refused that a person could have approved, and needless escalations.

For the records to keep and how to count coverage, see Appendix 1.6.

C1. Which requirements and guidance apply?

In the frameworks reviewed here, the recurring operating expectations are similar: clear accountability, defined authority and limits, controlled changes, human oversight where required, monitoring of live operation, evidence of what happened, and a response when things go wrong.

First determine, with risk, compliance and legal teams, which requirements apply to the AI-enabled process, given the organisation's role and jurisdiction, the system's intended use and classification, and application dates.

Requirements and guidance fall into four kinds:

• Binding legal and policy requirements, such as the EU AI Act and, for Canadian federal institutions, the Directive on Automated Decision-Making, apply by role, system and date.
• Supervisory guidance, such as OSFI's E-23, PRA SS1/23 and the US model-risk guidance, applies to regulated firms within its scope; SR 26-2 expressly leaves generative and agentic AI outside its scope. OSFI's July 2026 technology risk bulletin sets out sound practices institutions can consider.
• Voluntary frameworks, such as NIST AI RMF and ISO/IEC 42001, name functions and management-system requirements.
• Professional guidance, such as the IIA's, sets expectations for those who review the work.

The detailed table is in Appendix 2. It is a scoping aid, not a conformance assessment: each entry states what the regulation, standard or guidance contributes and what it leaves to you.

C2. Questions to ask management

Use these questions before agents are given more authority and at periodic reviews. Ask management to answer for a specific process, using recent work and supporting evidence.

Appendix 1. Control and operating reference

Contents: 1.1 Oversight and the workflow · 1.2 Control domains · 1.3 Human review capacity · 1.4 Controls in live operation · 1.5 Responding when problems arise · 1.6 Evidence records and coverage · 1.7 Review arrangements and independence.

1.1 Oversight and the workflow

The main paper summarises management oversight in four steps.

The oversight loop is the organisation's management system: plan, do, check and act. It establishes policy and ownership, reviews risk and performance, and directs improvement.

The workflow loop follows one AI-enabled process from planning and design through build, verification and deployment, then operation, monitoring, assurance and response. Evidence from live operation informs decisions about continuing it and about changes to wider governance.

We infer the plan-do-check-act cycle from ISO/IEC 42001's clause structure. The workflow loop adapts the model lifecycle in OSFI's Guideline E-23 and extends it through operation, assurance and response. NIST's Govern, Map, Measure and Manage functions sit across both loops.

1.2 Control domains

For each control domain, the table names the established control in generally accepted terms, the additional controls we propose for agent work, and the regulations and standards that map to it; Appendix 2 sets out their scope. Use it to check whether the controls you already run cover the work agents now do, and where they need extending.

Items 2 to 6 adapt the AI lifecycle stages in NIST AI RMF 1.0; the other domains group established control disciplines.

General IT controls continue to apply to the systems, data and infrastructure agents use; the table sets out the additional controls proposed for agent work.

1.3 Human review capacity

The work that reaches people changes. A team used to handle a mix of cases, many of them simple. Once agents take the routine cases, what reaches people is mostly what the agents escalate: the unusual, the ambiguous and the disputed, often with a customer already waiting. The remaining cases may take longer and require more judgment, and a full day of them is more tiring than a mixed workload. Plan staffing and support for that changed workload. The risk is that reviewers under that load, and under queue pressure, approve what is in front of them: the tendency the EU AI Act calls automation bias.

Plan the people in the loop like any other capacity:

• Roles. Who directs the agents, who reviews their work, who approves consequential actions and who corrects errors. Each needs the authority to say no, enough evidence, the knowledge to judge it and the time to do it.
• Load. Size teams for the harder cases that now reach them. Build in breaks, and rotation between demanding and lighter work, so that a reviewer at the end of a shift can still refuse.
• Skills and tools. Train people to handle escalated cases and to question an agent's output. Keep them doing some of the work themselves, so their judgement stays in practice. Give them tools that help: case summaries that show they were AI-generated, what they drew on and their limits; grouping of similar exceptions; and quick routes to specialists.
• Limits. How much work can wait for review, and for how long, before the agents' work is slowed, narrowed or escalated to the business process owner.
• Measures. A few that show whether oversight is working: the review queue against its limit, the age of the oldest case, how often people override the agent, and recurring problems.

1.4 Controls in live operation

The five controls below change most when agents act in live operation. They build on controls most organisations already test, such as accountability, access and change control, and checks on each transaction, rather than adding a separate framework.

1. Limits on acting alone (Governance and risk appetite). Define what an agent may do without further approval, including the consequences of its actions, and enforce those limits through the system.

2. Access and segregation of duties (Security, identity and observability). Give each agent identity a named owner and only the access it needs. Make sure an agent cannot combine access or actions that would be kept apart for a person, and test this through the credentials and tools the agent actually uses.

3. Change control (Deployment and change management). Approve and record changes to models, prompts, tools, data and permissions. Where an action must be prevented, restrict the capability that enables it; a written instruction alone does not enforce that restriction.

4. Approval for one action (Deployment and change management). Our approval mechanism is designed to identify a specific action and a recorded version of its permitted scope. Reassess it when the relevant conditions change; a standing approval may no longer reflect the circumstances of the action.

5. Checks on completed work (Assurance). Check whether the completed work was accurate, complete, valid and within the approved scope. Authorisation is part of that assessment; it does not establish that the intended business outcome was achieved.

1.5 Responding when problems arise

Problems surface through poor outcomes, failed checks, customer or staff reports, supplier changes or an unavailable checking service. When outcomes fall short or something material changes, the person authorised decides whether the work continues, is restricted or stops. The evidence informs that decision; a person makes it.

Weigh the impact. Base the response on the harm the problem could do to customers and the business. Weigh:

• how much damage one wrong action can do, and whether it can be reversed;
• how many customers or transactions it could reach, and how quickly;
• whether the problem is still happening or was a single event;
• what regulators, customers and contracts require, since some rules specify the response and its timing (see the box below);
• what it costs customers and the business to slow or stop the work.

Choose a response. Distinguish between a check that found a problem and a checking service that was unavailable.

• Continue and watch more closely. Check more cases, tighten alerts and follow the trend. Suits a contained, low-impact problem.
• Hold the action for a person. The system automatically holds the action if a required pre-action check fails, is inconclusive or returns no result. This includes a warning, a result marked not assessed or an abstention.
• Restrict. Lower limits, remove tools or send certain cases to people until the cause is fixed.
• Use an approved manual or fallback path. If the checking service is unavailable, pause the work. If the work cannot wait, use a manual or fallback path that management has approved, provided the risk stays within the limits it has set. Record the evidence gap and reconcile the work when checking resumes. An outage is not a passed check. A fallback is not permission for an agent to continue unchecked.
• Pause or stop the agents. Use this when harm could spread quickly or cannot be undone.

When a problem is found after an action, assign someone to correct or further check the completed work. An authorised person decides whether subsequent work should continue.

After a stop. The recovery plan assigns who reconciles unfinished work, corrects affected records and addresses the cause, accounting for unfinished actions and the people affected. The authorised person approves restart under the scope re-established in Figure 5 or a narrower one.

An authorised person decides how to resolve each exception; neither the agent nor the checker makes that disposition. The permitted decisions are reject; approve once, with conditions and an expiry; require remediation; narrow scope; escalate; suspend; or another disposition the organisation's mandate allows.

Confirm closure through a later check, preferably by someone other than the person who performed the remediation. Where a reviewer concludes that an exception was raised in error, retain the original finding and record it as “disputed as raised”, with the reason.

Record when the failure occurred separately from when it was discovered. Preserve the required route for review or recourse where a decision continues to affect a person.

1.6 Evidence records and coverage

Evidence is what lets someone check, later, whether something is or was true. Figure 13 shows what a reviewer asks about any check, and what to keep to answer each.

The five answers help a reviewer judge whether evidence is relevant, reliable and sufficient, the tests the IIA's Global Internal Audit Standards set for the information internal auditors gather (Standard 14.1).

Assign responsibility for applying the organisation's retention and deletion requirements to these records, including the tool settings. Keep the material a record identifies for as long as the record itself, within data protection limits, so a reviewer can still follow the check.

1.7 Review arrangements and independence

What assurance involves

Assurance means two things. It is a conclusion: a statement of how far leaders can rely on what was done, judged against suitable criteria. It is also the work that produces that conclusion: examining and testing what actually happened against those criteria.

Delivering assurance takes four things together:

• People. Competent, objective reviewers: management checks the work, risk and compliance oversee it, and internal audit or an external provider reviews it independently.
• Process. Criteria suited to the risks and objectives, a defined scope and period, and an agreed method for testing the work and recording the conclusion.
• Technology. Systems that capture evidence as the work happens, keep it intact and let a reviewer reach it.
• Evidence. Relevant, reliable and sufficient records, each showing where it came from and what it covers.

Management's own checks support an assurance assessment. Independent assurance needs a reviewer who is independent of the work and reaches their own conclusion from the evidence. Management, risk and compliance, and the independent reviewer each test the evidence before relying on it.

Embedded controls and independent assurance are distinct disciplines. An embedded checker can be useful and still not be independent. An operational checking service used by management remains part of management's controls, even when an external supplier operates it. An external assurance engagement is a separate arrangement, assessed against its mandate and independence requirements. Internal audit can provide independent assurance where it is positioned independently, with the chief audit executive reporting directly to the board, as can an external assurance provider within its mandate and independence requirements.

The reviewer weighs all the evidence for relevance, reliability and sufficiency: the operator's own records, model-provider tests and platform controls, each with its origin and scope visible. The reviewer also needs evidence about the checker itself: test results, false alarms, known failure modes and the classes of problem it has not seen. In our own testing, all findings in one overnight run traced to one checker bug.

Internal audit can rely on management's checks and on other assurance providers once it has evaluated them: who did the work, how independent and competent they are, how they did it, and whether the evidence supports their results. It records the basis for that reliance. The reviewer must assess the evidence and reach their own conclusion about what it supports. Each finding then needs an owner and a date for correction.

Appendix 2. Regulations and standards in detail

The Section column shows where Appendix 1 discusses each one: Oversight and the workflow (1.1), Control domains (1.2), Human review capacity (1.3), Controls in live operation (1.4), Responding when problems arise (1.5), Evidence records and coverage (1.6), and Review arrangements and independence (1.7).

Appendix 3. Glossary

Agent

Software that can choose steps or tools and take business actions.

AI-enabled work

Business work in which an AI system decides, recommends or acts.

Assurance

Both a conclusion and the work that produces it: a statement, measured against agreed criteria, of how far leaders can rely on a piece of work, and the examining and testing behind it (after the IIA Global Internal Audit Standards). Independent assurance needs reviewers independent of the work and of the checks built into it.

Continuous assurance, as used in this paper

Checking AI-enabled work against its intended business outcome and agreed requirements, using the evidence to inform decisions on whether the work continues and within what limits.

Coverage

The work assessed over a stated period, with the in-scope population and selection method made clear. State when the total population is unknown.

Failed / inconclusive / not assessed

Failed: examined, and the evidence shows the requirement was not met. Inconclusive: examined but unresolved, for example because evidence was missing. Not assessed: not examined.

Exception and disposition

An issue raised by a check, and the decision an authorised person makes about it.

Evidence package

A bundle of evidence and verification material prepared for review.

Reperformance

Running a check again. It is distinct from an independent re-execution by an outside party.

Non-human identity

The credential or workload identity an agent uses, with a named owner.

Appendix 4. Sources

Regulation and supervisory guidance

• European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). OJ L, 2024/1689, 12 July 2024. eur-lex.europa.eu
• European Parliament and Council of the European Union. Regulation (EU) 2026/1744 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 (Digital Omnibus on AI). OJ L, 2026/1744, 24 July 2026; entered into force 27 July 2026. eur-lex.europa.eu
• Office of the Superintendent of Financial Institutions (Canada). Guideline E-23 – Model Risk Management (2027). Published 11 September 2025; effective 1 May 2027. osfi-bsif.gc.ca
• Office of the Superintendent of Financial Institutions (Canada). Guideline B-13 – Technology and Cyber Risk Management. Effective 1 January 2024.
• Office of the Superintendent of Financial Institutions (Canada). Generative and Agentic Artificial Intelligence: Implications for Technology, Cyber Security, and Operational Resilience. Technology Risk Bulletin, July 2026. osfi-bsif.gc.ca
• Board of Governors of the Federal Reserve System. SR 26-2: Revised Guidance on Model Risk Management. 17 April 2026. federalreserve.gov
• Board of Governors of the Federal Reserve System, Federal Deposit Insurance Corporation and Office of the Comptroller of the Currency. Supervisory Guidance on Model Risk Management. SR 26-2 attachment, 17 April 2026.
• Prudential Regulation Authority. SS1/23 – Model risk management principles for banks. Supervisory statement, current version 16 April 2026 (updating 17 May 2023). bankofengland.co.uk
• Treasury Board of Canada Secretariat. Directive on Automated Decision-Making. Published 29 March 2021; page modified 24 June 2025. tbs-sct.canada.ca
• US Office of Management and Budget. M-25-21 – Accelerating Federal Use of AI through Innovation, Governance, and Public Trust. 3 April 2025. whitehouse.gov
• US Office of Management and Budget. M-25-22 – Driving Efficient Acquisition of Artificial Intelligence in Government. 3 April 2025. whitehouse.gov
• National Association of Insurance Commissioners. NAIC Model Bulletin: Use of Artificial Intelligence Systems by Insurers. Adopted 4 December 2023. content.naic.org

Standards and frameworks

• National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, January 2023. nist.gov
• National Institute of Standards and Technology. Secure Software Development Framework (SSDF) Version 1.1: Recommendations for Mitigating the Risk of Software Vulnerabilities. NIST SP 800-218, February 2022.
• National Institute of Standards and Technology. Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile. NIST SP 800-218A, July 2024.
• The Institute of Internal Auditors. Global Internal Audit Standards. Issued 9 January 2024; effective 9 January 2025. Glossary; Standard 14.1. theiia.org
• The Institute of Internal Auditors. The IIA's Artificial Intelligence Auditing Framework. 2024.
• The Institute of Internal Auditors. Global Technology Audit Guide (GTAG) 3: Coordinating Continuous Auditing and Monitoring to Provide Continuous Assurance. 2nd edition, March 2015.
• The Institute of Internal Auditors. The IIA's Three Lines Model: An Update of the Three Lines of Defense. Position paper, 2020; updated September 2024. theiia.org
• US Government Accountability Office. Standards for Internal Control in the Federal Government (Green Book). GAO-25-107721, May 2025.
• HM Treasury. The Orange Book – Management of Risk – Principles and Concepts. 2023 edition.
• International Organization for Standardization and International Electrotechnical Commission. Information technology – Artificial intelligence – Management system. ISO/IEC 42001:2023, first edition, December 2023. iso.org
• Committee of Sponsoring Organizations of the Treadway Commission (COSO). Internal Control – Integrated Framework. May 2013. coso.org
• Committee of Sponsoring Organizations of the Treadway Commission (COSO). Enterprise Risk Management – Integrating with Strategy and Performance. June 2017. coso.org
• CEN-CENELEC Joint Technical Committee 21. Artificial intelligence – Quality management system for EU AI Act regulatory purposes. EN 18286:2026, available 22 July 2026. cencenelec.eu
• OWASP Gen AI Security Project. OWASP Top 10 for Agentic Applications for 2026. Published 9 December 2025.
• FINOS (Fintech Open Source Foundation). FINOS AI Governance Framework. Version 2, 20 October 2025.
• Cloud Security Alliance, AI Safety Initiative. The AI Agent Governance Gap: What CISOs Need Now. 3 April 2026.

Research on language models

• Sharma et al. Towards Understanding Sycophancy in Language Models. arXiv:2310.13548, 2023. arxiv.org
• Huang et al. Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024; arXiv:2310.01798. arxiv.org

Research on automation and human factors

• Bainbridge, L. Ironies of automation. Automatica 19(6), 775–779, November 1983. doi.org

Cases

• Civil Resolution Tribunal of British Columbia. Moffatt v. Air Canada. 2024 BCCRT 149, February 2024. civilresolutionbc.ca