The Bank of England AI Consortium is urging financial institutions to change how they oversee GenAI models, as legacy model risk frameworks are reaching a breaking point.
Generative AI adoption is accelerating across UK and US financial services, and the consortium is recommending a pivot from static model validation to system-level risk oversight.
From Model Risk to System-Level Governance
GenAI applications are often classified as high-risk systems, but they rarely rely on a single model, instead combining multiple components like foundation models and third-party APIs.
To address these vulnerabilities, the consortium proposed managing risk from an AI model system perspective, including whole-system testing and outcome-based explainability.
This approach would define transparency by whether the system behaves as intended, rather than trying to dissect individual model components.
For instance, whole-system testing involves evaluating individual components alongside the end-to-end AI system, taking into account third-party models that update independently, to ensure the entire system functions as expected.
Outcome-based explainability, on the other hand, focuses on defining transparency by whether the system behaves as intended, and maintaining auditable decision logs, rather than attempting to dissect black-box weights.
A New Approach to Risk Management
The consortium outlined a four-step failure containment framework to catch disruptions before they impact operational resilience.
This framework includes identifying failure types, detecting signals, gathering diagnostics, and deploying circuit breakers to trigger automated controls or hand control to human operators.
According to the consortium, greater standardisation of AI incident reporting could support cross-firm learning and improve visibility of failures.
The four-step framework is designed to help institutions respond to AI-related incidents in a timely and effective manner, and to prevent minor issues from escalating into major disruptions.
Related: UX Boosts Efficiency in Accounts Payable Operations
By identifying failure types, institutions can categorise breakdowns by root cause, such as data corruption, prompt injection, model hallucination, or API latency, and develop targeted responses to each type of failure.
Broader Systemic Concerns
The consortium also detailed concerns around autonomous agentic payments, which could outpace existing governance frameworks, and the heavy reliance on a small group of cloud and frontier model providers.
The consortium recommended strengthening third-party vendor oversight and requiring auditable documentation.
For example, artificial intelligence systems can be designed to include human oversight and continuous testing to mitigate risks.
CISOs and CROs should audit AI supply chains, adopt system-level frameworks, and build automated fallbacks to isolate hallucinations before outputs reach execution layers.
They should also prepare for regulatory shifts ahead of stricter enforcement mandates on critical technology providers in the UK and US.
The Bank of England AI Consortium’s recommendations are aimed at helping financial institutions handle the complexities of GenAI and ensure that they are able to manage the associated risks effectively.
By adopting a system-level approach to risk oversight, institutions can better understand the interconnectedness of their AI systems and take steps to mitigate potential vulnerabilities.
This may involve developing more sophisticated testing and validation protocols, as well as implementing more robust governance frameworks to oversee the use of GenAI.
Ultimately, the key to successful GenAI adoption will be the ability to balance the benefits of these technologies with the need to manage the associated risks, and to ensure that institutions have the necessary frameworks and expertise in place to do so.
Leave a Reply