Dario Amodei is right that we need to pace the AI frontier. Banking offers a tested discipline for doing it: never let capability outrun the capacity to control it.
By Sridhar Aiyangar, Founder & Managing Director · September 2026
A few weeks ago, following the OpenAI–Hugging Face incident, I wrote:
Agentic AI can initiate, adapt, pursue objectives and escalate in ways no one instructed or anticipated. It is becoming critical infrastructure without being governed as critical infrastructure.
Dario Amodei's new essay, We Must Pace the Frontier, takes this further. Investing in AI safety is no longer enough on its own: if capability advances faster than alignment, interpretability and controls, safety may never catch up. The pace of capability development itself must be managed.
I agree, particularly given the combined risks of recursive self-improvement and loss of control. Where I would build on his argument is in the framing.
A bank cannot grow simply because profitable opportunities exist. Its growth is constrained by its capacity to absorb and manage risk. Capital absorbs losses. Liquidity protects against funding stress. Risk appetite sets boundaries. Stress testing asks what happens when assumptions fail, and recovery planning asks what management can actually do when things go wrong. AI needs an equivalent discipline.
A Capability Appetite would define how much autonomy, access, decision-making authority and potential impact an AI system may have, relative to the organisation's demonstrated ability to understand, govern and contain it. The principle:
That shifts the governance question from "What can this model do?" to "What can we safely allow this model to do?"
An AI that drafts an email is one category of risk. An AI that can access systems, execute code, move data, initiate transactions or alter infrastructure is an entirely different one. Greater autonomy increases productivity, but also the blast radius of failure.
That calls for capability-triggered governance: as capability rises, requirements should automatically tighten around independent validation, interpretability, access controls, human authorisation, monitoring, containment, cyber resilience and recovery.
Amodei's proposal points the same way. He calls for capability checkpoints, where a model reaching a defined capability level must show corresponding evidence of alignment, interpretability and control before deployment, and for independent evaluators embedded inside frontier AI companies, citing the precedent of bank supervisors. Self-certification is not sufficient when failure can extend beyond the institution creating the risk.
Testing asks: "Does the system operate as intended?" Stress testing asks: "What happens when it doesn't?"
AI stress testing should go beyond accuracy and red-teaming to scenarios such as circumvention of safeguards, autonomous escalation, collusion between agents, manipulation of monitoring and failure of human intervention.
In banking, a meaningful stress test also assesses management actions and counterbalancing capacity: the buffers a firm can draw on to absorb a shock and restore stability. For AI, if a system behaves unexpectedly:
AI is concentrated in a small number of frontier-model developers, chip suppliers, cloud platforms and data-centre providers, while models, agents and infrastructure are increasingly interconnected. Anyone who has studied financial crises will recognise the pattern:
Concentration + Interconnectedness + Opacity + Common Dependencies = Rapid Contagion
Individually rational behaviour does not guarantee a resilient system. AI governance must move beyond "Is this model safe?" to "Is the system resilient?" That may ultimately mean identifying systemically important AI providers, with supervisory expectations proportionate to the externalities their failure could create.
Amodei notes that AI is increasingly helping to build the next generation of AI, potentially compressing development cycles dramatically. Boards meet periodically, policies take months and regulation takes years, yet capability can now change materially between governance cycles.
This is not an argument for stopping AI. Its benefits for medicine, science, education and productivity could be extraordinary, and the debate should not be reduced to acceleration versus regulation. A better approach: move quickly where capability remains within demonstrable control capacity; apply stronger safeguards as autonomy and systemic importance increase; require independent assurance where failures create externalities; and where capability materially outruns control capacity, slow down until the gap closes. That is not anti-innovation; it is how complex systems scale sustainably.
My earlier work set out seven AI governance guardrails: validation, explainability, auditability, approval, controlled deployment, monitoring and accountability. All still stand. But agentic AI needs another layer above them: control capacity that includes not just preventative controls, but resilience, counterbalancing capacity and recoverability.
We do not make banks resilient by assuming losses will never occur. We build them to absorb stress, contain failure and recover before instability becomes systemic. AI will require the same discipline.
Dario Amodei calls on us to pace the frontier. I would put the principle more broadly:
That may prove to be one of the most important principles for governing the next phase of AI.
Join the discussion on LinkedIn
Part of The Balance Sheet Doctor — Sterling Consulting's Building Financial Resilience series.
The principles that keep a balance sheet resilient — appetite, stress testing, counterbalancing capacity and recovery — are the same ones that keep capability within control. Talk to us about applying that discipline across your risk and your AI.
Request a Diagnostic Review