We Never Trusted Humans. Why Would We Trust AI?

The Financial Stability Board recently warned G20 finance ministers and central bank governors about the growing risks posed by AI. Its most immediate concern is AI’s potential to increase the speed, scale and economics of cyberattacks. But the FSB has also been examining another side of the issue: the risks created as financial institutions themselves adopt increasingly capable and autonomous AI.

That second issue interests me. Financial institutions are beginning to use AI to perform work historically performed by employees: reviewing fraud alerts, investigating cases, assessing risk, recommending actions and potentially making decisions. That raises a question we’ve dealt with before: How much should we trust an actor we’ve given authority to make decisions on our behalf?

I’ve served as an expert witness in several embezzlement cases where millions of dollars were lost because controls designed to prevent a trusted employee from abusing their authority failed. Financial institutions learned long ago that trust is not a control.

We didn’t simply trust employees because they were hired, trained and given authority. We surrounded them with separation of duties, dual approvals, supervisory reviews, spot checks, audit trails, second- and third-line oversight, mandatory vacations and whistleblower programs. Why? Because an employee could be careless, corrupt, compromised or deceived.

So why would we approach AI differently? Could a criminal manipulate an AI fraud investigator into closing an alert? Fool it with a deepfake, synthetic identity or altered document? Manipulate contact information so the agent calls the criminal instead of the customer? Could repeated interactions reveal why transactions are being flagged, effectively turning the AI into an oracle for probing the bank’s fraud controls?

And if any of that happened, would anyone notice? So how do we test it? Start the same way we test controls around humans. Identify what the AI can do, what could go wrong and which controls are supposed to prevent it. Then deliberately try to make those controls fail. Social-engineer the AI. Give it manipulated documents, conflicting information, synthetic identities and deepfakes. Try to make it disclose why something was flagged. Attempt to bypass escalation rules. Test whether separation of duties still works. Determine whether decisions can be reconstructed, challenged and reversed.

Governance tells us how AI is supposed to behave. Red teaming asks what happens when someone deliberately tries to make it fail. If we’re going to trust an AI agent with decisions that once required a controlled and supervised human, shouldn’t we subject the AI, and the controls surrounding it, to the same scrutiny?

We learned the hard way not to trust humans without controls. We shouldn’t have to learn the same lesson again with AI.