The New Compliance Frontier: Managing AI Outputs Under Regulation
I was in a meeting with a regulator when they asked a question nobody in the room could answer: 'If an AI helped form this advice, who is accountable for the output?' The room went quiet. Not because the answer is complicated — it's the adviser, obviously — but because nobody could explain how they'd evidence that. How do you prove a human reviewed the AI's work? How do you show what the system surfaced, what the adviser accepted, and what they overrode? That meeting changed how I think about AI in regulated markets.
The accountability gap
Most AI tools in insurance today produce an output — a summary, a comparison, a recommendation — and the adviser forwards it to the client. There's no record of what the AI produced versus what the adviser changed. No audit trail showing the human reviewed clause 14.3 and agreed with the system's interpretation. No evidence that the adviser considered an alternative and rejected it for a stated reason. That's the accountability gap. The adviser is legally responsible for the advice, but they can't prove how they formed it. When things go well, nobody asks. When a claim is disputed or a complaint is filed, everyone asks.
Traceability is the answer
After that meeting, I became convinced that the single most important feature of any AI system in regulated advice isn't intelligence — it's traceability. Every answer should point to the clause it came from. Every comparison should show its working. Every decision should be logged with what the system surfaced and what the adviser did with it. This isn't about bureaucracy. It's about building the evidence that proves your advice process was sound, whether a regulator asks in a routine review or a client challenges it in a dispute. The firms that build this into their workflow now won't need to retrofit it when regulation catches up.
Where accountability sits
My position on this is straightforward: accountability sits with the adviser, always. The AI is a tool. It reads, compares, and surfaces — but it doesn't decide. The adviser reviews, judges, and recommends. The system's job is to make that process visible and audit-ready, so when someone asks 'how did you reach this recommendation?', the answer is already documented. That's what I mean by showing your working. Not as a feature. As the foundation.
