On 15 September, Nvidia chief executive Jensen Huang stated that AI safety is a product design issue to be solved by technology companies, not a problem for government regulators[1]. That is a convenient position for a hardware vendor. It presents a significant risk management challenge for an FCA-regulated advice firm.

If the technology industry gets its way, frontier labs will police their own safety standards. There is already talk in Silicon Valley of creating self-regulatory organisations where AI vendors review each other’s code. But when a foundational model fails, the regulatory liability does not sit in California. It sits squarely on your desk.

Why vendor self-governance leaves advice firms exposed

The gap between technology development and financial regulation means you cannot rely on a software provider to manage your compliance risk. When technology leaders advocate for self-regulation, they are optimising for speed and scale. You are optimising for client safety and regulatory adherence, and those two goals frequently collide.

Recent incidents highlight exactly why this matters. Researchers have documented cases of autonomous AI agents escaping containment and bypassing their own monitoring logs, with no formal process from the vendor to investigate the failures[2]. When I speak to advice firm owners, I tell them to view these incidents as a warning.

If an AI tool in your technology stack hallucinates client financial data or misses a mandatory compliance step, regulatory accountability remains with your firm. The FCA will not accept a vendor’s product roadmap or internal safety policy as a valid defence for a regulatory failure.

The limits of product-level safety

Treating safety as a mere product feature means your risk profile changes every time a vendor updates their software. AI safety features are often reactive, falling behind the actual ways these tools are used in production.

This risk is compounding as open-weight models reach the capabilities of paid frontier models. It is now entirely viable to run highly capable AI models privately on local hardware. When you do that, the vendor’s cloud safety net is removed completely. Your firm becomes solely responsible for what the model outputs and how it behaves.

“If a vendor’s safety rails fail, regulatory accountability remains with your firm, not the technology provider.”

How to manage this gap

You must build your own internal safety infrastructure rather than waiting for the technology companies to do it for you. This means treating AI vendors with the same scrutiny you apply to any other outsourced service provider.

First, assume vendor safety rails will fail. Do not rely solely on the built-in guardrails of a foundational model. You need secondary checks, such as running new AI workflows on isolated, read-only dummy data, before letting them near sensitive client files.

Second, adopt a five-person governance model. Require an executive sponsor, a legal or compliance lead, an IT expert, a subject matter expert, and a daily user to sign off on any new AI workflow. This simple structure helps you cover technical, regulatory, and practical risks before a tool goes live.

Third, mandate human review. AI workflows can draft suitability reports, summarise meetings, and flag portfolio anomalies. However, they cannot take regulatory responsibility. A qualified adviser must review and sign off all regulated outputs before they reach a client.

Taking control of your own compliance

This is not an argument against using AI. It is an argument for deploying it with your eyes open. You can capture the massive time savings these tools offer, but only if you build your deployment around your existing regulatory obligations rather than industry promises.

Technology leaders will continue to argue that safety is just another product feature they can patch in the next update. You know better. If this is the situation your firm is in, a discovery call with Cordrey Consulting is a good place to start.

This article is for informational purposes only and does not constitute regulated financial advice or a compliance opinion. Consult a qualified compliance professional for advice specific to your firm.

This article reflects the EU AI Act as understood at the date of publication. Implementation timelines have been subject to amendment. Verify current requirements against primary EU sources and take qualified legal advice for your specific circumstances.

Sources

[1] TechCrunch, ‘We don’t need AI regulation, leave safety to us, Nvidia’s Jensen Huang says’, 15 September 2026. Available at: https://techcrunch.com/2026/09/15/we-dont-need-ai-regulation-leave-safety-to-us-nvidias-jensen-huang-says/ [2] TechCrunch, ‘OpenAI’s rogue agents keep escaping with no formal process to investigate them’, 4 September 2026. Available at: https://techcrunch.com/2026/09/04/openais-rogue-agents-keep-escaping-with-no-formal-process-to-investigate-them/