Moral machines: Artificial intelligence and governance

5 minute read

Artificial intelligence (AI) is moving from experiment to infrastructure. It now sits inside products, research, customer service, risk models and everyday workflows. Yet despite the excitement surrounding AI, the most important question for boards and leadership teams is no longer whether they are using it. It is whether they understand where it is being used, what decisions it shapes and who is accountable when something goes wrong.

These were the questions at the heart of a recent investee company engagement exercise by Quilter Cheviot with major AI users including AstraZeneca, HSBC, JPMorgan Chase, Merck, RELX, Wolters Kluwer and Experian. Across sectors as diverse as healthcare, financial services and professional information, one theme emerged consistently: AI is becoming a governance issue before it becomes a technology one. These companies were selected because they are significant AI users in high-impact sectors. While attention often focuses on developers, most investment portfolio exposure sits with companies using AI across ordinary business processes, products and decisions. These deployer companies sit closest to clients and often face the largest liability risk related to any AI failures.

Ambition is outpacing accountability

Adoption is racing ahead of oversight. Research from AICDI covering almost 3,000 companies found that while 44% publicly communicate an AI strategy, only 13% align that strategy with a formal governance framework. Even fewer disclose policies ensuring human oversight of AI systems. In other words, many organisations can describe their ambition more readily than their control environment.

The risks are not theoretical. In 2019, Apple’s credit card programme came under scrutiny after allegations that women were offered significantly lower credit limits than men with similar financial profiles. More recently, UnitedHealth faced legal action over claims that an AI tool was used to deny elderly patients access to extended care, with many of those decisions later overturned by human reviewers. These examples are reminders that AI can amplify flawed decisions at scale just as easily as it can improve efficiency.

This is why governance matters.

Governance is the leading indicator

The central lesson from our engagement was that governance may be the most useful leading indicator of responsible AI. Unlike climate change, which benefits from relatively clear metrics such as emissions, AI has no universally accepted measure of success or failure. Technologies evolve rapidly and risks differ depending on how systems are deployed. What matters most is whether organisations have the structures, accountability and oversight needed to manage those risks.

The strongest companies did not treat AI ethics as a specialist project hidden away in the technology department. Instead, they embedded it within existing risk, compliance, privacy and operational resilience frameworks. Some use formal AI impact assessments. Others have dedicated oversight committees or external ethics advisers. The common feature is that responsibility sits with leadership, not just developers.

Keeping humans in the loop

Human oversight emerged as perhaps the most important safeguard. The most mature organisations see AI as an expert-support tool rather than a replacement for judgement. In healthcare, for example, AI can help identify potential treatments or summarise clinical evidence, but doctors remain responsible for the final decision. In investment management, AI can accelerate research, while accountability for decisions remains firmly with investment professionals.

The principle is straightforward: the higher the stakes, the greater the need for meaningful human involvement.

The transparency gap

Transparency, however, remains a work in progress. Many organisations appear stronger at governing AI internally than explaining externally how those controls operate. Stakeholders increasingly want more than broad commitments to responsible AI. They want evidence that systems are tested, monitored and capable of being challenged when things go wrong.

That challenge extends beyond algorithms themselves. AI is already reshaping workplaces, raising questions about skills, reskilling and the future of entry-level roles. It depends on vast quantities of data, creating new concerns around privacy, consent and human rights. It also relies on physical infrastructure that consumes significant amounts of energy and water. Governance therefore becomes the thread connecting a much wider set of risks and opportunities.

The questions every board should ask

For trustees, board members and senior executives, the implication is clear. You do not need to become an AI expert. But you do need to ask the right questions. Do we know where AI is being used? Who is accountable for its outputs? Can decisions be explained and challenged? Are impacts on people, privacy and trust being assessed before systems are deployed?

AI itself is neither ethical nor unethical. It is a tool. A machine-learning model that helps identify drug targets under expert supervision is not the same ethical proposition as an opaque tool that affects credit access, healthcare coverage or employment outcomes without explanation or appeal. What determines the outcome is the governance surrounding it. Based on our engagements, across all sectors, the recurring weaknesses are similar: outcome-based reporting, impact assessments, formal model inventories, incident transparency and external assurance. These are not reasons to reject AI. They are reasons for sustained stewardship where standards and expectations can be set and reinforced.

Trust as a competitive advantage

The organisations that ultimately succeed with AI will not necessarily be those making the boldest claims. They will be those able to demonstrate that accountability remains human, oversight remains effective and trust has been designed into the system from the beginning. In the race to harness AI’s potential, governance may prove to be the most important competitive advantage of all.