AI in financial services: why your transformation strategy needs a measurement layer

Large banks have been deploying AI longer than most industries. They have the data infrastructure, the regulatory discipline, and the risk management frameworks that make AI adoption tractable. In 2025, two thirds of banks and insurers use AI models, and 94% of large banks use GenAI in at least one function.

The adoption numbers are high. The value numbers are not keeping pace.

More than two in five financial institutions say they have slowed use case development because of disappointing outcomes. McKinsey's research on AI in the credit business found a consistent pattern: institutions that focused on early ROI were more likely to give up on AI, while those that committed through the early disappointment started to see compounding returns. The differentiator was not the technology. It was the measurement infrastructure that told them what was working and what was not.

What will distinguish market leaders in financial services is not adoption rate, but how well they redesign processes, organisational setup, and people change management to benefit from the efficiency gains AI can provide. That redesign requires visibility. You cannot redesign around what you cannot measure.

The financial services AI deployment map

Financial institutions are deploying AI across more functions simultaneously than almost any other sector. The complexity is significant.

Trading floors have AI copilots for market research, portfolio analysis, and credit memo drafting. McKinsey found that a US bank that used AI agents to change how it creates credit risk memos experienced a 20 to 60% increase in productivity and a 30% improvement in credit turnaround. Compliance teams use AI for AML, KYC, and regulatory review. A large Dutch financial institution achieved a 90% reduction in onboarding time and cut staff workload by 30% through AI-assisted compliance processes.

HR, legal, finance, and operations have each launched their own AI tools, often independently of each other and with different vendors. Customer service teams run AI agents handling queries, escalations, and account support at scale.

Each of these deployments has different users, different success definitions, and different risk profiles. The compliance team that uses AI to review regulatory documents has a different accuracy bar from the trading team using it for research synthesis. The customer service agent handling product enquiries has different governance requirements from the internal HR chatbot.

The measurement challenge is that most financial institutions are trying to assess this portfolio of deployments through a combination of system dashboards, manual reviews, and periodic surveys. None of these tell them what they actually need to know.

What system monitoring cannot see

Every AI deployment in a financial institution generates infrastructure metrics: uptime, latency, token consumption, API call volume. These are necessary for operations. They are not sufficient for transformation.

When trading analysts ask an AI copilot for research and then immediately follow up with three clarifying questions, the infrastructure dashboard records four successful API calls. The signal — that the initial response consistently fails to address what analysts actually need — is invisible.

When compliance staff include sensitive data in prompts more frequently than policy permits, no infrastructure alert fires. The behavioral pattern only becomes visible when conversation data is analyzed at the aggregate, anonymized level.

When a global bank deploys AI copilots across 80,000 employees and discovers that one region's staff routinely includes more PII in prompts than others, that insight does not come from system monitoring. It comes from behavioral analysis of conversation patterns, surfaced before a regulatory audit rather than during one.

These are the signals that determine whether AI transformation in financial services produces the outcomes leadership is trying to achieve, and whether governance frameworks are actually working in practice.

The three measurement gaps that stall transformation

Financial institutions stalling on AI transformation typically share three measurement gaps.

No visibility into adoption depth by function. Login counts and session volume tell you AI tools are being accessed. They do not tell you whether compliance officers are using AI for routine document triage only, or for the complex regulatory interpretation work that would generate real productivity gains. They do not tell you whether trading analysts are relying on AI for straightforward research queries and continuing to handle complex scenario analysis manually. Adoption depth — whether employees are using AI for the tasks that generate real value, and whether that usage is deepening over time — requires behavioral data that infrastructure monitoring does not provide.

No connection between AI activity and business outcomes. Executive stakeholders in financial institutions want to know whether the AI investment is improving analyst productivity, reducing compliance review time, or improving customer resolution rates. Session counts do not answer those questions. Connecting AI activity to business outcomes requires measuring task completion, hours saved per function, error rates on AI-assisted work, and — for customer-facing deployments — commercial signals like resolution rates, escalation patterns, and retention indicators.

No early warning on governance and compliance risk. Financial services AI deployments face a distinctive governance requirement: the consequences of a compliance failure are not just operational. They are regulatory. McKinsey's 2026 AI Trust survey found that in the agentic era, organizations must contend with systems doing the wrong thing, not just saying the wrong thing — making oversight and behavioral visibility foundational requirements, not optional enhancements. Behavioral patterns in AI conversations — employees testing guardrails, including restricted data types, asking for outputs that approach policy limits — are the early warning signals that allow governance teams to intervene before they become regulatory incidents. (Fortune)

What the measurement layer enables in practice

A global financial institution that deploys behavioral analytics on its internal AI copilots gains a different kind of intelligence from the one available through infrastructure monitoring.

At the department level, it can see that compliance teams show higher AI engagement than expected, while treasury departments are generating high volumes of follow-up queries — a signal that the AI is not performing well enough on specialized financial queries for that function. The fix for treasury is targeted: improve the training data and knowledge coverage for the specific query types treasury needs. That insight is only possible with function-level behavioral data.

At the governance level, it can see emerging patterns before they become incidents. A cluster of employees in a specific region including more sensitive data in prompts than policy permits is visible in aggregate conversation data. A pattern of queries approaching the boundary of what the AI is authorized to advise on is detectable before it produces a non-compliant output.

At the executive level, it can report in transformation language rather than infrastructure language. Not "the AI had 99.9% uptime last quarter" but "trading analysts are asking 40% more complex research questions and completing analysis tasks 25% faster. Compliance review time for standard documents is down by an average of 18 minutes per document. Three departments show strong adoption depth and are ready for expanded use cases. Treasury requires targeted improvement before expansion."

This is the reporting that connects AI investment to business performance and enables leadership to make informed decisions about where to scale, where to invest in improvement, and where to pause.

The regulatory dimension

Financial services AI governance is not voluntary. DORA in Europe creates technology risk disclosure requirements that extend to AI systems. The FCA and PRA in the UK have issued clear expectations around AI oversight and explainability. Basel frameworks are beginning to address AI model risk in credit applications. The direction across all regulatory frameworks is consistent: financial institutions need to demonstrate that they understand how their AI systems are being used in practice, not just how they were designed.

That demonstration requires visibility into user behavior, not just system performance. An institution that can show regulators not only what its AI systems do, but how employees use them, what governance controls apply to specific interaction types, and how those controls are validated through behavioral monitoring, is in a materially different compliance position from one that can only report system uptime and API metrics.

The behavioral measurement layer is, in this sense, not just a tool for improving AI performance. It is part of the governance infrastructure that regulated financial institutions need to operate AI at scale.

Nebuly

Nebuly is the ROI platform for enterprise AI. It connects to the AI agents your business runs on, the assistants your customers interact with, and the tools your employees use every day, including Claude, ChatGPT, and Copilot, and translates that activity into business value. How much time is being saved across teams. What revenue your AI is influencing. What adoption and AI proficiency look like in practice, across departments and geographies. All aggregated at the organizational level, never tied to individuals.

Nebuly supports self-hosted deployment on AWS, Azure, and GCP, keeping all interaction data within your own infrastructure. It meets SOC 2 Type II, ISO 27001, and ISO 42001 standards.

If you need clarity on what your AI investment is actually delivering, book a demo.