AI adoption stories #3: Global bank transforms risk management with early user analytics
AI adoption stories #3: Global bank transforms risk management with early user analytics
TLDR
→ A global bank with over 80,000 employees deployed AI copilots across trading, equity research, HR, finance, and legal, with no initial visibility into how those conversations were actually going. → Three risks were invisible without conversation analytics: employees unknowingly including PII in prompts, users asking questions outside policy boundaries, and the AI returning incorrect or misleading information in complex financial scenarios. → Deploying analytics within the bank's own VPC provided real-time visibility across all departments. In the first 60 days, risk flagging prevented dozens of potential compliance violations before they escalated. → Behavioral analysis revealed that legal teams had the highest error rates, not due to technical failure, but because the AI needed better training on regulatory language. HR showed the strongest adoption, with consistent daily usage and high satisfaction. → The case demonstrates that in regulated AI deployments, monitoring employee AI interactions is not optional. Understanding how employees use AI tools is what makes it possible to manage risk while enabling productivity.
A top-tier bank with over 80,000 employees launched a AI program to boost internal productivity. What started with copilots for traders and equity research quickly expanded to HR, finance, and legal teams, each using their own assistant.
The challenge was clear: thousands of employees were interacting with AI tools across departments, but the bank had no visibility into how these conversations were actually going.
Why they needed visibility from the start
Traditional risk management wasn't built for conversational AI. The bank faced three critical blind spots.
→ First, employees were unknowingly including personal and confidential information - including PII - in prompts. Without monitoring, sensitive data exposure could happen at scale.
→ Second, users asked questions outside policy boundaries with no system to flag risky behavior. In a regulated environment, this created compliance gaps.
→ Third, the AI sometimes returned incorrect or misleading information, especially in complex financial scenarios. These hallucinations created confusion and risk in regulated functions.
Real-time insights that changed everything
Nebuly's deployment within the bank's VPC provided immediate visibility across all departments. The analytics revealed patterns that would have taken months to surface through traditional methods.
Risk flagging identified prompts containing PII, restricted terms, and non-compliant usage in real-time. The team could now spot issues before they escalated.
Failure detection surfaced wrong or hallucinated answers before user trust eroded. This became crucial for maintaining confidence in AI tools across sensitive departments.
Most importantly, they could track how different departments used their copilots, revealing distinct conversation patterns between traders, HR staff, and legal teams.
Early impact on operations
The results were immediate. Risk detection prevented dozens of potential compliance violations in the first 60 days.
User behavior analysis showed that legal teams had the highest error rates, but not due to technical issues. The AI needed better training on regulatory language and legal terminology.
HR teams showed the strongest adoption, with consistent daily usage and high satisfaction scores. This insight helped the bank expand similar tools to other people-focused departments.
What this means for other financial institutions
Start with visibility. When you deploy AI tools in regulated environments, monitoring isn't optional. Understanding how employees interact with AI tools helps you manage risk while enabling productivity.
The bank here avoided potential compliance issues by seeing the full picture of employee AI usage, not just system performance metrics.
Nebuly
Nebuly is the ROI platform for enterprise AI. It connects to the AI agents your business runs on, the assistants your customers interact with, and the tools your employees use every day, including Claude, ChatGPT, and Copilot, and translates that activity into business value. How much time is being saved across teams. What revenue your AI is influencing. What adoption and AI proficiency look like in practice, across departments and geographies. All aggregated at the organizational level, never tied to individuals.
If you need clarity on what your AI investment is actually delivering, book a demo.
Read more Gen AI adoption stories:
- #1: What user analytics revealed about a GenAI copilot rollout
FAQs
What were the three compliance risks the bank identified through AI conversation analytics?
The bank identified three distinct risk patterns in the first weeks of deployment. Employees were unknowingly including personally identifiable information in prompts — a data exposure risk that occurred through normal use, not deliberate circumvention of policy. Users were asking questions that approached or crossed policy boundaries, creating compliance gaps in a regulated environment. And the AI was returning incorrect or misleading information in complex financial scenarios — hallucinations that created confusion and risk in departments where accuracy is a regulatory requirement.
How did the bank deploy Nebuly while maintaining data privacy requirements?
Nebuly was deployed within the bank's own virtual private cloud, meaning all interaction data stayed within the bank's infrastructure. No conversation data was transmitted to external servers. This deployment model allowed the bank to maintain full control over sensitive employee interaction data while still accessing the behavioral analytics needed to understand usage patterns, detect compliance risks, and measure adoption across departments.
What did the bank learn from comparing AI usage patterns across departments?
The department-level analysis revealed distinct and actionable differences. Legal teams showed the highest error rates in the system, but analysis of the specific query types showed this was a training data gap rather than a technical failure — the AI needed better coverage of regulatory language and legal terminology. HR teams showed the strongest adoption, with consistent daily usage and high satisfaction scores. This insight gave the bank a clear direction: improve the AI's legal training data, and use HR's successful adoption model as a blueprint for expanding to other people-focused departments.
What is the lesson for other financial institutions deploying AI at scale?
The case shows that in regulated environments, visibility into how employees actually interact with AI tools is a governance requirement, not an optional enhancement. System monitoring tells you the AI is running. Behavioral analytics tells you whether employees are using it safely, whether they are encountering accuracy issues that create risk, and where adoption is genuine versus nominal. The bank in this case avoided potential compliance violations because behavioral patterns were visible before they escalated — a form of proactive risk management that infrastructure monitoring cannot provide.
How can banks use AI adoption data to guide rollout decisions?
Adoption data at the department level reveals which functions have genuinely embedded AI tools into daily workflows and which have nominal access without real use. That distinction informs where to invest in training and support, which use cases to expand, and which need redesign before scaling. In this case, the bank used HR's strong adoption pattern to identify the characteristics of a successful internal AI deployment — consistent use, high satisfaction, low error rate — and used that as a model for expanding to other departments. Without behavioral data, these decisions would have been made on assumption rather than evidence.
Stay up to date on what we're learning, building, and seeing as enterprise teams deploy and measure AI agents in production.