Managing risk in GenAI chatbots: technology risk vs. user risk

When a global financial services organization deployed an internal AI assistant across 80,000 employees, they had strong technical security in place. Prompt guards, encryption, API monitoring, access controls. The infrastructure was correctly governed.

Within the first 60 days, something unexpected appeared in the conversation data. The trading department showed employees repeatedly asking whether the AI would accept restricted market data. Not because they were trying to circumvent policy, but because they were genuinely uncertain about what was and was not permitted. The legal department had multiple employees asking variations of the same compliance question, a clear signal of insufficient training rather than malicious intent. The HR department showed PII inclusion spikes during hiring season, driven by employees under deadline pressure taking shortcuts. And the compliance team's conversations included requests for outputs the company's policies explicitly prohibited.

None of these patterns triggered a single alert in the technical security layer. Every interaction was logged as successful. The system was working. The governance problems were invisible.

Why technical security addresses only half the problem

Enterprise AI risk sits on two distinct levels. Most governance investments cover the first. The second is where most incidents actually originate.

The first level is technology risk: vulnerabilities in the AI system itself. Prompt injection attacks, where users manipulate prompts to override system instructions or extract sensitive data. Model poisoning, where training data is compromised to corrupt model behavior. Data exfiltration, where sensitive information leaks through AI outputs or API responses. Supply chain vulnerabilities from third-party models and integrations.

These are real threats that require strong technical controls: prompt guards, input validation, output filtering, access controls, and continuous model integrity verification. Most mature enterprise AI deployments have invested in this layer.

Fewer than one quarter of IT leaders are very confident their organizations can manage governance when rolling out AI tools. The gap is not in technical security investment. It is in what technical security cannot see.

The second level is user risk: how employees actually interact with AI systems, and what that behavior reveals about compliance exposure, training gaps, and policy confusion. User risk is not about attackers exploiting your system. It is about employees using it in ways that create business, regulatory, or reputational risk, often without any awareness that they are doing so.

What user risk looks like in practice

User risk manifests in conversation patterns that are behaviorally invisible to technical monitoring because they do not constitute a system-level failure. The interaction completed successfully. The model responded. No error was logged. The risk exists in the content and context of what was exchanged.

Several patterns are consistently significant across enterprise AI deployments.

Unintentional PII exposure occurs when employees paste customer names, account numbers, or identification data into AI prompts as part of routine task completion. They are not attempting to create a security incident. They are trying to work efficiently. The data enters the AI system through normal use and may be retained in logs, processed in ways not covered by the organization's data agreements, or exposed through outputs. Technical security tools monitor for external attacks on the system. They do not monitor what employees voluntarily input.

Policy confusion patterns appear when employees ask the AI whether certain actions are permitted, test whether the system will accept restricted inputs, or ask variations of the same compliance question repeatedly. These patterns indicate that training has not landed effectively, that policies are unclear, or that employees lack confidence in how to use the AI within the organization's guidelines. Left unaddressed, confusion patterns are a leading indicator of eventual policy violations.

Compliance violations in outputs occur when employees request content that violates regulatory requirements, company policy, or ethical standards, and receive it. A customer service agent that provides different information to different customer segments, an HR tool that generates job descriptions with embedded bias, or a financial assistant that makes recommendations outside its sanctioned scope all create exposure that registers as successful interactions in technical dashboards.

Departmental and seasonal variation reveals that risk is not uniformly distributed across the organization. Certain teams, under specific types of pressure, show elevated risk patterns at predictable times. Employees in functions with high-volume deadlines, teams handling sensitive data categories, and departments that received less onboarding support consistently show higher risk concentrations. Knowing where risk is concentrating, before it manifests as an incident, is what makes governance proactive rather than reactive.

Why the distinction changes the governance response

The same behavioral pattern requires a completely different response depending on its cause. That is the practical value of distinguishing technology risk from user risk.

In the financial services deployment described earlier, the trading department's guardrail-testing behavior looked superficially like a security threat. Technical security tools, designed to catch attackers, would flag it as such. User analytics revealed it was confusion. The right response was targeted training on which data types the AI system was configured to handle, not enforcement action.

The HR department's PII spike during hiring season was predictable and addressable. It reflected employees under deadline pressure taking shortcuts, not a structural policy failure. The right response was process redesign during high-pressure periods, not additional access restrictions that would have reduced productivity across the organization.

The compliance team's requests for prohibited outputs were genuine violations requiring enforcement. But without behavioral analytics showing the pattern across conversations over time, the organization would not have known whether these were isolated incidents or a systemic issue in a specific function.

Gartner's research found that organizations providing persona and role-based guidance for AI users are twice as likely to report higher AI value, and those that conduct regular AI system assessments are three times more likely to achieve high AI performance. The causal relationship runs in both directions: governance practices that distinguish between training gaps, process failures, and genuine violations produce better outcomes than those that apply uniform enforcement to all risk signals. (McKinsey & Company)

Building the two-layer governance framework

Effective governance requires both layers operating in parallel, each doing what it is designed for.

The technical layer handles system-level threats: input validation that catches prompt injection attempts, output filtering that prevents sensitive information disclosure, API monitoring that detects unusual access patterns, and model integrity verification that confirms the system has not been compromised. This layer is the domain of security engineering and is not the responsibility of business or compliance functions to operate, though they need visibility into its outputs.

The user risk layer handles behavioral patterns: aggregated, anonymized analysis of how employees are using the AI system, what they are asking, where policy confusion is appearing, which functions show elevated risk concentrations, and whether the patterns that precede most incidents are forming. This layer is the domain of the AI governance function, working with legal, compliance, and HR to interpret what the patterns mean and determine the right organizational response.

A Gartner survey of 360 organizations found that those that deployed AI governance platforms were 3.4 times more likely to achieve high effectiveness in AI governance than those that did not. The organizations in that cohort were not simply investing more in security tooling. They were building the organizational infrastructure to understand and respond to both levels of risk. (McKinsey & Company)

The practical implementation has four components. An AI asset inventory that documents which systems are deployed, what they are authorized to do, and who uses them, is the foundation. Without knowing what is deployed, governance cannot be applied consistently. Usage monitoring that analyzes behavioral patterns at the aggregated, anonymized level gives the governance team the visibility to identify risk concentrations before they become incidents. A response framework that distinguishes between training gaps, process failures, and genuine violations ensures that governance interventions address root causes rather than symptoms. And regular review cycles, not point-in-time audits, give organizations the continuous signal that reflects how AI usage is actually evolving as employees find new applications for the tools they have been given.

The governance requirement for regulated industries

For organizations in financial services, healthcare, manufacturing, and other regulated sectors, the two-layer approach is not optional. Regulators are beginning to require that organizations demonstrate not just what AI systems are deployed but how employees are using them and what governance controls apply to that use.

The EU AI Act creates conformity assessment requirements for high-risk AI systems that include documentation of how the system is used in practice, not just how it was designed. Financial regulators under DORA and equivalent frameworks are beginning to require technology risk disclosures that extend to AI systems. Healthcare regulators have issued guidance on AI use that extends to how clinical staff interact with AI tools.

The practical requirement is the same: organizations need to be able to show what their AI systems do, how employees use them, what risk controls are in place, and how those controls are validated over time. An organization that can answer these questions from both the technical and behavioral layer is in a significantly stronger compliance position than one that can only report on system performance metrics.

Nebuly

Nebuly is the ROI platform for enterprise AI. It connects to the AI agents your business runs on, the assistants your customers interact with, and the tools your employees use every day, including Claude, ChatGPT, and Copilot, and translates that activity into business value. How much time is being saved across teams. What revenue your AI is influencing. What adoption and AI proficiency look like in practice, across departments and geographies. All aggregated at the organizational level, never tied to individuals.

If you need clarity on what your AI investment is actually delivering, book a demo.