Use cases

A global health and wellness technology company partnered with Nebuly to better understand how people use its GenAI assistant.

At launch, the company previewed the assistant to 1.5–2.0 million eligible users, with Nebuly analytics designed to support up to 500,000 daily interactions at full scale.

GenAI Assistant – User Analytics

Nebuly provides analytics for the company’s customer-facing GenAI assistant.
‍
The platform analyzes chat interactions to uncover user intents, satisfaction levels, friction points, and risk signals, giving teams visibility into how people actually experience and engage with the assistant.

This helps identify when users are frustrated, confused, or disengaged, enabling continuous improvement of conversation quality and user satisfaction.

GenAI Assistant – Behavioral Analysis

‍Nebuly also enables behavioral segmentation to understand how different types of users interact with the assistant over time.

By grouping users based on their engagement patterns, for example:

• Light users (1–2 conversations, <5 turns each)

• ‍Medium users (several conversations, 5–10 turns)

• Deep users (multiple conversations, long sessions)

teams can uncover usage habits, engagement depth, and behavioral trends across cohorts (e.g. morning-only users vs. power users).

This helps product and data teams profile user behavior, identify what drives adoption or drop-off, and design more personalized and effective chatbot experiences.

Challenges

Limited visibility on user behavior

The company could not systematically identify what people discuss with the GenAI assistant or where it performed below expectations. Earlier analysis relied on manual CSV exports and spreadsheets, missing quiet but important user signals.

Measurement challenges

Users often struggled with frustrating chatbot experiences, receiving irrelevant, incomplete, or incorrect answers. The company lacked visibility into when and why users were dissatisfied, since existing tools couldn’t measure LLM-specific issues like poor response quality or confusing follow-ups.

Without clear user-centric metrics such as user satisfaction, error rate, or perceived risk, the company couldn’t identify what made conversations fail or improve them over time.

Compliance and privacy needs

Because the product handled sensitive health topics, privacy and compliance were non-negotiable. Teams required a solution that controlled access, kept data local, and excluded user-specific or regulated content.

The Nebuly solution

Data integration and privacy-first setup

A self-hosted Nebuly instance on AWS brought together assistant and experience data. The solution provided unified analytics across business units and supports custom tagging, clustering, and demographic analysis.

Defined KPIs and behavioral signals

Nebuly introduced consistent metrics such as user satisfaction, error rate, and engagement, tracked by topic and user cohort. Dashboards now track risk and satisfaction trends in real time.

Privacy, compliance, and auditability

Granular permissions, retention policies, and user-level deletion ensure compliance. The self-hosted option gives internal stakeholders full control over healthcare-related data.

Self-service insights

The platform enables product managers, analysts, and behavioral scientists to access information directly. This replaced manual analysis and improved collaboration across functions.