User behavior

Autocapture the behavior of your LLM users

Empower your team to gain instant visibility into massive amounts of LLM user data.

Understand if your users are engaged and satisfied when interacting with your LLM

Gain complete visibility into engagement and satisfaction with Nebuly's engagement analytics. With Nebuly, you can track key metrics such as:
  • Total, New, and Returning Users: Track overall engagement and break it down between first-time and users who revisit the LLM, indicating sustained interest.
  • Number of Interactions: Quantify overall interactions, analyzing trends over time to spot patterns.
  • Average Daily Interactions per User: Assess the typical frequency of user interactions, providing insights into engagement habits.
  • Users by Average Daily Interactions: Classify users based on average daily engagement, revealing distinct user behaviors.
  • User Satisfaction: Monitor the evolution of user satisfaction as they interact with your LLMs over time

Understand how frequently your users are interacting with your LLM

The Retention tab allows you to assess how long users continue to interact with your LLM product.
It enables to answer questions such as:
  • How many users are still active two weeks after they started interacting with the LLM? How many after two months?
  • Of the users who actively interacted with the LLM during a particular week of the year, e.g. when I announced a major product release, how many were still active a week later?

Retention parameters

At the top of the Retention page, you can select 3 time parameters. By changing the first two, you define the time frame you want to analyze the retention for, which by default is set to the last 7 days. The third parameter, which is set to days by default, defines the granularity at which the retained users are visualized.
For example, if you select a 3-week time frame and a daily granularity, the Retention charts will provide insight into how many users were retained for each day over the past 3 weeks.

Retention definition

The retained user is defined as a user who has had at least one interaction with the LLM.
The Retention tab on Nebuly platform provides two visualizations of the user retention:
  • Retention cohorts
  • Average retention across cohorts

Retention cohorts

The Retention cohorts allow you to understand whether the users who were active in a given time window continued to interact with the LLM during the rest of the time frame.
The first column of the Retention cohorts table shows the time window, for example, the week between December 11 and December 17. The second column shows how many users are in the cohort of users who had at least one interaction during that time interval. The following columns show the percentage of users in that cohort who had at least one interaction in subsequent time periods, such as the following weeks.
The first row of the graph shows the average number of users and their weighted average retention. This is evaluated as described below.

Average retention

The Average retention, shown both as line chart on top of the Retention page and as tabular way in the first row of the Retention cohorts table, is evaluated as the weighed average retention.
For example, let's say we want to analyze weekly retention for the two weeks of December 10-16 and December 17-25. In the week of December 10-16, 10 users had at least one interaction with the LLM, and of those, 6 users also had at least one interaction in the following week, which we will call week 1. In the week of December 17-25, 20 users had at least one interaction with the LLM, and 10 of them had at least one interaction in the following week (week 1). The average number of users in Week 1 is evaluated as the arithmetic average: (6+10)/2 = 8 users. The average retention in Week 1 is calculated as a weighted average: (6+10)(10+20) = 53% average retained users.

Explore tasks, questions and actions users are doing

The best way to easily understand what your users are doing with your LLMs is using the User Flow chart. This chart identifies the most frequent User Actions (tasks or questions) done by your users. You can use User Flow to understand how your users sequentially ask questions to your LLM-product and analyze drop-offs or unsuccessful behavior.

Understand the most discussed topics by your users

Topic reveals the subjects users frequently discuss in your LLM product. By identifying these popular topics, you can enhance your LLM product by enriching content in areas most discussed by users. For clear visualization, use the Analytics Chart, filtering Users or Interactions by Topic. To track trends over time, utilize the Line Chart or Vertical Bar Chart.

Understand why LLM users are dissatisfied

Nebuly allows you to automatically understand why some of your users are not satisfied with the answers they are getting from your LLMs. This can help you understand where to improve your LLMs.

Frustration reasons

Explore why some LLM answers are insufficient according to your users. To track the causes of frustration, use the Analytics chart, filtering Users or Interactions by Frustration Reason. To track trends over time, utilize the Line Chart or Vertical Bar Chart.

Challenging questions

Pinpoint user actions and questions that lead to dissatisfaction. Use the Analytics chart, break down Users or Interactions by Action and filter by all Frustration Reason. To track trends over time, utilize the Line Chart or Vertical Bar Chart.

Topic deficiencies

Determine topics where LLMs' answers are inadequate for your users. Use the Analytics chart, break down Users or Interactions by Topic and filter by all Frustration Reason. To track trends over time, utilize the Line Chart or Vertical Bar Chart.

RAG effectiveness

Evaluate the efficacy of RAG data sources for your users. Use the Analytics chart, break down Interactions or Users by Data Source and filter by all Frustration Reason. To track trends over time, utilize the Line Chart or Vertical Bar Chart.

Dissatisfied user behavior

Analyze user behavior after (or before) receiving an unsatisfactory answer. Use the User Flow chart and filter by all Frustration Reason.

Time to value

Understand how long it takes your users to get the answers they were looking for. Select two actions in the Time To Value chart to learn how long it took users to complete the second action after performing the first.

Understand RAG data sources commonly utilized by your users ​

Data Sources show the most accessed RAG sources by your users. To view this clearly, utilize the Analytics chart and filter by Data Source for Users or Interactions. To track trends over time, utilize the Line Chart or Vertical Bar Chart.
Cohorts

Understand user cohorts of your LLM products

User cohorts are groups of users who share a common characteristic, such as date of the first interaction, engagement with the LLM, or behavior.
Leverage user cohorts in charts to gain a better understanding of the user journey and behavior of specific groups of users and identify opportunities to increase user engagement or reduce churn.

Gain insights on first-time and repeat users

Nebuly helps to identify the value perception of users who had their first interaction in the chosen timeframe and compare it with those who have repeatedly engaged with the LLM.
  • Assess user value perception for first-time interactions.
  • Track onboarding satisfaction trends and analyze first-time user interaction frequency.
  • Compare trends between first-time and repeat users.

Identify the most engaged LLM users

Nebuly helps you highlight the users who find the LLM's answers useful.
  • Review satisfied user profiles.
  • Track satisfaction trends over time.
  • Monitor changes in their interaction frequency.

Identify the users that are likely to churn

Nebuly helps pinpoint which are the users dissatisfied with your LLMs' answers, enabling timely action.
You can gain insights into these user groups by:
  • Examining neutral and dissatisfied user profiles.
  • Tracking users' frustration trends over time.
  • Observing changes in their interaction frequency.

Create custom cohorts

Nebuly empowers you to build customized cohorts based on user frequency, behavior, or satisfaction.
Unlock insights within these user groups by:
  • Applying the created cohort in charts and reports.
  • Compare the engagement and behavior of different types of user cohorts
Cost tracking

Track the cost of your LLMs

Understand how cost correlates with user engagement and satisfaction to measure ROI.

Track how much each LLM user is costing you

Nebuly allows you to link user satisfaction levels to LLM costs.
For a clear cost-benefit analysis, it addresses:
  • The cost of serving dissatisfied users.
  • The cost of serving satisfied users.
Security & compliance

Enterprise-grade security

At Nebuly we take security and compliance extremely seriously. Whether you’re a fast-growing startup or one of the largest companies in the world, your data is in good hands with us.

We maintain strong regulatory compliance for data protection

We adhere to protocols that meet the certification standards required to comply with key data protection legislation. We are actively working towards obtaining SOC2 Type 2 and ISO 27001 certifications and undergoing audits.

We ensure PII removal for enhanced data privacy

We automatically identify all data fields that contain personally identifiable information (PII) and replace all PIIs with pseudo values or codes that do not reveal any personal information about the individual.

We employ high-level encryption for maximum data security

Secure your data with the same encryption that banks use and ensure that no authorized parties can view information. All data in transit is encrypted using TLS/SSL.

We offer robust identity and access management for secure data control

Create granular permissions that dictate who can access and work with your data.
  • Support for role-based permissions
  • Logs can be deleted upon requests

We prioritize infrastructure security for data protection and scalability

We take steps to ensure that our infrastructure is both secure and scalable.
  • Private endpoints enforced across system infrastructure
  • IP whitelisting and private VPC enforced

We provide thorough security training to ensure compliance and data protection

All Nebuly staff undergo comprehensive security training when they join and periodically thereafter to maintain compliance standards.
Craft bespoke LLMs for your users
Request a demo and you'll get an overview of Nebuly's capabilities and how it can help understanding your users.