Nebuly product Changelog

July 18, 2024

Upgrade

New Overview page

We’ve restructured the overview page to be like a control panel, providing a comprehensive view of your LLMs’ performance. Special emphasis has been placed on the insights section, where insights can now be sorted by “Trending” and “Most Discussed.”

For all charts, we’ve added a comparison feature against the previous time period. For instance, if you select “Last 30 Days” as your time range, data from the 60 to 30 days prior will also be displayed as a dashed line.

Additionally, we've introduced a view dedicated to the most relevant Topics and Behavioral Alerts. The topics view provides a snapshot of what your users are discussing, while the Behavioral Alerts view allows for continuous monitoring of negative user intents that could potentially impact your business.

July 16, 2024

Upgrade

User intelligence update

We've updated the User Intelligence page with a refreshed UI, enhancing visual clarity in displaying coverage (above the total number of interactions) for each topic and intent.New metrics have been introduced for improved visibility into the conversations involving various topics and intents.
These include:• ConversationsTotal number of conversations in which a topic, intent, or behavioral alert appears.

• Average interactions per conversationsThe average number of interactions in conversations that included a specific topic, intent, or behavioral alert.
• Average conversation timeThe average duration of user conversations involving the topics or intents listed in a table row.

With the addition of more metrics, we've also incorporated the ability to personalize your view on the User Intelligence page. Simply click the "edit columns" button to select and arrange the columns that best meet your needs.

July 12, 2024

New Feature

New Global filters

We’ve updated the UI for global filters to enhance their visibility. Now, you can always see which filters are currently active.

New & Improved
• We've renamed "User Feedback" to "Problems Identified." You can now view all negative user feedback detected by the platform using the same metrics introduced for user intelligence.

• We've updated the behavior of the external endpoint. Now, for each aggregate, you can select "topics" as a variable to receive a list of topics to which the aggregate belongs. For example, this applies when the aggregate is a 'type_of_problem' and you want to know the topics affected by a model issue.
‍
• Report It is now possible to add filters at the report level. Filters added at this level will not be displayed in the created charts.
‍
• Report Sharing You can now share your reports with edit permissions. Users will be able to edit individual charts but won't have the ability to modify filters set at the report level. This ensures that your stakeholders can customize their data views without the risk of accessing data from other projects they are not permitted to see.

Fixes
• Fixed an issue not showing the full intents when the sentence was too long.

July 04, 2024

Upgrade

New favorites feature

You can now select the topics and user intents you want to save as favorites.
All the favorites you previously saved will be visible on a new page in “User Intelligence”. You can save existing topics and intents as favorites and also define new keywords or sentences you would like to track. The custom-defined favorites will be searched in the user requests (queries).

June 25, 2024

Upgrade

New Retention Page & retention metric

To improve visibility on how topics, user intents, and negative user feedback impact product retention, we have redesigned the retention feature. This update allows you to select a topic, intent, or user feedback and analyze user retention based on those specific categories.

Additionally, topics, intents, and user feedback now display retention metrics within the selected time frame. Given a time range, retention for a topic is calculated as the percentage of users who, after discussing the topic, return to converse with the LLM in a separate session.

June 20, 2024

New feauture

Duplicate charts and reports

We have added the ability to duplicate charts and reports. To do this, simply click the button in the top left corner of a report and select the duplicate option. For individual charts, you can also select the report in which you want to duplicate the chart.

New & Improved
• Updated the external endpoints. Now multiple `group by`s can be selected over
• Implemented new external endpoint to create custom group bys from pre-exisiting ones. Docs can be found at https://docs.nebuly.com/tracking/api-reference/embedded-pages/interactions/get-interaction-multi-aggregates
‍
Fixes
• Fixed a bug preventing the process of all interactions in some edge cases.
‍

June 18, 2024

Enhancement

Merge topics and explore “Other”

Now, it is possible to merge multiple topics together.
Additionally, we have added the option to explore the “Other” category. This allows you to delve into various other topics and move the ones you’re interested in tracking to the “main” topics section.
The main topics are defined as the top 9 by volume within a given time range, but we also retain all the topics you moved from the “Other” section to the main topics. It is possible to move topics both from “Other” to “Most Relevant Topics” and vice versa.

New & Improved
• The default time range is now set to 7 days.
• Filter values are now searchable, and we have added the option to select all filters.
• "Read Only" permissions have been added for sharing individual pages.
• When selecting filter values, you can now select all values by simply clicking on “Select All.”
‍
Fixes
• Fixed a bug in the pagination of user intelligence, showing the wrong page after a filter was applied.
• Fixed a bug preventing to visualize the merged intents and topics.
• Fixed a bug preventing to visualize the first option in the filters in the “Audiences” page.
‍

June 14, 2024

Update

Update filters UI

We have released simplified versions of our filters across the entire platform.
Now, you can easily choose to show or hide custom intents or rules directly from the filters. Additionally, we’ve made it much simpler to delete previously selected filters. Now, when you have filters selected, clicking the filter button will first display the selected values, allowing you to easily delete them if needed.

June 12, 2024

New feature

De-merge intents and rules previously created

We have added one of the most requested features!

Once you merge a group of intents or implicit user feedback, you can now roll them back to their individual values. Merged intents and feedback are identified by the “merged” tag, and selecting them will show the unmerge details.

June 10, 2024

New feature

Share reports, user intelligence and user feedback data outside the platform

We have introduced a new feature that allows you to share specific snapshots of your user intelligence, user feedback, and reports pages with individuals who are not registered on the platform.

Now, you can easily share the usage reports you create on the Nebuly platform with your colleagues and customers. They will have access only to the data you choose to share, and their access will be restricted to the specific page you selected.

Sharing a view of a specific page is simple:
‍
1. Click on the share button in the top-right corner of the page.

2. In the pop-up window, select the type of permissions you want to grant.
‍
3. Click on the copy link button to copy the link to your clipboard.
‍
4. Share the link with the desired recipients.

June 07, 2024

Enhancement

Group by tags as a view in your User Intelligence page

You can now view data grouped by tags on the user intelligence page. Sort by different group by options, selecting your preferred one as predefined and disabling the ones you’re not interested in.

New & Improved
• We have completely redesigned the APIs for embedded pages. The updated documentation can be found at: https://docs.nebuly.com/tracking/api-reference/embedded-pages/quickstart
• We added new examples for the usage of our apis. See the docs for further info.
‍
Fixes
• Fixed a bug that prevented the merging of negative implicit feedback with previously merged negative implicit feedback.

Deprecated features
• We are currently deprecating all old embedded APIs, including all endpoints previously used for retrieving warnings and intents. If you are still using the old APIs, please refer to the new APIs.

May 27, 2024

New feature

Semantic search and merge negative implicit feedback

We have added new functionality to the user feedback page, enhancing how you search and manage negative implicit feedback. With our semantic search feature, you can now easily find issues without needing the perfect keyword. Simply search for the problem you’re interested in, such as “missing capabilities,” and you’ll discover all related feedback where customers have reported that the assistant is lacking specific capabilities.

Additionally, you can now merge negative implicit feedback for broader tracking. This feature is especially powerful when used with semantic search. For example, if you want to monitor a general issue, like the assistant replying in the wrong language, you can search “replying in the wrong language,” select the relevant feedback, and merge it to create a generalized implicit feedback category. From then on, the platform will track this generalized feedback instead of individual detailed instances.

May 24, 2024

Enhancement

Trends for intents and feedback

Now, both user feedback and user intents illustrate the changes in the number of interactions, users, and feedback within the selected time frame compared to the previous period. This is particularly useful to have a quick understanding on how the behaviour of your users is changing over time. For example, if you select the last 30 days as your time range, you’ll see the relative improvement compared to the period from 60 to 30 days ago.

We have also included information on the percentage of total interactions that correspond to a specific intent or user feedback to give a better understanding on how relevant they are to your customer base.

May 21, 2024

New feature

User frustration

We have added the negative interactions trend to the overview page, providing you with a quick insight into your LLM product’s performance and customer satisfaction levels. According to our data, best-in-class products have less than 5% negative interactions, while the most problematic user experiences exceed 40% negative interactions. This powerful tool offers a concise snapshot of your performance.

New & Improved
• Enhanced the UI across various features, optimizing the user experience. The user intelligence and user feedback pages now display the same information as before, but with significantly fewer elements, resulting in a cleaner and more efficient interface.
‍
Fixes
• Fixed a bug that occasionally caused duplicated interactions to appear on the intent details page.

May 20, 2024

New feature

Four distinct types of user feedback

You now have a new, more comprehensive visibility into LLM user feedback. You can view four distinct types of LLM feedback:1. Negative Implicit FeedbackUsers express dissatisfaction during their conversation when the LLM doesn't meet their expectations.2. Negative Explicit FeedbackUsers give a thumbs down to an interaction with the assistant.3. Positive Implicit FeedbackUsers commend the assistant during the chat for a well-generated answer.4. Positive Explicit Feedback:Users give a thumbs up to an interaction with the assistant.

Analyze feedback grouped by intent and drill down into the ones that interest you most. For negative implicit feedback, you can define your own custom rules. Additionally, we've added a toggle feature on the platform that allows you to view only the rules you've created.

New & Improved
• Not LLM related issues have been moved in user behaviour section as negative intents
• You can now mark multiple implicit warnings as irrelevant simultaneously. Simply select their checkboxes and click the button that appears at the bottom.
‍
Fixes
• Fixed a bug preventing the creation of custom rules

May 14, 2024

Update

More focus on user feedback

We have updated our User interface to better separate the analysis of User Behavior and User Feedback. The User Warnings page now called User Feedback provides enhanced visibility into Feedback about the LLM, and you can create custom rules for specific categories of warnings. Additionally, with the "Group by type" or "Group by user intent" features, you can organize the feedback in the way that suits you best.

New & Improved
• Updated the model proposing the suggestions. Now a larger number of suggestions is proposed with a better quality on the selected warnings.
• Added meaningful empty states showing the estimated time for the end of the interaction processing
‍
Fixes
• Fixed major bug in the MAU computation in the overview.

May 06, 2024

Enhancement

Customize your end-user intents

Select one of the predefined suggestions

You can now fully customize the intents generated by the Nebuly platform. You have the option to create a new intent to track a specific user behavior, or you can merge two detailed intents into a new intent that better suits your specific needs.Merged intents inherit all the warnings and interactions of the original intents. Newly created intents look back over the last 60 days of interactions to find occurrences of the defined intent.

New & Improved
•The UI of the warnings page has been refreshed, including the addition of a new table for the user-defined rules, providing a much clearer picture of the different warning detection rules.
• Added the possibility to filter the interactions in the user intent details page.

Fixes
• Fixed bug not showing any result when searching over filter values for custom tags
• Now global filters apply on retention chart as well.

May 03, 2024

New feature

Custom Warning Categories 2/2

Define your own warning category

You can now decide what type of warnings are relevant to your business and define your own warning category. This gives you full control over the type of warnings tracked on the platform.

Apr 29, 2024

New feature

Custom Warning Categories 1/2

Select one of the predefined suggestions

In addition to the custom warning category, we are also releasing the warning suggestion feature. Warning suggestions give you an overview of the most impactful warnings that the platform has automatically detected from your user interactions. You can then decide to accept the suggestion and turn it into a warning category, or mark the suggestion as irrelevant. This allows the platform to adapt to your needs and provide more relevant suggestions in the future.

New & Improved
• New simplified external endpoints to get the warnings generated by the platform

Fixes
• Fixed a bug in the global filters that displayed the wrong number of interactions next to the suggested filter values.

Apr 22, 2024

Enhancement

Increase visibility on trending intents

Now you can sort the user intents in the user intelligence page by most trending other than by number of users and number of warnings. We also have refreshed the UI, with a cleaner view on the user intents.
‍
New & Improved
• In the overview page you can now see how the trends evolved respect the previous month
• It is now possible in the overview page to change the time-range you want to get the overview on

Fixes
• Fixed a rare issue showing the same warning both as LLM related and not LLM related

Apr 16, 2024

Enhancement

Global filters

Filters that stay on as you navigate

With our latest update you can use global filters that are shared across the platform pages. These filters are particularly useful when you want to analyse your data about a specific user group.

New & Improved
• The external endpoints that can be used to embed our overview, user-intelligence and warnings pages have been updated considering the latest platform UX. Find more info on our Docs.
• The overview, user-intelligence and warnings pages have a refreshed UI
• The Navbar and the main platform components feature a redesigned UI

Fixes
• We fixed a bug in the intent computation that was creating new intents for newer interactions, even when the same intent already existed.

Apr 12, 2024

Enhancement

Intent search and a new user warning category

Search your users intents

Now you can simply search over all the intents and apply filters. You can easily find the information you are looking for in a matter of seconds.

We also added a new “Harassment” category to user warnings. This category shows when the user is trying to harass the model or cover some not-allowed topics like asking for explixit sexual content or asking for dangerous information (like building a bomb).

Apr 05, 2024

Feature

Enhancement

Update on user warnings

We grouped user warnings into 4 categories

1. Business warnings: valuable customer insights on what you can improve in your product and business
‍
2. Lack of personalization: users do not find the model responses tailored to their specific user profile. This may be because the answers are too complex or simply not relevant to their needs.
‍
3. Wrong information: users complain about information being wrong.

Changelog

Get the latest product updates straight to your inbox

Get the latest product updates
straight to your inbox