AI Adoption Stories #2: How early analytics shaped a financial AI assistant
AI Adoption Stories #2: How early analytics shaped a financial AI assistant
TLDR
→ A European media company built a financial AI assistant for subscribers, giving readers fast answers on markets, stocks, and investment concepts. They instrumented behavioral analytics before launch, not after. → Early user data revealed the three query categories that dominated usage: specific stock information, current market movements, and complex financial instruments. The roadmap focused on these rather than assumed priorities. → Nebuly also served as an early warning system for model failures and hallucinations, allowing issues to be caught and fixed during beta before they reached a wider audience. → The clearest technical result was latency: average response time dropped from approximately 35 seconds to 17 seconds, guided by real usage data showing where performance was falling short. → The broader lesson: instrumenting analytics from day one means building around actual user behavior rather than assumptions. Teams that do this iterate faster and avoid building features nobody needs.
A large European media company set out to build a financial AI assistant for subscribers. The goal was simple: give readers fast answers on markets, stocks, and investment concepts.
They began with an autonomous agent that pulled market data, educational content, and news. What they needed next was clarity on how people would actually use it, plus confidence on accuracy and stability, and a plan to drive engagement.
Why they added analytics on day one
Before launching to real users, the team wanted full visibility into how the assistant would perform in the wild.
A demo of Nebuly showed clear, ready-to-use dashboards, options to keep data fully in their own infrastructure, and compliance standards fit for a media environment.
They decided to instrument analytics from the very start so they could build with real behavior in mind.
Building around real usage
With analytics instrumented from the start, the team could see exactly what early users were asking for. Patterns appeared fast.
Specific stock information
Current market movement updates
Details on complex financial instruments
This view of real behavior let them focus their build on the features people actually used. Nebuly also gave them an early warning system for model failures, so issues could be spotted and fixed before they reached more users. The same data became the baseline for testing new ideas and tracking their impact during the beta.
Early results
The clearest technical win was response time. Average latency dropped from about 35 seconds to about 17 seconds. Analytics also worked as an early warning system for bugs and hallucinations.
Real usage data informed the roadmap. Planned work focused on stock specific search, guided education for novice investors, and decision support for experienced readers.
What this means for other teams
Instrument early. When analytics are in from day one, you see real jobs to be done, avoid features no one needs, and iterate faster while the product is still in beta. The team here built around actual behavior instead of assumptions, with clear signals for both experience and reliability.
Nebuly
Nebuly is the ROI platform for enterprise AI. It connects to the AI agents your business runs on, the assistants your customers interact with, and the tools your employees use every day, including Claude, ChatGPT, and Copilot, and translates that activity into business value. How much time is being saved across teams. What revenue your AI is influencing. What adoption and AI proficiency look like in practice, across departments and geographies. All aggregated at the organizational level, never tied to individuals.
If you need clarity on what your AI investment is actually delivering, book a demo.
FAQs
What did early usage data reveal about how subscribers used the financial AI assistant?
Three query categories dominated from the first sessions: requests for specific stock information, questions about current market movements, and queries about complex financial instruments. This real usage data was different from what a product team might have assumed or prioritized based on the original design. It focused development resources on the features that users were actually seeking, and allowed the team to deprioritize functionality that looked important in planning but showed minimal user demand in practice.
How did analytics reduce latency by 50% for the financial AI assistant?
Usage analytics revealed where the assistant's response time was creating friction, specifically which query types were producing the longest latency and where the experience was degrading noticeably. With that data, the engineering team could target optimization at the specific parts of the system that were creating the most user-facing delay, reducing average response time from approximately 35 seconds to approximately 17 seconds. Without usage analytics showing where the latency problem was concentrated, optimization would have been less targeted and less effective.
What is the practical value of using analytics as an early warning system during beta?
During beta, a model failure or hallucination that reaches a small number of users is a manageable problem. The same failure reaching a large audience is a trust and reputational problem that is much harder to recover from. Analytics that surface model failures, incorrect outputs, and unusual patterns in real time during beta allow the team to identify and fix issues before they scale. This is particularly important for financial content, where incorrect information can have direct consequences for users making investment decisions.
Stay up to date on what we're learning, building, and seeing as enterprise teams deploy and measure AI agents in production.