How one of Italy's largest telcos cut unhandled AI requests by ~90% and made its support team self-sufficient

At a glance

• ~90% drop in unhandled requests over six months

• Error rate falling continuously for six months

• Fully self-hosted, cleared the operator's AI Act risk assessment

• Support team now runs its own reporting, with no analyst in the loop

The challenge

The operator runs several AI assistants across the business, including an IT help desk assistant employees use inside Microsoft Teams to resolve issues and open tickets. The team had no structured way to see what employees actually asked, where the assistant failed, or whether it was getting better. Reporting meant a developer exporting a CSV each week for the business team to read by hand. Quality issues went unnoticed, and no one could say with confidence the assistant would still be earning its place in six months.

The solution

Nebuly was deployed self-hosted on the operator's own cloud and connected via API to the assistant's conversation logs, with user data anonymised before analysis. That met the operator's data residency and AI Act requirements from the start. Nebuly analyses every conversation and surfaces what employees bring to the assistant, where it breaks down (unhandled requests, task failures, negative signals), and a single error-rate KPI the team tracks over time. A custom taxonomy maps everything to the operator's own categories, and the team builds its own reports, drilling from any metric straight into the underlying conversations.

The results

Over six months the error rate fell steadily. Unhandled requests dropped by roughly 90%. Just as telling, the business team stopped waiting on manual reports. They now run their own analysis in Nebuly, independent of the data science team.

Forward look

The operator plans to extend Nebuly across its customer-facing chatbots and voicebots, with AI ROI and AI proficiency reporting next.