With LLM-powered applications, gathering user feedback as soon as possible is crucial. Post-launch, user feedback is essential for building a diverse evaluation dataset, ensuring continuous improvement of both the model and user experience. But how do you collect feedback, and how do you decide between collecting explicit feedback and implicit feedback? This post explores both methods, their differences, and evaluates their pros and cons.

Methods of Collecting LLM User Feedback

There are two primary methods for collecting LLM user feedback: explicit and implicit.

What is Explicit LLM user Feedback?
Explicit feedback is direct and intentional feedback collected from users after interacting with your product. Examples include rating systems such as ⭐ rating, thumbs up/down 👍 👎, surveys, and written comments. Users are aware that they are providing feedback, and this data can be structured, such as a numerical rating, or unstructured, like free-form text comments.

What is Implicit LLM user Feedback?
Implicit feedback is indirect feedback gathered from user prompts, behaviour and interactions with the application. One example is tracking user reactions to generated outputs, such as monitoring the frequency of followup questions around a certain topic, or request to improve the output in some specific way. Implicit user feedback can also include engagement data like time on site and drop-off rates. This data provides insights into areas where the model's performance needs improvement, revealing patterns in user behaviour that can inform model adjustments and enhancements. Without specifically asking for consent, users may not be aware that their actions are being monitored for implicit feedback purposes.

Pros and Cons of Explicit LLM User Feedback

Pros:

• Clarity: Explicit feedback provides clear and direct insights into user opinions and satisfaction.
• Specificity: Users can point out specific issues, making it easier to identify and address problems.
• Transparency: Users know how their feedback is being used, which can build trust.

Cons:

• Volume: The amount of explicit feedback collected is often extremely limited, which can skew results. Response rates to Explicit LLM user feedback requests are commonly only 1-3 %.
• Subjective: Users providing feedback may have extreme opinions (either very positive or very negative), which may not be representative of the broader user base.

Pros and Cons of Implicit Feedback

Pros:

• Volume: Implicit feedback can be gathered from all users, providing a larger and more comprehensive dataset.
• Cost-effective: It’s generally cheap to collect since it doesn’t require active participation from users or the service provider.
• Objectivity: Implicit feedback is less likely to be biased as it captures data from a larger user sample and is based on actual user behavior rather than subjective opinions.

Cons:

• Interpretation: Implicit feedback can be challenging to interpret. For example, a user quickly leaving a conversation can mean they got their answer or they thought it was worthless.
• Privacy: If users are not made aware of how their data is being used, implicit  can raise privacy concerns.
• Incompleteness: Implicit feedback may not capture the full context of user interactions, making it harder to understand the "why" behind certain behaviors.

The Power of Intelligent Implicit LLM user Feedback

While explicit feedback provides valuable insights, its limitations make it less scalable for large-scale applications. Implicit feedback, on the other hand, offers incredible potential when collected and analyzed intelligently. Here’s why:

1. Scale: Implicit feedback can be collected from the entire user base, offering a more comprehensive view of user interactions and preferences.
2. Efficiency: Automated collection of implicit feedback reduces the need for active user participation, which can save time and resources.
3. Depth: When analyzed correctly, implicit feedback can reveal deep insights into user behavior and preferences that might not be captured through explicit feedback alone. For instance, tracking the followup requests users make to generated outputs can provide rich data on areas where the model and the business more broadly needs improvement.
4. Continuous Improvement: Implicit feedback enables continuous learning and adaptation. By monitoring user behavior in real-time, models can be updated more frequently and accurately, leading to better performance and user satisfaction.

Combining Both Methods for Optimal Results

While implicit feedback offers significant advantages, combining it with explicit feedback can provide the best of both worlds. Explicit feedback can validate and contextualize the insights gained from larger implicit feedback data set, offering a fuller picture of user satisfaction and areas for improvement.

For example, an LLM-powered resume assistant could use implicit LLM user feedback to track which sections of the generated resumes users edit and/or ask on their next prompt for the model to improve most frequently. This data can highlight areas where the model’s performance needs enhancement. Simultaneously, explicit feedback collected through direct ratings can provide additional context and specific suggestions from users.

Conclusion

Both explicit llm user feedback and implicit LLM user feedback have their places in evaluating and improving LLM-powered applications. While explicit feedback offers clarity and specificity, its scalability issues make it less viable for forming a evaluation data that represents the whole user base. Implicit feedback, on the other hand, provides a wealth of data from the entire user base, though it requires consideration of privacy concerns.

Ultimately, the most effective approach often involves a combination of both methods, leveraging the strengths of each to continuously refine and enhance the application’s performance. By intelligently collecting and analyzing implicit feedback, businesses can gain deep insights into user behavior, leading to better models, higher user satisfaction, and greater overall business success.

About Nebuly

Nebuly is dedicated to enhancing LLM-based products by gathering valuable user feedback, both explicit and implicit. If you're interested in discovering more about how we can help optimize your AI solutions, we’d love to chat. Schedule a meeting with us today, HERE!

‍

‍