Feature #03

User-rated dataset for evals

Automatically generate a user-rated dataset from implicit feedback embedded in interactions to use in your LLM evaluation pipeline.

Harness both implicit and explicit in-production user feedback to automatically generate datasets for model refinement.

This feature leverages the implicit and explicit feedback provided by users directly engaged with your language models in a production environment, turning them into invaluable data labelers.

By capturing both explicit user feedback (👍/ 👎) and the more subtle cues embedded within user interactions (implicit feedback), this system automatically constructs a dataset that is rich with real-world insights. This method allows for continuous improvement of model performance based on direct user input, bridging the gap between AI-as-a-judge and practical, user-driven feedback.