Docs
Scores & Evaluation
Annotation in UI

Human Annotation for LLM apps

Collaborate with your team and add scores via the Langfuse UI. You can add scores to both traces and observations within a trace.

Annotate

Common use cases:

  • Collaboration: Enable team collaboration by inviting other internal members to annotate a subset of traces and observations. This manual evaluation can enhance the overall accuracy and reliability of your results by incorporating diverse perspectives and expertise.
  • Annotation data consistency: Create score configurations for annotation workflows to ensure that all team members are using standardized scoring criteria. Hereby configure categorical, numerical or binary score types to capture different aspects of your data.
  • Evaluation of new product features: This feature can be useful for new use cases where no other scores have been allocated yet.
  • Benchmarking of other scores: Establish a human baseline score that can be used as a benchmark to compare and evaluate other scores. This can provide a clear standard of reference and enhance the objectivity of your performance evaluations.

Create score configurations

To use annotation, you need to create a score configuration. You can create multiple score configurations for different types of scores. Score configurations are immutable. However, you can archive configs if you no longer want to use them in annotation. Archived configs can be restored at any time.

To create a score configuration:

  • Navigate to Settings, locate the Score Configs table, and click Add new score config.
  • Specify the name of the score configuration and the type of score you want to create. You can choose between Categorical, Numeric, and Boolean score types.
  • Optionally, add a description to provide additional context for your team members.

Your configs are now available for annotation of traces and observations.

Create config

Data Labelling on LLM traces or observations

To annotate a trace or observation:

  • Navigate to the trace detail view.
  • Click on the Annotate button.
  • Select the scores you want to add.
  • Fill in the score values. The scores will be saved automatically. Annotation scores can be edited or deleted at any time.
  • Optionally, add comments to individual scores.

Annotate

View scores on trace or observation

Upon completing annotation click on the Scores tab to view a table of all the scores that have been added to the trace or observation.

Detail scores table

Was this page useful?

Questions? We're here to help

Subscribe to updates