This research explores the effect of dialogue context on the quality of crowd-sourced labels used in evaluating task-oriented dialogue systems (TDSs). The study finds that providing only a portion of the dialogue context can produce more favorable ratings from annotators, although it risks losing critical details necessary for comprehensive evaluations. Conversely, presenting the full context improves relevance ratings but complicates usefulness assessments. The findings advocate for refined task designs that optimize annotation efficiency while ensuring quality, suggesting large language models may assist in generating concise summaries for better annotator performance.
Crowdsourced labels are essential for evaluating task-oriented dialogue systems, yet challenges exist in obtaining consistent groundtruth from annotators, especially regarding dialogue context comprehension.
Our study examines the impact of truncated dialogue context on annotation quality, revealing that limited context can yield more favorable ratings albeit risking the richness of evaluations.
Collection
[
|
...
]