This research explores the impact of dialogue context on crowdsourced labeling quality. Conducted in two phases, the study manipulates both the amount and type of contextual information to assess its effects on annotators' relevance and usefulness ratings. In Phase 1, the experiment varied the length of dialogue context provided to participants, while Phase 2 focused on different types of context. The researchers designed Human Intelligence Tasks (HITs) without revealing the study's purpose to minimize bias in labeling, gathering valuable insights into contextual influences on annotators' judgments.
Our study investigates how the amount and type of dialogue context influence annotators' judgments of relevance and usefulness in crowdsourced labeling tasks.
In Phase 1 of our experiments, we discovered that varying the amount of dialogue context provided significantly impacts the quality and consistency of relevance labels.
By varying the contextual information in our tests, we aim to provide deeper insights into how these changes affect annotators' perceptions and judgments.
The study refrains from disclosing the research angle to crowdworkers to minimize bias, ensuring the integrity of the labels collected.
#crowdsourcing #dialogue-context #annotation-quality #human-intelligence-tasks #research-methodology
Collection
[
|
...
]