← Back to papers
2018 ACL ACL 2018

Efficient Online Scalar Annotation with Bounded Support

Abstract

AbstractWe describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels) proposed here. Our proposal leads to increased correlation with ground truth, at far greater annotator efficiency, suggesting this strategy as an improved mechanism for dataset creation and manual system evaluation.

🧭 Keyword Pioneer — dataset construction
🐣 Hot Topic Early Bird — human annotation
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Keisuke Sakaguchi , Benjamin Van Durme

Topics

Machine Learning > Core Methods > Regression Machine Learning > Learning Types > Active Learning Machine Learning > Optimization & Theory > Statistical Learning Machine Learning > Learning Types > Evaluation Machine Learning > Core Methods > Optimization

Keywords

online learning pairwise ranking human annotation annotation efficiency dataset construction scalar annotation ground truth correlation
Download PDF

Related papers

Economic Event Detection in Company-Specific News Text 2018
Investigating Effective Parameters for Fine-tuning of Word Embeddings Using Only a Small Corpus 2018
SemAxis: A Lightweight Framework to Characterize Domain-Specific Word Semantics Beyond Sentiment 2018
Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer 2018
Affordances in Grounded Language Learning 2018