← Back to papers
2018 ACL ACL 2018

Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning

Abstract

AbstractWe present a study on reinforcement learning (RL) from human bandit feedback for sequence-to-sequence learning, exemplified by the task of bandit neural machine translation (NMT). We investigate the reliability of human bandit feedback, and analyze the influence of reliability on the learnability of a reward estimator, and the effect of the quality of reward estimates on the overall RL task. Our analysis of cardinal (5-point ratings) and ordinal (pairwise preferences) feedback shows that their intra- and inter-annotator α-agreement is comparable. Best reliability is obtained for standardized cardinal feedback, and cardinal feedback is also easiest to learn and generalize from. Finally, improvements of over 1 BLEU can be obtained by integrating a regression-based reward estimator trained on cardinal feedback for 800 translations into RL for NMT. This shows that RL is possible even from small amounts of fairly reliable human feedback, pointing to a great potential for applications at larger scale.

🌉 Interdisciplinary Bridge — Machine Learning and Natural Language Processing and Reinforcement Learning
🧭 Keyword Pioneer — reward estimator
🐣 Hot Topic Early Bird — bandit feedback
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Julia Kreutzer , Joshua Uyheng , Stefan Riezler

Topics

Machine Learning > Optimization & Theory > Stochastic Processes Natural Language Processing > Applications > Machine Translation Reinforcement Learning > Methods > Policy Learning Machine Learning > Learning Types > Reinforcement Learning

Keywords

reinforcement learning neural machine translation bandit feedback human feedback sequence-to-sequence learning bandit learning reward estimator human bandit feedback
Download PDF

Related papers

Economic Event Detection in Company-Specific News Text 2018
Investigating Effective Parameters for Fine-tuning of Word Embeddings Using Only a Small Corpus 2018
SemAxis: A Lightweight Framework to Characterize Domain-Specific Word Semantics Beyond Sentiment 2018
Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer 2018
Affordances in Grounded Language Learning 2018