2018
EMNLP
EMNLP 2018
Automatic Pyramid Evaluation Exploiting EDU-based Extractive Reference Summaries
Abstract
AbstractThis paper tackles automation of the pyramid method, a reliable manual evaluation framework. To construct a pyramid, we transform human-made reference summaries into extractive reference summaries that consist of Elementary Discourse Units (EDUs) obtained from source documents and then weight every EDU by counting the number of extractive reference summaries that contain the EDU. A summary is scored by the correspondences between EDUs in the summary and those in the pyramid. Experiments on DUC and TAC data sets show that our methods strongly correlate with various manual evaluations.
🌉
Interdisciplinary Bridge
— Interdisciplinary and Natural Language Processing
🧭
Keyword Pioneer
— reference summarization
🐣
Hot Topic Early Bird
— evaluation metrics
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Security & Privacy, Speech & Audio