Evaluating Automatic Topic Segmentation as a Segment Retrieval Task

Several evaluation metrics have been proposed for topic segmentation. Most of them rely on the paradigm that segmentation is mainly a task that detects boundaries, and thus are oriented on boundary detection evaluation. Nevertheless, this paradigm is not appropriate to get homogeneous chapters, which is one of the major applications of topic segmentation. For instance on Broadcast News, topic segmentation enables users to watch a chapter independently of the others. We propose to consider segmentation as a task that detects homogeneous segments, and we propose evaluation metrics oriented on segment retrieval. The proposed metrics are experimented on various TV shows from different channels. Results are analysed and discussed, highlighting their relevance.