Error Analysis of ToTTo Table-to-Text Neural NLG Models

AbstractWe report error analysis of outputs from four Table-to-Text generation models fine-tuned on ToTTo, an open-domain English language dataset. We carried out a manual error annotation of a subset of outputs (a total of 3,016 sentences) belonging to the topic of Politics generated by these four models. Our error annotation focused on eight categories of errors. The error analysis shows that more than 46% of sentences from each of the four models have been error-free. It uncovered some of the specific classes of errors; for example, WORD errors (mostly verbs and prepositions) are the dominant errors in all four models and are the most complex ones among other errors. NAME (mostly nouns) and NUMBER errors are slightly higher in two of the GeM benchmark models, whereas DATE-DIMENSION and OTHER categories of errors are more common in our Table-to-Text model. This in-depth error analysis is currently guiding us in improving our Table-to-Text model.