A Comparison of Deep Learning Methods for Language Understanding

In this paper, we compare a suite of neural networks (recurrent, convolutional, and the recently proposed BERT model) to a CRF with hand-crafted features on three semantic tagging corpora: the Air Travel Information System (ATIS) benchmark, restaurant queries, and written and spoken meal descriptions. Our motivation is to investigate pre-trained BERT’s transferability to the domains we are interested in. We demonstrate that neural networks without feature engineering outperform state-of-the-art statistical and deep learning approaches on all three tasks (except written meal descriptions, where the CRF is slightly better) and that deep, attention-based BERT, in particular, surpasses state-of-the-art results on these tasks. Error analysis shows the models are less confident when making errors, enabling the system to follow up with the user when uncertain.