Scaling Intent Understanding: A Framework for Classification with Clarification using Lightweight LLMs

AbstractDespite extensive research in intent classification, most task-oriented dialogue systems still rigidly assign intents to user utterances without addressing ambiguity, often leading to misrouted requests, irrelevant responses, and user frustration. Proprietary large-language models (LLMs) can generate effective clarifying questions but are too costly for large-scale deployment. Smaller open-source LLMs are more economical, but struggle to ask appropriate clarifying questions. This paper introduces a domain-agnostic framework that equips lightweight, production-ready open-source LLMs with the ability to perform intent classification alongside precise ambiguity resolution via clarifying questions. We validate our framework on both proprietary and public intent classification datasets, demonstrating its ability to perform intent classification as well as generate clarification questions in case of ambiguity. To compare models, those trained with our framework and external baselines, we also propose an evaluation methodology that jointly assesses the accuracy of intent classification and the timing and quality of clarifying questions. Our instruction-tuned models achieve performance comparable to leading proprietary LLMs while offering an 8X reduction in inference cost, enabling broader, cost-efficient deployment. When deployed in the customer-care system of an e-commerce enterprise, our model reduced the misrouting rate by 8%, resulting in a significant improvement in automation rates, which potentially translates in dollar savings by reducing escalations to human agents.