Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based Question Answering

AbstractOpen-ended question answering requires mod- els to find appropriate evidence to form well-reasoned, comprehensive and helpful answers. In practical applications, models also need to engage in extended discussions on potential scenarios closely relevant to the question. With augmentation of retrieval module, open-source Large Language Models (LLMs) can produce coherent answers often with different focuses, but are still sub-optimal in terms of reliable ev- idence selection and in-depth question analysis. In this paper, we propose a novel Chain-of- Discussion framework to leverage the synergy among multiple open-source LLMs aiming to provide more correct and more comprehensive answers for open-ended QA, although they are not strong enough individually. Our exper- iments show that discussions among multiple LLMs play a vital role in enhancing the quality of answers.