SBU-NLP at SemEval-2025 Task 8: Self-Correction and Collaboration in LLMs for Tabular Question Answering

AbstractThis paper explains the submission of the SBU-NLP team at SemEval-2025 Task 8: question-answering over tabular data. We present a novel algorithm for this task, aimed at systems capable of interpreting large tables and providing accurate answers to natural language queries. The evaluation uses the DataBench dataset, which covers a wide range of topics and reflects the complexity of real-world tabular data. Our approach incorporates a self-correction mechanism that iteratively refines LLM-generated code to address errors and prevent common mistakes. Additionally, a multi-LLM collaborative strategy is employed to generate answers, where responses from multiple LLMs are compared, and the majority consensus or a valid alternative is selected. The method relies exclusively on open-source models, avoiding costly processes like training or fine-tuning. Experimental results demonstrate that combining multiple LLMs with self-correction leads to significant performance improvements. However, challenges arise with list-based answers and responses involving multiple numerical, string, or boolean values, where further refinement is needed. The proposed simple system was among the top performers in both Subtask A and Subtask B among open-source models in the competition.