S3Prompt: Instructing the Model with Self-calibration, Self-recall and Self-aggregation to Improve In-context Learning

AbstractLarge language models achieve impressive results by inferring conditional probability distributions in the context of user input to generate responses. However, they still have the following limitations in practical applications: 1) User queries are often colloquial and do not conform to the conditional probability distribution of LLM. 2) Unsupervised generation and recall of in-context examples(compared to random sampling) remains an open problem. To alleviate the above problems, we propose a novel Self-calibration, Self-recall and Self-aggregation prompt pipeline (S 3Prompt). Specifically, we first design a question calibration prompt to align colloquial queries with LLM context. Secondly, we construct a candidate recall prompt that allows LLM to generate potential background information, which is different from traditional retrieval-based QA. Finally, we design an information aggregation prompt to generate the final answer by aggregating the recalled information. Notably, we find that the self-generated information by LLM has a smaller gap when fused with LLM. We conducted comprehensive experiments on various datasets, including numerical reasoning, common sense reasoning, logical reasoning, and reading comprehension. The results showed that the performance of LLM can be significantly improved by using question calibration, candidate recall, and information aggregation, without requiring annotated datasets and model parameter updates.