Core Concepts
Concise Chain-of-Thought (CCoT) prompting can reduce response length by 48.70% for GPT-3.5 and GPT-4 without significantly impacting problem-solving performance.
Abstract
The researchers introduced Concise Chain-of-Thought (CCoT) prompting, which combines the effectiveness of Chain-of-Thought (CoT) prompting with the efficiency of concise prompting. They compared the response length and correct-answer accuracy of standard CoT and CCoT prompts using GPT-3.5 and GPT-4 on a multiple-choice question-and-answer (MCQA) benchmark.
The key findings are:
CCoT reduced average response length by 48.70% for both GPT-3.5 and GPT-4 compared to standard CoT, with a negligible impact on problem-solving performance.
For GPT-4, CCoT did not decrease performance in any problem domain compared to standard CoT.
For GPT-3.5, CCoT incurred a 27.69% reduction in accuracy on math problems (AQUA-RAT and SAT Math) compared to standard CoT, but had minimal impact on other problem domains.
The cost savings of using CCoT over standard CoT were 21.85% for GPT-3.5 and 23.49% for GPT-4, due to the reduced response length.
These results have practical implications for AI engineers building LLM-based solutions, as CCoT can reduce costs, energy consumption, and response times without sacrificing performance. Theoretically, the findings raise new questions about which specific aspects of a CoT are necessary for an LLM's problem-solving capabilities.
Stats
The rectangular solid is 3 x 4 x 15.
The diameter of the inscribed sphere is 15.8113.
Quotes
"CCoT reduced average response length by 48.70% for both GPT-3.5 and GPT-4 while having a negligible impact on problem-solving performance."
"For GPT-3.5, CCoT incurred a 27.69% reduction in accuracy on math problems (AQUA-RAT and SAT Math) compared to standard CoT, but had minimal impact on other problem domains."
"The cost savings of using CCoT over standard CoT were 21.85% for GPT-3.5 and 23.49% for GPT-4, due to the reduced response length."