Kernekoncepter
The authors explore how language-informed program sampling can generate informative questions efficiently, highlighting the importance of grounding and cognitive resource constraints in question-asking tasks.
Resumé
The study investigates human question generation using a grounded task based on Battleship. It compares models' ability to generate informative questions by leveraging large language models (LLMs) and probabilistic programs. The research emphasizes the significance of integrating linguistic competence with reasoning about possible worlds for effective question-asking.
The authors introduce a new approach, LIPS, that translates natural language questions into symbolic programs to evaluate their information gain. They find that LLMs play a crucial role in generating questions but struggle with grounding them effectively in the board state. The study demonstrates how Bayesian models can capture human-like priors while revealing limitations of pure LLMs as grounded reasoners.
By analyzing data from human participants and various models, the research shows that Monte Carlo sampling can approximate human performance in generating informative questions. However, challenges remain in translating natural language questions into meaningful programs and ensuring effective grounding in the task environment.
Overall, the study sheds light on the complex interplay between language use, cognitive resources, and information-seeking behavior in question-asking tasks like Battleship.
Statistik
EIG = 4.67 (Human)
EIG = 1.36 (CodeLlama)
EIG = 1.36 (GPT-4)
EIG = 4.67 (Grammar)
Citater
"Asking informative questions requires integrating linguistic competence with representing and reasoning about possible worlds."
"Our model leverages large language models to pose questions in everyday language and translate them into symbolic representation."
"Our results illustrate how cognitive models of informative question-asking can leverage LLMs to capture human-like priors."