Asking goal-oriented questions and learning from answers (@ CogSci 2019)

Anselm Rothe, Brenden M. Lake, and Todd M. Gureckis

This paper presents a model for how people ask goal-oriented questions and update their beliefs from answers. They work in the context of a battleship game, where the set of questions that a person can ask are pre-defined, as are the answers (e.g. the color on a given board cell, or whether the ship of a given color is horizontal or vertical). The hypothesis space $\mathcal{H}$ of boards is bounded ($\approx 1.6 million$), so it's feasible to have an oracle that answers questions deterministically.
Under this modelling, you can be Bayesian and maximize expected information gain to reduce uncertainty for the particular goal in mind. The user computes the probability of a certain answer by using a uniform prior over all boards for which the answer would be consistent. This model seems to explain people's behavior quite well. The model's scores correlates strongly ($r = 0.84$) with people's performance. If the goal is inverted for the model, its performance correlates negatively ($r = -0.14$) with the person's performance (with flipped goals), showing that the goal is in fact a very significant component of this question-asking procedure. It also predicts well the follow-up questions a person asks.