Twenty Questions is a game in which one player thinks of a person, place or thing, and the other tries to identify it using as few YES/NO questions as possible.
Deep20Bench lets LLMs play this game. Finding the answer requires world knowledge, but the model must also remember the previous questions and strategically choose the next one.
Claude Opus 5 currently leads with an average of 12.34 questions, followed by Kimi K3 with 12.74. I was surprised that the inexpensive GPT-5 Nano also performed well, ahead of several more expensive models.
This first version uses seven subjects with five trials each: 35 trials per model and 385 games across 11 models. I know that seven subjects is not much, but the tests already cost around $150 ;-).
The code and all results are public, and every game can be inspected question by question.
Twenty Questions is a game in which one player thinks of a person, place or thing, and the other tries to identify it using as few YES/NO questions as possible.
Deep20Bench lets LLMs play this game. Finding the answer requires world knowledge, but the model must also remember the previous questions and strategically choose the next one.
Claude Opus 5 currently leads with an average of 12.34 questions, followed by Kimi K3 with 12.74. I was surprised that the inexpensive GPT-5 Nano also performed well, ahead of several more expensive models.
This first version uses seven subjects with five trials each: 35 trials per model and 385 games across 11 models. I know that seven subjects is not much, but the tests already cost around $150 ;-).
The code and all results are public, and every game can be inspected question by question.
Feedback is very welcome.