A new study, TextQuests, has emerged as a benchmark to evaluate how well large language models (LLMs) can perform in text-based video games. These games, often rich in narrative and puzzles, require players to navigate through storylines, interact with characters, and solve problems using only text commands. The research tests various LLMs, including GPT-4 and Claude, on a curated set of classic and modern text adventures.
Preliminary results show that while LLMs excel at understanding language and generating coherent responses, they struggle with long-term planning and maintaining game state. For instance, models often forget inventory items or fail to follow complex branching narratives. However, some models demonstrated impressive creativity in dialogue and puzzle-solving, occasionally exploiting flaws in game logic.
The TextQuests benchmark aims to push AI beyond simple question-answering into interactive, goal-oriented tasks. As LLMs improve, such challenges could inform the development of more capable AI for gaming, virtual assistants, and interactive storytelling.