Measuring General Intelligence with Generated Games
We present gg-bench, a collection of game environments designed to evaluate
general reasoning capabilities in language models. Unlike most static
benchmarks, gg-bench is a data generating process where new evaluation
instances can be generated at will. In particular, gg-bench is synthetically
generated by (1) using a large language model (LLM) to generate natural
language descriptions of novel games, (2) using the LLM to implement each game
in code as a Gym environment, and (3) training reinforcement learning (RL)
agents via self-play on the generated games. We evaluate language models by
their winrate against these RL agents by prompting models with the game
description, current board state, and a list of valid moves, after which models
output the moves they wish to take. gg-bench is challenging: state-of-the-art
LLMs such as GPT-4o and Claude 3.7 Sonnet achieve winrates of 7-9% on gg-bench
using in-context learning, while reasoning models such as o1, o3-mini and
DeepSeek-R1 achieve average winrates of 31-36%. We release the generated games,
data generation process, and evaluation code in order to support future
modeling work and expansion of our benchmark.
68236b86102b1d3069ebafab
Vivek Verma
extracted_pending
2025-05-13T15:55:50.678Z
64b88247e436bbca16603baf
/avatars/7bde6b0f75bccc3195fb72cbe5860a7e.svg
Vivek Verma
user
vivekverma
68236b86102b1d3069ebafac
David Huang
68236b86102b1d3069ebafad
William Chen
68236b86102b1d3069ebafae
Dan Klein
admin_assigned
2025-05-14T10:09:04.358Z
632be88b3690fb57e70e0bf1
/avatars/74ff8f30b3662db2602495bdf493d397.svg
Dan Klein
user
danjklein
68236b86102b1d3069ebafaf
Nicholas Tomlin
claimed_verified
2025-05-14T07:35:26.733Z
6269d074a6a7bba9e46d8d50
https://cdn-avatars.huggingface.co/v1/production/uploads/1651101782106-noauth.jpeg
Nicholas Tomlin
user
nickatomlin
Measuring General Intelligence with Generated Games
We present gg-bench, a collection of game environments designed to evaluate
general reasoning capabilities in language models. Unlike most static
benchmarks, gg-bench is a data generating process where new evaluation
instances can be generated at will. In particular, gg-bench is synthetically
generated by (1) using a large language model (LLM) to generate natural
language descriptions of novel games, (2) using the LLM to implement each game
in code as a Gym environment, and (3) training reinforcement learning (RL)
agents via self-play on the generated games. We evaluate language models by
their winrate against these RL agents by prompting models with the game
description, current board state, and a list of valid moves, after which models
output the moves they wish to take. gg-bench is challenging: state-of-the-art
LLMs such as GPT-4o and Claude 3.7 Sonnet achieve winrates of 7-9% on gg-bench
using in-context learning, while reasoning models such as o1, o3-mini and
DeepSeek-R1 achieve average winrates of 31-36%. We release the generated games,
data generation process, and evaluation code in order to support future
modeling work and expansion of our benchmark.
gg-bench is a dynamic benchmark for evaluating general reasoning in language models by generating new game environments and assessing winrates against reinforcement learning agents.
game environments general reasoning capabilities language models data generating process natural language descriptions Gym environment reinforcement learning RL agents self-play winrate in-context learning reasoning models GPT-4o Claude 3.7 Sonnet o1 o3-mini DeepSeek-R1