GLEE

A unified framework and benchmark for language-based economic environments, with large-scale datasets of LLM and human interactions.

GLEE (Games in Language-based Economic Environments) is a unified framework and benchmark for studying how agents behave in two-player, sequential, language-based economic games (Shapira et al., 2024). It was developed with my co-authors Omer Madmon, Itamar Reinman, Samuel Joseph Amouyal, Roi Reichart, and Moshe Tennenholtz.

Why We Built GLEE

My earlier work centered on a single game: the language-based persuasion game behind the Travel or Trouble dataset, where a human traveler decides whether to trust a bot’s hotel reviews. That dataset is rich, but a pattern that holds in persuasion need not hold in bargaining, in negotiation, or under different information and incentive structures. Rigorous, general claims about how LLMs reason and act in economic interactions call for evidence from more than one game.

To compare strategic behavior across games, incentives, and information structures, we needed a framework that spans many settings at once, with consistent parameterization and economic measures that make results comparable between them. That is exactly what GLEE provides.

What GLEE Includes

GLEE standardizes research on language-based economic games around three core contributions:

A framework. An open-source engine for simulating and analyzing two-player, sequential, language-based games. Inspired by the economic literature, GLEE defines three base families of games, bargaining, negotiation, and persuasion, each with consistent parameters, degrees of freedom, and economic measures. Self-gain scores an agent’s own performance, while efficiency and fairness score the outcome of the game. Because each family is defined along six to seven axes of variation, the framework spans a large combinatorial space of game configurations, and researchers can plug in new agents and evaluate them across any of these standardized settings.
An LLM-vs-LLM dataset. A large collection of roughly 80K LLM-vs-LLM games, comprising 587K individual agent decisions, spanning 1,320 game configurations (384 bargaining, 576 negotiation, and 360 persuasion).
A human-vs-LLM dataset. A complementary dataset gathered from 3,405 human participants who played against LLM agents, enabling direct comparison between human and machine behavior in the same economic environments.

An example negotiation interaction in GLEE.

An example negotiation game in GLEE: a buyer and a seller exchange messages and price offers until they reach an agreement or the game ends.

Because every game shares a common interface and a common set of measures, the same configuration grid can be used to benchmark any new agent. The experimentation platform lets you sweep over game parameters, prices, discount factors, information conditions, and message settings, and launch thousands of standardized matchups.

Research Built with GLEE

GLEE’s environments, datasets, and evaluation infrastructure carry a research program of our own. Within GLEE itself, we used them to ask whether LLM agents behave rationally, whether they mimic human behavior, whether they reach efficient and fair outcomes, what role natural language plays in the interaction, and how the economic characteristics of the environment shape these dynamics.

Beyond GLEE, each of our follow-up papers builds on a different part of the project:

Predicting Decisions of AI Agents from Limited Interaction (Shapira et al., 2026a). Can an agent predict an unfamiliar counterpart’s next decision after seeing only a few of its games? Using GLEE’s framework to run new agents and its LLM-vs-LLM dataset for generalization, we formulate this as a target-adaptive text-tabular task with an “LLM-as-Observer” representation.
Alignment Makes Language Models Normative, Not Descriptive (Shapira et al., 2026b). Does alignment make a language model a better model of human behavior? Using GLEE’s human-vs-LLM dataset, we show that aligned models predict human behavior far worse than their base counterparts in multi-round strategic settings, while doing better in one-shot, normative ones.
The Poisoned Apple Effect (Shapira et al., 2026c). Can releasing a new AI technology reshape a market even when nobody ends up using it? Using GLEE’s LLM-vs-LLM dataset, we study strategic manipulation of mediated markets through technology expansion of AI agents, and show that simply releasing a technology can shift equilibrium payoffs.

Other groups have begun to build on GLEE as well. ARIA (Yang et al.; NeurIPS 2025 Spotlight), a reinforcement-learning method that trains language agents by aggregating rewards in an intention space, evaluates its agents across 48 GLEE game configurations, playing both sides of the bargaining and negotiation games against fixed opponents.

The GLEE Competition

To put the framework to work, we are organizing the GLEE Competition, the official competition of the First Workshop on Interpreting Agent Behavior (IAB) at NeurIPS 2026. It runs online from August 1 to 29, 2026, with players competing across all three GLEE families over many different configurations.

The GLEE Competition: build an agent, or step into the games yourself.

There are two tracks, with $6,000 in prizes between them:

Agent track: Participants build an agent that plays through the competition’s REST API. Nothing constrains how it decides: a prompted LLM, a fine-tuned one, a classical game-theoretic solver, or any mix of them. What the leaderboard ends up comparing is therefore strategies, not just the model underneath. The winning agent takes $2,000, with prizes down to fifth place.
Human track: Participants play the games themselves, in the browser. Human players are scored on the same scale as the agents, which makes the leaderboard a running comparison between people and models on identical games. The top-ranked human takes $500, with prizes down to third place.

Both tracks draw on the same matchmaking pool, so an opponent may be another agent, a human, or one of our baseline bots, and you only learn which after the game ends. Ratings update after every game, comparing your payoff against every payoff earned in the same role on the same configuration, and the overall ranking averages a player’s three family ratings, so it rewards agents that handle all of GLEE rather than specializing in one game.

Everything happens online, and prizes and award certificates go to the winners whether or not they make it to NeurIPS in person. The IAB workshop also invites short competition papers describing how an agent was built; the most interesting methods, together with the top-ranked agents, will be presented at the workshop.

We have already run a successful pilot of the competition, which produced the data behind our Predicting Decisions of AI Agents (Shapira et al., 2026a).

References

2026

Predicting a target agent’s next decision from K prior games. (A) LLM-as-Predictor directly prompts a large LLM for the decision. (B) A text-tabular formulation feeds game features and text representations to a tabular foundation model (TabPFN). (C) Our model adds LLM-as-Observer: the hidden state of a small frozen LLM that reads the game state and dialogue becomes an additional decision-oriented representation.
Predicting Decisions of AI Agents from Limited Interaction through Text-Tabular Modeling

Eilam Shapira, Moshe Tennenholtz, and Roi Reichart

arXiv preprint arXiv:2605.12411, 2026a

S T A Abs Bib PDF

AI agents negotiate and transact in natural language with unfamiliar counterparts: a buyer bot facing an unknown seller, or a procurement assistant negotiating with a supplier. In such interactions, the counterpart’s LLM, prompts, control logic, and rule-based fallbacks are hidden, while each decision can have monetary consequences. We ask whether an agent can predict an unfamiliar counterpart’s next decision from a few interactions. To avoid real-world logging confounds, we study this problem in controlled bargaining and negotiation games, formulating it as target-adaptive text-tabular prediction: each decision point is a table row combining structured game state, offer history, and dialogue, while K previous games of the same target agent, i.e., the counterpart being modeled, are provided in the prompt as labeled adaptation examples. Our model is built on a tabular foundation model that represents rows using game-state features and LLM-based text representations, and adds LLM-as-Observer as an additional representation: a small frozen LLM reads the decision-time state and dialogue; its answer is discarded, and its hidden state becomes a decision-oriented feature, making the LLM an encoder rather than a direct few-shot predictor. Training on 13 frontier-LLM agents and testing on 91 held-out scaffolded agents, the full model outperforms direct LLM-as-Predictor prompting and game+text features baselines. Within this tabular model, Observer features contribute beyond the other feature schemes: at K=16, they improve response-prediction AUC by about 4 points across both tasks and reduce bargaining offer-prediction error by 14%. These results show that formulating counterpart prediction as a target-adaptive text-tabular task enables effective adaptation, and that hidden LLM representations expose decision-relevant signals that direct prompting does not surface.
@article{shapira2026predictingdecisionsaiagents, title = {Predicting Decisions of AI Agents from Limited Interaction through Text-Tabular Modeling}, author = {Shapira, Eilam and Tennenholtz, Moshe and Reichart, Roi}, year = {2026}, eprint = {2605.12411}, archiveprefix = {arXiv}, primaryclass = {cs.LG}, url = {https://arxiv.org/abs/2605.12411}, journal = {arXiv preprint arXiv:2605.12411}, }
Pearson correlations of base models and human decisions (x-axis) vs. aligned models and human decisions (y-axis) across four game families. Points below the diagonal indicate base advantage.
Alignment Makes Language Models Normative, Not Descriptive

Eilam Shapira, Moshe Tennenholtz, and Roi Reichart

arXiv preprint arXiv:2603.17218, 2026b

S Abs Bib PDF

Post-training alignment optimizes language models to match human preference signals, but this objective is not equivalent to modeling observed human behavior. We compare 120 base-aligned model pairs on more than 10,000 real human decisions in multi-round strategic games - bargaining, persuasion, negotiation, and repeated matrix games. In these settings, base models outperform their aligned counterparts in predicting human choices by nearly 10:1, robustly across model families, prompt formulations, and game configurations. This pattern reverses, however, in settings where human behavior is more likely to follow normative predictions: aligned models dominate on one-shot textbook games across all 12 types tested and on non-strategic lottery choices - and even within the multi-round games themselves, at round one, before interaction history develops. This boundary-condition pattern suggests that alignment induces a normative bias: it improves prediction when human behavior is relatively well captured by normative solutions, but hurts prediction in multi-round strategic settings, where behavior is shaped by descriptive dynamics such as reciprocity, retaliation, and history-dependent adaptation. These results reveal a fundamental trade-off between optimizing models for human use and using them as proxies for human behavior.
@article{shapira2026alignmentmakeslanguagemodels, title = {Alignment Makes Language Models Normative, Not Descriptive}, author = {Shapira, Eilam and Tennenholtz, Moshe and Reichart, Roi}, year = {2026}, eprint = {2603.17218}, archiveprefix = {arXiv}, primaryclass = {cs.CL}, url = {https://arxiv.org/abs/2603.17218}, journal = {arXiv preprint arXiv:2603.17218}, }
Illustration of the "poisoned apple" example, in which Alice increases her payoff at Bob’s expense by releasing a new technology—without the players actually using that technology in practice.
The Poisoned Apple Effect: Strategic Manipulation of Mediated Markets via Technology Expansion of AI Agents

Eilam Shapira, Roi Reichart, and Moshe Tennenholtz

arXiv preprint arXiv:2601.11496, 2026c

S A Abs Bib PDF

The integration of AI agents into economic markets fundamentally alters the landscape of strategic interaction. We investigate the economic implications of expanding the set of available technologies in three canonical game-theoretic settings: bargaining (resource division), negotiation (asymmetric information trade), and persuasion (strategic information transmission). We find that simply increasing the choice of AI delegates can drastically shift equilibrium payoffs and regulatory outcomes, often creating incentives for regulators to proactively develop and release technologies. Conversely, we identify a strategic phenomenon termed the ”Poisoned Apple” effect: an agent may release a new technology, which neither they nor their opponent ultimately uses, solely to manipulate the regulator’s choice of market design in their favor. This strategic release improves the releaser’s welfare at the expense of their opponent and the regulator’s fairness objectives. Our findings demonstrate that static regulatory frameworks are vulnerable to manipulation via technology expansion, necessitating dynamic market designs that adapt to the evolving landscape of AI capabilities.
@article{shapira2026poisonedappleeffectstrategic, title = {The Poisoned Apple Effect: Strategic Manipulation of Mediated Markets via Technology Expansion of AI Agents}, author = {Shapira, Eilam and Reichart, Roi and Tennenholtz, Moshe}, year = {2026}, eprint = {2601.11496}, archiveprefix = {arXiv}, primaryclass = {cs.GT}, journal = {arXiv preprint arXiv:2601.11496}, url = {https://arxiv.org/abs/2601.11496}, }

2024

GLEE: A Unified Framework and Benchmark for Language-based Economic Environments

Eilam Shapira, Omer Madmon, Itamar Reinman, Samuel Joseph Amouyal, Roi Reichart, +2 more authors, and Moshe Tennenholtz

arXiv preprint arXiv:2410.05254, 2024

S A Abs Bib PDF Code Poster

Large Language Models (LLMs) show significant potential in economic and strategic interactions, where communication via natural language is often prevalent. This raises key questions: Do LLMs behave rationally? Can they mimic human behavior? Do they tend to reach an efficient and fair outcome? What is the role of natural language in the strategic interaction? How do characteristics of the economic environment influence these dynamics? These questions become crucial concerning the economic and societal implications of integrating LLMbased agents into real-world data-driven systems, such as online retail platforms and recommender systems. While the ML community has been exploring the potential of LLMs in such multi-agent setups, varying assumptions, design choices and evaluation criteria across studies make it difficult to draw robust and meaningful conclusions. To address this, we introduce a benchmark for standardizing research on two-player, sequential, language-based games. Inspired by the economic literature, we define three base families of games with consistent parameterization, degrees of freedom and economic measures to evaluate agents’ performance (self-gain), as well as the game outcome (efficiency and fairness). We develop an open-source framework for interaction simulation and analysis, and utilize it to collect a dataset of LLM vs. LLM interactions across numerous game configurations and an additional dataset of human vs. LLM interactions. Through extensive experimentation, we demonstrate how our framework and dataset can be used to: (i) compare the behavior of LLM-based agents to human players in various economic contexts; (ii) evaluate agents in both individual and collective performance measures; and (iii) quantify the effect of the economic characteristics of the environments on the behavior of agents. We believe that our framework can contribute to the growing intersection of LLMs, ML, and economics, and we encourage researchers to explore it further and build on its foundation.
@article{shapira2024gleeunifiedframeworkbenchmark, journal = {arXiv preprint arXiv:2410.05254}, title = {GLEE: A Unified Framework and Benchmark for Language-based Economic Environments}, author = {Shapira, Eilam and Madmon, Omer and Reinman, Itamar and Amouyal, Samuel Joseph and Reichart, Roi and Tennenholtz, Moshe}, year = {2024}, eprint = {2410.05254}, archiveprefix = {arXiv}, primaryclass = {cs.CL}, url = {https://arxiv.org/abs/2410.05254}, }