Travel or Trouble

2023

1. Human Choice Prediction in Language-based Persuasion Games: Simulation-based Off-Policy Evaluation

   Eilam Shapira, Reut Apel, Moshe Tennenholtz, and Roi Reichart

   arXiv preprint arXiv:2305.10361, 2023

   Abs Bib PDF Code

   Recent advances in Large Language Models (LLMs) have spurred interest in designing LLM-based agents for tasks that involve interaction with human and artificial agents. This paper addresses a key aspect in the design of such agents: Predicting human decision in off-policy evaluation (OPE), focusing on language-based persuasion games, where the agent’s goal is to influence its partner’s decisions through verbal messages. Using a dedicated application, we collected a dataset of 87K decisions from humans playing a repeated decision-making game with artificial agents. Our approach involves training a model on human interactions with one agents subset to predict decisions when interacting with another. To enhance off-policy performance, we propose a simulation technique involving interactions across the entire agent space and simulated decision makers. Our learning strategy yields significant OPE gains, e.g., improving prediction accuracy in the top 15% challenging cases by 7.1%. Our code and the large dataset we collected and generated are submitted as supplementary material and publicly available in our GitHub repository

   @article{shapira2023human,
     title = {Human Choice Prediction in Language-based Persuasion Games: Simulation-based Off-Policy Evaluation},
     author = {Shapira, Eilam and Apel, Reut and Tennenholtz, Moshe and Reichart, Roi},
     journal = {arXiv preprint arXiv:2305.10361},
     year = {2023},
   }