ExpressivityArena: Can LLMs Express Information Implicitly?

Joshua Tint, Som Sagar, Aditya Taparia, Caleb Liu, Kelly Raines, Bimsara Pathiraja, Ransalu Senanayake
Findings of the Association for Computational Linguistics: EACL, 2026

In one sentence. ExpressivityArena is an information-theoretic framework for evaluating how well large language models communicate tone, emotion, identity, and intent implicitly, across nine tasks — revealing that models handle affective content well but lag behind human baselines on sociolinguistic signals.

picture_as_pdf PDF

Abstract

Human communication is often implicit, conveying tone, identity, and intent beyond literal meanings. While large language models have achieved strong performance on explicit tasks such as summarization and reasoning, their capacity for expressivity, or implicit communication, remains underexplored. We introduce a framework for evaluating the expressivity of LLMs using information-theoretic communication models. Our approach quantifies how well LLM-generated text communicates target properties without explicit mention, across nine tasks spanning emotion, identity, and tone. To enable scalable and reproducible evaluation, we employ LLM-based graders validated against human judgments. Our results reveal that while models are adept at expressing affective content, they struggle with sociolinguistic signals, lagging behind human baselines. This study provides a necessary step to evaluate human-like implicit communication, with implications for applications such as education, mental health support, and socially-aware dialogue systems.

Note. This work is also referred to as ExpressivityBench in the latest arXiv revision; the two names describe the same benchmark. An earlier version appeared at the NeurIPS 2024 Workshop on Behavioral Machine Learning.

BibTeX

@inproceedings{tint2026expressivityarena,
  title     = {ExpressivityArena: Can LLMs Express Information Implicitly?},
  author    = {Tint, Joshua and Sagar, Som and Taparia, Aditya and Liu, Caleb and Raines, Kelly and Pathiraja, Bimsara and Senanayake, Ransalu},
  booktitle = {Findings of the Association for Computational Linguistics: EACL},
  year      = {2026}
}

Topics

large language model evaluation, benchmark, implicit communication, expressivity, information theory, LLM-as-a-judge