The Rational Agent Benchmark for Data Visualization
IEEE Trans. Vis. Compt. Graph. (Proc. VIS) 2023
The estimated payoffs by rational agent framework in Kale et al. for 100 simulated experiments in which behavioral agents make decisions (behavioral decision score, green) and report PoS judgments (PoS raw score, purple, and adjusted calibrated PoS score,orange) by visualization condition with means added and without. The rational agent benchmark and the rational agent baseline are shown as dotted lines.
Abstract
Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.
Citation
BibTeX
@article{wu2023rational,
title={The rational agent benchmark for data visualization},
author={Wu, Yifan and Guo, Ziyang and Mamakos, Michalis and Hartline, Jason and Hullman, Jessica},
journal={IEEE transactions on visualization and computer graphics},
volume={30},
number={1},
pages={338--347},
year={2023},
publisher={IEEE}
}