Anthropocentric bias in language model evaluation

Abstract

Evaluating the cognitive capacities of large language models (LLMs) requires overcoming
not only anthropomorphic but also anthropocentric biases. This article identifies two types of
anthropocentric bias that have been neglected: overlooking how auxiliary factors can impede
LLM performance despite competence (auxiliary oversight), and dismissing LLM mechanistic
strategies that differ from those of humans as not genuinely competent (mechanistic chauvinism).
Mitigating these biases requires an empirical, iterative approach to mapping cognitive tasks to
LLM-specific capacities and mechanisms, achieved by supplementing behavioral experiments
with mechanistic studies.