Design and Evaluation of Metaphor Processing Systems

Abstract

System evaluation methodologies receive significant attention in natural language processing (NLP), with the systems typically being evaluated on a common task and against shared datasets. This enables direct system comparison and facilitates progress in the field. However, computational work on metaphor is considerably more fragmented than similar research efforts in other areas of NLP and semantics. Despite the growing interest in the phenomenon, metaphor processing community still lacks a common task definition, dataset and evaluation strategy. The goal of this paper is to bring together the insights from existing work, theoretical considerations and real-world requirements for system applicability, in order to devise the desired properties of metaphor processing systems and their evaluation.