The Unified and Holist Method Gamma for Inter-annotator Agreement Measure and Alignment

Abstract

Agreement measures have been widely used in Computational Linguistics for more than 10 years, in order to check the reliability of annotation processes. While considerable effort has been made concerning categorization, fewer studies address unitizing (or more specifically segmentation) tasks and when both paradigms are combined even fewer methods are available and discussed. The aim of this paper is threefold. First, we advocate that to deal with unitizing, alignment and agreement measure must be considered as a unified process, since a fair measure should rely on an alignment of the units from different annotators, while this alignment should be computed according to the principles of the measure. Second, we propose the new versatile measure Gamma which fulfills this requirement and copes with both paradigms, and we provide an algorithm and an implementation which closely approximates it. Third, we show that this new method performs as well as, or even better than, other more specialized methods, devoted to specialized paradigms (categorization and segmentation), while combining the two paradigms at the same time. Moreover, we demonstrate that Gamma is a large generalization of two current measures.