Information-Theoretic Compositional Distributional Semantics

Abstract

In the context of text representation, compositional distributional semantics models aim to fuse the distributional hypothesis and the principle of compositionality: Text embedding is based on the contexts in which linguistic items appear, in such a way that items with a similar distribution have similar representations. Furthermore, these representations are in turn combined by compositional functions taking into account the text structure. However, the theoretical basis of compositional functions is still an open issue. In this paper we define and study the notion of Information-Theoretic Compositional Distributional Semantics (ICDS): i) We first establish formal properties for embedding, composition and similarity functions based on Shannon’s Information Theory; ii) we analyze the existing approaches under this prism, checking whether or not they comply with the established desirable properties; iii) we propose two parameterizable composition and similarity functions that generalize traditional approaches while fulfilling the formal properties; and finally iv) we perform an empirical study on several textual similarity data sets that include sentences with a high and low lexical overlap, and on the similarity between words and their description. Our theoretical analysis and empirical results show that fulfilling certain formal properties affects the effectiveness of text representation models.