From Words to Instances and Back: A Survey of Approaches to Meaning Representation and Interpretation

Abstract

The knowledge that is encoded in the representations generated by pre-trained language models is currently under intense exploration, since it can provide insights into the models’ impressive performance. Information about linguistic structure is directly accessible in these token-level representations, but knowledge about lexical meaning is situated at a higher level of abstraction, that of concepts or word types. We thus currently witness a resurgence of interest towards word-level representations in the lexical semantics field, with works that seek to derive higher-level embeddings from token level ones, or to infuse into them information from static word vectors and other knowledge sources. This survey will investigate current developments, putting them into perspective in the light of distributional methods for meaning representation. We will analyse the motivation behind this come-back of classical knowledge integration and transformation approaches, tracing the challenges posed by the contextualised approach to meaning representation, and the interest of the two paradigms for interpretation studies and downstream tasks.