Dual Attention Model for Citation Recommendation with Analyses on Explainability of Attention Mechanisms and Qualitative Experiments

Abstract

Based on an exponentially increasing number of academic articles, discovering and citing comprehensive and appropriate resources have become non-trivial tasks. Conventional citation recommendation methods suffer from severe information losses. For example, they do not consider the section of the paper that the author is writing and for which they need to find a citation, the relatedness between the words in the local context (the text span that describes a citation), or the importance of each word from the local context. These shortcomings make such methods insufficient for recommending adequate citations to academic manuscripts. In this study, we propose a novel embedding-based neural network called ``dual attention model for citation recommendation (DACR)'' to recommend citations during manuscript preparation. Our method adapts the embedding of three semantic pieces of information: words in the local context, structural contexts, and the section on which the author is working. A neural network model is designed to maximize the similarity between the embedding of the three inputs (local context words, section, and structural contexts) and the target citation appearing in the context. The core of the neural network model comprises self-attention and additive attention; the former aims to capture the relatedness between the contextual words and structural context, and the latter aims to learn their importance. Recommendation experiments on real-world datasets demonstrate the effectiveness of the proposed approach. To seek explainability on DACR, particularly the two attention mechanisms, the learned parameters from them are investigated to determine whether the information on relatedness and importance are appropriately extracted and represented. In addition, considering that appropriate citations potentially exist and are not found by the authors, we conduct a qualitative analysis with expert annotators on selected citing contexts and the associating recommendations from DACR, thereby validating the associating ability of our approach.