Summarizing Information Graphics Textually

Abstract

Information graphics (such as bar charts and line graphs) play a vital role in many multimodal documents. The majority of information graphics that appear in popular media are intended to convey a message and the graphic designer uses deliberate communicative signals in order to bring that message out such as highlighting certain aspects of the graphic. The graphic, whose communicative goal (intended message) is often not captured by the document's accompanying text, contributes to the overall purpose of the document and cannot be ignored. This article presents our approach to summarizing the high-level content of a non-scientific information graphic, which provides the intended message and the salient features of the graphic via a brief textual summary. This approach brings together insights obtained from empirical studies and applies methodologies typically used in processing text in order to extract information (such as generating referring expressions for graphical elements) required for summarizing this form of non-linguistic input data. This work also presents a novel bottom-up generation approach to simultaneously construct the discourse and sentence structures of textual summaries by leveraging different discourse related considerations such as the syntactic complexity of realized sentences and clause embeddings. Evaluation studies validate our bottom-up generation approach and the effectiveness of the overall methodology of summarizing information graphics.