BeNeCLA Blog

Discourse op Dinsdag

Date & time: 23 November, 13:00-14:30

Location: Utrecht University, Kromme Nieuwegracht 80, Stijlkamer van Ravesteyn 1.06

Maite Taboada

Simon Fraser University (Vancouver, Canada)

Coherence and cohesion in multimodal documents

In this talk I present preliminary results of an ongoing project on the discourse characteristics of multimodal documents. This is joint work with Christopher Habel, carried out under an Alexander von Humboldt Foundation Fellowship at the University of Hamburg.

A great deal of work in the last few years has focused on the relationships between text and material presented through other modalities, be it visual, audio, or a combination of the two. Research on document design and learning has been trying to elucidate what kind of impact multimodal material has on the reader. Much research has studied whether to use multimodal material or not, where to place it, and what effect captions or other verbal information surrounding such material have on the reader (e.g., Acartürk et al., 2008; Mayer, 2009).

Less frequently discussed is the nature of the relationship between graphical material and the text itself. The point of departure for this work is that multimodal documents, just like any other form of discourse, exhibit coherence and cohesion relations. In particular, we are examining coherence relations between text and graphical material (pictures, diagrams, figures and tables), and cohesive ties that establish cross‐reference between the two modes. In order to understand and categorize the types of relations between figures and text, we are making use of Rhetorical Structure Theory (Mann and Thompson, 1988), and discuss whether RST relations are sufficient for describing these types of relations.

Ours is a corpus study. Because we believe genre constraints the types of figures present and the way they are introduced, we study three different genres: newspaper articles (New York Times), magazine articles in a scientific magazine (Communications of the Association for Computing Machinery), and scientific articles (Journal of Computational Linguistics). The corpus consists of about 1,500 pages of material, containing over 700 figures, tables and graphs. We show that figures stand in both presentational and subject matter relations to the text they accompany, and that cross‐referencing varies widely across genres, with newspaper articles showing little or no reference to the graphical material, and scientific articles marking the reference to the figure explicitly in the text.

The Discourse op Dinsdag discussion group is intended for researchers working on discourse from a language use perspective, and offers a platform to discuss their work (in progress). For more information check our website http://www.let.uu.nl/vici.