Presented by Tiago Torrent, Federal University of Juiz de Fora – FrameNet Brasil
Abstract: Multimodal data analysis is at the core of several research initiatives in (Computational) Linguistics and related fields. Since the second half of the 20th century, scholars have been proposing theoretical and methodological approaches to tackle the problem of meaning construction in a multimodal setting. Such approaches have contributed to the understanding how modalities interact, how comprehenders experience them and how their interaction is organized in terms of grammar. In this talk, I present FrameNet as a fine-grained semantic representation that can be used for annotating multimodal corpora. I start by showing the changes in database structure implemented in FrameNet Brasil for enhancing its representational capacity. Then, I demonstrate how multimodal data can be annotated within this framework using de FN-Br WebTool. Finally, I present two ongoing applications of the framework to the analysis of video comprehension and to multimodal machine translation.