Les gestes d’attention conjointe dans les interactions multimodales / Gestures of joint attention in multimodal interactions

Discourse is inherently interactive since it is necessarily addressed to another person and the representation of others influences the nature of discourse itself and thus the structure of the linguistic message. This is directly observable in face-to-face interactions in which the state of shared knowledge between interlocutors is constantly changing and influences the linguistic content of the message. But it is also the case in monologues in which speakers rely on the construction and evolution of the information delivered to shape their future discourse. In the linguistic framework of multimodality, however, the term interaction takes on different meanings and if it may offer a cover term for an ‘exchange between different individuals’, it may also mean the way in which the modalities (verbal, oral and visual) do not only participate in the elaboration of the linguistic message, but also the way in which these different modalities interact with each other. According to Adami (2017), a multimodal analysis does not a priori privilege one modality over another, even though one of the modalities may be emphasized in a social practice. Each semiotic system has its own systemic constraints and affordances. What can be communicated with language or voice will not necessarily be communicable visually and vice versa. The talk describe in this abstract will present gestures which play a role in establishing joint attention, especially beats and points in diverse types of interactions, and the manner in which they are inserted in multimodal ensembles (Enfield, 2009).

For a number of years now, researchers in Gesture Studies have shown that coverbal gestures play a cognitive role in both the production of the linguistic message and its perception. For instance, Morsella & Krauss (2004) have shown that hand gestures have an impact on the production of speech insofar as the rate of speech is slowed down when gestures are restricted and they observed that gestures help retrieve the phonological form of words. Krauss et al (2000) had also previously mentioned that iconic gestures facilitate lexical retrieval. But more recent studies have shown that there are also links between lexical recall and other types of gestures which are therefore also likely to improve lexical production. For example, So and colleagues (2012) have shown that there is a link between the production of hand beats and the recall of lexical items. This is very much related to the rhythmic role that can facilitate training as well as the emphasis role of beats that highlight discourse items.

Morsella & Krauss (op. cit.) also associate gestures with the memorization of spatial information. This confirms Kita's work on the links between verbal production, associated gestures and the information encoded in the linguistic message. For this scholar (2000:180), “gesturing helps the speaker organize information in a way suitable for linguistic expression. When a person speaks and gestures, analytic thinking and spatio-motoric thinking are collaboratively organizing information.” Later, Kita et al. (2017:1) add that representational gestures allow “activating, manipulating, packaging, and exploring spatio-motoric information for speaking and thinking.” More generally, gestures make it possible to highlight certain information present in the uninterrupted flow of the linguistic message (Alibali & Kita, 2010), but also to organise discourse units (Kendon, 2004, McNeill, 2005) and interactions (Bavelas & Gerwing, 2007).

In terms of perception, gestures also play an important role. Facial gestures not only improve the perception of segmental speech units (lip-reading is known to be important for people with a hearing impairment), but also facilitate the perception of prosodic units - whether lexical or contrastive accents (Dohen & Lœvenbruck, 2004a and b, 2005, 2009, Swerts & Krahmer, 2008). But this is also the case for head movements (Swerts & Krahmer, 2007, Al Moubayed et al., 2010a and b) and hand gestures, which Ferré (2018) has even shown can induce the perception of a contrastive accent where an utterance does not contain one acoustically. Biau & Soto-Faraco (2015) also show that the production of beats “may facilitate the anticipation of associated acoustic cues relevant for prosodic/syllabic-based segmentation in speech perception.

References

Adami, E., 2017. Multimodality, in: García, O., Flores, N., Spotti, M. (Eds.), The Oxford Handbook of Language and Society. Oxford University Press, Oxford, pp. 451-472.

Al Moubayed, S., Beskow, J., Granström, B., 2010a. Auditory visual prominence. From intelligibility to behavior. Journal of Multimodal User Interfaces 3, 299-309.

Al Moubayed, S., Beskow, J., Granström, B., House, D., 2010b. Audio-Visual Prosody: Perception, Detection, and Synthesis of Prominence, in: Esposito, A.M. (Ed.), Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces: Theoretical and Practical Issues. Springer Verlag, Heidelberg, pp. 55-71.

Alibali, M.W., Kita, S., 2010. Gesture highlights perceptually present information for speakers. Gesture 10(1), 3-28.

Bavelas, J., Gerwing, J., 2007. Conversational hand gestures and facial displays in face-to-face dialogue, in: Fielder, K. (Ed.), Social communication. Psychology Press (Frontiers of Social Psychology Series), New York, pp. 283-308.

Biau, E., Soto-Faraco, S. 2015. Synchronization by the hand: the sight of gestures modulates low-frequency activity in brain responses to continuous speech. Front. Hum. Neurosci. 9:527. doi: 10.3389/fnhum.2015.00527

Dohen, M., Lœvenbruck, H., 2004a. Identification des corrélats visibles de la focalisation contrastive en français, in: Proceedings of XXVèmes Journées d'Étude sur la Parole, Fès, Maroc, 185-188.

Dohen, M., Lœvenbruck, H., 2004b. La focalisation contrastive est-elle visible ? Une étude perceptive visuelle de la focalisation contrastive en français, in: Proceedings of XXVèmes Journées d'Étude sur la Parole, Fès, Maroc, 189-192.

Dohen, M., Lœvenbruck, H., 2005. Audiovisual Production and Perception of Contrastive Focus in French: a Multispeaker Study, in: Proceedings of Interspeech/Eurospeech, Lisbonne, Portugal, 2413-2416.

Dohen, M., Lœvenbruck, H., 2009. Interaction of audition and vision for the perception of prosodic contrastive focus. Language and Speech 52(2-3), 177-206.

Enfield, N.J., 2009. The Anatomy of Meaning. Speech, Gesture, and Composite Utterances. Cambridge University Press, Cambridge.

Ferré, G., 2018. Gesture/speech integration in the perception of prosodic emphasis, in: Proceedings of Speech Prosody, Poznan, Poland, 35-39.

Kendon, A., 2004. Gesture. Visible Action as Utterance. Cambridge University Press, Cambridge.

Kita, S., 2000. How representational gestures help speaking, in: McNeill, D. (Ed.), Language and Gesture. Cambridge University Press, Cambridge, pp. 162-185.

Kita, S., Alibali, M. W., & Chu, M. 2017. How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245–266.

Krauss, R.M., Chen, Y., Gottesman, R.F., 2000. Lexical gestures and lexical access: a process model, in: McNeill, D. (Ed.), Language and Gesture. Cambridge University Press, Cambridge, pp. 261-283.

McNeill, D., 2005. Gesture and Thought. University of Chicago Press, Chicago and London.

Morsella, E., Krauss, R.M., 2004. The Role of Gestures in Spatial Working Memory and Speech. The American Journal of Psychology 117(3), 411-424.

Swerts, M., Krahmer, E., 2007. Acoustic Effects of Visual Beats, in: Proceedings of Auditory Visual Speech Processing (AVSP), Hilvenbareek, Pays-Bas, On CD-Rom.

Swerts, M., Krahmer, E., 2008. Facial expression and prosodic prominence: Effects of modality and facial area. Journal of Phonetics 36, 219-238.

So, W.C., Sim Chen-Hui, C., Low Wei-Shan, J., 2012. Mnemonic effect of iconic gesture and beat gesture in adults and children: Is meaning in gesture important for memory recall? Language and Cognitive Processes 27, 665-681.
Diaporama