Jonas Nölle, Oliver G. B. Garrod, Chaona Chen, Meng Liu, Robin A. A. Ince, Philippe G. Schyns and Rachael E. Jack

Human facial expressions are a powerful tool for social communication. In addition to the salient morphology of the human face (Tomasello et al., 2007; Vick et al., 2007), its powerful capacity to generate many complex dynamic face movement patterns enables it to convey myriad nuanced social messages. Though used frequently for everyday interactions, facial expressions have mainly been studied as displays of affect in nonhuman primates (Van Hooff, 1972; Waller & Micheletta, 2013) and humans (Ekman, 1994) and less so as a pragmatic tool for social communication. Thus, while multimodal accounts of communication continue to gain importance (e.g., Perniss, 2018; Holler & Levinson, 2019), including in discussions of language origins (Zlatev et al., 2017; Fröhlich et al., 2019), the contribution of facial expressions remains understudied. One major challenge to this endeavour is the sheer number and complexity of human facial expressions (Jack et al., 2018).
Here we showcase a novel data-driven psychophysical method that can navigate the complexity of facial expressions (Yu et al., 2012), and an information-theoretic framework (Ince et al., 2017) that can characterize the social information they convey. Specifically, we agnostically generate dynamic facial expressions, each composed of a random subset of individual facial movements called Action Units (AUs, e.g., Nose Wrinkler, Upper Lid Raiser; Ekman & Friesen, 1978), and ask participants to categorize them according to a given set of social messages – e.g., basic emotions such as ‘happy’ or ‘fear’ (Jack et al., 2012) or pragmatic messages such as ‘interested’ or ‘bored’ (Chen et al., 2015). We then build a statistical relationship between the dynamic face movements presented on each trial and the participant’s responses. This produces a precise, quantitative model of the dynamic face movement patterns that convey each social message to the participant. Analyses of the resulting facial expression models can reveal, for example, syntactical combinations of face movements (Jack et al., 2014), cross-cultural commonalities, variances, and accents that can facilitate or hinder cross-cultural communication (Jack & Schyns, 2017).
Using this data-driven method, we have examined human facial expressions as a tool for social communication and present new work addressing two key questions: Firstly, do facial expressions possess a similar iconic potential to manual gestures that can ground symbolic communication (Fay et al., 2014)? Does eye squinting express ‘small size’ akin to manual ‘pinching’ gestures (Woodin et al., 2020)? Our results suggest that they do – for example, facial movements that originally evolved to control sensory input (Susskind et al., 2008), such as narrowing the eyes in facial expressions of disgust, can be exapted to ground more abstract social signals for communication. Secondly, we examine the pragmatic role of specific face movements such as eyebrow raising and eye widening in supporting speech comprehension including emphasis, intonation, and modulating semantic content. Together, our results show specifically how human facial expressions serve a pragmatic function in social interaction, highlighting their central importance in theories of multimodal language origins. We anticipate that our data-driven method will facilitate further knowledge developments in this domain.

References

Chen, C., Garrod, O., Schyns, P., & Jack, R. E. (2015). The Face is the Mirror of the Cultural Mind. Journal of Vision, 15(12), 928–928. https://doi.org/10.1167/15.12.928
Ekman, P. (1994). All emotions are basic. In P. Ekman & R. J. Davidson (Eds.), The nature of emotion: Fundamental questions (pp. 65–58). Oxford University Press.
Ekman, P., & Friesen, W. V. (1978). Facial action coding system: Investigator’s guide. Consulting Psychologists Press.
Fay, N., Lister, C. J., Mark Ellison, T., & Goldin-Meadow, S. (2014). Creating a communication system from scratch: Gesture beats vocalization hands down. Frontiers in Psychology, 5(APR), 1–12. https://doi.org/10.3389/fpsyg.2014.00354
Fröhlich, M., Sievers, C., Townsend, S. W., Gruber, T., & Schaik, C. P. van. (2019). Multimodal communication and language origins: Integrating gestures and vocalizations. Biological Reviews, 94(5), 1809–1829. https://doi.org/10.1111/brv.12535
Holler, J., & Levinson, S. C. (2019). Multimodal Language Processing in Human Communication. Trends in Cognitive Sciences, 23(8), 639–652. https://doi.org/10.1016/j.tics.2019.05.006
Ince, R. A. A., Giordano, B. L., Kayser, C., Rousselet, G. A., Gross, J., & Schyns, P. G. (2017). A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula. Human Brain Mapping, 38(3), 1541–1573. https://doi.org/10.1002/hbm.23471
Jack, R. E., Garrod, O. G. B., Yu, H., Caldara, R., & Schyns, P. G. (2012). Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences, 109(19), 7241–7244. https://doi.org/10.1073/pnas.1200155109
Jack, Rachael E., Crivelli, C., & Wheatley, T. (2018). Data-Driven Methods to Diversify Knowledge of Human Psychology. Trends in Cognitive Sciences, 22(1), 1–5. https://doi.org/10.1016/j.tics.2017.10.002
Jack, Rachael E., & Schyns, P. G. (2017). Toward a Social Psychophysics of Face Communication. Annual Review of Psychology, 68(1), 269–297. https://doi.org/10.1146/annurev-psych-010416-044242
Jack, Rachael E., Garrod, O. G. B., & Schyns, P. G. (2014). Dynamic Facial Expressions of Emotion Transmit an Evolving Hierarchy of Signals over Time. Current Biology, 24(2), 187–192. https://doi.org/10.1016/j.cub.2013.11.064
Perniss, P. (2018). Why We Should Study Multimodal Language. Frontiers in Psychology, 9. https://doi.org/10.3389/fpsyg.2018.01109
Susskind, J. M., Lee, D. H., Cusi, A., Feiman, R., Grabski, W., & Anderson, A. K. (2008). Expressing fear enhances sensory acquisition. Nature Neuroscience, 11(7), 843–850. https://doi.org/10.1038/nn.2138
Tomasello, M., Hare, B., Lehmann, H., & Call, J. (2007). Reliance on head versus eyes in the gaze following of great apes and human infants: The cooperative eye hypothesis. Journal of Human Evolution, 52(3), 314–320. https://doi.org/10.1016/j.jhevol.2006.10.001
Van Hooff, J. (1972). A comparative approach to the phylogeny of laughter and smiling. Non-Verbal Communication, 209–241.
Vick, S.-J., Waller, B. M., Parr, L. A., Smith Pasqualini, M. C., & Bard, K. A. (2007). A Cross-species Comparison of Facial Morphology and Movement in Humans and Chimpanzees Using the Facial Action Coding System (FACS). Journal of Nonverbal Behavior, 31(1), 1–20. https://doi.org/10.1007/s10919-006-0017-z
Waller, B. M., & Micheletta, J. (2013). Facial Expression in Nonhuman Animals. Emotion Review, 5(1), 54–59. https://doi.org/10.1177/1754073912451503
Woodin, G., Winter, B., Perlman, M., Littlemore, J., & Matlock, T. (2020). ‘Tiny numbers’ are actually tiny: Evidence from gestures in the TV News Archive. PLOS ONE, 15(11), e0242142. https://doi.org/10.1371/journal.pone.0242142
Yu, H., Garrod, O. G. B., & Schyns, P. G. (2012). Perception-driven facial expression synthesis. Computers & Graphics, 36(3), 152–162. https://doi.org/10.1016/j.cag.2011.12.002
Zlatev, J., Wacewicz, S., Żywiczyński, P., & van de Weijer, J. (2017). Multimodal-first or pantomime-first?: Communicating events through pantomime with and without vocalization. Interaction Studies, 18(3), 465–488. https://doi.org/10.1075/is.18.3.08zla