iMAP : Tracking Multimodal Communication in Humans and Agents

Max Louwerse, PI
Ellen Bard, Co-PI
Mark Steedman, Co-PI
Art Graesser, Co-PI
Xiangen Hu, Co-PI

The iMAP (Intelligent MapTask Agent) project investigates multimodal communication in humans and agents, focusing on two linguistic modalities - prosody and dialog structure, which reflect major communicative events, and two non-linguistic modalities - eye gaze and facial expressions. It aims to determine: 1.Which of the non-linguistic modalities align with events marked by prosody and dialogue structure, and with one another; 2.Whether, and if so when, these modalities are observed by the interlocutor; 3.Whether the correct use of these channels actually aids the interlocutor's comprehension. Click here for video.