SSPNetSocial Signal Processing Network

www.sspnet.euproject no. 231187

Annual Public Report 2011

Project Description

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of our cognitive abilities that has been argued to be indispensable and perhaps the most important for success in life. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis and synthesis of relevant behavioural cues like blinks, smiles, crossed arms, head nods, laughter, etc., the research efforts in machine analysis and synthesis of human social signals like empathy, politeness, and (dis)agreement, are few and tentative. The main reasons for this are the absence of a research agenda and the lack of suitable resources for experimentation.

The mission of the SSPNet is to create a sufficient momentum by integrating an existing large amount of knowledge and available resources in Social Signal Processing (SSP) research domains including cognitive modelling, machine understanding, and synthesizing social behaviour, and so:

enable creation of the European and world research agenda in SSP, provide efficient and effective access to SSP-relevant tools and data

repositories to the research community within and beyond the SSPNet, and further develop complementary and multidisciplinary expertise necessary

for pushing forward the cutting edge of the research in SSP.

The collective SSPNet research effort is directed towards integration of existing SSP theories and technologies, and towards identification and exploration of potentials and limitations in SSP. More specifically, the framework of the SSPNet revolves around two research foci selected for their primacy and significance: Human-Human Interaction (HHI) and Human-Computer Interaction (HCI). A particular scientific challenge that binds the SSPNet partners is the synergetic combination of human-human interaction models, and automated tools for human behaviour sensing and synthesis, within socially-adept multimodal interfaces.

Project Goal and Objectives

The goal of the SSPNet is to establish a virtual distributed institute for research on Social Signal Processing (SSP), i.e. on conceptual modelling, detection, interpretation, and synthesis of social signals:

Conceptual modelling means that social signals will be described by cognitive scientists based on the experimental observations of social interactions.

Detection means that social signals will be extracted automatically from sensor data (e.g., video and audio recordings).

Interpretation means that social signals will be mapped automatically into socially relevant information (e.g., presence or absence of conflict).

Synthesis means that social signals will be generated artificially with computers to elicit desired social perceptions (e.g. to make artificial agents more friendly).

The goal is articulated into 10 concrete objectives:

1. Building the framework for structuring and sharing the SSPNet knowledge. The SSPNet is defining and discussing the agenda for research on SSP, providing the resources for experimentation (see Objective 9), and establishing a virtual SSPNet knowledge repository through which the SSP-relevant information is accessible to the research community within and beyond the SSPNet (www.sspnet.eu).

2. Describing the principles and laws of social interaction and social relations. The SSPNet studies the laws underlying social interactions, makes them explicit, and investigates how are these laws expressed and influenced by social signals.

3. Detecting, interpreting, and synthesizing the phenomena observed and described by social scientists. Social scientists have studied social interactions for the last seventy years and they have shown the existence of social signaling patterns stable at least for specific cultures and social situations. The patterns are the observable evidence of interaction quality and as such they can be detected, interpreted, and synthesized automatically. The development of such automatic, multimodal tools is a very important objective of the SSPNet.

4. Focusing on group interactions. The investigation of group interactions is particularly challenging from the technological point of view because it requires algorithms capable of taking into account multiple signal sources involved in complex interaction patterns. The SSPNet addresses such a major challenge for all modelling, detection, interpretation, and synthesis.

5. Assessing the effect of the context on social interactions. The SSPNet addresses several problems relevant to the effect of context on social signals, including the discrimination between posed and spontaneous behaviour (e.g.,

politeness vs. irony vs. enjoyment), the communication effectiveness, the persuasiveness, and the quality of information delivery.

6. Improving technology with social signals. The SSPNet investigates how social signalling patterns can be used as a-priori information in technologies dealing with human-human and human-machine interaction (e.g., how to improve speaker diarization taking into account turn-taking regularities, how to make synthetic voices more socially appealing, etc.).

7. Addressing cross-cultural issues. The SSPNet fosters activities addressing explicitly cross-cultural issues to interpret correctly data collected in different cultures, produce synthetic social signals perceived in the same way in different cultures, and assess the effect of culture on social situations.

8. Identifying relevant applications. The SSPNet explores the identification of concrete applications. The efforts will include in particular the contact with companies active in domains like call centres, multimedia analysis, production and indexing, marketing analysis, communication, etc.

9. Collecting datasets annotated in terms of Social Signals. The SSPNet is collecting existing repositories of relevant data including TV and radio shows, meetings, human-agent interactions, etc., and makes the data available to the research community via the SSPNet portal (www.sspnet.eu).

10. Establishing performance assessment criteria. The SSPNet contributes to the definition of evaluation and benchmarking procedures accepted by the community via the organization of national and international SSP centred challenges.

The SSPNet Portal

The main outcome of the SSPNet is the web portal (www.sspnet.eu), an instrument that aims at providing any interested researcher with the most important and yet difficult to obtain resources for working in SSP, i.e., the knowledge, data, and tools1.

Figure 1. Some of the presentations currently available on the portal.

In particular, the portal hosts four repositories of publicly available material that allow virtually anybody to work on SSP related problems:

Bibliography: a collection of references (including the publications of the project) that provide not only the background of the domain, but also state-of-the-art works on the most important SSP trends.

Data: a collection of annotated corpora containing audio and video material annotated in terms of social signals.

Tools: a collection of software packages addressing some of the most important needs in SSP experiments and research.

The Virtual Learning Centre: a collection of lecture and presentation recordings covering a wide spectrum of SSP related subjects (see Figure above).

The repositories are enriched with continuity and they are fully searchable and browsable. In its first 30 months of life, the portal has been visited 55741 times by 27711 unique visitors.

1 The SSPNet portal has been featured in the “Best of Web” column of the IEEE Signal Processing Magazine (A.Vinciarelli and M.Pantic, “A Web Portal for Social Signal Processing”, IEEE Signal Processing Magazine, 27(4):142-144, 2010).

Dissemination and Training

The SSPNet organizes a large number of events aimed at gathering and fostering a vibrant SSP research community. The events of 2011 have been (in chronological order):

First International Workshop on Emotion Synthesis, Representation, and Analysis in Continuous Spacehttp://emotion-research.net/sigs/speech-sig/emospaceSanta Barbara (SA) – March 21st, 2011

FG 2011 Facial Expression Recognition and Analysis Challengehttp://sspnet.eu/fera2011/Santa Barbara (USA) – March 24-25, 2011

COST 2102 International Training School on Cognitive Behavioural Systemshttp://www.ias.et.tu-dresden.de/ias/cost-2102/cost-2102-final-meeting/Dresden (Germany) - February 21-26, 2011

International Workshop on Voice and Speech Processing in Social Interactions.http://sspnet.eu/2011/01/vspsi/Glasgow (UK) - April 18-19, 2011

Fourth International Workshop on CVPR for Human Communicative Behavior Analysis.http://ibug.doc.ic.ac.uk/~maja/static/cvpr4hb-11.htmlColorado Springs (USA) - June 25, 2011

International Summer School in Affective Scienceshttp://www.affective-sciences.org/node/2789Geneva (Switzerland) – August 22-31, 2011

First International Audio/Visual Emotion Challenge and Workshophttp://sspnet.eu/avec2011/Memphis (USA) – October 9th, 2011

Second IEEE International Workshop on Socially Intelligent Surveillance and Monitoring.http://sspnet.eu/2011/03/sism/Barcelona (Spain) - November 12, 2011

Third International Workshop on Social Signal Processing.http://sspnet.eu/2011/05/wssp/Scottsdale (USA) - December 1, 2011

The Consortium

University of Glasgow (UK)Alessandro Vinciarelli, Coordinator

Imperial College London (UK)Maja Pantic, Scientific Coordinator

Idiap Research Institute (CH)Fabio Valente

University of Edinburgh (UK)Steve Renals

University of Twente (NL)Dirk Heylen

University of Roma Tre (I)Isabella Poggi

Queen’s University Belfast (UK)Roddy Cowie

DFKI (D)Marc Schroeder

CNRS/TELECOM ParisTech (F)Catherine Pelachaud

University of Geneva (CH)Marc Mehu

Delft University of Technology (NL)Emile Hendriks

University of Gothenburg (S)Jens Allwood