Experiencing perceptual crossing Dr. Tom Froese IIMAS-UNAM

Experiencing perceptual crossingDr. Tom FroeseIIMAS-UNAM

The socially extended mind

Take-home message: Nonlinearly coupled agents form an extended dynamical system with novel properties.

Sociality as a complex system

“Brain-body-environment-body-brain system” (Froese, Iizuka and Ikegami, 2013)

Sociality: autonomous interaction process, co-regulated sensorimotor interaction, …

De Jaegher and Di Paolo (2007); Froese and Di Paolo (2011)

Embodied inter-subjectivity

What about the subjective experience associated with an extended body?

Froese and Fuchs (2012)

Can we measure people’s sensitivity to inter-bodily resonance (social contingency)?

Perceptual crossing paradigm

Froese, Iizuka and Ikegami (2014a)

Virtual reality environment

Lenay and Stewart (2012)

Participants can encounter 3 different objects in the virtual space:1) Avatar of other player, 2) Shadow of other player (like instantaneous replay of double TV paradigm),3) Static object

The objects produce the same instantaneous tactile feedback (on/off), but their affordances for engaging in embodied interaction differ.

Task: subjects should click whenever they encounter the other’s avatar.

Time series of a trial

Auvray et al. (2009)

Summary of clicking results

Auvray et al. (2009)Shadow object

Analysis of experimental results

• Collective success:• “The results on the cause of clicks showed that participants

clicked significantly more often on the other participant’s avatar than on the fixed object and mobile lure.”

• Individual failure:• “participants did not seem able to discriminate between stimuli

due to the avatar and those due to the mobile lure: the probability of a click for these two types of stimuli was not significantly different.”

• Conclusion: • “sensitivity to the situations of perceptual interaction […], instead

of being perceived by each of the participants, arises from the dynamics of the interaction itself.”

Auvray et al. (2009)

Debating meaning of the results

• “An experiment by Auvray, Lenay and Stewart shows a constitutive role for interaction.

• The variation in the number of clicks is attributable only to the differences in the stability of the coupling and not to individual strategies.”• De Jaegher, Di Paolo and Gallagher (2010, pp. 444-5)

• Yes, but is it constitution of social cognition? This interpretation of the results has also been criticized.

• Individuals’ social cognition seems to be unaffected.

Toward individual recognition• Lenay and Stewart (2012) tried to remedy this situation.

• Their experimental setup is the same as that of the first perceptual crossing experiment, except that this time the sensory feedback is no longer a tactile stimulus, but a sound which is different for each of the objects which can be encountered.

• Twenty participants took part in their experiment.

• Each time the receptor field overlaps a virtual object, a sound is emitted which varies according to the nature of the object. • (1) the horn of a car, • (2) the horn of a big lorry, and • (3) the tinkling of a bicycle bell.

• These sounds were chosen to be easily differentiated and named.

Experimental procedure• Participants were told that they could freely explore the 1D

space containing three types of auditory object:1. The receptor field of the partner2. Fixed objects3. Mobile objects

• However, the nature of the dynamics of the mobile object was not explained. • Condition C1: mobile object attached by link (100 units away)• Condition C2: mobile object follows recorded trajectory from the

previous trials (condition C1 trials)• The instruction was to associate a sound to each of the three

types of objects.

Experimental procedure• There were 4 trials in total: first two condition C1 trials and

then two condition C2 trials.

• The sounds are reattributed differently and randomly for each session.

• At the end of each session, each participant associates each type of of the three types of object with a sound.• What sound was made by the fixed object?• What sound was made by the mobile object?• What sound was made by the partner’s object?

ResultsFor the set of all 80 trials, 60 were perfect (3/3; 100%).


Individual recognition• What explains the individual success here is the ability of the

participants to distinguish the dynamics of perceptual crossing from the dynamics of interacting with other objects.

• Since the participants have access to different intrinsic properties for the three objects, they can recognize different occurrences of the same object.

• The different intrinsic properties of the objects can be associated with properties characteristic of the dynamics of the interaction.

• Thus, the situation of perceptual crossing is now recognized as a property of an object already identified by other means.


Specialized neural systems• If we apply this explanatory scheme to the development of the new-

born infant, we may suppose that the dynamics of perceptual crossing with the care giver is associated with the visual perception of the intrinsic properties of their face.

• The logical point which is crucial here is that the individual neuronal structures which participates in the association can be the result and not the primary cause of this dynamics of interaction.

• If, on the contrary, the inter-individual interactions had to be the effect of prior internal structures – if it were necessary to already have the means of recognizing partners before engaging in interaction with them – then the process of learning, or the evolutionary scenario, which account for the appearance of these structures would be almost impossible to imagine.

Necessity of the explicit trace?• Lenay and Stewart (2012) have found one solution to the

individual recognition and thereby provided evidence for the possibility of individual social cognition.

• But were the sounds “intrinsic properties of the objects”?

• And if the sounds were actually extrinsic properties, can we speak about the interactive constitution of social cognition?

• Individual recognition has been enabled by the introduction of an additional objective factor that is independent from the real-time dynamics of interaction themselves.

The “social” as mutual responsiveness

Example of co-regulation: Someone’s act of giving can only be successfully realized if it is complemented by someone else’s act of receiving.

Froese and Di Paolo (2011)

Experimental setup• Same experimental setup as in original perceptual crossing

study by Auvray et al. (2009).

• Instructions to participants were modified as follows:• You are a team competing against other teams; try to help each

other to accomplish the task of locating each other.• There are 15 trials, each of 1 minute duration.• Each player only has one click per trial. • Your overall team score will be calculated as follows:

• For each correct click, plus 1 point.• For each wrong click, minus 1 point.• For each absent click, no change.

• We tested 17 teams (total of 34 participants).

Virtual reality space

• Each player can encounter 3 objects of equal size:• A static object• The other player’s avatar (embodiment of sensor)• The other player’s shadow (same movement, w/o sensor)


Example of co-regulation


Evidence of individual sensitivity

• In our study players’ clicks were specifically sensitive to contact with the other’s avatar:

• A click was almost six times as likely to occur after contact with the other’s avatar (1.17% probability) when compared to the static object (0.20% probability).

• A click was nearly twice as likely to be made after a player was stimulated by the other’s avatar (1.17% probability) compared to the other’s shadow object (0.65% probability).

• Turning the individual epistemic task of agency detection into a social pragmatic task of coordination has for the first time revealed an individual sensitivity to the other’s presence in the perceptual crossing paradigm.

Co-regulation of co-awareness• Was this enhanced individual sensitivity dependent on the

distributed social interaction process as a whole?

• Turn-taking (TT) was significantly higher during the 10 seconds before clicks in Joint Success trials compared to correct clicks in Single Success trials (P = 2.6x10-4), as well as compared to wrong clicks (P = 3.3x10-7).

• Even though it might be expected that TT performance was significantly higher before correct clicks in Single Success trials compared to wrong clicks, this was not the case (P = 0.15).

• TT is therefore not an indication of objective clicking success per se, but rather of the extent of cooperative interaction.

Quantifying user experience (PAS)

• Based on Perceptual Awareness Scale (PAS)• Sandberg et al. (2010). … Consciousness and Cognition

• After each trial: “Please select a category to describe how clearly you experienced your partner at the time you clicked”:

PAS 4: Clear experiencePAS 3: Almost clear experiencePAS 2: Vague impressionPAS 1: No experience

• Verbal reports: After each trial participants could also choose to briefly describe their experience in their own words.

Subjective evidence for co-awareness

Froe

se, I

izuk

a an

d Ik

egam

i (20

14a)

Subjective difference between types of success.

Co-regulation of co-awareness• Amount of turn-taking before clicks was significantly higher

for PAS 4 than for PAS 3, and especially compared to PAS 2.• There was no significant difference between PAS and PAS 1, but

this may due to an outlier and insufficient samples of PAS 1.

• This finding supports our hypothesis that co-regulation is correlated with intersubjective experience.• And it is evidence for constitution of social cognition.

• But is this social awareness reducible to an individual alone?

• Or can an experience be genuinely shared across subjects?

Evidence for mutual awareness

Froe

se, I

izuk

a an

d Ik

egam

i (20

14a)

Most jointly successful clicks occurred within seconds of each other.

Inter-bodily resonance• Did players experience their partner’s emotional state via inter-bodily

resonance?

• Given the scarcity of free-text descriptions, it was usually impossible to evaluate the experience at this level of detail. However, in at least one experiment there was a clear emotional correlation.

• After trial 10 one player somewhat confidently remarked: • “I think I am pretty sure that I could communicate about my intention”

(E10T10Pa), • and two trials later he writes: • “Same as before, but I felt that the partner is anxious” (E10T12Pa).

• After the next trial his partner writes: • “I think my click was correct but if this response was autonomous object’s, I

will get anxious” (E10T13Pb).Froese, Iizuka and Ikegami (2014b)

Redd

y (2

003)

Hypothesis and evaluation• We proposed that the perceptual crossing paradigm can fill a

gap in experimental psychology: to study the development of social awareness, but from an adult’s perspective.

• Following enactive theory, we hypothesized that we should find something akin to the main stages of development of social awareness in infants to be recapitulated in adults • If they are forced to implicitly relearn the interactive skill of social

perception via the human-computer interface.

• As a first attempt to evaluate this hypothesis we re-analyzed the data of our perceptual crossing experiment, but now as a diachronic sequence of 15 trials.

Froese et al. (2014b)


When averaged over all teams, there was no significant increase in turn-taking Performance. But when we look at the best performing team:



Recapitulation of development?

• We defined three phenomenological categories (A-C):A. Individual awareness of being the object of another’s attentionB. Mutual awareness of being each other’s objects of attentionC. Mutual awareness of specific aspects of the interaction being

the object of joint attention

• Qualitative methodology:1. Identify some defining examples for each category2. Code all samples and calculate inter-observer reliability3. Determine relationships between categories of agreed codings

• Clarity of the other’s presence• Temporal sequence




This gives an interobserver-reliability kappa of 0.51, which can be interpreted as “moderate agreement.”


Clarity of other’s presence

• The average PAS clarity for category C was significantly higher than for category B (mean-C = 3.62; P = 3.71x10-6).

• The PAS clarity of social awareness associated with categories A and B was not significantly different.

• These categories may not be experienced as qualitatively distinct from the first-person perspective.

• We therefore grouped A and B as AB for diachronic analysis of developmental trends.



Conclusions• Appropriate task behavior can become unconsciously self-

organized via the dynamics of interaction.

• Interactions can become consciously recognized as a social encounter during co-regulation of their dynamics.

• This awareness of the other person appears to be based on a genuinely shared awareness of each other.

• The interactive constitution of intersubjective experience is a social skill that is acquired by practice and develops over time.

Homework• Read before next class:• Lenay, C., & Stewart, J. (2012). Minimalist approach to perceptual

interactions. Frontiers in Human Neuroscience, 6(98). • Focus on Experiment 3!

• Froese, T., Lenay, C., & Ikegami, T. (2012). Imitation by social interaction? Analysis of a minimal agent-based model of the correspondence problem. Frontiers in Human Neuroscience, 6(202).

• Due date of second essay: to be determined…

References• Auvray, M., Lenay, C., & Stewart, J. (2009). Perceptual interactions in a minimalist virtual environment.

New Ideas in Psychology, 27(1), 32-47• De Jaegher, H., & Di Paolo, E. A. (2007). Participatory sense-making: An enactive approach to social

cognition. Phenomenology and the Cognitive Sciences, 6(4), 485-507• De Jaegher, H., Di Paolo, E. A., & Gallagher, S. (2010). Can social interaction constitute social cognition?

Trends in Cognitive Sciences, 14(10), 441-447• Froese, T., & Di Paolo, E. A. (2011). The enactive approach Theoretical sketches from cell to society.

Pragmatics & Cognition, 19(1), 1-36• Froese, T., & Fuchs, T. (2012). The extended body: A case study in the neurophenomenology of social

interaction. Phenomenology and the Cognitive Sciences, 11(2), 205-235• Froese, T., Iizuka, H., & Ikegami, T. (2013). From synthetic modeling of social interaction to dynamic

theories of brain-body-environment-body-brain systems. Behavioral and Brain Sciences, 36(4), 420-421• Froese, T., Iizuka, H., & Ikegami, T. (2014). Embodied social interaction constitutes social cognition in

pairs of humans: A minimalist virtual reality experiment. Scientific Reports, 4(3672). doi: 10.1038/srep03672

• Froese, T., Iizuka, H., & Ikegami, T. (2014b). Using minimal human-computer interfaces for studying the interactive development of social awareness. Frontiers in Psychology, 5(1061). doi: 10.3389/fpsyg.2014.0106

• Lenay, C., & Stewart, J. (2012). Minimalist approach to perceptual interactions. Frontiers in Human Neuroscience, 6(98). doi: 10.3389/fnhum.2012.00098

• Reddy, V. (2003). On being the object of attention: Implications for self-other consciousness. Trends in Cognitive Sciences, 7(9), 397-402

Documents

Experiencing perceptual crossing Dr. Tom Froese IIMAS-UNAM