Painting an experience?How aesthetics might assist a

neuroscience of sensory experience

Ron ChrisleyCentre for Research in Cognitive Science and

School of Informatics, University of Sussex

Neuroesthetics: Where Art and the Brain CollideESF Workshop, IULM, Milan 24-25 September 2009

5 relevant areas of my research

1. Embodied creativity2. Enactive models of experience3. Synthetic phenomenology4. Interactive empiricism5. Art works/installations

(The title of my talk concerns area 3)

My research and neuroesthetics

• Not a neuroscientist• Much of the work I am reporting on is

trying to provide a bridge from art/aesthetics into the cognitive sciences, which then permits a connection with neuroscience

• Re: this workshop:– How can the models of aesthetic processes I am

investigating be informed by neuroscience/aesthetics?– Can the models suggest future directions for

neuroscience? Aesthetics/art?

1. Embodied creativity• Goal: Design a robot/environment system likely to

exhibit creative behaviour:– Novel (at least for the robot)– Of (aesthetic) value (for humans, if possible)

• Engineering approach:– No direct modelling of human creativity– But exploit what is known about creativity in humans (and

animals?), when expedient– Allow for possibility that insights into the human case may accrue

anyway• Manifesto only: No implementation yet

– Set of "axioms"– Assume case of musical output for examples

Principles of embodied (aesthetic) creativity1. If you make your robot pleasure-seeking, and make creativity

pleasurable, you'll make your robot creative2. To be a good creator, it helps to be an appreciator3. Let the robot experience output in the real world, as we do4. We won’t like what the robot likes unless it likes what we like5. An important motivator is the approval or attention of others6. Novelty can be achieved by trying to produce outputs on the

subjective edge of chaos (that lie just beyond the robot’s ability to explain/predict)

7. Let dynamics play a role in appreciation8. Patterns in one's own states can be the objects of appreciation9. The best way to make outputs in the real world is to be embodied in

the real world

Underlying architecture• CNM:

– Recurrent neural network– Forward model of environment

• Learns to anticipate/predict the sensory input it will receive if it performs a given action in a given context

• In conjunction with motivators can enable the robot to select actions that carry an expectation of "pleasure"

Underlying architecture

Key:

Recurrent Connection (Copy)Full Inter-Connection Between Layers Of Units

Action

Expected Sensations

Predicted State

Previous Predicted State

(Context Units)

D-map

T-map

Generalisation to aesthetic experience• Perhaps the content of aesthetic

experience can be understood using the same framework, with two extensions:1. Affective expectations2. Aesthetic expectations

Aesthetic experience: affect

• Just as a system can have expectations concerning its actions and resulting sensory inputs, it can also have expectations concerning its actions and resulting desirable or undesirable states

• Thus, affect, and in particular the affective character of aesthetic experience, may be accommodated in EBA

Aesthetic experience: artifice

• When we experience a visual work of art we do not just experience its visual properties (explicable in terms of expectations to receive input x if we move our eyes thus)

• We also experience it as a work of art: we possess expectations for how the work would change if we were to make (or the artist were to have made) this or that brushstroke

• Set of all such expectations is the content of the aesthetic experience of the work

3. Synthetic phenomenology

• A science of consciousness needs a way to refer to or specify the content of conscious experiences

• Standard means: e.g., "Mary is having a visual experience of a red bike leaning against a white fence"

• Problem: Can only specify experiences with linguistic, conceptual, content

• Yet several good reasons to believe that some of content of experience is non-conceptual

Do I have to draw you a picture?• An obvious alternative is to use non-linguistic, non-symbolic

specifications• E.g., for the case of visual experiences, use images• Can't just take a picture of the scene the subject is seeing

(literalism)• Even in the case of a robot model of experience, can't just use

the raw video camera output– For example; the current "output" of a human retina contains gaps or

blindspots that are not part of experience.– Furthermore, our visual experience, as opposed to our retinal output,

at any given time is stable, encompassing more than the current region of foveation, and is coloured to the periphery

• But what alternatives are there?

Depictive specifications of the content of visual experience• If the set of expectations determines experiential content,

then displaying those expectations (in the right way) will count as a specification of that content

• "Filled-in" areas specify what input the robot would expect to receive if it moved its head so that it is looking in that location

• Grey areas do not indicate an expectation to receive grey input; they indicate to you the absence of any expectation for that location– "Absence of expectation is not an expectation of absence"– Alternative architectures (e.g., generalising neural networks)

would have no such undefined regions of state space

Depictions• Need not explicitly occur anywhere in the agent; are of

experience corresponding to current expected inputs, not actual inputs

• Generated off-line by the theorist:– Sample the action space– Feed actions into forward model– Arrange the resulting expected sensory patterns in a spatial array

according to the spatial relations of the sampled actions• Theorist's experience of the depiction (plus

interpretation/bracketing instructions) is the same as (or shares crucial properties with) the experience to be specified

Depiction of anexpectational state

Depictive specification and art• In a sense, content

specification is what (at least some) artists have been trying to do for millenia

• Thus, a clear role for – artists– graphic designers– sound engineers– directors– etc.

Depictive specification and art• E.g., creating a film of a car

chase in the desert• Not just a matter of "objectively

recording"• Requires artistic insight into

how film will affect viewers, e.g.:– Pacing– timing, number, kinds of cuts

and edits– what is in the frame– steadiness

• Cf Picasso's portraits

Problem: Subjectivity of Art?• Science strives to be

objective• But content of a work of

art is highly subjective• Yes, but artisitic insights

can be objectively investigated; e.g.:– Perspective– Moving images:

“persistence of vision”– Effect of colour on mood\– Et al

Problem: Subjectivity of Art?• Is subjectivity always at odds

with science?• Objective science is not the

elimination of the scientist's subjectivity

• Rather the negotiation of it• Perhaps subjectivity of theorist

can be exploited• Lessons from interactive Art?

A two-way interaction• Often, it has been thought that

the interaction was one-way:– cogntive scientists informing

the work of designers, artists, and other "creative" types

• But the converse interaction is needed as well:– Artistic input into radically

different means of specifying the contents of experience

4. Interactive empiricism

• Some problems concerning (aesthetic) experience are philosophical; require conceptual breakthroughs/progress

• Such breakthroughs (e.g. new concepts) may not be achievable by reason alone, but require experiential activity– “…Modelling of consciousness… requires some clarifications and

refinements of our concept of consciousness. Design of, construction of, and interaction with artificial systems can itself assist in this conceptual development.” (Sloman and Chrisley 2003)

– Sensory augmentation (e.g., work with Froese and Spiers on how using the “Enactive Torch” might alter concepts of perception)

– Art works are also “artificial systems”: creation of and interaction with

5. Art works/installations

5. Art works/installations

Recap

1. Embodied (artistic) creativity2. Enactive models of (aesthetic) experience3. Synthetic phenomenology: specifying the

content of the sensory, affective and aesthetic components of experiencing art works

4. Interactive empiricism: changing our concepts of experience through creating or interacting with art

5. Art works/installations based on the above

5. Art works/installations

Thank you.

Comments welcome:ronc@sussex.ac.uk