Prototype GINGA

PROTOTYPE GINGATHE DANCER BECOMES MUSIC

ADVANCED INTERACTION STUDIO.

Sebastián Alvarado Grugiel

Costa Rica

June 2014

Master in Advanced Architecture2013-2014

PROTOTYPE GINGATHE DANCER BECOMES MUSICADVANCED INTERACTION STUDIO.

STUDENTSebastián Alvarado Grugiel

Costa Rica.

FACULTY

Carlos Gómez

Xavi González

Master in Advanced Architecture2013-2014

‘‘I would only believe in a God who could dance.’’

Friedrich Nietzsche

INDEX

Pre-studyAdvanced ArchitectureSound, Color and Interaction

1 9

Introduction 7

References

Special thanks

42

44

2 17Interaction

Input/Output - Looking for LoopsAbout HumansSynesthesiaDesigning Behaviors for New Humans

3 28Prototype GINGA

Context - CapoeiraReading the BodySensor SetupTurning Data into Sound

4 36ExhibitionsSONAR 2014IAAC

The dancer becomes music. Ginga is a wearable technology that turns body movement into sound.

Our body is the result of thousands of gen-erations of intense evolution. When we use modern technologies our bodies are reduced to a chair and a finger click. De-sign creates behaviours. A new approach is needed to develop the technologies of the future, starting from our physical re-ality, giving emergent technologies a hu-man shape.

Prototype Ginga is an experiment con-ceived in Capoeira, the Brazilian dance/martial arts.

Introduction

IaaC MAA_01 2013-2014

11 2013-2014 MAA_01 IaaCIaaC MAA_01 2013-2014

PRE-STUDY1

Advanced Architecture

Now a days different disciplines of knowledge are find-ing points of coincidence as they develop and get more specific. There is no more an expert in architecture, but an architect who is an expert in some area of the field, within asymmetric groups of experts. Disciplines mix as they get complex and creativity and intelligence result in hybrid new fields of knowledge, linking and under-standing in new ways, as sciences look for answers in-side each other.

The first personal interest in interaction came from an interest in music, trying to find a link between personal passions. Architecture in evolving incredibly fast now, and is demanding of new links between knowledge. If we compare the evolution of architecture and the evo-lution of life, we see that architecture have evolved in form, while life evolves in form and intelligence. Will ar-chitecture become intelligent?... Advanced Architecture is intelligent, through interaction and communication.

12 IaaC MAA_01 2013-2014

Prototype GINGA

13 2013-2014 MAA_01 IaaCIaaC MAA_01 2013-2014

14 IaaC MAA_01 2013-2014

Prototype GINGA

Sound, Color and Interaction

Sound is essential for the interaction of the most com-plex kind of life. Language is one of the most important elements of human interaction and intelligence. It could have evolved from music, as exposed by Rousseau, Herder, Humboldt, and Charles Darwin. Complex organ-isms like humans use also color/light as information to interact. Can intelligent Architecture use color/light and sound to interact and communicate?

Color and sound can be measured, and turned into bits of information. Bits can be transmitted and under a cer-tain code, they can become messages for interaction. This way, color becomes sound and sound becomes color. They are both just channels for information. This can be used to develop Intelligence in Architecture de-signing new lightscapes and soundscapes.

15 2013-2014 MAA_01 IaaCIaaC MAA_01 2013-2014

16 IaaC MAA_01 2013-2014

Prototype GINGA

Experiment 1

Using seven basic colours and assigning them to each musical note, we can trans-late sound information into colour information. How would that change how we ex-perience music, space, atmospheres?

17 2013-2014 MAA_01 IaaCIaaC MAA_01 2013-2014

IaaC MAA_01 2013-2014 18 IaaC MAA_01 2013-2014

Prototype GINGA

MORE information at

https://www.youtube.com/watch?v=C-jtWJiO6SI

19 2013-2014 MAA_01 IaaCIaaC MAA_01 2013-2014 IaaC MAA_01 2013-2014

INTERACTION2

Input/Output - Looking for Loops

Interaction is the action that occurs as two or more ob-jects have an effect upon one another. Object 1 sends an output to Object 2, which receives it as an input. Ob-ject 2 reacts sending an output that Object 1 receives as input. This modifies the behavior of Object 1 and its next output. Interaction sets a loop of inputs and out-puts that modify the behavior between objects.

Interaction Design focuses on the way people interact with each other using external objects. Interaction is as good as the experiences that people can share. Tech-nology is a tool to make this interaction more complex and intriguing.

20 IaaC MAA_01 2013-2014

Prototype GINGA

Kids playing in a square. The metallic pyramid is noth-ing particularly special, but the people’s interaction gives it an important meaning. The interaction between people, its complexity, and the experiences we learn and share, is what makes interaction meaningful.

21 2013-2014 MAA_01 IaaCIaaC MAA_01 2013-2014

A loop of inputs and outputs.

Input

IaaC MAA_01 2013-2014 22 IaaC MAA_01 2013-2014

About Humans

Interaction is as good as what people share and learn from each other. People should be the target of any in-teraction design. Technology improved how we commu-nicate today and proved to be a great tool for exploring new ways of interaction, making it more complex by linking new interests.

Modern humans have been existing for 200 000 years. This means that there are around 6000 generations of humans between us and our first ancestros. We are the result of such evolution, resulting in the biological tech-nologic marvel that is our body. We are capable of in-credible things: we play music, dance, sing, paint, build, swim, glide, surf, you name it; and our brains backup all our creativity and understanding of the world.


The human body is a powerful result of evolution.

IaaC MAA_01 2013-2014 24 IaaC MAA_01 2013-2014

Synesthesia

This project of interaction was based on how to add complexity to the human natural behavior. The main in-terest was in the human factor of interaction. We define reality using several channels, we call senses, to regis-ter information from what we call the world around us.

“This music crept by me upon the waters”

From “The Tempest” by William Shakespeare.


The data from the environment is picked up by recep-tors that are specialized to respond to a specific physi-cal stimulus. The data is transferred to the brain as sig-nals with the same amplitude but different frequencies. The brain doesn’t determine the type of stimuli since all arrive in the same form, but from the place the stimuli came from. This means that any stimuli received by the photoreceptor will be decoded in the visual center of the brain as light information; if by any chance it came to the sound center, it would be interpreted as sound. This ability/weakness of the brain to mix sensory sig-nals some times lead to SYNESTHESIA.

Understanding interaction through the human experience.

26 IaaC MAA_01 2013-2014

Prototype GINGA

Synesthesia is a neurological phenomenon that occurs when the stimulation of one sensory or cognitive path-way is experienced in a second sensory or cognitive pathway. People with synesthesia may see each letter in a particular color when they are actually black and white and even if they are blind, experiment taste in their mouth when they listen to a particular word, or count by arranging number in space and color. Studies on synesthets reveal that our brain establish sensory associations during our childhood, all the mixed sense experience of synesthesia is defined when we are very young. Synesthets appear to have connections between different areas of the brain that are typically completely separate.

Although synesthesia is present in a minority of people and with very diverse manifestations, it might help to explain the complexity of human intelligence. Everyday we use synesthetic principles to understand complex concepts, like number sequences. Synesthesia helps to explain another critical skill of the human condition: creativity.

27 2013-2014 MAA_01 IaaCIaaC MAA_01 2013-2014

Genetic studies on synesthesia have found that synes-thetic experiences are 8 times more common among artists, poets and novelists. This increased creativity could be caused by the perception of a sense disguised as another and the ability to perceive hidden informa-tion. The basis of creativity is finding unexpected links.

The use of metaphors in language evidence our nat-ural ability to link our senses to create and describe concepts. We might say things as loud green, sharp cheese, bright sound, where we mix sound, color, shape and physical properties together. Synesthetic processes could even explain the origins of human language, ac-cording to several researchers. In a way, we are all syn-

Some kinds of synesthesia allow people to see letters in color, even thought they are printed in black and white.

IaaC MAA_01 2013-2014 28 IaaC MAA_01 2013-2014

Behaviors for new humansDesign makes behavior. Designers have to be conscious that every time we design a product, we are also de-sign how people are going to behave around it. This is a huge responsibility, challenge and opportunity.

Modern technologies, even though powerful, tend to constrain the body into not natural behaviors. Mod-ern humans suffer from diseases related to the lack of movement and forced sedentary live styles. Future technologies will take a human form so that instead of limiting its potential, it will amplify it.

How should technology attach to the body?The picture shows the work of Heather Hansen and how she creates several drawings based on the body move-ment and proportions. New technologies can improve the human interaction by finding new ways of under-standing the relationship between the human reality and the technological interfaces.


Physical Movement Translated into Symmetrical Charcoal. Emptied

Gestures by Heather Hansen.

Neil Harbisson uses this color sensor that detects color frequency

and produces a sound in his head. This way he can listen to color.

The paintings shown are his interpretation for Für Elise by Beethoven

and Triumphal March from Aida by Giuseppe Verdi, from left to right

IaaC MAA_01 2013-2014 30 IaaC MAA_01 2013-2014

PROTOTYPE GINGA3

Choosing a context - Capoeira

Based on the previous research, this experiment was run to clarify the viability for technology to take a hu-man form, reading its natural behavior to develop new interfaces between humans and machines/computers.A context was chosen in order to define limits and direct the project.

There was a special interest since the beginning of the studio in music for interaction. This was added to the interest in the human experience and its physical real-ity. The perfect context would be that in which the body could express freely and creatively in relation to mu-sic, using all the body potential. Several contexts were considered specially contemporary dance, break dance and martial arts. The perfect combination was found in Capoeira, a Brazilian dance/martial art.

GINGA is the name of the main movement in the Ca-poeira dance/martial art.


Capoeira is a Brazilian martial art that combines ele-ments of dance, acrobatics and music. It originated in Brazil in the 16th century. When Portugal reclaimed Bra-zil, they brought slaves from Africa to work the lands in inhumane conditions. Slaves developed several fighting techniques hoping to fight their way to freedom. They trained with music trying to disguise their fighting tech-niques as a dance or a game, in order to avoid punish-ments.

Modern Capoeira is practiced as a performance dance, using the traditional instruments and songs. Even though it still uses several fighting movements, it is not a combat martial art. It is a dance that improves your body abilities and reinforce social relations.

IaaC MAA_01 2013-2014 32 IaaC MAA_01 2013-2014

Reading the bodyThe project developed a set of sensors capable of read-ing the body movement, in this case, the Capoeira dance. The traditional music of Capoeira will play and the ‘‘players’’ will start dancing. Each player will have a set of sensors. As they move following the music, the sensor will read their movements and translate them into data that will alter the base music, adding sounds or modifying it.

The resulting music and sound will alter again the be-havior of the dancer, which will react to it with new movements that will again alter the base music. In this way, there is a loop of action-sound-reaction in which the dancers not only listens to the music but make it. The set of sensors amplify the human output adding complexity to the experience. This results in an interac-tion that adds complexity to the experience.

The device amplifies the human power. The dancer listens to the music and dances, the sensors turn the dance into data, the data is transmitted to a computer that uses it to modify the music. The new music is rein-terpreted by the dancer, who adapts the movement to the new sounds. The loop is closed and flowing.


Human

Flex sensorAccelerometer

Wireless micro controller.Arduino Fio + Xbee

Computer + MAX/MSPMusic programming

Sound/Music

Flowing loop of interaction.

IaaC MAA_01 2013-2014 34 IaaC MAA_01 2013-2014

Sensor SetupThe device works by attaching a flex sensor in the el-bow to detect the extension of the arm and the act of opening the body. An accelerometer in the wrist reads the start and end of any wrist displacement in space. It reads in x,y,z axis.

The sensors send those inputs to an Arduino Fio board and turns it into numbers. The data is sent wirelessly via Xbee radios to a computer. The numbers are put into a programming language for music called MAX/MSP, which creates several frequencies with each input.

The real achievement of the prototype was to turn the body movement into data that can be used, remapped and modified as parameters in different computer ap-plications. This means that the device can be used in several fields such as music composition, music perfor-mance, dance performance, medical uses, virtual real-ity interfaces, robotics and sports.

All the technologies used to develop this prototype are open source.


First sketch.

Accelerometer Flex sensor Arduino FIO + LIPO battery

Prototype GINGA

IaaC MAA_01 2013-2014 36 IaaC MAA_01 2013-2014

Turning movement/data into sound

Numbers are information that can be used to do any-thing. Movement is turned into data using a sensor set-up. The numbers are used to produce sound and mod-ify music, through a programming language for music (max/msp).

Arduino Fio+XbeeRadio signalto computer

Flex Sensor

AccelerometerX, Y, Z axis


Data received in four channels. Shown as a wave.

GINGA - Basic movement in Capoeira

X

Y

Z

Flexion

IaaC MAA_01 2013-2014 38 IaaC MAA_01 2013-2014

Exhibitions

As part of the study programme for the Interaction Studio we had the chance to present our project in SONAR+D, a international conference for creativity and technology, designed to promote talent and business opportunities in the digital culture environment. It was also shown in the final presentation of the master in IAAC.

People had the chance to try out GINGA and the feed-back got from them was of great value. The sound of GINGA is very experimental, so as people move they change basic frequencies. People immediately recog-nized the sound changing as they moved, which proved that when technology turns into human shape we can instinctively adapt it to our natural behavior.

Several people described the sound as robotic, and sug-gested different kinds of sound like nature sounds of water, rain, or more recognizable sounds. Even though, the reaction was generally very positive and many of the visitors really engaged into the experience of listen-ing to their own body as it moved.


From the feedback and the experiences learned, this project could grow in several directions such as medi-cal applications for rehabilitation, dance performances, music composition and robotics. It could also comple-ment alteration to traditional musical instrument or even become one by itself.

MORE information at:

instagram.com/gingasonar2014

facebook.com/gingasonar2014

40 IaaC MAA_01 2013-2014

Prototype GINGA

SONAR 2014

41 2013-2014 MAA_01 IaaCIaaC MAA_01 2013-2014

42 IaaC MAA_01 2013-2014

Prototype GINGA

IAAC Final Presentation

43 2013-2014 MAA_01 IaaCIaaC MAA_01 2013-2014

44 IaaC MAA_01 2013-2014

REFERENCES

Connecting – Trends in

UI, Interaction, & Experience

Design

by Bassett & Partners

youtube.com

2014

What color is Tuesday? -Exploring Synesthesia

ted.com

2013

I Listen to color Ted Talk by Neil Harbisson

ted.com

2012

How architecture helped music evolveTed Talk by David Byrne

ted.com

2010

The 4 ways sound affects us

Ted Talk by Julian Treasure

ted.com

2009

When Senses Collide

-Synesthesia Origins

BBC

2008

Software (as) artTed Talk by Golan Levin

ted.com

2004

45 2013-2014 MAA_01 IaaCIaaC MAA_01 2013-2014

IMAGES REFERENCE

Ear by Jerry Wang

Page 25

The Noun Project

Eye by Jean-Philippe Caba-

roc

Page 25

The Noun Project

Nose by Rachel Healey

Page 25

The Noun Project

Hand by Maico Amorim

Page 25

The Noun Project

Mouth by Sergi Delgado

Page 25

The Noun Project

SynesthesiaPage 28

wikipedia.org

Emptied Gestures By Heather Hansen

Page 30

heatherhansen.net

I Listen to color By Neil Harbisson

Page 30

ted.com

CapoeiraPage 32

By Jacek Dyląg

46 IaaC MAA_01 2013-2014

Special thanksCarlos Gómez

Xavi González

Guillem Camprodon

Apostolos Mouzakopoulos

John Giraldo

Giombattista Areddia

Rodolfo Parolin

Pablo Marcet

Raphael Libonati

Michele Braidy

Mary Katherine Heinrich

Esther Marquez

Carmen Aguilar

Daniel Giraldo

Marina Castan

Rhys Duindam

Tomico Plasencia

IaaC MAA_01 2013-2014

www.iaac.net

Accredited by:

www.upc.edu

fablabbcn.org www.valldaura.net

valldauraSelfS ufficientLab

The dancer becomes music. Ginga is a wearable technology that turns body movement into sound.

Our body is the result of thou-sands of generations of intense evolution. When we use mod-ern technologies our bodies are reduced to a chair and a finger click. Design creates behav-iours. A new approach is need-ed to develop the technologies of the future, starting from our physical reality, giving emergent technologies a human shape.

Prototype Ginga is an experi-ment conceived in Capoeira, the Brazilian dance/martial arts.

Documents

Prototype GINGA