1

2

3

Summary Acknowledgements Introduction

1.1 Motivation 1.2 Project goal

1.2.1 Aim 1.2.2_Minimum_requirements

1.2.3 Possible further enhancements 1.2.4 Project scope 1.3 Methodology 1.4 Project Schedule & Budgets 1.5 Creation of a prototype

2 Requirement Specifications

2.1 Software Requirement 2.2 Hardware Requirement 2.3 Functional requirement 2.4 Non-Functional requirement

3 Implementations 3.1 Performance 3.2 Motion Detection 3.3 Settings 3.4 Programming 3.6 Interactivity

4 Testing & Debugging 5 Problem Disclosure & Solving 6 Research & References

4

Summary This project is going to implement computer vision and motion graphic interacts with dancer. Seems programming and Stage performance are difficult to combine together. In previous year, artist experiment the possibility of combine program and performing art together. However, technical problems limited their creation. It also makes general public hard to understand the output of interactive performance. This project presents the new generation of stage performance to professionals, art lover, and general public. They will enjoy such innovative performance, and thru to know more about New and Interactive media art.

Acknowledgements This project would not have been possible without the help, one way or another, of the following people. Prof IP, Ho Shing Horace my project supervisor for his continuing support throughout the project’s life cycle. Dr. WONG, Kam Wah, my project assessor for her feedback on my mid-project report and during the progress meeting of this project. Ms. Koala Yip, for her valuable advice on stage performance. My family and friends.

Introduction 1.1 Motivation Performing art is a traditional media. Interactive media is used as communicate human and computer. This project is trying to investigate and improve the aesthetic value of programming on stage. Showing potential ability of interact with stage performer and director, It trys to integrate program into performance stage. In the traditional performance no matter dancing or drama, performer has to prepare. In this project Performance can be entended the meaning by different kind of media, especially programming. 1.2 Project goal In this section the aim, minimum requirements, possible further enhancements and scope of the project are given. This section acts mainly as a reference point. How the above were derived will become clearer in the oncoming sections of the report.

1.2.1 Aim The aim of this project was defined as being: “To create and implement programming into stage performance, discovering the possibility of the Real-Time communication between performer, programmer and audience.”

1.2.2 Minimum requirements In order to realize this aim, the following minimum requirements were derived:

• To direct live performance with dancer • To manage stage production suitable for live performance • To develop a software tools which will construct a virtual space by allowing dancer to interact • To carry out the communication between dancer and program • To report on the processes of developing, testing and evaluating the live performance.

5

1.2.3 Possible further enhancements Further enhancements to this project were focused around the live performance that would be created. During the requirements analysis of the project, requirements were prioritized. Some of the requirements were deemed as essential in order for the project to succeed an others were deemed as non-essential. When the essential requirements were implemented, focus turned to the non-essential requirements and some of these were implemented. A list of which non-essential requirements were implemented and which were not will not be given here due to the number of non-essential requirements and because such a list at this stage would be out of context. However, in order to give an idea of the nature of any enhancements it can be said that possible further enhancements for this live performance fell into one of the following categories: a) They were oriented in providing interaction between audience, dancer, and software tools. b) They provided hardware item into the stage. For example, various lighting, moveable backdrop, which do software tools control.

1.2.4 Project scope The data for this project contains position and gesture form computer vision, motion graphic design, Sound design and choreography (movement design). The live performance management includes casting, show control, motion tracking hardware ,software research ,event planning and investigating the possibility of implemention program in lve peformance .

1.3 Methodology Choosing a suitable methodology for a software project of this type is imperative. This, the author Knew from the outset and this definitely became more apparent as the project progressed. The main criterion when deciding upon development methodologies was a methodology that would be the priority of each part of the project scope. Due to the main idea of the project is the communication of dancer and computer. The first priority is research for motion detection. At the same time the project prepare the story broad and movement. Program and live performance are developed synchronal. The project will use dancer action (speed, moving direction, angle of their four limbs, jump), performer position on the stage as the interaction mode. E.g. if performer punch in the center of the stage, computer generate an explosion in center of the screen. Furthermore the project wants to use the distance between two performers to interact with computer. Response (output) Visual The project would like to use both abstract graphic such as ink or solid graphic such fish, 水袖 and addition shape of the dancer shadow. Sound Music sample from difference song and loops For example, the program defines separated part of the stage. When dancer move to different part that will cause sound looping 4 circles.

6

1.4 Project Schedule & Budgets

1.4.1 Schedule 2007 Dec

Program development 2008 Jan Casting Confirm Storyboard Print ad Production 2008 Feb Choreography 2008 Mar Stage Management Studio testing 2008 Apr On Stage testing 2008 May Report / Presentation / Oral Defense Rehearsal On Show 2008 June Follow up work On stage Preparation 2008 July Exhibition Setting Exhibition

Task Duration (month) Motion detect application development

3

Motion detect device research

1

Stage management 2 Music composition 1 Visual and sound element development

3

Story broad 1

7

1.4.2 Budgets & Recourses Item Quantity

IR Light (I cant found the ID form

production center) 6

VEM Ppanmin PANASONIC MULTI

MEDIA PROJECTOR (PT-P1SDEA)

WITH REMOTE X1, VGA CABLE X 1

PT-P1SDEA

3

IR Pass filter 3

MiniDV 3

Computer Laptop/PC 3

WIFI router 1

Century Stand 8

Projection Screen 1

Costume 2

8

1.5 Prototype

1.5.4 Story Broad Hexic Actor: a Girl A Boy Introduction Program comes from Chinese philosophy, 1 comes 2, 2 comes 4, 4 comes 8 comes the world, that's what are the world's structure. Elements go where they come. Also disappear from it. Scenes 1 Time morning Location forest: Girl weak up, dancing is her hobbit. She is happy when she is dancing. She dances on the mountain, forest and the lake; she dances with wind and rain. Her dancing makes birds sing and follow. However nobody can share her happiness. She feel sad because she always alone. She is alone. Scenes 2 Time sunset Location River One-day girl pass a river, she found interesting. The shadow from the water can follow her movement, this moment girl is happy, she comes to the river/wall everyday. Someone can dance with her. But she discover her shadow only follow her. She feels sad again. Scenes 3 * Time afternoon Location River Girl still goes to river/wall for her shadow; something special power makes her shadow become live. Shadow has I own mind. Evolution, a kind of connectivity between shadow and girl, when times go by, shadow has his owns shape, a kind if boy's model. Girl is happy. They create their own thing on the river/wall. Scenes 4 finale Mid night Location River However girl are not satisfying enough. She wants boy become real and come to her would. Boy rejects. Girl uses force. She breaks the wall/river. Pull boy out, boy becomes a human. He is not a spirit anymore. They can come together. When they stay longer. Boy disappear more. Boy only can exit in his world. He cannot go back to his world. Girl tries to stop it. But it is too late. Last boy only leave a bit of light. And disappear in girl's hands. Girl dies together.

2 Requirement Specifications 2.1 Software Requirement 2.1.1 Development tools At the very beginning, Opens and InfoMus Lab’s EyesWeb, after researching the technique and theory will cost the project for long time on coding. Also balancing the project scope and aims, the project chooses the following development tools. Software detection: Processing + Java language + JMyron Library Processing is an open source programming language and environment for people who want to program images, animation, and interactions. Students, artists, designers, researchers, and hobbyists for learning, prototyping, and production use it. It is created to teach fundamentals of computer programming within a visual context and to serve as a software sketchbook and professional production tool. Artists and designers develop processing as an alternative to proprietary software tools in the same domain. (http://www.processing.org ,05/06/2008)

9

Myron is the cross-platform, cross-language, open source, video capture and computer vision plug-in. One core C++ object gets cross-compiled as a handful of high-level language "wrapper" libraries. The wrapper for Java and Processing is called JMyron. The wrapper for Macromedia Director is called WebCamXtra. The aim of the project is to keep computer vision free and easy for the new media education and arts community.(http://webcamxtra.sourceforge.net/ , 05/06/2008)

Software Computer Graphic OPENGL library

Computer vision technique Frame differencing, Color detection, Edge detection Sound analysis and MIDI Generate jmetude, sonia sound library for processing

Core development environment Processing 135Beta Computer vision library JMyron library

Music and loops composition Apple Soundtrackpro Texture design Adobe Illustrator Texture design Adobe Photoshop Draw Stage Set Goggle Sketchup Draw Gant chart Excel

Technically speaking, the project process both computer vision , computer graphic ,sound analysis and sound generate. Due to this is one-man project, C++ and OpenCV development too will spend lot of time on program developing. Thus processing plus various java and processing library and increase project productivity. It is lightweight and open-source, it is available to cross OS platform.

2.2 Hardware Requirement Actually computer vision technique can handle most of the situation and environment. The problem or hardware is that the camera which capture the dancer in front of screen. Consequently camera also capture the projection to computer, feedback will arise. To prevent this situation, the project chooses ir technology to filter visible light. Helping Computer capture infrared vision. Fire wire has length limited. However RCA cable can deliver video signal for long distance without signal drop. Before input video into computer, converting analog signal to digital do by fire wire adaptor. Those hardware can decrease workload of computer calculation. Thus improve interactive

Hardware Inferred light inferred pass filter RCA cable to fire wire adaptor Projector Camera Speaker Project screen(infrared absorb surface)

2.3 Functional Requirement Vector3D public float x; public float y; public float z;

10

Vector3D(float x_, float y_, float z_) Vector3D(float x_, float y_) Vector3D() +setX(float x_) +setY(float y_) + setZ(float z_) + setXY(float x_, float y_) + setXYZ(float x_, float y_, float z_) + setXYZ(Vector3D v) + float magnitude() + Vector3D copy() public Vector3D copy(Vector3D v) +add(Vector3D v) +sub(Vector3D v) +mult(float n) +div(float n) +normalize() +limit(float max) + float heading2D() + Vector3D add(Vector3D v1, Vector3D v2) + Vector3D sub(Vector3D v1, Vector3D v2) + Vector3D div(Vector3D v1, float n) + Vector3D mult(Vector3D v1, float n) + float distance (Vector3D v1, Vector3D v2)

2.4 Non-functional Requirement The team developing Hexic is fully aware that a stage performance purpose, as a result, one of our first Nonfunctional Requirements would be performance of main program. Instead of releasing software that is slow and buggy, the project is committed to making the release version of Hexic without any bugs that would interfere with the instance reaction of the software in any way. Also, the project had determined that the software must be high quality of graphic design, the project would design for many kind of motion graphic. Sound design is the concern from the start of Hexic’s development, the project also design for many kind of loops and mix many song for the show. Lastly, Hexic is based heavily on its show control, therefore the project planned to control no only software but also hardware.

ParticleSystem

ArrayList particles; ParticleSystem(int num,Vector3D v) + run() + addParticle() + addParticle(Particle p) boolean dead()

11

3 Implementations 3.1 Performance

3.1.1 work flow

Scenes Sound/Song Duration Start time Acting Program

1.1 Generate sound 0:01:00 0:01:00 Weak up

1.2 Piano solo 0:01:00 0:02:00 Miss lover Ripple Myron

1.3 Teardrop 0:00:40 0:02:40 Upset Shadow

2 Violin 0:01:20 0:04:00 Trap in the box

String and Myron glob Box

3 Human (Calexico) 0:01:30 0:05:30 In fantasy

Edge + string

4 Human 0:01:30 0:07:00

Back you reality and depress Water fish

3.2 Motion Detection Although the project can pre-render or self create image. The project trys to connect program and live performance. JMyron does help the program capture the real time object. JMyron can output a sort of data from the real time image. Through the data program can get the dancer position (x,y,z),speed, and movement gesture each of the x-axis, y-axis, and z-axis . According these data can form more three-dimensional of the stage.

12

3.3 Setting ISO view

Top view

13

Stage design diagram

3.4 Programming The program capture the real-time movement form each camera. Computer gets the x-axis, y-axis, and z-axis image separately. Then calculate useful data from JMyron afterward sending the data through network library to main computer. Main computer use such data to generate graphic also it uses sonia library analysis input song/ music.

3.5 Interactivity While dancer perform on the stage, all of the movement can generate scenes related effect, it can be sound , computer graphic, collisions. Furthermore ,at specific scenes , dancer needs to fulfill some pre-set statement to trigger the event. Same-time director can control the character inside the virtual world. Through these director can interact with dancer. Director can communicate with dancer through the program. Also Dancers can interact with each other , in one-sconce dancers perform with others silhouette.

14

4 Testing & Rehearsal

4.1 Testing In this section , aim to test the integrate with movement and computer performance.

The first prototype is very base techniques apply into the movement. The outcome in not very well. No scenario, only have simply motion tracking with some fly and bubble appear. After intrim report the computer software waes designed base on the story board.

15

Choreographic This is the most important part of this project, What has this part done is directing the choreographer design the movement base on the story broad, After that, we discuss the possibility of the interaction, what new feature we can add into the dance. Should we use lifting? Should we dance outside the stage? Which kind of movement is suitable for the show? As a result we decide the dancer will dance with a virtual character that will combine with another dancer and computer effect. Reference: http://www.vimeo.com/920818

16

4.2 Rehearsal & onShow scene1

scene 2

17

scene 3

scene 4

18

scene 5

19

finale

20

5 Problem Disclosure & Solving 5.1 Venue The project aims to produce a live , real-time performance, the biggest consideration is space for performance. This tried to deal with this limitation at the very beginning. Venue priority was Osage Gellary,Fringe Club, Y-space dance Studio, CCDC Studio, and SCM TV studio. However first 4 venues are not available at ideal period. Last performance place was chosen to be SCM TvStudio. TV Studio is not the idealess place for dance, include lighting , and floor sound system. However this is impossible to fight for dance space from other organization. Also testing place is other difference from the project. Program development can be done in indoor laboratory. For the actual environment is totally different from desktop. Enough spaces can help the project do smoothly.

Solving To prevent waste time on fine tune program while rehearsal and before final presentation. The project demonstrates some of the situation of the stage environment. Part f the scenes have been executed several time. Also recording the on stage reference video for in laboratory testing. According to the reference video we can test the effect and the program many and many times.

5.2 Hardware To prevent projection screen feedback and lighting affected, we use infared pass filter to filter visible light.When demonstrating the hard ware, we found that one infrared light is not enough to light up all the stage. In the final presentation, the project will use 4 or more infrared to assist to light the stage. This avoids capture image light flare. Firewire has length limitation , however this helps to improve program performance. Comparing with USB webcam and CCTV. Fire wire has internal chipset, which can reduce system workload. In a Stage performance can not have any fatal error. The solution is link up RCA and Fire Wrie by 150DVCAM for fire-wire adaptor(3.3 setting “Stage design diagram”). Because RCA can have very long length without signal drop.

5.3 Content After intrim report , we discover only graphical element appear is not enough. the solution is designing the narration first, after that discuss with choreographer what should the movement be, last base on the movement the design the sonic and graphic elements.

21

6 Research & References

6.1 References

6.1.1 book references Laban Movement Studies and actor training [electronic resource] : An experiential and theoretical course for training actors in physical awareness and expressivity. Author: Casciero, Thomas. Processing: a programming handbook for visual designers and artists / Casey Reas, Ben Fry. Author Reas, Casey. Publisher Cambridge, Mass. : MIT Press, c2007. ISBN 9780262182621 (hardcover : alk. paper) Works. 1998. L'oeuvre musicale [sound recording] / Pierre Schaeffer. Author Schaeffer, Pierre, 1910-Publisher Paris [France] : INA-GRM ; Albany, NY : EMF ; [S.l.] : distribution, Musicdisc, p1998. On sonic art / by Trevor Wishart.Author Wishart, Trevor. Publisher Amsterdam : Harwood Academic Publishers, c1996.Edition New and rev. ed. / edited by Simon Emmerson.

6.1.2 online reference Drop Spin Fade http://www.youtube.com/watch?v=XOWyAlgfMFY Shadow Monster

http://www.worthersoriginal.com/viki/#page=shadowmonsters a string http://tokage.cafe24.com/string.htm TENORI-ON http://www.global.yamaha.com/tenori-on/ http://www.youtube.com/watch?v=_SGwDhKTrwU Chris O’shea http://www.chrisoshea.org/projects/muon/video/

6.1.3 Programming Reference http://www.flight404.com/version8/index.html http://www.processing.org http://webcamxtra.sourceforge.net/ http://en.wikipedia.org/wiki/Flocking_%28behavior%29 http://en.wikipedia.org/wiki/Ripple_%28fluid_dynamics%29 http://falstad.com/ripple/ http://www.fliht404.com/blog

6.1.4 Music and sound Reference Human (Calexico Mix) –Goldfrapp Human -Goldfrapp Teardrop -Massive Attack Track 04 -Buffalo66 soundtrack

22

6.2 Research 漂移: Interactive media creative team (Shanghai and Singapore Arts Festivals)

6.3 Observation Osage Siren 藍胡子城堡 http://www.kungmusic.net/