1

ImmersivePipelineresearchreport

byDr.BlancaRegina.June2018.

LastreviewJan,10th2019

SupervisedbyProf.AtauTanaka.

Abstract

Thisreportexaminesimmersiveexperiencesinthecontextofsocialsharedspaces.ItpresentstheImmersivePipelineresearchprojectledbyProf.AtauTanakaandfundedbytheAHRC/EPSRCResearchandPartnershipDevelopmentcallfortheNextGenerationofImmersiveExperiencesfromJanuarytoJune2018.Itcoversahistoricalandtechnologicaloverviewofstateoftheartonthisfield,casesofstudyandinterviewsindepthwithsomeoftheparticipants.

https://vimeo.com/303504093

2

Tableofcontents

1.Introduction:WhatisanImmersiveExperience?

1.1Spatialcinema,introductiontotheelementsofasharedimmersiveaudiovisualexperience.

1.2FromsingleuserVRtosharedimmersiveexperience. 2.ImmersivePipelineresearchobjectives. 3.ImmersivePipelineresearchdevelopment.

3.1ImmersivePipelineworkshop–February12-13th,2018,Goldsmiths,London.

3.2ImmersivePipelineperformanceeventatS.I.M.L.–April12th,2018,Goldsmiths,London.

3.3VisittoHexadome,anewimmersivefacility–April,2018,MartinGropiousBau,Berlin.

3.4CreatingS.I.M.L.SimulatorandpresentationatSpliceFestival-May2018,RichMix,London.

3.5Casestudies:conversion/translationofformats.

3.5.1FromCineChambertoS.I.M.LwithNautHumonandBlancaRegina.3.5.2FromSatospheretoS.I.M.LwithLillevan.3.5.3FromS.I.M.LtoSatospherewithAlbaCorral.

4.Conclusions.

5.Addendum.

5.1InterviewtoTarikBarri.

5.2Weblinktowebsitedocumentationandfilmoftheresearchproject.

http://sonics.goldsmithsdigital.com/immersive-pipeline

3

1. Introduction:WhatisanImmersiveExperience?

Soundandimageconstantlysurroundus,theirpresenceisubiquitousthereforeweareallpartofanimmersion,whetherornotweareawareofit.Immersiveexperiencesintroduceaudiencestodigitaljourneyswheretheyaretransportedthroughanysortofdevicetoanykindofcontentfromforexample,asocialmediathreadtoaneducationalscienceproject,anarchitecturalsimulation,afilm,anartinstallationoramediatedperformance.Immersiveexperiencestakeonformsofaudiovisualcommunicationconfiguringmediatedenvelopedenvironments.Fromexperiencesontelephonestomultichannelscreensinphysicalorvirtualspaces.IntheImmersivePipelineproject,weinvestigatedthedifferentkindsofimmersionandnarrativityandaskwhethertheyareindividualorcollectiveexperiencesandhowareandcanbetransportedfromonetoanotherscenario.

1.1Spatialcinema,introductiontotheelementsofasharedimmersiveaudiovisualexperience.

Whiledigitaltechnologiesarerelentlessinpursuingnovelty,acenturyofdevelopmentinthehistoryoffilm,expandedcinema,andtheevolutionofnarrative,haveintroducedaudiencestoanumberofimmersiveaudiovisualstorytellingprojectsandspatialcinema.

Ontheseexperiences,wehavewitnessedconstantformatchangesinmusicandvisualartsandthetransformationandadvancesinthehistoryofdigitalvideoandsound,linkedtotheevolutionofcomputingdevelopments.Theseinterrelatedpracticesandmediatedformsofpresentationconnectwiththepre-cinemainventionssuchasdiorama,panoramaandthemultimediaperformancesbyGeorgesMéliès‘phantasmagorias’developedattheendoftheXIXcenturythatledtomanydevelopmentsinnarrative,filmlanguageandtechnology.

Throughoutthelastcentury,wewillwitnessthetransformationfromthemechanicaltothedigitalera,enteringintheXXICenturywithanewtechnologicandmedialandscape,thedigitalandimmersiveculture.

BuckminsterFullercarriedoutearlyexperimentswithdomesandimmersivearchitectureinthe60’sandwrotetheprologuetoGene’sYoungblood,ExpandedCinema(1970).Thisseminaltextintroducestheconceptsofintermediaartsandmultimediacultureasnewartisticcontextsforfilmandvideo.TheExpandedArtsDiagram(1966)byGeorgeMaciunasincludesExpandedCinemaandmanyinterrelatedartformsfromverbaltheatretoanti-art,linkingcinemaandotherart-forms.TheseideaswerelaterexpandedattheLondonFilmmakersCooperativeintheUKsincetheearly70’sonwards.

AccordingtoDrKarelDoing1,fundamentaleventsonthehistoryofimmersiveexperiencesare‘TheVortexConcerts’heldbyJordanBelsonandHenryJacobsin1957to1959attheMorrisonPlanetarium,SanFranciscoandStanVanDerBeek'sMovie-DrometheatreatStonyPoint,NewYorkin1965.StanVanderBeek(1927-1984),whowasanamericanstudentofBuckminsterFulleratBlackMountainCollege,imaginedthattheatreswouldbeconnectedviasatellite1 Karel Doing presentation at Immersive Pipeline Workshop, from Expanded Cinema to the Immersive Pipeline, April 2019, Goldsmiths London

4

transmissionandthatimageswillbesentacrossthenetwork,connectingpeopleallovertheworldhelpingthemtounderstandeachotherandeliminatingthereasonforwar.Pioneersofimmersiveexperiences,VanderBeekcreatedthe‘Movie-Drome’,adometheatrebuiltbyhimselfwithmultipleprojectorsthatsurroundedtheaudience.Thenoveltyofthesituationrequiredhimtogivedirectionstotheaudienceexplainingtothemwhatwillbethebestwaytoexperiencetheevent.Atthe‘Movie-Drome’theaudiencewillbelyingonthefloorwiththeirfeettowardsthemiddleofthespace.Vanderbeekwasgoingaroundthespaceperformingwithslide-projectionsthatcompletedthemulti-projectionexperiencewhilehewasalsodrawingonoverheadprojectors.Thispositioningoftheaudienceinthespaceintroducedanewconceptofexperience.Thesesamedesignideaswillsetaprecedentandafield,developedfurtheratthecurrentimmersivespacesaroundtheworldandinourpresenttimes.

1.2FromsingleuserVRtosharedimmersiveexperience.

Notionsofrealityandspacehadbroadenedexperientiallythroughdigitalmedia.Accordingtothepioneerinmediastudies,MarshallMcLuhan,oursensesandexperiencesareexpandedbythemediathatsurroundusandthatthrougheverydayuse,ourperceptionandconsciousnessincludesthesemediationsnaturallyandas‘byosmosis’transferredtonewgenerations.

Withcomputer-basedstereoscopicdisplaytechnologieshaveappear,virtualreality,augmentedreality,mixedreality(aswellasmediatedrealityanddiminishedreality2andothernewterminologiesthatemergeasnewtechnologiesarise).Theserelatedconceptshavebeendevelopedtothepointthatthelimitsthatseparatethembecomeincreasinglyinvisible–theyrepresentformsofimmersiveexperience.Asdigitalmediaobjects,theycanexistinvariableforms,potentiallyininfiniteversions.3Thesedefinitionsaredifferentpointsalongaspectrumofimmersivecomputing,spanningdigitalandphysicalspace.

Theimmersivecultureconnectswithnewaudiencesexperiencingmultimediacontentathighspeed.VRheadsetsprovideanindividualsimulationoftotalimmersionbydeliveringhigh-resolutionaudiovisualexperiences.Inisolation,anyreferencetothephysicalspaceistransposedtothevirtualspace,body,spaceandtimeawarenessdecreasesasthereisn’tanycontactwithanyexternalandenvironmentalfactors.Thisisolationisconnectedtotheuseoftechnologyonasingleuserexperiencesetting.

Incontrast,theproductionofimmersiveexperiencesinaphysicalaudiovisualenvironmentconnectsuswiththesocialandvisceralexperienceanditscollectiveexpansionthroughtechnology.Theeventbecomesofasharedimmersiveexperiencewithmultichannelvideoand

2 - VR/ Virtual reality creates a simulated world once ones place the VR headset or eyewear. An experience that transports the user to a different place that is generated entirely by technology. - AR/ Augmented reality overlays generated images or video on top of the reality. Is integrated into video games, marketing strategies in the form of catalogue apps... - MR/ Mixed reality (MR) combines both generated images and real-world object, providing a real-world presence to intelligent virtual objects with human interaction.

3 According to Lev Manovich the principles of New Media are numerical representation, modularity, automation, variability and transcoding. The Language of New Media, 2011, The Mit Press, Cambridge, Massachusetts.

5

surroundsoundcomposition.Inthesevenues,wecanenjoywithouttheneedforVReyewearorheadphones.

Therearenownumerousimmersivecinemaprojectsandaudiovisualperformancescreatedforsurroundcinema.IntheUK,examplesofthisspacesaretheIMAXcinemas,whichscreensizeiscomparativelymoresignificantthanaregularcinema.Othervenuespresentevenbiggersurroundscreensandsoundsystemsfacilitiesincludingspaceswithroundedvaults,forexample,thedomeattheNationalSpaceCentreinLeicesterandInternationalSpaceInnovationCentreintheUKwith9.5Kand28soundsystemchannelvideowallandsatelliteapplications.

Forthelast20yearswehavewitnessedthecreationofneweducationprogrammes,audiovisualcompanies,venues,festivalsandnewaudiencesforthedigitalmediaenteringintherealmofnewmediaimmersivecontent,languageandlegacyoftheelectronicmediaandculture.Newtechnologiesareconnectedtonewnarrativesshapingthedigitalcultureandfuture.Theunderstandingofthisculturalshiftanditsimplicationsisvitaltounderstandhowtheycanhelpdrivesocialandeconomicprosperity.Immersivetechnologiesareconnectedtothesharedimmersiveandsocialexperience,bringingtheseaudiovisualimmersionstoaphysicalspaceandasacollectiveexperienceratherthananindividualoneusinggogglesorscreendevices.

2.ImmersivePipelineresearchobjectives.

TheImmersivePipelineresearchproject4wasfundedbytheAHRCundertheNextGenerationofImmersiveExperiencesprogrammeandlookedattheproductionofpipelinesandtranslatorsfortheauthoring,sharing,andtouringofimmersiveaudiovisualperformativeartworks.

Partnersoftheprojectincludedthesomeofthemostrelevantandpioneerinstitutionsinthisfield,SociétédesArtsTechnologiques,InstituteforSoundandMusic,RecombinantMediaLabs,SonicWomb,GarageCube,publisherofMadMapperandModul8,Derivative,publishersofTouchDesigner,NSCCreative,UKdistributorsofFrontPictures.

Theresearchprojectaimedtocreatecontentproductionpipelinestoallowartiststoconcentrateonthecreativeaspectsoftheirimmersiveworks,andfacilitatetheconversionandtranslationofexistingworksfromoneformattoanother.Theprojectgoalswerealsotoprovidearesearchplatformandtoanswerthefollowingmainquestions:

• Canweidentifybestpracticeinimmersiveproductiontechniquestodefineanauthoringworkflowfornewimmersiveaudiovisualperformanceworks?

• Canwedrawuponthewealthofexistingpracticeforittodrivetechnologydevelopment,insteadoftechnologydeterminingcreativity?

• Canwecreatetechnologiesforthetranslationofcreativeworkfromoneimmersiveformattoanotherinsuchawaytomaintainoriginalcreativeintent,andliberateworksfrombeinginadvertentlysitespecific?

4 Immersive Pipeline Research project http://sonics.goldsmithsdigital.com/immersive-pipeline

6

Theprojectraninthesix-monthperiod,JanuarytoJune2018,andconsistedofaworkshop;aperformanceevent;sitevisitstopartnersites;festival/conferencepresentations;andthedevelopmentofhardwareandsoftwareprototypes.Theprojectconductedethnographiesofprominentartists,curators,andtechnologies,gatheringdataandtestimonialsfromnationalandinternationalexpertsonthefield.TheprojectresearchwascarriedoutintheEmbodiedAudiovisualInteraction(EAVI)researchunit,anditseventstookplaceintheSonicsImmersiveMediaLabs(SIML)facility,bothatGoldsmiths,UniversityofLondon.

3.ImmersivePipelineresearchdevelopment.

3.1ImmersivePipelineworkshop–February12-13th,2018,Goldsmiths,London.

Ascopingtwo-dayworkshopwasheldonFebruary12-13th,2018Goldsmiths,Londonandgatheredprojectpartners,nationalandinternationalacademics,artistsandpractitionersspecializedinaudiovisualexperiencesandimmersivemedia.Thesessionspresentedhistoricalandtechnicaloverviews;descriptionsofcurrentandbestpractices;mappedoutinternationalnetworksconnectingeducation,researchcentres,andindustry.Inadditiontotheprojectteam,coreparticipantsincluded:LucCourchesne(SAT),NautHumon(RmlCineChamber),DrKarelDoing,NickMeehan(Hexadome),PaulMowbray(NSCCreative),FrancoisWunschel(MadMapper),ProfJulianHenriques(Goldsmiths)andDrBlancaRegina.Inadditiontoworkshoppresenters,discussionsessionsincludedtheparticipationofstudents,artists,andprogrammersinattendance.

DrKarelDoing5introducedthehistoryofExpandedCinemaasaprecursorofthecurrentwaveofimmersiveexperiencesandspaces.Introducingthepioneersofexperimentalcinema,Doingintroducedfilmperformanceanditstransformationsascentralelementofimmersiveaudiovisualpractices.

LucCourchesneafoundermemberofS.A.T(SocietedesArtsTechnologiquesinMontreal)presentedhislectureviaSkype,somehowasaglimpsetopioneervideoartist,fatherofvideostreamingNamJunePaik.FromtheSATinMontreal,Canada,hestartedhisinterventionbyrecountinghowhehadwitnessedBuckminsterFuller’sgeodesicspacesasachildinthesixties

5 Karel Doing’s thesis 'Ambient Poetics and Critical Posthumanism in Expanded Cinema' enquires the possible overlaps between these different fields. In conclusion he proposes new forms of humility, doubt and listening to be advanced in today's over-confident and exploitative human culture.

7

duringWorldExpoMontreal.Artistandacademic,Lucisanewmediaartistwhohasproducedmulti-screeninteractivenarrativessincethe1990s.Startingwithfourrectangularscreens,heeventuallydevelopedbespokesystemsforliveactionfilmingin360º.TheSATopsherewasbuiltin2010andisapublicfulldomevenuehousedintheS.A.T.ArtsCentre.In2014,theybegantoorganizetheannualImmersiveExperience(IX)symposiumseries.IXreferstoimmersiveexperiencedesignasaparalleltoUX,oruserexperiencedesign.ForLuc,coiningtheterm,immersiveexperience,providedtheSATwithawaytheVRcommunitywithoutbeinglimitedVR.6

NautHumonpresentedtheCineChamber7aswellasaseriesofimmersiveprojectshecarriedoutsincethe1980sand90s.Underthename,SoundTrafficControl,theybeganwithmultichannelsoundandimmersivetheatreandeventuallyaddedmultichannelimageprojection.AsRecombinantMediaLabs,Nauthasworkedasacuratorandorganizer,presentingKurtHentschläger&UlfLangheinrich’sduoGranular_Synthesisgroundbreakingwork,+MODELL5.WithCineChamber,Nautmovedfromspatialsoundtospatialcinema.In1999,theypresented,9hours9minutesand9secondsonSeptember9of1999atArsElectronicainAustria,Linzconsistingof61timedmodulescreatedby33participants.

Whenyougetalargecanvas,youreallyhavetothinkandrethinkofhowareyougoingtodealwithspatiality.Wewereworkingforyearswithspatialsoundputtingspeakerineachcornerbutinthisenvironmentyoudohavespeakersineachcornerandyoucanseeitsoyoucanworkwithsurroundsound.Butwhenyouaredealingwithspatialimagehowdoyoucoordinatethat?Howdoyoucomposethat?Whatarethechallenges8?

TheCineChamberstabilizedasaten-screenconfiguration(8mx12m)inarectangularshapewith16.8.2surroundsound.InitiallysetupinSanFranciscoithassincebeensetupindifferentvenuessuchastheatrestages,concerthalls,andauditoriumsintheUSandEuropeincludingin2011atCTM9atHausderKulturenderWeltinBerlin.Theyhavecommissionednumerousmodules,installations,performancesandworkshopsincludingworksbymorethan50artistsincludingMaryanneAmacher,EdwinVanDerHeide,RyoichiKurokawa,ChristianMarclay,Fennesz,Lillevan,MasakoTanaka,BlixaBargeld,CarstenNicolai,FrankBretschneiderandRyojiIkeda.RecombinantMediaLabshostsartistresidenciesandorganizesanannualfestivalinSanFrancisco.

6 Luc Courchesne talk at Immersive Pipeline Workshop , February 12th 2018, Goldmiths London.

7 CineChamber http://recombinantfestival.com

8 Naut Humon Immersive Pipeline workshop April 12th 2018. Director of RML Cinechamber http://rml-cinechamber.org

9 Cinechamber at CTM https://archive.ctm-festival.de/archive/ctm11/transmediale11.html

8

PaulMowbraypresentedtheworkofNSCCreative,aLeicester-basedcreativeandtechnologystudio.NSCCreativehasaccesstothelargestplanetariumdomeintheUKwith18meterdiameter360°domeenvironment,theNationalSpaceCentreinLeicester.AsmallerversionoftheDome,threemetersofdiameternegativepressurestudiodomecreatedfordemosandprototypingandisavailablefortouring.Mowbrayisco-producerofFullDomeUK,afestivalfocusedinimmersiveexperiencescreatedin2012.NSCCreativeproducescontentforVRinstereoscopic3D.NSCCreativeisUKdistributorforUkrainiansoftwarecompany,FrontPictures,publishersoftheScreenberry360.Thismulti-projectionandauto-calibrationprojectionmappingsoftwareisamulti-calibrationsystemthathasamulti-stageauto-alignmentsystemthatdoesedgeblendandintensitymapping.Thissystemallowsafastturnaroundforcomplexprojectionenvironmentlikeacurvespaceandallowstheperformerandartiststofocusontheexperienceandallowscaptureoftheimageandtransfertothesystemthatiscalibratedtothearea.Pauldescribedthatthechallengesonimmersiveexperiencesareonhowtoexpandandtodesigntoaspecificeyepoint,gettingawayfromthatavertedvisionor‘sweetspot’andhowtherearealsootheroptionsforartiststovisualize,testandprototypeaprojectinVR.Mowbrayintroducedcompositionstrategiesforcreatingvisualsforvirtualrealityprojectsandimmersivespaces

FrançoisWunshelarchitect,artistandprogrammercofounderofMadmapper10discussedtheessentialrelationshipsbetweenthebuiltenvironment,itssightandsurroundings.Hisworkisbetweenthephysicalanddigitalwork,basedontheunderstandingthatarchitectureisimmersive,asonehastogoinsideit,andisexperiencedasonelivesinit.Thedigitalelementiscombinedasanotherlayerthataddsontopofthatreality.Thereforethepixelsoftheimagecanbeunderstoodintwoways,oneasacharacterofthestoryandtheotherastheenvironment,space.Onthetoolsandelementsthatenablethese,fordesignandcreationofspace,theyworkaddinglightingtothearchitectureandvideousingvideomapping.Forthat,theydesignedthetoolMadmappertobeabletoaddressthesedifferentrealitiesatthesametimesoftwarethatallowstransformingthelightingandthevideoprojectionofaspecificspaceinrealtime.Theycanmatchthephysicalandthedigitalwithit.InarchitecturetheyuseAutocadbutwithMadmapperallowsyoutodrawinthespacedirectlyinrealtime.Regardinghisexperienceswithfulldomeperformancehepointedthefollowing:

• Immersive=allyouknowiswrong.• Immersive=constraints,asadirectaccesstocreativityovercomethechallenges.

10 Madmapper https://madmapper.com

9

Regardingaudiencestudies,inimmersiveexperiencesthemovementoftheaudienceitsdynamicnotstatic.Lookingattheideaofthecontextofplaceandhowcanweassociatethiswiththeexperienceitself.Theideaofplaceisasignificantpartoftheimmersiveexperience.Therefore,whenyouwalkintospace,thereisanexperienceexpectationsogoingintoacinema,amuseum,acommunitywherethereisahighconcentrationofvisualandsoundart.

Onthesubjectofspaceanditsimportance,NickMeehan,ArtisticDirectoroftheInstituteforSoundandMusic(ISMBerlin),11presentedthenewimmersiveaudiovisualfacilityandprojectHexadome.

Initiallyfocusedonsound,immersiveartandelectronicmusic,asthethreepointsofdemonstration,theirprojectHexadomeisanexampleofimmersiveexperienceinsitespecific,comprisingastructurewithsixscreensandamultichannelsoundsystem,presentedinMay2018wheretheycommissionedmorethantenartistsparingsoundandvisualartistsincludingpioneersasBrianEnoandTarikBarri,HollyHendonandotherseminalartistsofaudiovisualimmersiveexperiences.

ProfJulianHenriquesfromtheMedia&CommunicationsdepartmentatGoldsmithsmadetwopresentations,asciencecommunicationproject,andamulti-screengraphicalstory.TheSonicWombproject,carriedoutincollaborationwithmedicalresearchersatUniversityCollegeLondon,looksatthemostintenseimmersiveexperience,thebabygestationinthemother’swomb.Lookingatwhattheauditoryworldinsidethewombislike,theirworkfocusesonfindingawaytodemonstratewhatthebabyhearsinitially,presentingtheresultstothemedicalprofessioninordertounderstandthedamagethatisdonetothebabiesinincubators.Thisprojectisinitiallydesignedonmedicalgroundstounderstandhowtheexperienceofthenewbabiescanbeimproved.Whatwasfoundisthatthebabyhasthreechannelsofsounds.TheyinitiallydesignedtheSonicWombSoundOrbpresentinganindividualexperienceandtheirfuturelooksattheimmersivecollectiveexperience.WiththeImmersivePipelineproject,theyseektoaddavisualdimensiontothecurrentimmersiveauditoryexperience.

Dr.BlancaReginapresentedtheinitialcasestudyrelatedtotheconversionandtranslationofimmersiveprojects,fromconversionoftechnologiestoadaptationtodifferentspaces.

Thesessionincludedtwodiscussionsaroundthestateoftheart.ParticipantsincludedChrisAllen(ThelightSurgeons,VickyBennet(Peoplelikeus)NickRothwell(RavensbourneUniversity)SteveBeresford(musician),FranciscoBernardoandotherPhDstudents.

Thediscussionintroducedthenewtopicsandprovideddataforourresearchthatisdevelopedinourconclusions,

Whattoolneedstobedevelopedorprovidedinanimmersivefacility?

Whataretheconditionsandaccessibilitytoworkonthesespaces?

11 ISM, Hexadome https://berlin-ism.com/en/about

10

Ifartistsmayconfrontthesameproblems,whatcanwemakeavailablesowhereanartist?

Whatcankindofworkscanwecreateinthesefacilities?

Howquicklyyougetadjustedtothespaceasanaudience,inthedarknessandwiththesoundscomingtoyou?

3.2ImmersivePipelineperformanceeventatS.I.M.L.–April12th,2018,Goldsmiths,London.

TheImmersivePipelineperformanceeventwasapresentationonimmersiveaudiovisualperformancesatSonicsImmersiveMediaLabs(SIML),atGoldsmithsUniversityofLondononApril12th,2018.Theresearchhostedamicro-residencywithlocalandinternationalartistswhereweproducedpresentationsoftheirworkandtheirworkinprocessatasurroundedaudiovisualfacilitywithfloortobottomscreens,sixprojectorsandasurfaceareaofninemeterslongandfivemetersandahalfwide.

Theeventpresentedfourdifferentliveperformanceswithenvelopedsoundandvideoofatotaldurationoftwentyminuteseach.Eachworkpresenteddifferentaesthetic,narrativeapproaches,andmethodsforthecreationofenvelopedsonicandvisualcompositionsforimmersiveexperiences.Fromfilm-basedcontentmodifiedwithreal-timeeffectstoproceduralgraphicsandsounds,generatedbycodeandcomputeralgorithms.Thepresentationoftheworkslastedanhouranaccommodatedsixtypeopleineachpresentation.Throughouttheday,threepresentationswereshowcased,oneforacademicsinashortversionandtwoforthepublicinthehourversion.Thepublicperformancesandmicro-residencyprovidedanopportunityforartiststoreflectupontheirpracticesandmethodsandintroducedanewcatalogofimmersiveaudiovisualworkstoresearchersandaudiencesintheUK.

LillevanRechercheandAtauTanakapresentedthreeshortaudio-visualimmersivepieceslookingatthebodyasasourceofthesound,image,andphysicalexpression.Atau’ssoundperformancewasgeneratedwithasensorsystemdetectingtheelectromyogram(EMG)signal,electricalimpulsesfromthenervoussystemcausingmuscletension,andrendersamusicalinstrumenttheperformer’sownbody,allowinghimtoarticulatesoundthroughconcentratedgesture.Lillevan’sabstractmovingimagescreatedimmersiveorganicscenariosofsomenaturalphenomenawhereAtauwasframed.ThemovementofAtauproducedandmodifiedthesoundsthatwereextendedthroughthegraphicsemulatingfire,waterandgas.Lillevan’sorganiccompositionsandpoeticmetaphorswerefilmsprocessedinrealtimerespondingtotheaudioperformancewiththetoolforvisualperformance,Modul812,connectingthegraphicstothe12 Modul8 is a software for live visual performance developed by GarageCube, a company established in 2005 by Yves Schmid and Boris Edelstein, based in Geneva, Switzerland. Modul8 was started with the intention of providing a tool for VJing in openGL www.garagecube.com/modul8

11

sonicperformance.TheaudienceswereonthefloordirectingtheirsighttoAtau’sbodyandarmsthatweregeneratingandcontrollingthesoundwithhismovements.Loup,asingleshortrecordedsampleofhowlingwolvesthatarestretchedandtransposed,creatingacontinuousgrowling.Lifting,wherewhistlingoscillatorsaremodulatedinfrequencyandamplitude,recallingtheclassicalTheremin.Myogram,adirectsonificationofmuscleactivitywhereweheartheneuronimpulsesofmuscleexertionasdata.

AlexAugierandAlbaG.Corralpresentedtheirworkinprocessend(O)animmersiveaudiovisualperformancethatcomestolifeeachtimedifferently.

TheirworkatSIMLprovidedanopportunityforthedevelopmentoftheirfirstimmersiveaudiovisualworkinprogressthatamonthlaterwasexpandedandtranslatedtothedomeatS.A.T.forapresentationatElectraFestivalinMontreal,June2018.TheduowithAlbacreatingthevisualsandAlexworkingonthesoundimmersedtheaudienceinadreamlikeexperiencefortwentyminutes.Thatwasofasurroundedcanvasforareal-timepaintingthatevolvedintoamovingimagecompositionwithsound,configuringanabstractexpressionistaudiovisualimmersiveworkcreatedwithproceduralgraphicsusingaself-madetoolcreatedinProcessing13thatwasreceivingdatafromthesoundviaOSC14.

AllOSCdataarefloat,from0.to1(normalize).

OSCdatacomefromAlexsounddevices,-AbletonLive,usingaMaxForLivepatch: -Towritevisualcontrolsaccordingwiththemusic.Forexample,tochangeaglobalvisualpresetandallowsAlbatofocusonanother(andmoreinteresting)control.Inthiscase,itlookslikeanenvelope/automatization. -Towritesometriggersaccordingwiththemusic.Forexample,eachkickisconnectedwithatriggerandAlbacanuseittomakewhatshewants.Forexample,tochangethecolour…Inthiscase,itlookslikeaMIDInote.

13 Processing is an open-source graphical library and integrated development environment (IDE) designed by Casey Reas and Ben Fry for the electronic arts, new media art, and visual design communities with the purpose of teaching non-programmers the fundamentals of computer programming in a visual context. Processing uses the Java language, with additional simplifications such as additional classes and aliased mathematical functions and operations. http://processing.org 14 OSC stands for open Sound Control a digital protocol for networking sound synthesizers, computers, and other multimedia devices for purposes such as musical performance or show control.

12

-Maxpatch.AlexusesthepatchforspatializationandFXcontrol(monome,arc): -Tosynchronizevisualparameterswiththespacepositionofsound -Tosynchronizevisualparameterswithmycontrollers((asfilter,drive…).Forexample,theycouldsynchronizethesoundfilterwiththebrightness….OSCdatawassentwith60HzfrequencytobesynchronizedwithProcessingfpsandrefreshrateofscreen.Alexusesa[speedlim]objectinMaxtodothata[change0.]objecttosentonlydatathatchanging(object).Fortheartists,theconceptionofthisworkfollowedtheconceptofGesamtkunstwerk,implyingthatonedimensiondidnottakeprecedenceovertheotherbutiscompleted.TheelectronicsoundswherespatializedconnectingitsmovementtotheabstractlandscapesandcolorfulmovingimagescreatedbyAlbausingreal-timepaintinggesturesonhertablet,andchanginghercompositionwithmidisignals.

BryanDunphypresentedasoloperformancewithproceduralsoundsandgraphicscreatedinOpenFrameworks.15Withaminimalaestheticinsoundandvisualsspacewastransformedinanelectroniclandscapewithmoving3Daudiovisualobjects.Fortenminutes,theaudiencewasimmersedinacomputergraphicsspaceasifwerefloatinginanabstractvideogameenvironmentthatwasmodifiedbyBrianpulseschanginginreal-time.Followingthevisualmusictradition,Bryan’screatedgenerative3Dproceduralgraphicswiththreedifferentcolorgroupsthatrepresentedadifferentfrequencyband(low,midandhigh)Thegreentonesrepresentedthelowfrequencies,thegreytonestothemid-rangeandthepurpletonestothehighfrequencies.Thegraphicsandaudiowerenotaffectingeachotherbutcomingfromthesamesource,atrainedcomputerneuralnetworkthatdisplayedaudioandvisualoutputsthatBryanchoseanddisplayedforhisartisticpurposes.Thesoundcompositionofthepiecewaswiththeaudiosynthesisofadditiveandfrequencymodulationsynthesis.Whilsttheaudioandvisualsarenotdirectlyaffectingeachotherincode,theartisticmappingofaudioandvisualmaterialinthetrainingphasewasbasedonthephenomenonofcrossmodalventriloquismwheresimilarmovementinsoundandvisualmaterialhelptobindthetwomediaintheaudience'sperception.Diagram of the performance setup for Ventriloquy 2 by Bryan Dunphy

15 OpenFrameworks is an open source toolkit designed for "creative coding" founded by Zachary Lieberman, Theo Watson and Arturo Castro. OpenFrameworks is written in C++ and built on top of OpenGL. http://openframeworks.cc

13

ThroughouttheweekweconductedonetooneinterviewswiththeartistsinvolvedAlbaCorralandAlexAugier,LillevanandAtauTanakaandGoldsmithsPh.D.student,BryanDunphy.

Theimmersiveaudiovisualperformanceeventwasademonstrationoftheprimarygoalsoftheproject,alsofortheconversionofworksfromonespacetoanotherwithadifferenttechnicalsetup.

14

3.3VisittoHexadome,anewimmersivefacility–April,2018,MartinGropiousBau,Berlin

‘Hexadome’ifoneofthelatestnewspacesforimmersiveaudiovisualdesignedbyISM,InstituteforSoundandMusic,presentedatthemuseumMartinGropiusBauinBerlininGermany,inApril2018.Itisanaudiovisualspacewithfifty-sixspeakers,andsixscreensallsurround.ItwasdevelopedbytheISMincollaborationwithPfadfindereiandZKM,CenterforArtandMediaandincorporatestheKlangdomsoundsystem,developedbyZKM,InstituteforMusicandAcousticsTheydidnotgiveheadsetsandheadphonetotheaudiencebecausespecifically,theyweretryingtopromotetheideaoftheshareddirectexperience,theimportanceofthespaceiskeytothecreationofthework.

OntheartistsworkprocessBrianEno,theorganizerspointedhowhestartedhisworkwithafinishedcompositionandthenwhenheenteredthespaceandherealizedthearchitectureandalsothereverberationsofthespace,hestartedworkinginanewpieceandtheninanothernewpiece.Theypointedthatitisessentialtoknowortoexplorethearchitecture,spacewherethiskindofworksispresented.

Hexadome’smaininterestwastomakeanimmersivecollectiveexperienceintegratedintothemuseum,MartinGropiusBauasthecontextiscrucialforthisworksandespeciallywhatwearetryingtoputforwardwhichistoassociatethisworkswiththemuseumexperience.TheunfortunatethingisthatinmanycasesMuseumswherenotnecessarilydesignedwherethesoundisaconsideration.Sothathadtocreateastructurethatfortunatelycanworkwithinthebeautyofthespaceitself,butthathasactivecontroloverthecharacteristicsofsound.

Moreover,ontheoppositesidethewaythevisualstructurewasdesignedtonotonlyprovidetheaudiencewiththeimmersivevisualexperiencebutalsotoremindofthemajesticspace,veryornateanduniqueinitsownrightthatwasoccupying.

Sotohaveanaudiencewalkingandtohaveanexperienceofbeenenvelopedinsoundbutalsosurroundedbythiswindowsintotheworldthatthevisualartistsarecreatingforitandstillunderstandthatwearewithinthecontextofthespaceitisessential.

Onthecuratorialtheme,theycreatedaprogramme,inmultiplewithartistscomingfromdifferentbackgrounddifferentaestheticsofsound,immersiveart,andelectronicmusiccommunity.Theyworkedwithartiststhatwerelocallybased,soFrankSchneider,TarikBarri,HollyHendon,MarcelMFO,PeterVanHoesenandalsolookatarightbalancebetween

15

establishingandemergingartists.BrianEnowasinvitedbecausehisinterestinnewtechnologiesandbecausehewaspreviouslyatZKMveryimpresseddeKlangdomsoundsystemandwasanexcellentopportunityforhimtoworkwithimmersivesoundandlight

WewerealsocurioustoseeoncheckhowanartistwhoisalreadyexperiencedwithmultichannelandsoundenvironmentswillworkwithKlangdomsound

Theorganizationmissionistocreateawarenessforthiskindofworksandeventuallyfindahomeanewhomeanewmuseumexperienceforthiskindofworks.

3.4CreatingS.I.M.L.SimulatorandpresentationatSpliceFestival-May2018,RichMix,London

ThetechnicalsetupoftheimmersiveaudiovisualfacilityatGoldsmithsUniversity,S.I.M.L.comprisessixprofessionalHDvideoprojectors,screens,andamultichannelsurroundsystem.

TheImmersivePipelineteamdevelopedthecanvasresolutioncalculationsandprovidedsoftwaresetupwithamiddlewareapplicationforthegraphicsandvisuals,anadvancedtoolforvideomappingMadmapper,whichallowedartistsaccesstothemultiscreencanvasusingtheirtoolsforcontentproductionandvideoperformance.

ThisS.I.M.Lsystemallowedmultipleconfigurationswithatotalcanvassizeof7680x800pixelsbeenabletoreachahigherresolutionof11520x1200pixels.ToprovideanongoingsystemforthedevelopmentoftheImmersivePipelineresearchtheprojectcreatedtheS.I.M.LSimulator,consistingofaphysicalonascalesimulatoroftheS.I.M.L,SonicsImmersiveaudiovisualfacilityatGoldsmithsUniversity.Thisminiatureonscalesimulatorprovidesaphysicalsimulatorforresearchersandartiststocreate,performandplaytheircompositionsinrealtimewithouttimeandspacerestrictions.Thesimulatorisa1:10scalesystemoftheS.I.M.Lspaceofatotalsize1,30metersx60cm.DesignedbyDr.BlancaReginaandProfAtauTanakaithassixminiprojectorsHDwithtripods,afoldableminiatureofdouble-sidedprojectionPerspex.

ItiscontrolledwiththesamemiddlewaresoftwarethantheactualfacilityanddrivenbyMAClaptopwithtwoTripleHeadtogo,Matroxgraphicscards,providingacompletecanvasresolutionof7680x800pixelsandframerateof60Hz.

AtSpliceFestival16,May2011,ProfAtauTanaka,Dr.BlancaReginawithPhDstudentBryanDunphypresentedthesimulatortothepublic.Forthesimulator,BryanDunphyadaptedthe

16 Splice Festival http://www.splicefestival.com/line-up/immersive-pipeline/

16

codetousefewerresourcesofthelaptopcomputer,to1280x800pixels.Hisprojectconsistedofmirroringandinvertingtheselectiontotransferittotheminiatureasitwaspreviouslydoneintheoriginalspace.

3.5Casestudies:Conversion/Translationofformats

Oneofthefocusesofthisresearchhasbeenthetranslationofimmersiveaudiovisualprojectsfromonevenuetoanothervenuewithdifferentspatialconfigurationsandformats.Focusingintheimageconversion,likeinpaintingindigitalimagedifferentcanvasesexistwithdifferentsize,aspectratio,resolutionandshape.

ThespacesthatinvolvedtheconversionwereSIMLasixscreen16:10aspectratiorectangularspace,Cinechamberattenscreens,16:9aspectratiorectangularspaceandSatsosphereadomewithModularscreenconfiguration:180∞,210∞andvertical230∞by360∞,18mindiameter,11.5to13mhigh,8videoprojectors,157speakers.

Fromoneimageformattoanotherconversionpossiblesolutionsinclude:

• Stretching-tolengthenbypullingfromoneofthesidesoftheframeofthevideohorizontalorvertical

• Cropping-theprocessusuallyconsistsoftheremovalofsomeoftheperipheralareasofanimagetoremoveextraneoustrashfromthepicture,toimproveitsframing,tochangetheaspectratio,ortoaccentuateorisolatethesubjectmatterfromitsbackground.

• Theletterbox-isthepracticeoftransferringfilmshotinawidescreenaspectratiotostandard-widthvideoformatswhilepreservingthefilm'soriginalaspectratio

• Zoom–imagechangesfromalongshottoaclose-uporviceversa.

Toverifyandstudytheconversionofworksfromonespacetoanother,wefocusedonthreeexamplestoextractsomeconclusions:

3.5.1FromCineChambertoS.I.M.LwithNautHumonandBlancaRegina3.5.2FromSatospheretoS.I.M.LwithLillevan3.5.3FromS.I.M.LtoSatospherewithAlbaCorral

3.5.1FromCineChambertoS.I.M.LwithNautHumonandBlancaRegina

OurfirstcasestudywastoworkwithaseminalpiecefromCineChamber,Recombinant-FestivalcatalogandtranslateittoS.I.M.Lspace.TheselectedpiecewasfromMasakoTanaka,apanoramicpioneerofCineChamber’sformatwithasoundtrackbyOval.

17

Theoriginalprojectwasof10filmsthathadtobestitchedwithtenprojectors.EachoriginalfilmsizewasHD1920x1080pixelsandwasdistributedin10projectors.TheobjectivewastoconvertthespatialcinemaworkofMasakoTanakadevelopedforCineChamber,arectangularspacewithten-screen,tenvideoprojectorswith16:9aspectratiotoS.I.M.L,arectangularfacilitywithsixscreens,sixvideoprojectorswith16:10aspectratio.Thecalculationsoftheresolutionwereasfollow:

CineChamberCanvasSize=(1920x10)Hx1080V=19200x1080pixelsS.I.M.LCanvasSize=(1920x6)Hx1200V=11520x1200pixelsS.I.M.LfinalCanvasSizeforperformance=(1280x6)Hx800V=7680x800pixels

ConvertingCineChamber→SIML19200x1080→7680x800pixels

TheoutcomeoftranslationoftheworkwascompletedbyBlancaReginaandAtauTanakaandsupervisedbyNautHumonandBarryThrew.ThecasestudyforthisconversionhadadurationoffiveminutesandwaspresentedinpublicSIMLperformances.TheendtoolforthetranslationwasaprojectcreatedMadMapperthatinvolvedareal-timeconversionandoutputtosixprojectorswithoutanyeditsusingtheoriginalvideoandsoundsources.Thefinalconversionmethodimpliedscalingdown,stretchingwithoutcreatingadifferentialdistortionandletterboxtheoriginalcontenttodisplaythecontent.Thesoundwasstereoanddistributedacrossthefourplusonespeakersintheroom

18

3.5.2FromSatospheretoS.I.M.LwithLillevan

Asignificantdifference,bothforcreatingandviewing,inthistwospacesdependsonitsspacedesign.Thedifferentformsofviewingimplieddifferencesinimagining,creating&testingthevisualcontent.Lillevanusedsomeofthesamesourcematerialforbothvenues,adaptingfortechnicalreasonsaswellasartisticandpoeticconcerns.

S.I.M.LCanvasSize=(6projectors)

LillevanresolutionatSIMLwasof7680 × 800pixels

SatospherecanvasSize=(8projectors)

Satosphereresolutionforliveperformances=2048x2040pixels

SIMLdisplaysmovingimagewrappedaroundtheviewer-moreorlessateyelevel.SIMLisclosetoanaquarium.SIMLcalledfortherectangularbaseclip,whileallowingforcollagefragmentsofanydimensionsdesired

19

SATdisplaysvideoaroundANDoverhead.Fromtheaudiencepointofview,theexperienceismoreofacathedral-thevisitorautomaticallylooksup

Regardingtheliveperformanceaspect,SATrequiredalargesquarematerial,size&codecwerepredetermined.

ForSAThehadtopre-rendermaterial,duringtheperformancehecouldonlymixsomepre-renderedlayers,changecolors,butcouldnotactuallycreateorstronglymanipulatehismaterialduringtheperformance.

AtSIMLhecouldcreateandmanipulatematerialduringtheperformance,alsobecauseSIMLwasdevisedtoallowtopluginanduseownsoftware.SIMLprovidedamoreexcitingspaceregardinglaboratory/studio/rehearsal–asitallowedgreaterfreedomforreal-timeinteractionandexperimentationinthedaysprecedingtheperformances.

Inconclusion,forsomebodyfocusedonmovingimageryintheworldofsingle-channelvideo(16:9or4:3)theimmersivespacespresentanincrediblerevolution;wearenolongerlimitedtoafixedframeaspect.However,thenewfreedomcanonlybeexplorediftheartisthasplentyoftimetoplay,experiment,make“mistakes”,findanewmovingimagelanguage.Ofcourse,wecanenjoymerelydistributingourimagesaroundtheroom,butthatwouldbetoolittleandtoosimpleforthesenewimmersivepossibilities.

Inimmersivespacesandaudiovisualsystems,wedonothavearealstandard.Wearestilldiscoveringnewwaystodothis.Therearequiteafewplacesintheword.Eachplacehastobecalibratedandconfigured.Thisisanewformofpresentation,anewlanguage,abigimmersivepoeticmoment.Wehaveaveryhightechspacewherewearedoingreal-timeorganiccompositions.17.

3.5.3FromSIMLtoSatospherewithAlbaCorral

AlbaCorralpreparedEnd(0)animmersiveaudiovisualperformancewithAlexAugierinApril2018atSIML,London,arectangularfacilityofsixprojectorsandsixscreensfloortosealing.AcoupleofmonthslatertheytravelledtoSATMontrealandadaptedtheworktoadomefacility

17 Interview to Lillevan Recherche by Blanca Regina at a Goldsmiths University April 12, 2018. ‘Lillevan is an animation, video and media artist. He is perhaps best known as founding member of the visual / music group Rechenzentrum (1997-2008). Lillevan, has performed and collaborated with many artists from a wide array of genres, from opera to installation, from minimal electronic experimentalism to dance and classical music; performed and exhibited all over the globe, and at all the major media festivals’ http://www.lillevan.com/biography/

20

atSAT,Montreal.ConvertingSIML→SatosphereS.I.M.LCanvasSize=(6projectors)AlbaresolutionatSIMLwasof3840 × 700pixelsSatospherecanvasSize=(8projectors)Satosphereprovidedtworesolutionoptions:Finalresolutionforliveperformances=2048x2040pixelsFinalresolutionforscreenings=4096x4096pixels

Bothspacesaredifferent;thedomeisaroundedvaultformingtheroofofabuildingorstructurewithacircularbaseandSIMLisarectangularspacewithoutascreenonthetop.TheSatsospherehasavariableconfigurationonitsspaceallowingthreedifferentinternallayouts.Thegeometrychangesinbothspacesastheyaredifferent,fromasquaretoasphereorsemi-sphere.Thechangeofshapeitsessentialthereforethecompositionchanges.Inthedome,youplaywiththetoppartthatcannotbeexploredintherectangularspace.Inthedome,thetopandthehorizonareessential.Thehorizonisthesameinbothspaces.Forexample,thecontentofthelinesworksinbothprojectsfortheverticalmovement.TechnicallytheresolutionthatAlbausedwasdifferentaccordingtohowthecompositionwillbeconfiguredtospace.

TheFulldomeisaspaceintotaldarkness,whichproducesaveryintensespatialexperience,whereyouquicklylosethereferenceofspaceandtime.ForexampleinVRthelightofthegogglesisperfect,butthelightcontaminationoftherealisspaceismuchdifferent.Also,thefactorofthespeedandsizeofthemovement.

21

Consideringtheworkanditscontentsameelementsweresharedonthesamecases,oncethetechnicalsidewasconveyedtheartistsestablishedthatmusicwouldmodifythevisuals,sotheycanloopandimprovisethroughoutthescenesthatlastedaroundfourorfiveminuteseachbutwhenthepointoftransitionwasestablishedthenewscenewasreconnected.Atastudiowithmultichannelsoundsystemsthesoundspatializationcompositionscanbeproducedbuttoproducethevisualsinanyotherspacepresentsmorecomplications.

Consideringtheaudiencesinbothvenues,eachshowwasdifferentwiththesamescoreanddifferentinterpretation,somecommonelementsofsonicorvisually.Theaudiencetookdifferentpositionssometimessitting,layingandothersstanding.

Conclusion

- Immersiveenvironmentshadbeenconsideredafieldofresearchsincethe50softheXXCentury,fromarchitecture,scienceandhumanitiesstudiestotheactualdevelopmentsofvirtualrealityandaugmentedreality.Manyaspectsandelementsarecentraltothedevelopmentsofnewsoftwareanddigitalprojectorstothepresenttimessincetheearly70softheXXthCenturyalsoaconcernofearlycinema.

- Thereisaculturalcontentshift.Theelectronicmusicculturehasreachedthemuseumandmuseumneedstoadapttothesenewworksthatrequiretechnologyandspacetobepresented,enactedanddisplayed.Thereisaneedformoretimeusingfacilitiesfortheresearchandproductionofimmersiveaudiovisualcompositions.

- Categoriesintheimmersivecomputingspectrumarecontinuallyappearing,andthenewglossaryoftheacronymsofthefutureiscontinuouslyexpanding.Withinourresearch,wefocusonthemostsocialandembodiedcategoryofthisspectrum.Immersivecollectiveexperiencesareattheoppositesideofthecomputingspectrumofreality,samemindsetbutwithoutaheadset.Thiswayofshowcasingandproducingfilmsandperformancesintroducesanewlanguageandcompositingmethodsinaudiovisualproductionanddigitalstorytelling.Audiencesadaptthecontentwiththeirimagination.Asaudience,youhavearangeofview,andalisteningcapacity.

- Theauthoringworkflowforimmersiveaudiovisualcompositionvariesdependingonthefacility.Theseproductionsdonotfollowthesameconventionsofsingle-screencinemaasaresultinnolongershowcasedinonescreen.Eachartisthasitswayand

22

methodtoproducecontent.Imagecontentproductiontools,differentlanguages,andaesthetics.Theproductionofimageandsoundinthedigitalhasmultiplerangesofapproaches.Audiovisualcontentproductioncanbedonewithanalogueanddigitalmediaandcanbedrivenbyphysicalinteractionandmachinelearning.Visceralandmovingimageperformanceswithdigitalmedia.

- Thereisaneedaccesstothehistoryoftheexistingpracticesandtotransfertheneedsofthecreatorsittoguidetechnologycanservethenewconceptsandideasthatreplytotheactualconcernsanddevelopmentofthelanguage.Thecreatorsusedifferenttoolsandhavedifferentideasofwhattheformatcando.Formanyartists,thereisaneedtohaveatechnicianhelpingsupportingthetranslation.Fromwhatwecanfindthereisnostandardonthetools.Standardsaregoodbutwecannotmakeastandardasitemergesfromthepractice.So,theconstrainsonimmersivespacesindicateasolution.

- Fromthecompositionpointofviewofimmersiveexperiencesessentialelementsareframing,resolutionandpointofviewoftheexperience.Clearspecsofthesystem,mapsanddimensionofthespaceandtemplatesareessentialelementsforthetranslationandadaptationoftheworktoanewspace.Whatfinallymattersfortheworkisthesmoothnessoftheanimation,forexample,toplay60framesforseconds.

- - Technicalvitalelementsinsurroundedcinemaincludecreateaseamlesscanvasby

usingmultipleprojectors(blending),werewecannotappreciatethestitchingbetweenthem,avoidorcorrectdistortionoftheimageandcolourdifferencebetweenthem.

- - Technologyisconstantlyevolving,therefore,imageresolution,typesoflight,

brightness,contrastandaspectratio.Anotherelementthathasbeencentraltosoftwaredevelopmenthasbeentobeabletocontrolthecontent(input)andtransportittotheprojectorsandspace(output).Syphon,Spout,andNDIareessentialprotocolsthatallowtransferringandtransportvisualcontentinrealtimefromoneinputtomanyoutputsofvice-versa.Thetoolsarechangingfastsothereisademocratisationoftechnology.Stillthecostofthetechnologiesishigh.

Afewoforganizationshaveapermanentsetup,evenSIMLinGoldsmithsisasharedspace.CineChamberandHexadomedon’thaveapermanentsetupeither.Thedomeshaveitbutthereisalimitedaccesstothesespaces.Therefore,thereisalackoftimeforartistsinimmersivespacesfordevelopingtheirwork.Spaceavailabilityisrestricted.Mostofthefacilitiesaresharedspacesthataccommodateanumberofmanifestationsandactivities.VRandsimulatorscanprovideasolutionbuttheycan’tprovideaccesstothevisceralexperience.Thisisrealityandwearelivinginavirtualrealitynottheotherwayaround,statesVickyBennet,consideringthecreationofworksforthisimmersivespace.Formakingherpieceaftercreating10minutesofthepieceshetesteditintherealphysicalspace,shecouldn’tbuilditinthevirtualspace,sheneededtherealspacetounderstandhowtheimaginationoftheaudiencewillprocesstheexperiencesheneededatestforthephysicalforthevisceralexperience.Youcanmakedifferentassociationswhenyouaredesigning.

23

Inallthespaces,thereislimitedtimetorehearse.Thereisalackofspacesandimmersivethesefacilities.Simulatorsforimmersiveaudiovisualprojectsfacilitatethecreationandunderstandingofworksprovidingasolutiontooneofthemainissuesaddressedbyartistsandresearcherstheneedformoretimefortheproduction,creation,anddevelopmentofimmersiveprojects.Asit’snotpossibletoalwaystestsinthefinalscenariostheVRprovidessomesolutionsforcertainstagesontheprojectsuchasthecompositionofmaterials.It’sarevolutionthatprovidessomesolutions.Also,VRisatechnologythatstillsevolvingconstantly.Virtualrealityisreallyimportantbecauseitcansimulatetheenvironment.Youcangetalotofworkdonethere.Asapreparationforthebigscalespace.Butthereareproblemswhenyouchangescales.

Therearealsoscalemodel’sphysicalmodelsthatcanbedonewithanaffordablebudget.Forexample,theminiSimlortheDomedevelopedbyFrontPictures.Butthatdoesnotprovideenoughfeedback.Themodelworksinaverydifferentwaysoitwillprepareforsomebutwon’tavoidterriblesituationsthatmayhappenintheactualfacilities.Technologiesarefragileandyouneedtopushthesignalinbigdimensionspaces.

AnotherimportantfactorregardinglonghoursworkinginthesespacesisthelackoflightWhenspaceiscompletelydarkafterlonghoursworkingthebodyresentsduethelackofnaturallight,sotheproductivityresents.Therefore,tofollowupdevelopingtheimmersiveworksalternativespacesarerequested.Thatiswhensimulatorsandotherstudiosthatwillhelptovisualiseandtesttheworkbecameanimportantfeature.

Video-mappingtechnologiesareconsolidatedasthetechniqueofassigningvisualcontentintoarchitectureandbi-dimensionalortri-dimensionalspacesOntools,therearedifferenttoolsforproductionofcontentandattheendofthechain,pixelpushingtodistributethecontentacrossmultipleoutputs.Forexamplemadmapper

24

Addendum

InterviewbyBlancaReginafortheImmersivePipelineResearch,April21st2018

I’mTarikBarri,I’manaudiovisualartistandweareintheMartinGropiusBauinBerlininGermany.

Howwastheexperiencetocreateforthesix-screenspacethatISMHexadomecreated?18

Icantellyoualittlebitofhowthiscameup.Nick,oneofthemainorganizersoftheHexadomeprojectwasawareofmyworkforquitesometimeandIhadbeeninvitedtosomeotherworksthatheorganized,andthoseweremorestandardeventswithascreenandtwospeakers,buthewasveryinterestedinthatwholeideaofamorespatialexplorationofsoundandvisualsandthatledhimatsomepointtocreatethiscrazyHexadomeprojectwiththewhole54speakers6screensallsurround.

ThatcompletelyfitsthewayIcreatemusicbecauseIcreatemusicbycreatingavirtual3DspaceinprogramminglanguageandwithinmyownprogramthatImadeIflythroughthisvirtualspace.AndbyflyingIflypassmusicalmelodiesandrhythmsetc.sothepathdetermineswhatwehearsodeterminesbasicallythestructureofthecomposition.Thatwholeprincipleisinherentlybuiltaroundthisideaofbeingsurroundedbyspace,beensurroundedbysound.ThissystemthattheISMsetupjustenablesmetoreallyshowitinthatexactlythatway,thatyoudon’tjusthavea2Dscreenandyouimaginethespacearoundyoubutyoucanactuallyseethespacearoundyouandwithallthespeakersaroundyoucanhearthewholespacearoundyou.

25

Pleaseletusknowalittlebitaboutyourtool,software.Areyouusingitforvisualstoo?

BothforsoundandvisualsImademytool.OriginallyImadeitmainlyforsoundbutitwasveryevidentveryearlyon,thatwillonlyworkemotionallyifthevisualsaregoingtomatchtheemotionalintensityofthesounds.SowheneverthesoundsbecamemorebeautifulIfeltokthevisualscanstaybehindsothat’showIbasicallylearntoprogrammebothinawaythatI’mnowveryhappywithbecausebothsidesconstantlyinspiredeachother.

DidyoumanagetoperformliveyourworkatHexadomeandhowwastheexperience?

ThissoftwareismadeaslivesoftwaresoImostoftendomystuffwithmytridimensionalworldliveandthisisagreatwaytogetamoreplayfulnaturalintuitivefeeltoyourownmusicifyoucanactuallyplaywithitratherthanifyouhavetoreallythinkitallthroughandletthecomputerrenderfortwodaysandthenyouhearseetheresults,Ilikethatismoredirect.

However,rightnowwhathappenedisthatImadeaninstallationversionthatisnottotallyliveatallbecausewiththesixscreensaroundandwiththeamountofdetailsthatIstillwanttohaveitsimpossibletoreallygetaproperreallyniceresultdoingitliveandIdon’twanttocompromiseallresultsthat’swhyIchoosetomakeitasainstallations.

Howwasthisideaofthecollaborationanddidyouknowtheartistbeforethecommission?

SureIknewTombecauseIwasalreadydoingvisualswithhisperformances.Iworkalot.Basically,IveryoftenaccompanymusicianswhenthenmakemusicandthenIdothevisuals.I’vedonethatwithRobertHenke,NicholasJaar,PaulJebanasamsoIhavethesecollaborationswithpeoplewheretheydoallthemusicandIdoallthevisuals.

ThisisoneoftheraretimeswhereIalsodobothmusicandvisualsatthesametime.I’mnotsureifIcansaythatImostwanttodothatbutI’mveryhappythatIcanincludethisinmypracticecausemywholesystemcamefromthatidea.AndIusethatsystemsometimespurevisualswithotherpeoplebutactually,IliketouseitinamorecompleteholisticwayandIthinktherearelotstoexploretherealsoinawaythatyouoftenseeintheworldandIdliketogofurtherandexplore.Basically,it’sadoorthatisnowthankstotheexhibitionsandthecollaborationswithTomisabitmoreopensoletsopentoseewhatwecanexplorethere.

WhatwillyourecommendfornotreinventingthewheelonAVimmersiveorimmersiveSound?Togoonestepfurther?

Ohwow,Ithinkonethingistolookcloselyatotherpeoplesworkandseewhatyouseetoooften.Verymanyartistsinmanyfieldstheyrepeateachotheroverandoveragainanditbecomeslikeastraightjacket,Iguessishowfashionworksveryoften.Lotsofpeoplewearthesamethings.Ithinkisokthattherearetraditionsandthatwecanagreeoncertainthingsasasociety.Butinartespecially,Ithinkandforsomereason,inthiswholeelectronic,technologicalartthereisthissortoftoooftenusedaesthetic,Idon’tknowifit'slikeamaleaesthetic,likeamanlikeorsomething.Oksoitstechnologyandisstraightandhardanditneedstobeconnectedwithscientificdata.IrespectthatbutalsoIwasmockingitalittlebitbecauseI’veseenittoomanytimessoI’mgettingsickofit.Ithinkit’snicetosometimeshaveabitmorerandomnessandalittlebitmoreplayfulness.Becausetechnologycanbecomplicatedandallthatkindofstuffbutatsomepoint,ifyouwanttogoalittlebitfurtherjustthanjustlookingattechnologyyoucangothroughitand

26

discoverotherworldswhichdon’tblindsocloselytothezerosandonesandgotothecolorsthatarebeyondthat.