Computer Music and Sonology at the University of São Paulomqz/CORFU2015.pdf · Observations on the...

Computer Music and Sonology at the

University of São PauloMarcelo Queiroz

Associate Professor – Computer Science DepartmentComputer Music Research Group

NuSom - Sonology Research Center9th Festival of Audiovisual Arts – Corfu – May 29, 2015

2

NuSom@USP: Sonology and Computer Music

Outline of this talk

● Team● Research Projects and Recent Work● Technical Contributions● Artistic Contributions● Future Perspectives

3

NuSom@USP: Sonology and Computer Music

Two constantly interacting research groups:

● Sonology & Music Technology Research Group● @Music Dept (School of Communications & Arts) ● 3 Profs and 2 post-docs● About 20 students (undergrad, MA, PhD)

Computer Music Research Group @Computer Science Dept (Inst of Math & Stats) 3 Profs and 1 post-doc About 16 students (undergrad, Msc, PhD)

4

NuSom@USP: Sonology and Computer Music

Joint medium-large projects● AcMus: room acoustics● Mobile: interactive processes and systems

Many subprojects:● Andante & Ensemble● Medusa● OpenAudience● NetConcerts● ¿Música? Concert Series● ...

5

Large projects: AcMus (2002-2006)

Goal: open source software for room acoustics Measurement: obtaining several acoustic parame-

ters from a recorded impulse response of a room Simulation: obtaining the impulse response from a

geometrical model (ray tracing, phantom sources) Acoustic tools: modal room response, reverberation

times, Schroeder diffusor design, auralization, optimization-based design

http://compmus.ime.usp.br/acmus

6

Large projects: Mobile (2009-2013)

Goals: Promote research on interactive processes within

technologically-mediated music production Deepen the discussion around theoretical and

conceptual aspects of interactive musical processes Develop interactive software systems for

composition, performance, and auralization Create artistic products as results of theoretical,

technological, and artistic cross-influences Promote concerts, events and seminars http://www.eca.usp.br/mobile

7

The NuSom Research Center

NuSom (Sonology Research Center) Created in 2013 Coordinator: Fernando Iazzetta (Music Dept) Vice-coordinator: Marcelo Queiroz (Comp Sci Dept) Partners in several Brazilian institutions, and also at

SARC (Belfast) and McGill (Montreal) Funded by the Pro-Rectory of Research at the

University of São Paulo http://www.eca.usp.br/nusom

8

The NuSom Research Center

NuSom Goals Explore connections between music, art & science View artistic production, technological research and

critical reflection as an unified process Creation of collective artistic works focused on an

experimental perspective Blur boundaries between academic / institutional

environment and informal music production

9

NuSom Recent Works

Experimental Music, Philosophical Issues, Urban Etnographies

Circuit bending in three acts: opening, touching, and multiplying (Alexandre M. Fernandez – MA)

Observations on the role of the digital instruments in Brazilian contemporary experimental music (José G. L. Allen – MA)

Gambiarra and sonic experimentation (Giuliano O. Lamberti – PhD)

¿Music?: processes and practices for creation and performance in a Sonology research environment (Vitor K. Miskalo – PhD)

Contradictions in the aestheticization of noise in music (Lilian Campesato – PhD)

Considerations on the musical silence after Cage's 4'33” (Carlos A. P. Avezum – PhD)

10

NuSom Recent Works

Digital Signal Processing and Interactive Systems

Universal design for interactive systems (Ana L. Basalo – MS)

Realtime audio processing on alternative computational platforms (André J. Bianchi – MS)

Interactive lighting systems for theatrical plays (Gilmar Rocha Dias – MS)

Automatic accompaniment systems (Roberto Piassi Bodo – MS)

Intelligent interactive environments (Tiago Andrade Togores – MS)

Nonlinear sound synthesis techniques (Antonio José H. Goulart – PhD)

User interfaces for music multiagent systems (Pedro Henrique Rocha Bruel – undergrad)

11

NuSom Recent Works

Music Interaction / Internet Music / Mobile Music

Musica Mobilis: possibilities of dialogues between sound and space (André Damião bandeira – MA)

Compositional strategies and the use of new technologies in Ecuatorian music (Jonathan Andrade – MA)

Audio transmission over residential Internet providers (Marcio Tomiyoshi Masaki – MS)

Sound processing on mobile platforms (Antonio D. C. Junior – PhD)

Medusa: an environment for distributed audio (Flávio Luiz Schiavoni – PhD)

Network Music: collaborative creation and musical performance in the context of information networks (Julián Jaramillo Arango – PhD)

12

NuSom Recent Works

Contemporary Music; Audio Visual

Sound organicity: A Bridge between Scenic Contemporary Music and the Workcenter of Jerzy Grotowski and Thomas Richards (Luciano Mendes de Jesus – MA)

The use of scenic and visual resources as boosters of music creation and performance” (Gregory Silva – MA)

Building sonorities: Compositional Techniques of an timbre–driven aesthetics (Magno Caliman Sposito – MA)

Spatial aspects of sound art: Bernhard Leitner and Christina Kubisch (Borys Duke – undergrad)

13

NuSom Recent Works

Improvisation / Performances

The piano in interactive musical processes: new gestures and new dialogues (Alexandre Z. Almeida – Post-Doc)

The electric guitar as improvisation and composition environment: growing opportunities from the use of technological devices (André Martins – MA)

The use of viewpoints technique in free improvisation (Fábio Martinelli – MA)

The free improvisation as a tool for novice guitar players (André Campos – PhD)

Collaborative Processes: composer–performer relationship in current musical practices (Mario del Nunzio – PhD)

14

NuSom Recent Works

Multimedia / Games / Audiovisual

OS support for game design (Marcos Almeida – PhD)

Machine learning applied to racing games (Vinicius Daros – MS)

Information Flow in intelligent interactive museums (Claudia A. A. Josimar – Post-Doc)

The influence of electroacoustic music in the creation of film soundtracks (Marcelo Nicolino – MS)

15

NuSom Recent Works

Acoustics / Auralization / Spatial sound modeling

Statistical validation of applying the Ambisonics system in room acoustics (Fabio Leão Figueiredo – Post-Doc)

Modeling spatial musical instruments using Wave Field Synthesis (Márcio José da Silva – MS)

The influence of the vocal tract configuration in the sound of the flute (Fabiana Coelho – PhD)

Modeling the effect of trumpet mute types (Alexandre C. G. Suetholz – undergrad)

Advanced sound field synthesis techniques (Thilo Koch – PhD)

16

NuSom technological contributions

OpenAUDIENCE (Regis Faria & collaborators) Software for audio immersion, sound scene design,

and auralization Modularization of sound processing chain and 2D

and 3D sound field representation Written as a Pd library Multichannel rendering of sound scenes Spatialized and interactive sound reproduction http://www.lsi.usp.br/neac/openaudience

17

NuSom technological contributions

Auralization of moving sound sources using HRTF interpolation (Gustavo H. M. Sousa)

HRTF filters model directional information for static sound sources; interpolation methods are proposed in time and frequency domains, and also in pole-zero representations of reformulated filters

Implementation in Pd. User/composer may interactively control sound source positions and influence auralization in realtime.

MSc dissertation, papers, and code at compmus.ime.usp.br

18

NuSom technological contributions

Music structure segmentation (André S. Pires) Goal is to obtain timestamps for structural

transitions (large-scale musical sections) and labels for corresponding sections. Uses statistical modeling of the input signal via descriptors.

May be used in realtime analysis as input to interactive processes, for instance, triggering different algorithms for each musical section

MSc dissertation, papers and code available at compmus.ime.usp.br

19

NuSom technological contributions

Crosslingual voice conversion (Anderson F. Machado)

Goal is to convert voice timbre in recorded sentences, keeping prosody without changing text

Involves statistical modeling of source and target speakers, and defining mappings between artificial phonetic classes and prosodic models.

Applications: dubbed films, TTS systems, security PhD thesis, papers and code at

compmus.ime.usp.br

20

NuSom technological contributions

Ensemble: a framework for musical multiagent systems (MMS) (Leandro F. Thomaz)

First general MMS framework to include audio communication and processing, spatialization and realistic sound propagation in the virtual world

Allows interactions among agents, between virtual agents and external musicians, and also between musicians (represented as Avatar agents)

PhD thesis, papers and code at compmus.ime.usp.br

Related work: XML scripting (Santiago Benavides), Pd interfaces for Ensemble (Pedro Bruel)

21

NuSom technological contributions

Medusa: a distributed sound environment (Flávio L. Schiavoni)

Provides multichannel network connections for interactive networked musical jams

Goal: musician-friendly interface for transparent access to network audio resources and remote reconfiguration

PhD thesis, papers and code at compmus.ime.usp.br

Related work: JackTrip/CELT integration for audio compression and delayed local feedback (Márcio Tomiyoshi)

22

NuSom artistic contributions

Hyperflute by

Cesar Villavicencio Sensors on the keys

and instrument body Max/MSP patch for

real-time processing Visual feedback from

screen monitor attached to the flute's head

23

NuSom artistic contributions

Compro Auri (2005-6)

by Giuliano Obici

The piece refers to “men-plate”, a common characterin major urban areas of Brazil.

In Compro Auri, people are invited to sell their listening,thus reversing the common contract between those whocreate and those who listen to music

24

NuSom artistic contributions

Happy Cube (Julián Jaramillo) Interactive musical object exploring geometry for

musical creation Depending on the leaning side of the cube, a

different generative process is triggered Turning the cube around one side, a variation of the

musical pattern is produced Uses an Arduino and a gyroscope Presented in MIS/SP 2010

25

NuSom artistic contributions

Happy Cube (Julián Jaramillo)

26

NuSom artistic contributions

MOBILE board / Tábua MOBILE (Henrique Iwao, Marcelo Muniz, Borys Duque)

Experimental musical instrument Two wooden boards connected by four wooden

columns

→ amplified upper board is used for manipulating several objects and electronic devices

→ lower board contains the electronic machinery, a sound board and other objects

Presented in MOBILE concert Por Trás das Coisas (2010)

27

NuSom artistic contributions

Tábua MOBILE (Henrique Iwao, Marcelo Muniz, Borys Duque)

28

NuSom artistic contributions

Tábua MOBILE (Performers: Henrique Iwao, Victor Kisil, Marcelo Muniz)

29

NuSom artistic contributions

LanHouse Concert (Giuliano Obici, Flávio Schiavoni)

Networked audiovisual installation/composition First part: interplay of sounds, motion, and

synchrony Second part: color variations and chaining of

horizontal textures Programmed in Pd (using netclient/netserver) Presented in MIS/SP (2010), LAC/Maynooth (2011),

FILE (2011) and PDCon(2011)

30

NuSom artistic contributions

LanHouse Concert (Excerpt) (Giuliano Obici, Flávio Schiavoni)

31

NuSom artistic contributions

Conexões dispersas/dispersões conexas (Lílian Campesato, Julián Jaramillo, Vitor Kisil)

Interactive musical performance that explores realtime generation of audiovisual material from physical interactions

A performer manipulates objects and images on a table monitored by cameras, microphones, and other sensors; two other artists control realtime interactive processes using those inputs

Presented in EIMAS/Juiz de Fora (2010), MOBILE concert Por Trás das Coisas (2010), ¿Música? 3 (2011) and Festival Internacional de la Imagen/Bogotá/Colômbia (2011)

32

NuSom artistic contributions

Conexões Dispersas/Dispersões Conexas (excerpt, performer: Lilian Campesato)

33

NuSom artistic contributions

Por Trás das Coisas(Behind Things, 18-19/10/2010)

Multimedia performance, directed by Fernando Iazzetta and presented in SOFTBORDERS: New Media Art International Conference

Collaborative creation by over 40 MOBILE participants Multimedia environment: music, dance, images, and

interactivity 6 scenes: new instruments, interactive games, dance

and free improvisation, physical and perceptual boundaries, disclosure and unveiling of things, and their fading away

34

NuSom artistic contributions

35

NuSom artistic contributions

Folder of the concert Por Trás das Coisas

(Behind Things)

October, 2010

36

NuSom artistic contributions

MOBILE team in Por Trás das Coisas

37

NuSom artistic contributions

Por Trás das Coisas: interactive games (scene 3, excerpt)

38

NuSom artistic contributions

Por Trás das Coisas: Physical and perceptual boundaries (scene 4, excerpt)

39

NuSom artistic contributions

Por Trás das Coisas: Fading away (scene 6, excerpt)

40

NuSom artistic contributions

Transparency (2013): one hour performance created bysix researchers/artists (European tour April 2013)

41

NuSom artistic contributions

¿Música? series 10 concerts between 2005 and 2014 The ¿Música? series comes from the often-heard

(and often-pertinent) questioning of whether something is or isn't qualified as music

It seeks to critically occupy ordinary spaces with sound performances (¿Música? 6 was presented at a student cafeteria)

It also invites people to reevaluate their listening habits and behaviors, proposing interactions between the daily activities and sonic environment

www.eca.usp.br/nusom/producoes-musica

42

NuSom artistic contributions

43

NuSom artistic contributions

NetConcert series 6 telematic concerts between 2011 and 2014 The NetConcert series explores the virtual space of

the Internet as a venue for music creation, a modified acoustic medium

Mostly between USP (São Paulo) ↔ SARC (Belfast), but also Atico-PUJ & Universidad Caldas (Colombia), CCRMA (Stanford) and IRCAM (Paris)

www.eca.usp.br/nusom/producoes-net-concert

44

NuSom artistic contributions

Rogério Costa and Manuel Falleiros (USP)and Franziska Schroeder (SARC)

45

NuSom artistic contributions

Julián Jaramillo and Pedro Paulo Kohler (USP)

Manuel Falleiros andMigue Diaz (USP),Robert Casey and

Franziska Schroeder (SARC)

46

NuSom artistic contributions

2nd NetConcert(2012)

6th NetConcert(2014)

47

Future perspectives

International cooperation

SARC (Sonic Arts Research Centre, Belfast) McGill University, Montreal CMMAS (Centro Mexicano para la Música y las

Artes Sonoras) Maynooth University University of Michigan National and Kapodistrian University of Athens Aristotle University of Thessaloniki

48

Future perspectives

Current collaboration with AUTh(→Emilios Cambouropoulos, Maximos Kaliaktsos-Papakostas, Asteris Zacharakis)

Conceptual Blending x Harmony Cadences and Chord Progressions Representations based on Voice Leading General Chord Types (GCTs) from Audio 2nd-order Markov Models for harmonization

Funded by FAPESP (São Paulo/Brazil)

49

Questions?

Sonology Research Center:

http://www.eca.usp.br/nusom

Computer Music Research Group:

http://compmus.ime.usp.br

My e-mail: mqz@ime.usp.br

These slides: http://www.ime.usp.br/~mqz/CORFU2015.pdf