Computational Approach of Musical Orchestration

Computational Approach ofMusical Orchestration

Grégoire Carpentier - Ph.D. DefenseIRCAM - Music Representation Group

Constrained Multiobjective Optimization of Sound Combinations in Large Instrument Sample Databases

Supervisors : Gérard Assayag (IRCAM) & Emmanuel Saint-James (LIP6)

December 16th, 2008

Computational Approach of Musical Orchestration

Contents

1. Computer-Aided Composition / Orchestration

Contents

2. Stating the Orchestration Problem

Contents

3. Combinatorial Optimization Problem(the orchidée algorithm)

Contents

4. Constraint Solving Problem(the cdcsolver algorithm)

Contents

5. Prototype of Orchestration Tool - Musical Examples

Contents

6. Conclusions and Future Work

Contents

CAC / Orchestration

Computer-Aided Composition

CAC / Orchestration

• Formalizing of musical structures

CAC / Orchestration

• Formalizing processes

CAC / Orchestration

• Tackling different aspects of musical writing:

CAC / Orchestration

• Tackling different aspects of musical writing:Durations, rythm

CAC / Orchestration

Pitches, melody, harmony

CAC / Orchestration

Time representations

CAC / Orchestration

Spatialization

CAC / Orchestration

Spatialization

Sound synthesis

CAC / Orchestration

Spatialization

Sound synthesis

Instrumental timbre ?

CAC / Orchestration

Orchestration - timbre

Symbolicworld(score)

Soundworld

(timbre)

CAC / Orchestration

Orchestration : projection

CAC / Orchestration

Orchestration : projection

Soundworld

(timbre)Projection

CAC / Orchestration

Orchestration : induction

CAC / Orchestration

Soundworld

(timbre)

CAC / Orchestration

Soundworld

(timbre)

CAC / Orchestration

Soundworld

(timbre)

Induction

CAC / Orchestration

Soundworld

(timbre)

Induction

CAC / Orchestration

Orchestration in practice

CAC / Orchestration

• Personal knowledge

CAC / Orchestration

• Personal knowledge Necessarily restricted

CAC / Orchestration

• Personal knowledge Necessarily restricted

• Orchestration Treatises

- [Berlioz1855]

- [Koechlin1943]

- [Piston1955]

- [Rimski-Korsakov1912†]

CAC / Orchestration

Obviously outdated

Necessarily restricted

- [Berlioz1855]

- [Koechlin1943]

- [Piston1955]

CAC / Orchestration

Obviously outdated

- [Berlioz1855]

- [Koechlin1943]

- [Piston1955]

• Current computer orchestration tools

- [Psenicka2003]

- [Rose&Hetrick2005]

- [Hummel2005]

CAC / Orchestration

Obviously outdated

Monoobjective approach of timbre

- [Berlioz1855]

- [Koechlin1943]

- [Piston1955]

- [Psenicka2003]

- [Hummel2005]

CAC / Orchestration

Obviously outdated

Monoobjective approach of timbre

Do not handle combinatorial issues

- [Berlioz1855]

- [Koechlin1943]

- [Piston1955]

- [Psenicka2003]

- [Hummel2005]

Contents

Statement

Problem statement

Statement

Problem statement

How can I use an orchestra to reproduce a timbre target within a given compositional context?

Statement

Problem statement

How can I find an instrument sound combination that:

Statement

Problem statement

- Best matches a given target sound?

Statement

Problem statement

- Fits writing constraints specified by the composer?

Statement

Problem statement

In computer terms:

Statement

Problem statement

In computer terms:

- A combinatorial optimization problem defined on a timbre description scheme

Statement

Problem statement

In computer terms:

- A combinatorial optimization problem defined on a timbre description scheme

- A constraint solving problem on the variables of musical writing

Statement

Strategy

Statement

Strategy

- Combinatorial optimization problem: the orchidée algorithm

Statement

Strategy

- Constraint solving problem: the cdcsolver algorithm

Statement

Strategy

- Collaboration between the two methods

Statement

Strategy

targetcontext

Statement

Strategy

targetcontext

featurescontraints

Statement

Strategy

targetcontext

featurescontraints

orchidéecdcsolver

Statement

Strategy

targetcontext

featurescontraints

orchidéecdcsolver

contextualizedsolutions

SampleDB

Contents

Combinatorial Optimization (orchidée)

Timbre similarity (1)

• Timbre perception is multidimensional

• Correlation between perceptual dimensions and sound features [McAdams+1995] [Peeters2004]:

• Spectral centroid <=> brightness

• Attack time <=> percussive / sustained sounds

SampleDB

Sample

SampleDB

feature 1feature 2

feature K

Sample

SampleDB

• Hypothesis: Sound combination feature set can be predicted from the values of components features

d 1d 2

Prediction ofcombination

features

d 1d 2

Prediction ofcombination

features

d 1d 2

Dissimilarity measuresalong each perceptual

dimension

d 1d 2

Sound features

• Spectral centroid (brightness) [McAdams+1995]

• Spectral spread (volume) [Chiasson2007]

• Resolved partials (harmonic tone)

• Time and noise features are not considered

• Preliminary tasks:

• Sound combinations features prediction functions

• Perceptual dissimilarity functions

Multiobjective approach

• Relative importance of perceptual dimensions cannot be known without prior information on listening preferences

• Multiobjective optimization: Jointly minimize

• Pareto dominance:

• Set of optimal solutions (implicitly corresponding to different listening preferences)

Formalizing the optimization problem

• Orchestra composed of instruments: variables problem

• Domain of each variable:

• Problem:

A NP-hard problem

A NP-hard problemConsider a 5000 sample sound database. Consider an 11 instruments orchestra in which only 4 can play simultaneously a given 4-note chord. Thus:

A NP-hard problem

• There are around a billon a feasible combinations

Consider a 5000 sample sound database. Consider an 11 instruments orchestra in which only 4 can play simultaneously a given 4-note chord. Thus:

A NP-hard problem

• Computing their features takes around 20 minutes with 3 perceptual dimensions

A NP-hard problem

Taking into account that:

A NP-hard problem

• A big orchestra may contain around a hundred instruments;

A NP-hard problem

• There might be around ten perceptual features;

A NP-hard problem

• Instrument sound databases may hold hundred of thousands items;

A NP-hard problem

Complete resolution methods cannot help here.

Metaheuristics are required.

• Instrument sound databases may hold hundred of thousands items;

GA optimization: the orchidée algorithm

• Evolutionary algorithm (using a population of individuals)

• Each individual is represented by a set of genes (chromosome)

• Orchidée: integer tuple encoding (“orchestra” representation)

• Orchidée: user preferences guessing mechanism

Initialpopulation

CurrentPopulation

Initialpopulation

CurrentPopulation

Evaluation

Initialpopulation

CurrentPopulation

Evaluation Selection

Initialpopulation

CurrentPopulation

Evaluation SelectionGenetic

Operators

Initialpopulation

CurrentPopulation

OperatorsCurrent

Population

Initialpopulation

CurrentPopulation

OperatorsCurrent

Population

Preservingdiversity

Initialpopulation

CurrentPopulation

OperatorsCurrent

Population

Preservingdiversity

iter max ?

Finalpopulation O

Goals :

Initialpopulation

CurrentPopulation

OperatorsCurrent

Population

Preservingdiversity

iter max ?

Finalpopulation O

Initialpopulation

CurrentPopulation

OperatorsCurrent

Population

Preservingdiversity

iter max ?

Finalpopulation O

Goals :

• Convergence towards Pareto front

Initialpopulation

CurrentPopulation

OperatorsCurrent

Population

Preservingdiversity

iter max ?

Finalpopulation O

Goals :

• Convergence towards Pareto front

• Diversity along the Pareto front

User preferences (1)

orchidée

Intermediarysolutions

orchidée

User choice

orchidée

User choice

ImplicitPreferences

orchidée

User choice

ImplicitPreferences

orchidée

User choice

ImplicitPreferences

orchidée(+ preferences)

• Weighted Chebychev norm [Jaszkiewicz2002] :

soit :

• Each efficient solution corresponds to a weight set

• Fundamental property [Steuer1986] :

• Computing weights from criteria [Carpentier2008] :

• Orchidée computes configurations fitness thanks to a Chebychev norm

• Orchidée computes configurations fitness thanks to a Chebychev norm• When preferences are unknown weights are randomly drawn at each

generation (multiobjective optimization)

• Orchidée computes configurations fitness thanks to a Chebychev norm• When preferences are unknown weights are randomly drawn at each

generation (multiobjective optimization)• When preferences are known weights are fixed (monoobjective

optimization)

Contents

CSP / Context (cdcsolver)

Compositional context

• Any musical material comes within a musical gesture

• Depending on this gesture all configurations may not be feasible

• Depending on this gesture all configurations may not be feasibleSample

feature 1feature 2

feature K

attribute 1attribute 2

attribute N

Sample

feature 1feature 2

feature K

attribute N

Sample

feature 1feature 2

feature K

pitchesdynamicsplaying stylesmute

attribute N

Sample

feature 1feature 2

feature K

• Modeling context with global constraints on attributes :

attribute N

Sample

feature 1feature 2

feature K

- “Between 8 et 12 instruments”

attribute N

Sample

feature 1feature 2

feature K

- “No more than 3 fortissimo”

attribute N

Sample

feature 1feature 2

feature K

- “At least two different pitches”

attribute N

Sample

feature 1feature 2

feature K

- “All strings play with a mute”

attribute N

Sample

feature 1feature 2

feature K

- “All strings play with a mute”

• Local search / soft constraints approach: Iteratively update a single configuration to minimize a set of cost functions

Design / Conflict?

• Design constraints: Anything required in the orchestration

Design / Conflict?

• Conflict constraints: Anything to avoid in the orchestration

Design / Conflict?

• Design constraints may be satisfied by instantiating free variables

Design / Conflict?

• Design constraints may be satisfied by instantiating free variables

• Conflict constraints may be satisfied by freeing instantiated variables

CDCSolver

3 neighborhood heuristics:

CDCSolver

1. If a conflict constraint is violated, first update an instantiated variable.

CDCSolver

2. If a design constraint is violated, first update a free variable.

CDCSolver

3. There is a priority variable to change [Codognet2002]. Decision rule is based on a min-conflict heuristic.

CDCSolver

1 move heuristic:

CDCSolver

4. In the current neighborhood choose the configuration that minimizes the global cost function (min-conflict).

1 move heuristic:

CDCSolver

1 move heuristic:

1 cycle handling heuristic:

CDCSolver

1 move heuristic:

1 cycle handling heuristic:

5. Handle cycles and local minima with a short term memory scheme (tabu list [Glover1997])

Combining orchidée and cdcsolver

Combining orchidée and cdcsolver• Genetic algorithms are not well suited for constrained problems

• Repairing every inconsistent configuration is inefficient

Initialpopulation

Currentpopulation

operatorsCurrent

population

Preservingdiversity

iter max ?

Finalpopulation O

• Genetic algorithms are not well suited for constrained problems

• Repairing every inconsistent configuration is inefficient

Initialpopulation

Currentpopulation

operatorsCurrent

population

Preservingdiversity

iter max ?

Finalpopulation O

cdcsolver

Initialpopulation

Populationcourante

operatorsCurrent

population

Preservingdiversity

iter max ?

Finalpopulation O

cdcsolver

Penalize inconsistentsolutions

Initialpopulation

Populationcourante

operatorsCurrent

population

Preservingdiversity

iter max ?

Finalpopulation O

cdcsolver

Configurations comparing rules

Initialpopulation

Populationcourante

operatorsCurrent

population

Preservingdiversity

iter max ?

Finalpopulation O

cdcsolver

Configurations comparing rules1. Choose the more consistent configuration

Initialpopulation

Populationcourante

operatorsCurrent

population

Preservingdiversity

iter max ?

Finalpopulation O

cdcsolver

Configurations comparing rules1. Choose the more consistent configuration2. Choose the configuration with highest fitness according to the current

Chebychev norm

Contents

Prototype

Prototype - Examples

Prototype

• MATLAB code

• OpenSoundControl (OSC) interface [Wright&al.2003]

• Communicates with OpenMusic and Max/MSP

Prototype

• MATLAB code

• Sound target analysis (feature extraction)

• Constrained multiobjective orchestration search

• User interaction - Listening preferences inference mechanism

• Enhancing timbre exploration with multiple views of solution set

• Manual editing / Auto-repairing and auto-transforming procedures

Prototype

• MATLAB code

• Sound target analysis (feature extraction)

• Constrained multiobjective orchestration search

• User interaction - Listening preferences inference mechanism

• Enhancing timbre exploration with multiple views of solution set

• Manual editing / Auto-repairing and auto-transforming procedures

• Currently used by composers at IRCAM

• Standalone application packaging in progress

More examples

• Car horn

• Tibetan horn

• No pre-recorded sound?

• Dynamic target

• Writing electronics

Contents

Conclusions

• Multiobjective combinatorial optimization model for the discovery of efficient solutions that approach a timbre target with a combination of instrument sounds

Conclusions

• Local search symbolic constraint solver that addresses compositional context issues

Conclusions

• Collaborative strategy for combining both methods in a single search process that handles potentially conflicting objectives

Conclusions

• Collaborative strategy for combining both methods in a single search process that handles potentially conflicting objectives

• Operational orchestration prototype already used in real-world musical situations

Current limitations

Conclusions

Current limitations

• Instrumental knowledge

Conclusions

Current limitations

• Timbre description

Conclusions

Current limitations

• No model for unisons

Conclusions

Current limitations

• Some playing styles cannot be handled

Conclusions

Current limitations

• Some playing styles cannot be handled

• Global constraints only

Conclusions

Anyway ...

Speakings (Jonathan Harvey)[2008] For orchestra and electronics

Anyway ...

Harmonic background line(beginning of the 3rd part)

Anyway ...

Orchestration :- 22 ostinato repetitions- Temporal evolution controlled by constraints

Anyway ...

Anyway ...Speakings (Jonathan Harvey) - Created August 19th, 2008 in Royal Albert Hall, London (BBC Scottish Orchestra, director Ilan Volkov)

Future work

Conclusions

Future work

• Automatic criteria inference for high dimension problems

Conclusions

Future work

• Automatic constraint inference from target analysis

Conclusions

Future work

• Attack / sustain mechanisms

Conclusions

Future work

• Emergence (and disappearance)

Conclusions

Future work

• Temporal model for “segmented” targets

Conclusions

Future work

• Temporal model for “segmented” targets

• Temporal model for “articulated” targets

Conclusions

t h a n k y o u

Computational Approach of Musical Orchestration

Documents

Oracle Fusion Distributed Order Orchestration …• View orchestration orders and orchestration processes. • Identify fulfillment lines and orchestration processes with exceptions

1. Introduction - HARVEST...musical” describes a piece of chamber theatre that includes orchestration and live musicians. Unlike “chamber opera” which Donald Jay Grout described

L'armonia sonora: Continuo Orchestration in Monteverdi's ... · continuo. To achieve the desired effect of musical speech (in armonia ... in Cavalieri's association of the guitar

Orchestration Roles to Facilitate Networked Innovation … · Orchestration Roles to Facilitate Networked ... This study examines orchestration roles ... Orchestration Roles to Facilitate

ISMIR 2004: Tutorial on Musical Knowledge, …ismir2004.ismir.net/download/ec-2004ismirTutorial.pdfISMIR 2004: Tutorial on Musical Knowledge, Computational Models and Retrieval October

Solutions Architect, Product Development, Orchestration ...F5 Networks Nathan Pearce Solutions Architect, Product Development, Orchestration & Automation . Programmability & Orchestration

Principles of Orchestration and the Analysis of Musical

App Orchestration 2 - docs. · PDF fileGetting Started with Citrix App Orchestration 2.6 ... Documentation and support for App Orchestration • App Orchestration in Citrix eDocs:

Enterprise Orchestration of Containers with Policies … … · Enterprise Orchestration of Containers with Policies using TOSCA ... Senlin Container Clustering, ... Orchestration

Chapter 5 Computational Analysis of Musical Formleve/pdf/Computational_analysis... · Chapter 5 Computational Analysis of Musical Form Mathieu Giraud, Richard Groult, and Florence

Computational Aspects of Musical Rhythms

Orchestration Workflow Operations - Cisco...Provisioning Vblock Orchestration Workflows ThefollowingVblockorchestrationworkflowexamplesareprovidedtoshowhowapre-definedorchestration

· Web viewCraig is an accomplished multifaceted musician with professional experience in composing, arranging, orchestration, song writing, musical direction, conducting and live

Computational Modelling of Music Cognition and …mas03dm/papers/WigginsPearce... · Chapter 1 Computational Modelling of Music Cognition and Musical Creativity Geraint A. Wiggins,

Using Order Orchestration Orchestration Distributed · PDF fileOracle Fusion Distributed Order Orchestration Using Order Orchestration 2Process Sales Orders 22 Overview

Orchestration Using Supply Chain Financial€¦ · Financial Orchestration Category ..... 8 3 Monitor Financial Orchestration 9

BizTalk Orchestration · BizTalk Orchestration handles the complexity for the user. Concurrency and Synchronization. BizTalk Orchestration ... Complete Instrumentation

MUSIC PRODUCTION SYNTHESIZER - Yamaha Corporation · 2019-01-24 · MUSIC PRODUCTION SYNTHESIZER. A single musical motif can evolve to become a magnificent orchestration. The Yamaha

BMus (Hons) Music · 2020. 1. 9. · Creative Orchestration and Arrangement, Live Electronics, Advanced Topics in Music History, Musical Structure and Understanding C2 compose music

Perceiving Changes of Sound-Source Size Within Musical ... · Perceiving Changes of Sound-Source Size Within Musical Tone Pairs ... composer and orchestration treatise writer Charles