Computational Approach of Musical Orchestration

Computational Approach ofMusical Orchestration

Grégoire Carpentier - Ph.D. DefenseIRCAM - Music Representation Group

Constrained Multiobjective Optimization of Sound Combinations in Large Instrument Sample Databases

Supervisors : Gérard Assayag (IRCAM) & Emmanuel Saint-James (LIP6)

December 16th, 2008

1

Computational Approach of Musical Orchestration

Contents

2


Contents

1. Computer-Aided Composition / Orchestration

2


Contents


2. Stating the Orchestration Problem

2


Contents



3. Combinatorial Optimization Problem(the orchidée algorithm)

2


Contents




4. Constraint Solving Problem(the cdcsolver algorithm)

2


Contents





5. Prototype of Orchestration Tool - Musical Examples

2


Contents






6. Conclusions and Future Work

2


Contents







3

CAC / Orchestration

Computer-Aided Composition

4

CAC / Orchestration


• Formalizing of musical structures

4

CAC / Orchestration



• Formalizing processes

4

CAC / Orchestration




• Tackling different aspects of musical writing:

4

CAC / Orchestration




• Tackling different aspects of musical writing:Durations, rythm

4

CAC / Orchestration





Pitches, melody, harmony

4

CAC / Orchestration






Time representations

4

CAC / Orchestration







Spatialization

4

CAC / Orchestration







Spatialization

Sound synthesis

4

CAC / Orchestration







Spatialization

Sound synthesis

Instrumental timbre ?

4

CAC / Orchestration

Orchestration - timbre

Symbolicworld(score)

Soundworld

(timbre)

5

CAC / Orchestration

Orchestration : projection

6

CAC / Orchestration

Orchestration : projection

Soundworld

(timbre)Projection

6

CAC / Orchestration

Orchestration : induction

7

CAC / Orchestration


Soundworld

(timbre)

7

CAC / Orchestration


Soundworld

(timbre)

7

CAC / Orchestration


Soundworld

(timbre)

Induction

7

CAC / Orchestration


Soundworld

(timbre)

Induction

7

CAC / Orchestration

Orchestration in practice

8

CAC / Orchestration


• Personal knowledge

8

CAC / Orchestration


• Personal knowledge Necessarily restricted

8

CAC / Orchestration


• Personal knowledge Necessarily restricted

• Orchestration Treatises

- [Berlioz1855]

- [Koechlin1943]

- [Piston1955]

- [Rimski-Korsakov1912†]

8

CAC / Orchestration



Obviously outdated

Necessarily restricted


- [Berlioz1855]

- [Koechlin1943]

- [Piston1955]


8

CAC / Orchestration



Obviously outdated



- [Berlioz1855]

- [Koechlin1943]

- [Piston1955]


• Current computer orchestration tools

- [Psenicka2003]

- [Rose&Hetrick2005]

- [Hummel2005]

8

CAC / Orchestration



Obviously outdated


Monoobjective approach of timbre


- [Berlioz1855]

- [Koechlin1943]

- [Piston1955]



- [Psenicka2003]


- [Hummel2005]

8

CAC / Orchestration



Obviously outdated


Monoobjective approach of timbre

Do not handle combinatorial issues


- [Berlioz1855]

- [Koechlin1943]

- [Piston1955]



- [Psenicka2003]


- [Hummel2005]

8


Contents







9

Statement

Problem statement

10

Statement

Problem statement

How can I use an orchestra to reproduce a timbre target within a given compositional context?

10

Statement

Problem statement


How can I find an instrument sound combination that:

10

Statement

Problem statement



- Best matches a given target sound?

10

Statement

Problem statement




- Fits writing constraints specified by the composer?

10

Statement

Problem statement





In computer terms:

10

Statement

Problem statement





In computer terms:

- A combinatorial optimization problem defined on a timbre description scheme

10

Statement

Problem statement





In computer terms:

- A combinatorial optimization problem defined on a timbre description scheme

- A constraint solving problem on the variables of musical writing

10

Statement

Strategy

11

Statement

Strategy

- Combinatorial optimization problem: the orchidée algorithm

11

Statement

Strategy


- Constraint solving problem: the cdcsolver algorithm

11

Statement

Strategy



- Collaboration between the two methods

11

Statement

Strategy




targetcontext

11

Statement

Strategy




targetcontext

featurescontraints

11

Statement

Strategy




targetcontext

featurescontraints

orchidéecdcsolver

11

Statement

Strategy




targetcontext

featurescontraints

orchidéecdcsolver

contextualizedsolutions

SampleDB

11


Contents







12

Combinatorial Optimization (orchidée)

Timbre similarity (1)

13



• Timbre perception is multidimensional

14

http://127.0.0.1/~Gregoire/soutenance/multidim/





15






• Correlation between perceptual dimensions and sound features [McAdams+1995] [Peeters2004]:

15







• Spectral centroid <=> brightness

15








• Attack time <=> percussive / sustained sounds

15






SampleDB




15






Sample

SampleDB




15






feature 1feature 2

feature K

Sample

SampleDB




15



• Hypothesis: Sound combination feature set can be predicted from the values of components features



16

d 1d 2

d K




16

d 1d 2

d K




16

d 1d 2

d K

d 1d 2

d K

d 1d 2

d K

d 1d 2

d K

d 1d 2

d K




16

d 1d 2

d K

Prediction ofcombination

features

d 1d 2

d K

d 1d 2

d K

d 1d 2

d K

d 1d 2

d K

d 1d 2

d K




16

d 1d 2

d K

Prediction ofcombination

features

d 1d 2

d K

d 1d 2

d K

d 1d 2

d K

d 1d 2

d K

Dissimilarity measuresalong each perceptual

dimension

d 1d 2

d K




16


Sound features

• Spectral centroid (brightness) [McAdams+1995]

• Spectral spread (volume) [Chiasson2007]

• Resolved partials (harmonic tone)

• Time and noise features are not considered

• Preliminary tasks:

• Sound combinations features prediction functions

• Perceptual dissimilarity functions

17


Multiobjective approach

18



• Relative importance of perceptual dimensions cannot be known without prior information on listening preferences

18




• Multiobjective optimization: Jointly minimize

18





• Pareto dominance:

18





T


19





• Set of optimal solutions (implicitly corresponding to different listening preferences)

T


19

http://127.0.0.1/~Gregoire/soutenance/pareto/

http://127.0.0.1/~Gregoire/soutenance/pareto/


Formalizing the optimization problem

20



• Orchestra composed of instruments: variables problem

20




• Domain of each variable:

20




• Domain of each variable:

• Problem:

(P)

20


A NP-hard problem

21


A NP-hard problemConsider a 5000 sample sound database. Consider an 11 instruments orchestra in which only 4 can play simultaneously a given 4-note chord. Thus:

21


A NP-hard problem

• There are around a billon a feasible combinations

Consider a 5000 sample sound database. Consider an 11 instruments orchestra in which only 4 can play simultaneously a given 4-note chord. Thus:

21


A NP-hard problem


• Computing their features takes around 20 minutes with 3 perceptual dimensions


21


A NP-hard problem




Taking into account that:

21


A NP-hard problem





• A big orchestra may contain around a hundred instruments;

21


A NP-hard problem






• There might be around ten perceptual features;

21


A NP-hard problem







• Instrument sound databases may hold hundred of thousands items;

21


A NP-hard problem





Complete resolution methods cannot help here.

Metaheuristics are required.



• Instrument sound databases may hold hundred of thousands items;

21


GA optimization: the orchidée algorithm

22



• Evolutionary algorithm (using a population of individuals)

22




• Each individual is represented by a set of genes (chromosome)

22





• Orchidée: integer tuple encoding (“orchestra” representation)

22






22






• Orchidée: user preferences guessing mechanism

22

Initialpopulation



23

Initialpopulation

CurrentPopulation



24

Initialpopulation

CurrentPopulation

Evaluation



25

Initialpopulation

CurrentPopulation

Evaluation Selection



26

Initialpopulation

CurrentPopulation

Evaluation SelectionGenetic

Operators



27

Initialpopulation

CurrentPopulation


OperatorsCurrent

Population



28

Initialpopulation

CurrentPopulation


OperatorsCurrent

Population

Preservingdiversity



29

Initialpopulation

CurrentPopulation


OperatorsCurrent

Population

Preservingdiversity

iter max ?

N

Finalpopulation O



30

Goals :

Initialpopulation

CurrentPopulation


OperatorsCurrent

Population

Preservingdiversity

iter max ?

N

Finalpopulation O



31

Initialpopulation

CurrentPopulation


OperatorsCurrent

Population

Preservingdiversity

iter max ?

N

Finalpopulation O

Goals :

• Convergence towards Pareto front



32

Initialpopulation

CurrentPopulation


OperatorsCurrent

Population

Preservingdiversity

iter max ?

N

Finalpopulation O

Goals :

• Convergence towards Pareto front

• Diversity along the Pareto front



33


User preferences (1)

34



orchidée

34



orchidée

Intermediarysolutions

34



orchidée


User choice

34



orchidée


User choice

ImplicitPreferences

34



orchidée


User choice

ImplicitPreferences

34



orchidée


User choice

ImplicitPreferences

orchidée(+ preferences)

34



35


• Weighted Chebychev norm [Jaszkiewicz2002] :


35



soit :


35



soit :


35



soit :


35



36

• Each efficient solution corresponds to a weight set

• Fundamental property [Steuer1986] :



36


• Computing weights from criteria [Carpentier2008] :




36






36






36






36






36






36


• Orchidée computes configurations fitness thanks to a Chebychev norm





36


• Orchidée computes configurations fitness thanks to a Chebychev norm• When preferences are unknown weights are randomly drawn at each

generation (multiobjective optimization)





36


• Orchidée computes configurations fitness thanks to a Chebychev norm• When preferences are unknown weights are randomly drawn at each

generation (multiobjective optimization)• When preferences are known weights are fixed (monoobjective

optimization)





36


Contents







37

CSP / Context (cdcsolver)

Compositional context

38



• Any musical material comes within a musical gesture

38




• Depending on this gesture all configurations may not be feasible

38




• Depending on this gesture all configurations may not be feasibleSample

feature 1feature 2

feature K

38





attribute 1attribute 2

attribute N

Sample

feature 1feature 2

feature K

38






attribute N

Sample

feature 1feature 2

feature K

pitchesdynamicsplaying stylesmute

38






attribute N

Sample

feature 1feature 2

feature K


• Modeling context with global constraints on attributes :

38






attribute N

Sample

feature 1feature 2

feature K



- “Between 8 et 12 instruments”

38






attribute N

Sample

feature 1feature 2

feature K




- “No more than 3 fortissimo”

38






attribute N

Sample

feature 1feature 2

feature K





- “At least two different pitches”

38






attribute N

Sample

feature 1feature 2

feature K






- “All strings play with a mute”

38






attribute N

Sample

feature 1feature 2

feature K






- “All strings play with a mute”

• Local search / soft constraints approach: Iteratively update a single configuration to minimize a set of cost functions

38

Design / Conflict?


39

Design / Conflict?

• Design constraints: Anything required in the orchestration


39

Design / Conflict?


• Conflict constraints: Anything to avoid in the orchestration


39

Design / Conflict?



• Design constraints may be satisfied by instantiating free variables


39

Design / Conflict?



• Design constraints may be satisfied by instantiating free variables

• Conflict constraints may be satisfied by freeing instantiated variables


39

CDCSolver


40

CDCSolver

3 neighborhood heuristics:


40

CDCSolver

1. If a conflict constraint is violated, first update an instantiated variable.



40

CDCSolver


2. If a design constraint is violated, first update a free variable.



40

CDCSolver



3. There is a priority variable to change [Codognet2002]. Decision rule is based on a min-conflict heuristic.



40

CDCSolver





1 move heuristic:


40

CDCSolver




4. In the current neighborhood choose the configuration that minimizes the global cost function (min-conflict).


1 move heuristic:


40

CDCSolver






1 move heuristic:

1 cycle handling heuristic:


40

CDCSolver






1 move heuristic:

1 cycle handling heuristic:

5. Handle cycles and local minima with a short term memory scheme (tabu list [Glover1997])


40

Combining orchidée and cdcsolver


41

Combining orchidée and cdcsolver• Genetic algorithms are not well suited for constrained problems


41

Combining orchidée and cdcsolver• Genetic algorithms are not well suited for constrained problems

• Repairing every inconsistent configuration is inefficient


41


Initialpopulation

Currentpopulation


operatorsCurrent

population

Preservingdiversity

iter max ?

N

Finalpopulation O

• Genetic algorithms are not well suited for constrained problems

• Repairing every inconsistent configuration is inefficient


41

Initialpopulation

Currentpopulation


operatorsCurrent

population

Preservingdiversity

iter max ?

N

Finalpopulation O

cdcsolver



42

Initialpopulation

Populationcourante


operatorsCurrent

population

Preservingdiversity

iter max ?

N

Finalpopulation O

cdcsolver

Penalize inconsistentsolutions



43

Initialpopulation

Populationcourante


operatorsCurrent

population

Preservingdiversity

iter max ?

N

Finalpopulation O

cdcsolver


Configurations comparing rules



43

Initialpopulation

Populationcourante


operatorsCurrent

population

Preservingdiversity

iter max ?

N

Finalpopulation O

cdcsolver


Configurations comparing rules1. Choose the more consistent configuration



43

Initialpopulation

Populationcourante


operatorsCurrent

population

Preservingdiversity

iter max ?

N

Finalpopulation O

cdcsolver


Configurations comparing rules1. Choose the more consistent configuration2. Choose the configuration with highest fitness according to the current

Chebychev norm



43


Contents







44

Prototype

Prototype - Examples

45

Prototype

• MATLAB code

• OpenSoundControl (OSC) interface [Wright&al.2003]

• Communicates with OpenMusic and Max/MSP


45

Prototype

• MATLAB code



• Sound target analysis (feature extraction)

• Constrained multiobjective orchestration search

• User interaction - Listening preferences inference mechanism

• Enhancing timbre exploration with multiple views of solution set

• Manual editing / Auto-repairing and auto-transforming procedures


45

Prototype

• MATLAB code



• Sound target analysis (feature extraction)

• Constrained multiobjective orchestration search

• User interaction - Listening preferences inference mechanism

• Enhancing timbre exploration with multiple views of solution set

• Manual editing / Auto-repairing and auto-transforming procedures

• Currently used by composers at IRCAM

• Standalone application packaging in progress


45

Demo


46


More examples

• Car horn

• Tibetan horn

• No pre-recorded sound?

• Dynamic target

• Writing electronics

47

http://127.0.0.1/~Gregoire/soutenance/exemples/











Contents







48

Conclusions

Conclusions

49

Conclusions

Conclusions

• Multiobjective combinatorial optimization model for the discovery of efficient solutions that approach a timbre target with a combination of instrument sounds

49

Conclusions

Conclusions


• Local search symbolic constraint solver that addresses compositional context issues

49

Conclusions

Conclusions



• Collaborative strategy for combining both methods in a single search process that handles potentially conflicting objectives

49

Conclusions

Conclusions



• Collaborative strategy for combining both methods in a single search process that handles potentially conflicting objectives

• Operational orchestration prototype already used in real-world musical situations

49

Current limitations

Conclusions

50

Current limitations

• Instrumental knowledge

Conclusions

50

Current limitations


• Timbre description

Conclusions

50

Current limitations



• No model for unisons

Conclusions

50

Current limitations




• Some playing styles cannot be handled

Conclusions

50

Current limitations




• Some playing styles cannot be handled

• Global constraints only

Conclusions

50

Anyway ...

51

Anyway ...

Speakings (Jonathan Harvey)[2008] For orchestra and electronics

51

Anyway ...


Harmonic background line(beginning of the 3rd part)

51

Anyway ...



51

Anyway ...



Orchestration :- 22 ostinato repetitions- Temporal evolution controlled by constraints

51

Anyway ...




51

Anyway ...




51

Anyway ...Speakings (Jonathan Harvey) - Created August 19th, 2008 in Royal Albert Hall, London (BBC Scottish Orchestra, director Ilan Volkov)

52

http://127.0.0.1/~Gregoire/soutenance/harvey/



Future work

Conclusions

53

Future work

• Automatic criteria inference for high dimension problems

Conclusions

53

Future work


• Automatic constraint inference from target analysis

Conclusions

53

Future work



• Attack / sustain mechanisms

Conclusions

53

Future work




• Emergence (and disappearance)

Conclusions

53

Future work





• Temporal model for “segmented” targets

Conclusions

53

Future work





• Temporal model for “segmented” targets

• Temporal model for “articulated” targets

Conclusions

53

t h a n k y o u

54

Documents

Computational Approach of Musical Orchestration