Motivic Pattern Extraction in Music, And Application …arima.inria.fr/006/pdf/arima00602.pdf · 16 Reviewed Article — ARIMA/SACJ, No. 36., 2006 Motivic Pattern Extraction in Music,

16 Reviewed Article — ARIMA/SACJ, No. 36., 2006

Motivic Pattern Extraction in Music, And Application to the

Study of Tunisian Modal Music

O Lartillot∗, M Ayari†

∗Department of Music, PL 35(A), 40014 University of Jyvaskyla, Finland†Ircam, Place Igor-Stravinsky, 75004 Paris, France

ABSTRACT

A new methodology for automated extraction of repeated patterns in time-series data is presented, aimed in particular

at the analysis of musical sequences. The basic principles consists in a search for closed patterns in a multi-dimensional

parametric space. It is shown that this basic mechanism needs to be articulated with a periodic pattern discovery system,

implying therefore a strict chronological scanning of the time-series data. Thanks to this modelling global pattern filtering

may be avoided and rich and highly pertinent results can be obtained. The modelling has been integrated in a collaborative

project between ethnomusicology, cognitive sciences and computer science, aimed at the study of Tunisian Modal Music.

KEYWORDS: pattern extraction, time-series data, closed pattern, periodic pattern, music analysis, tunisianmodal music

1 INTRODUCTION

This paper introduces a new methodology for repeatedpattern (or motif) extraction in symbolic sequences,and is applied particularly to the analysis of musicalscores. Among the different approaches that can beconsidered for time-series data analysis, one domainof research that has received much attention is theproblem of extraction of motives, i.e. the discoveryof patterns appearing frequently in time-series data[1, 2, 3, 4]. Indeed, motives may characterize impor-tant aspects of the data, and help discovering newassociation rules. In music too, repeated sequencesof notes are easily perceived by listeners as importantstructures, forming the words of the musical structure.

Lots of research have been carried out in this do-main and numerous interesting solutions have beenproposed. One major problem stems from the struc-tural redundancies logically resulting from this task,which, if not carefully controlled, may provoke com-binatorial explosion and infringe the quality of theresults. Few researches have considered the patterndiscovery problem within this general context. Theapproach presented in this paper follows this ideaof closed pattern, which is defined here in a multi-dimensional parametric space. Another combinatorialredundancy problem, provoked by immediate succes-sion of same patterns, is solved by introducing theconcept of cyclic pattern. The model has been appliedto the automated motivic analysis of musical scores,and in particular to the study of Arabic improvisationsplayed by Tunisian masters.

Email: O Lartillot [email protected], M Ayari

[email protected]

Most music databases contain sound files of per-formance recordings, which correspond to the way mu-sic is commonly experienced. The underlying struc-ture of music, on the other hand, is represented ina symbolic form – the score – that describes musicalpieces regardless of the way they are performed. Thereexist numerous digital formats of symbolic music rep-resentation (MIDI, MusicXML, Humdrum, etc.). Thepattern discovery system described in this paper isapplied uniquely to symbolic representation. A directanalysis on the signal level would arouse tremendousdifficulties. A pattern extraction task on the sym-bolic level, although theoretically simpler, remains ex-tremely difficult to carry out, and its automation hasnot been achieved up to now. Indeed, computer re-searches on this subject hardly offer results close tolisteners’ or musicologists’ expectations. Hence thepattern discovery task is too complex to be under-taken directly at the audio signal, and needs rathera prior transcription from the audio to the symbolicrepresentations, in order to carry out the analysis ona conceptual level.

2 AN INCREMENTAL MULTIDIMENSIONALMOTIVIC IDENTIFICATION

2.1 Definitions

Music is expressed along multiple parametric dimen-sions. This paper will focus on two main dimensions(Figure 1):

• Melodic dimension (melo) defined by pitch dif-ferences between successive notes. (In scores,pitches are represented by the vertical positionof the notes.)

Joint Special Issue — Advances in end-user data-mining techniques 17

• Rhythmic dimension (rhyt) defined by durationsbetween successive notes, and expressed with re-spect to metrical unit. For instance, in a 6/8 met-ric (whose metrical unit is the 8th note) a dotted8th note correspond to the value 1.5.

Figure 1 shows a musical example in traditional mu-sical notation. Below the score is indicated the cor-responding numerical description of the musical ex-ample along the melodic and rhythmic dimensions.For instance, the first value of the melodic dimension(melo = +1) indicates that the second note of thescore is located one step higher than the first note ofthe score on the vertical dimension. In musical term,this means that the second pitch is one step higherthan the first pitch. The first value of the rhythmicdimension (rhyt = 1.5) indicates that the duration ofthe first note (or, equivalently, the temporal distancebetween the first and second note) is equal to 1.5 timesthe basic unit.

Figure 1: Multi-dimensional description of a musical se-

quence.

A repeated succession of descriptions forms a pat-tern, and the occurrences of the pattern correspond toits repetitions in the score. Figure 1 shows two rep-etitions of a pattern, which are squared in the score.Each pattern occurrence is represented as a chain ofstates, each successive state (a, b, c, d, e in figure 1) cor-responding to each successive note of the occurrence.The pattern itself – i.e., the abstract entity that uni-fies all these occurrences – is represented at the lowerpart of the figure by the same chain of state. Thetransitions between successive states, indicated belowthe chain of states, describe the successive intervalsbetween successive notes of the pattern.

2.2 Identification of similarities

Patterns are generally not exactly repeated but trans-formed in multiple ways. These patterns should there-fore be detected through an identification of theirdifferent occurrences beyond their apparent diversi-ties. Current approaches follow two different strate-gies. One is based on numerical similarity, and toler-ates a certain amount of dissimilarity between com-pared parameters [5, 6]. The main drawback of thisstrategy arises from the impossibility of fixing pre-cisely similarity thresholds, on which identification de-cision are based, and hence insuring relevant analyses.

Reference cognitive studies [7], on the other hand,assert that similarity does not come from numericaldistance minimization, and propose instead an alter-native strategy based on exact identification alongmultiple musical dimensions of various specificity lev-els. Several approaches to pattern discovery followthis second strategy of identification along differentmusical dimensions [8, 9] and search for repetitionsalong each different dimension and product of dimen-sions.

Nonetheless there exist patterns that are progres-sively constructed along variable successive musical di-mensions. These heterogeneous patterns cannot beidentified by traditional approaches. For instance,each line of the score in figure 2 contains a repeti-tion of a same pattern: in the first half, both melodicand rhythmic dimensions are repeated whereas, in thesecond half, only the rhythmic dimension is repeated.The model presented in this paper is able to discoverheterogeneous patterns.

3 COMBINATORIAL REDUNDANCY FIL-TERING

This section presents the basic problem of pattern dis-covery and introduces the notion of closed pattern.

3.1 Formalisation

Let S =< a1a2 . . . aN > be a sequence of elements ofsome set ai ∈ A. A subsequence Si,l of index i ∈ [1, N ]and of length l ∈ [1, N + 1 − i] is a sequence of theform:

Si,l =< aiai+1 . . . ai+l−1 > . (1)

A sub-sequence Si,k is included in another sub-sequence Sj,l, noted Si,k ⊂ Sj,l when j 6 i and i+k 6j + l. A pattern of length l, denoted P ∈ P(S), isdefined as a repeated sub-sequence:

P ∈ P(S) ⇐⇒ ∃(i, j) ∈ [1, N ]2, P = Si,l = Sj,l. (2)

The support of a pattern P , denoted σ(P ), is thenumber of occurrences of the pattern, i.e.

σ(P ) =∣

∣

{

i ∈ [1, N ], Si,l = P}∣

∣. (3)

3.2 Maximal patterns and closed patterns

The task of discovering repeated patterns leads tocombinatorial problems. Indeed each pattern of length

l contains∑l

i=1 i = l(l−1)2 = O(l2) sub-patterns.

that would be considered as distinct patterns by anybrute force algorithm. The problem can be avoidedby restricting the search to the patterns of highsupport and/or to patterns of pre-specified length[4, 8, 9, 10, 11]. These constraints, in return, sig-nificantly reduces the richness of the analysis.

One common way to solve this problem consists infocusing on the maximal patterns P of the sequenceS, denoted P ∈ M(S), which are patterns of S not


Figure 2: Repetition of a heterogeneous pattern.

included in any other pattern of S [12, 13]:

P ∈ M(S) ⇐⇒

{

P ∈ P(S)�Q ∈ P(S), P ⊂ Q.(4)

For instance, pattern aij (which can simply bedenoted by its last state j) in figure 3 is a simplesuffix of pattern abcde (or e). It does not need to beexplicitly represented, since the set of its occurrences(or pattern class) can be directly deduced from theclass of its superpattern e.

This heuristic enables a significant reduction ofthe number of discovered pattern, but leads also to aloss of information. Indeed, not all the sub-patternsmay be immediately reconstructed knowing the max-imal patterns. In figure 4, for instance, the patternclass of j cannot be directly deduced from the patternclass of e, and should therefore be explicitly repre-sented in the final analysis. This corresponds to theconcept of closed pattern [13]. A pattern P will becalled closed, denoted P ∈ C(S), if and only if thereexists no proper super-pattern Q of same support :

P ∈ C(S) ⇐⇒

P ∈ P(S)�Q ∈ P(S),

{

P ⊂ Q

σ(P ) = σ(Q).

(5)

The concept of closed pattern offers a compactand lossless description of the musical piece: it al-lows an exhaustive description of the pattern config-urations and avoids any global selection based, forinstance, on minimum support threshold, as in tra-ditional approach for mining association rules [1, 2].Even pattern with only two occurrences are includedin the description. Some constraints, however, havebeen added to the framework, that reduce the solutionspace following cognitive heuristics related to musicperception. For instance, a constraint takes into ac-count the limitations of short-term memory: namely,for a pattern to be detected, the temporal distance be-tween at least two of its occurrences should not exceeda given threshold.

4 MULTI-DIMENSIONAL CLOSED PAT-TERNS

The model presented in this paper looks for closedpattern in musical sequences. For this purpose, thenotion of inclusion relation between patterns found-ing the definition of closed patterns needs to be gen-eralized to the multi-dimensional parametric space ofmusic, defined in section 2.1.

4.1 Formalisation of the problem

Musical patterns are represented with the help ofa conceptual framework that defines objects associ-ated with different kinds of attributes [14]. These at-tributes consist not only of the different musical di-mensions, but also of the different sub-patterns andsuper-patterns. The objects of the pattern descrip-tions are the successive notes of the musical sequenceforming the set N (S). Each note ni ∈ N (S) relatesto a specific temporal context, defined by the part of

the musical sequence concluded by this note ni, that isto say, the subsequence < n1 . . . ni >.

Each note ni is described firstly by the differ-ent musical characteristics of the preceding interval:−−−−→ni−1ni.

D0,pdesc

(ni) : desc(−−−−→ni−1ni) = p,

desc ∈ {melo, rhyt}, p ∈ desc.(6)

Each note ni is also described by the musical charac-teristics of the older intervals:

Dj,pdesc

(ni) : desc(−−−−−−−→ni−j−1ni−j) = p,

desc ∈ {melo, rhyt}, p ∈ desc.(7)

Then the pattern description of the sequence S

may be expressed as a formal context (N (S),D, I) [14]where :

• the set of objects is N (S): the set of notes in S,

• the set of attributes is D: the set of elementarymusical descriptions defined by equations 6 and7,

• and I is the binary relation between N (S) andD, called incidence, defined by:

(ni, δ) ∈ I ⇐⇒ δ(ni) is true. (8)


Figure 3: On the score are squared the occurrences of two patterns abcde and aij. The occurrences of these patterns are

represented below the score. Pattern descriptions are represented over the score, using a tree representation justified in

section 6.1. Pattern aij, suffix of abcde with same support (2 occurrences), is a non-closed pattern that does not need to

be explicitly represented.

Figure 4: Same convention than in figure 3. Pattern aij, whose support (4 occurrences) is now greater than the support

of abcde or abcdefgh (2 occurrences each) is a closed patterns and needs to be explicitly represented.

The derived description C′ of a set of notes C ⊂N (S) is defined as the common description of all thesenotes:

C′ ={

δ ∈ D | ∀n ∈ A, (n, δ) ∈ I}

. (9)

The notes in C are therefore occurrences of a samepattern, which is maximally described by C′.

The derived class D′ of a complex description D ⊂D is dually defined as the set of notes complying withthis description:

D′ ={

n ∈ N (S) | ∀δ ∈ D, (n, δ) ∈ I}

. (10)

The pattern discovery task consists in finding exhaus-tive class D′ sharing a same description D. The trou-ble is, lots of different descriptions Di may lead tosame classes D′

i.

4.2 A representation of patterns as formalconcepts

The derivators operations defined by equation 9 and10 establish a Gallois connection between the powerset lattices on N (S) and D [14]. The Gallois connec-tion leads to a dual isomorphism between two closuresystems, whose elements, called formal concepts of theformal context (S(S),D, I) corresponds exactly to theclose patterns P = (C, D), verifying:

C ⊂ N (S), D ⊂ D, C′ = D, and D′ = C. (11)

For a close pattern P = (C, D), C is called the extent

of D and D the intent of C. We may simply call C

and D respectively the class and the description of P .Hence, for a set of patterns Pi = (D′

i, Di) of sameclass D′

i = C, the close pattern P = (C, D) is de-scribed using the derived operator C′ defined in equa-tion 9: it contains all the elementary descriptions com-mon to all notes of the class C. In other words, closedpatterns are described as precisely as possible.

Closed patterns, or formal concepts, are naturallyordered by the subconcept-superconcept relation de-fined by

(C1, D1) < (C2, D2) ⇐⇒ C1 ⊂ C2 ( ⇐⇒ D2 ⊂ D1).(12)

This subconcept-superconcept relation can also becalled specificity relation. In the equation above, forinstance (C1, D1) is more specific than (C2, D2) andvice versa.

In particular, a pattern is more specific than itssuffixes, since the description of the suffixes are strictlyincluded in the description of the pattern.

4.3 Illustration

For instance, pattern abcde (in figure 5) featuresmelodic and rhythmic descriptions, whereas patternafghi only features its rhythmic part. Hence pat-tern abcde can be considered as more specific than


pattern afghi, since its description contains more in-formation. The less specific pattern afghi is a closedpattern, since its third occurrence is not an occurrenceof the more specific pattern abcde, and is therefore notfiltered out.

5 CYCLIC PATTERNS

In this section, we present another important factor ofredundancy that, contrary to closed patterns, has notbeen studied in current general algorithmic researches.

5.1 Periodic sequences

Periodicity occurs when a given pattern is successivelyrepeated, such as the repetition of the pattern < 1, 2 >

in figure 6. Periodicity leads to combinatory explo-sion, since all possible periods (i.e. all the possiblerotations of one period, such as abc and pqr in theexample) can be considered as patterns, as well as allpossible concatenations of periods and their differentprefixes, such as patterns abcd, pqrs, abcde, etc. onthe left panel of the figure. These redundant struc-tural artefacts should be replaced by a compact repre-sentation that would explicitly describe the structuralproperties of such configuration. For this purpose,we propose to model periodic sequence through cyclicgraphs. A cyclic pattern chain (CPC) is constructedfrom an originally acyclic pattern chain (APC) repre-senting one period of the cycle, where a transition isadded from the last state to the first state. In thisway, the whole local periodicity can be represented bya single pattern occurrence chain where each succes-sive state is uniquely linked to the successive phasesof the CPC. An example of CPC is given on the rightpanel of figure 6.

Each successive state of a pattern chain is re-lated to each successive prefix of the pattern occur-rence. For this reason, concerning the pattern occur-rence chain representing the entire periodicity, the firststates that represent the first period should not be as-sociated with the CPC since they are already associ-ated with the APC of that period. On the contrary,the states of the pattern occurrence chain followingthe first period will be associated to the CPC, sincethey represent a configuration that is actually specificto the periodicity. In this way, the CPC may be con-sidered as a child of the APC, as can be seen in thefigure.

This additional concept immediately solves the re-dundancy problem. Indeed, each type of redundantstructure considered previously are non-closed suffix

of prefix of the long pattern chain, and will thereforenot be represented any more. For instance, in figure6, pattern abcd is a suffix of pattern b′ with same sup-port. Pattern abcd is therefore non-closed and canbe discarded. Idem for patterns abcde, abcdef , etc.Pattern pq, suffix of abc with same support, is alsonon-closed. Pattern pqr, suffix of b′ with same sup-port, is non-closed too. Idem for patterns pqrs, pqrst,etc. Hence only one pattern occurrence remains, that

cover the whole periodic sequence, as displayed on theright panel of the figure.

But this compact representation will be possibleonly if the initial period (corresponding to the APC) isconsidered and extended before the other possible pe-riods. That is to say, in figure 6, the APC abc shouldbe considered before pqr. This shows therefore thatthe sequence needs to be scanned in a chronological

way. This justifies therefore the incremental approachfollowed by the algorithmic realisation of the model-ing, presented in section 6.1.

5.2 Related works

Researches are dedicated to the automated discoveryof periodic patterns in time-series data [15, 16, 17, 18].But as the search is focused on periodic patterns only,no interaction is proposed with acyclic pattern dis-covery. Hence, although offering interesting descrip-tions of time series data, they cannot be used in or-der to solve the combinatory problem presented in theprevious paragraph. In our approach, on the otherhand, the periodic pattern problem is deeply articu-lated with the acyclic pattern discovery process, in-suring the compactness of the results.

A simpler solution to the combinatory problemconsists in forbidding overlapping between patterns[3]. But this heuristics presupposes that time-seriesdata are segmented into one-dimensional series of suc-cessive segments. Time-series data do not all fulfil thisrequirement: musical sequences, in particular, maysometimes be composed of multi-levelled hierarchy ofstructures. Another solution is to control the combi-natorial explosion by selecting, once the analyses com-pleted, patterns featuring minimal temporal overlap-ping between occurrences [8]. But as the selection isinferred globally, relevant patterns may be discarded.Besides combinatorial redundancy remains problem-atic since the filtering is carried out after the actualanalysis phase. Our focus on local configurations en-ables a more precise filtering.

5.3 The figure/ground rule

Another kind of redundancy appears when occur-rences of a pattern – such as pattern acd =< 1A, 2A >

in figure 7 – are superposed to a cyclic pattern (b′),such that the pattern acd is more specific than the cy-cle period (b′ simply representing the successive rep-etition of As). In this case, the intervals that followthese occurrences are identical, since they are relatedto the same state (b′) of the cyclic pattern. Logicallythe pattern could be extended following the successiveextensions of the cyclic patterns (leading to patterne, and so on). This phenomenon, which frequentlyappears, leads to another combinatorial proliferationof redundant structures if not correctly controlled byrelevant mechanisms. On the contrary, following theGestalt Figure/Ground rule, the pattern acd can beconsidered as a specific figure that emerges above theperiodic background. Following the Gestalt rule, thefigure cannot be extended (into d) by a description


Figure 5: The rhythmic pattern afghi is less specific than the melodico-rhythmic pattern abcde.

Figure 6: Multiple successive repetitions of pattern abc =< 1, 2 > induce a complex intertwining of non-perceived

structures (represented in grey, left panel) that can be filtered out using cyclic patterns (right panel).

that can be simply identified with the background ex-tension. This rules shows the interest of integratingcognitive rules into the model, as these rules concernas much the perceptive adequacy of the results thanthe computational efficiency of the process.

6 IMPLEMENTATION DETAILS

6.1 Incremental pattern construction

This paragraph describes the basic mechanism of pat-tern extraction. As explained in section 5.1, it consistsin an online process that progressively reads the musi-cal sequence in a chronological order and that extractsa list of patterns. This list of patterns can be repre-sented as a prefix tree, since two motives with sameprefix can be considered as two different continuationsof this prefix.

The basic principle of the algorithm refers to asso-ciative memory, i.e. the capacity of relating items thatfeature similar properties. The associative memory ismodeled through inverted lists related to the differentmusical parameters (i.e. melodic and rhythmic dimen-sions). A first set of tables store the intervals of thepiece with respect to their values along each differentmusical dimension. For instance, two tables (Figure3, line a) store the intervals of the score according totheir melodic and rhythmic values. The melodic tableshows that the first interval of each bar shares samemelodic value melo = +1, and, the rhythmic tableindicates another identity rhyt = 1.5.

Intervals sharing a same value form occurrencesof an elementary pattern that simply represents this

particular interval parameter. The elementary pat-tern is represented as a child (here b) of the root ofthe pattern tree (a). Each time a new pattern is cre-ated, new tables (at the right of node b) store all thepossible intervals that immediately follow the occur-rences of the new pattern (b). When any identity isdetected in these new tables, a new pattern is createdas an extension of the previous one (c, as an exten-sion of b), and is represented as a child in the patterntree, and so on. This algorithm enables a progressivediscovery of the successive extensions of each pattern,either homogeneous or heterogeneous, as defined insection 2.2: the selection of musical dimensions defin-ing each successive extension of a pattern may vary.For instance, in Figure 8, the last extension of patternabcde is simply melodic since the rhythm of the lastinterval in each occurrence is different. Besides, ad-ditional constraints have been integrated in order toinsure a minimal continuity along these variable suc-cessive musical dimensions.

6.2 Avoiding redundant description of patternoccurrences

We have prolonged the attempt to optimize patterndescriptions by adding a principle of maximally spe-cific descriptions of pattern occurrences: when a pat-tern occurrence is discovered (pattern e in Figure 5),all the occurrences of less specific patterns (pattern i)are not superposed on it, since they do not bring addi-tional information, and can be directly deduced fromthe most specific pattern occurrence (e) and from thespecificity relation (between e and i).


Figure 7: Pattern c is a specific figure, above a background generated by the cyclic pattern b′.

Figure 8: Progressive construction of pattern abcde.

The less specific description should be taken intoaccount implicitly though, because their extensionsmay sometimes lead to specific descriptions. For in-stance (Figure 9), groups 1 and 3 are occurrences ofpattern h, and groups 3 and 4 are occurrences of pat-tern d. Since pattern d is more specific, the less spe-cific pattern h does not need to be associated withgroup 4. However in order to detect groups 2 and 5 asoccurrences of pattern l, it is necessary to implicitlyconsider group 4 as an occurrence of pattern h. Hence,even if pattern h, since less specific than d, was notexplicitly associated with group 4, it had to be consid-ered implicitly in order to construct pattern l. Implicitinformation is reconstituted through a traversal of thepattern network along specificity relations.

6.3 General and specific cycles

The integration of the concept of cyclic pattern in themultidimensional musical space requires a generalisa-tion of specificity relations, defined in previous section,to cyclic patterns. A cyclic pattern C is considered asmore specific than another cyclic pattern D when thesequence of description of pattern D is included in thesequence of description of pattern C. For instance, fig-ure 10 displays four different cycles, the less specificcycle d′′ � f describes the alternation of 1 and 2, themost specific cycle b′′ � g′ describes the alternation of1A and 2B, and the two other cycles b′ � c′ and d′ � e′

are in-between in the specificy graph. All these fourcycles forms therefore an oriented graph called specific

graph (SG) whose root is the less specific cycle d′′ � f .

7 GENERAL RESULTS

This model was first developed as a library of Open-

Music [19], called OMkanthus. A new version will beincluded in the next version 2.0 of MIDItoolbox [20],a Matlab toolbox dedicated to music analysis. Themodel can analyse monodic musical pieces (i.e., con-stituted by a series of non-superposed notes) and high-light the discovered patterns on a score.

7.1 Experiments

The model has been tested with different musical se-quences taken from several musical genres (classicalmusic, pop, jazz, etc.). Table 1 shows some results.The experiment has been undertaken with version0.6.8 of OMkanthus on a 1-GHz PowerMac G4. Amusicologist expert has validated the analyses. Theproportion of patterns considered as relevant is dis-played in the table.

The analysis of a medieval song called Geisslerlied

– sometimes used as a reference test for formalisedmotivic analysis – gave quite relevant results. Theanalysis has been actually carried out on a slight sim-plification of the actual piece presented in [21], exclud-


Figure 9: Group 4 can be simply considered as occurrence of pattern d. However, in order to detect group 5 as occurrence

of pattern l, it is necessary to implicitly infer group 4 as occurrence of pattern h too.

Figure 10: More detailed analysis of the perceived cyclic configurations.

Table 1: Results of analyses, either melodic (M) or melodico-rythmic (M+R), performed by OMkanthus 0.6.8.

Musical sequence Anal. Pattern classes Comp.

Name Notes type Disc. Relv. Succ. time

Geisslerlied 108 M 6 5 83% 2.2 sec.medieval song

Au clair de la lune 44 M+R 21 5 24% 5.6 sec.folk song

Bach, Invention in D minor 283 M 49 34 69% 37.6 sec.BWV 775

Mozart, Sonata in A K331 36 M+R 14 10 71% 0.8 sec.1st theme, 1st half, melody

The Beatles 390 M 14 10 71% 28.1 sec.Obla Di Obla Da

ing local motivic variations out of reach of the currentmodelling.

The melodico-rhythmic analysis of the Frenchsong Au clair de la lune posed problems: 21 patternswere discovered from a 44-note long sequence. This isdue to the fact that the successive steps of progressivegeneralisation or specification of cycles are currentlymodelled using distinct intermediary cyclic patterns.The inference of these redundant cyclic patterns willbe avoided in further works.

The algorithm has been successfully applied on amelodic analysis of a complete two-voice Invention by

J.S. Bach. Figure 11 shows the analysis of the 21 firstbars. The cyclic patterns are represented by gradu-ated lines, the graduation representing each return ofone possible phase. Due to the nature of the cyclicpatterns, no preference is given by the model betweendifferent possible phases of the same cycle. The rhyth-mic analysis of the piece, on the contrary, failed, due tothe alternation of sequences of either quarter notes or8th notes, which will require a formalisation throughhierarchical pattern chains (where successive states ofhigher-level patterns are linked to distinct lower-levelpatterns).


Figure 11: Automated motivic analysis of J.S. Bach’s Invention in D minor BWV 775, 21 first bars. The occurrences of

each pattern class are designated in a distinct way.

The analysis of The Beatles’ Obla Di Obla Da

melody shows 14 relevant pattern classes, representingthe chorus, verses, phrases and motives inside each ofthese structures. The 4 irrelevant patterns are redun-dant patterns subsumed by the 14 relevant ones.

In all these pieces, some patterns are consideredas irrelevant because they cannot be perceived as suchby listeners. Additional mechanisms should be addedto prevent these irrelevant inferences, based on short-term memory, top-down mechanisms, etc.

7.2 About algorithm complexity

The algorithm complexity may be expressed first interms of discovered structures: proliferation of re-dundant patterns, for instance, would lead to com-binatorial explosion, since each new structure needsproper processes assessing its interrelationships withother structures, and inferring possible extensions.Hence a maximally compact description insures in thesame time the clarity and relevance of the results andthe limitation of combinatorial explosion. Concerningtechnical implementation, the prototype needs furtheroptimisations.

The overall computational modelling results in acomplex system formed by a large number of highlydependent mechanisms. Without a real synthetic vi-sion of the whole system, no general assessment of theglobal complexity of the modelling has been achievedyet. The complete rebuilding of the modelling cur-rently undertaken should enable a better awarenessand control of complexity.

8 COGNITIVE STUDY OF TUNISIANMODAL MUSIC

The project, presented in this paper, of modellingof musical pattern discovery processes has been in-tegrated into a more general collaborative project be-tween computer science, ethnomusicology and cogni-tive sciences. The main objective of this project is todesign a cognitive modelling, using a complex system,of the processes of music perception and understand-ing, in order to understand the perceptive, musicaland computational aspects of sequence segmentation

and patterns recognition.This study has been focused on Tunisian Modal

Music, and particularly on Tba’, a modal system thatpresents interesting configurations. A Tba’ is basedon a musical scale – i.e. a set of pitches, such as(C, D, E, F, G, A, B) for instance – subdivided intotwo or several sub-scales called genres. Each genre ischaracterised by pivotal notes (more important thanothers) and melodic profile. Hence in a specific genre,the pitches that it contains are played in a specific or-der. Genres themselves are hierarchically connectedone with the others. In the musical scale of the Tba’,some of the notes play particular role in the mode:some are mostly played at the beginning of the impro-visation, or at the end of phrases. Finally, to each Tba’

is also associated a set of characteristic melodic pat-terns. Figure 12 presents an example of Tba’ modalstructure.

The study has been focused on one particular im-provisation by the Nay flute player Mohamed Saada,along the Tba’ Istikhbar Mhayyer Sika. First, the im-provisation has been transcribed from an audio recordinto a musical score. Then the resulting symbolic se-quence has been analyzed by the modeling.

8.1 Psychological experiments

Psychological experiments have been carried out in or-der to obtain a detailed description of listening strate-gies, and to assess the role of cultural schemes in par-ticular. This study has been focused in particular onthe determination of the patterns that form the basicstructures of the musical genres. In order to under-stand the impact of cultural knowledge on this partic-ular task, two groups of subjects have been considered:one group formed by European subjects unfamiliar toArabic modal music, and another group formed byArabic subjects of various degree of expertise in thismusic. Subjects have been asked to performed sev-eral tasks successively: First of all, after hearing themusical piece, they have to recognise the most salientmusical structures and, using these structures, to re-duce the whole improvisation in order to exhibit thedynamic macro-structure of the piece.

The experiments have been first carried out inParis on European subjects, and then extended to


Figure 12: Description of the Tba’ Mhayyer Sika D(Re), in terms of a sets of Genres and pivotal notes.

Arabic subjects in Tunisia. Results shows the relativevariability and divergence of the judgements of Euro-pean subjects. This is due to the fact that they can-not follow their own cultural scheme when analysingArabic modal improvisations. They have to rely in-stead on the structural characteristics of the musicaldiscourse, and in particular the discovery of repeatedpatterns. The experimental results of the listeningtests have been used as a guideline in order to im-prove the model presented in the previous sectionsand to take into account the stylistic characteristicsof Arabic modal improvisations.

8.2 Improvement of the modelling

One major limitation of the first version of the mod-elling, as presented in previous sections, is that onlyrepetition of sequences of notes that are immediatelysuccessive could be detected. In music in general,and in modal improvisation in particular, repeatedpatterns are often ornamented: secondary notes canbe added whose purpose is to emphasise the primarynotes of the initial pattern. Figure 13 displays, for in-stance, a melodic phrase, and one possible ornamen-tation. To some of the notes of the original phrase areadded secondary notes (displayed with smaller size inthe score) that are located in the neighbourhood of theprimary notes, both in time and pitch dimensions.

In order to take into account these ornamentation,a set of mechanisms have been added to the modelling.Solutions have been proposed [6] based on optimalalignments between approximate repetitions using dy-namic programming and edit distances. We have de-veloped algorithms that automatically discover, fromthe rough surface level of musical sequences, musi-cal transformations revealing the sequence of pivotalnotes forming the deep structure of these sequences.These mechanisms induce new connections between

non-successive notes, transforming the syntagmatic

chain of the original musical sequence into a complexsyntagmatic graph. The direct application of the pat-tern discovery algorithm on this syntagmatic graphenables the detection of ornamented repetitions.

8.3 Results of the computational modelling

The analysis of Mohamed Saada’s improvisation of Is-

tikhbar Mhayyer Sika is displayed in figure 14. Thediscovered structures are represented below each lineof the score. Each line represents an occurrence of thepattern, designated by a sign (1, 2, 3, 4, 5, + and -)on the left of the line. The notes actually consideredby each pattern occurrence are represented by squaresvertically aligned to the notes. These squares repre-sents therefore the successive states along the patternoccurrence chain, as shown in Figure 1.

Pattern ’-’ represents a simple sequence of notesof continuously decreasing pitch heights, and pattern’+’ represents a sequence of notes of continuously in-creasing pitch heights. Patterns 1 to 5 are sequencesrepeated several times in the improvisation. Eachblack square represents the beginning of a new oc-currence, and each white square one successive statealong the pattern chain. Grey squares corresponds tooptional states that are not found in all the occur-rences of the pattern. Finally, multiple branches des-ignates multiple possible paths for one same patternoccurrence. The improvisation is built on the specificmode Tba’ Mhayyer Sika D, characterised by the useof a specific set of notes (D, E, F, G, A, Bb, C) and aspecific melodic figure, which corresponds exactly tothe pattern 2. The beginning of the improvisation isalso based on the successive repetition of pattern 1,which corresponds to a periodic melodic curve start-ing from note F and ending to the same note F, whichis therefore a pivotal note of the improvisation. This


Figure 13: A melodic phrase and an ornamented version of it.

Figure 14: Analysis of the beginning of Istikhbar Mhayyer Sika improvised at the Nay flute by Mohamed Saada.

pattern correspond to the mode Mazmoum F shownin figure 12. The second line of the improvisation ischaracterised by the successive repetition of pattern3, which is a little melodic line progressively trans-posed. Pattern 4 corresponds to another importantmelodic profile associated to pattern 2. Finally thetwo last lines of the improvisation are characterised bythe repetition of pattern 5. Patterns 2 and 4 may beconsidered as stylistic characterisations of the modeIstikhbar Mhayyer Sika whereas patterns 1, 3 and 5shows the characteristics of the individual style of theimproviser.

The integration of these new mechanisms is notcompletely achieved. The application of the pattern

discovery algorithm in the general syntagmatic graphleads to combinatorial explosion of redundant patternsnot fully controlled yet, which will need further works.

9 CURRENT RESEARCHES

9.1 Addition of segmentation principles.

The structures currently found are based solely on pat-tern repetitions. Segmentation rules based on Gestalt

principles of proximity and similarity [22, 8] need tobe added. Although this rule plays a significant role inthe perception of large-scale musical structures, thereis no common agreement on its application to detailed


structure, because it highly depends on the subjectivechoice of musical parameters used for the segmenta-tions. The study will focus in particular on the com-petitive/collaborative interrelations between the twomechanisms, in particular the masking effect of localdisjunction on pattern discovery.

9.2 From monody to polyphony.

Our approach is limited to the detection of repeatedmonodic patterns. Music in general is polyphonic,where simultaneous notes form chords and parallelvoices. Researches have been carried out in this do-main [9], focused on the discovery of exact repetitionsalong different separate dimensions. Our model willbe generalised to polyphony following the syntagmaticgraph principle. We are developing algorithms thatconstruct, from polyphonies, syntagmatic chains rep-resenting distinct monodic streams. These chains maybe intertwined, forming complex graphs along whichthe pattern discovery algorithm will be applied. Pat-tern of chords may also be considered in future works.

9.3 Applications to musical databases.

The automated discovery of repeated patterns can beapplied to automated indexing of musical content insymbolic music databases. This approach may begeneralised later to audio databases, once robust andgeneral tools for automated transcription of musicalsound into symbolic scores will be available. A newkind of similarity distance between musical pieces maybe defined, based on these pattern descriptions, of-fering new ways of browsing inside a music databaseusing pattern-based similarity distance.

9.4 Applications to non-musical domains.

In future works, we plan to apply the algorithm tocase studies other than music scores, such as DNAsequences.

ACKNOWLEDGMENTS

This modelling was designed by Olivier Lartillotpartly in the context of a collaborative project, withcognitive ethno-musicologist Mondher Ayari, com-puter music scientist Gerard Assayag and cognitivescientists Stephen McAdams and Petri Toiviainen, fo-cused on the study of arabic improvised music withthe help of cognitive modelling. The project has beenfinanced by the French CNRS within the context ofthe ACI Complex Systems for Human and Social Sci-ences.

REFERENCES

[1] R. Agrawal, T. Imielinski and A. Swami. “Min-ing association rules between sets of items in largedatabases”. In Proceedings of the International Con-ference on Management of Data (SIGMOD 93). 1993.

[2] H. Mannila, H. Toivonen and A. I. Verkamo. “Effi-cient Algorithms for Discovering Association Rules”.In Proceedings of the AAAI Workshop on KnowledgeDiscovery in Databases. 1994.

[3] Y. Tanaka, K. Iwamoto and K. Uehara. “Discoveryof Time-Series Motif from Multi-Dimensional DataBased on MDL Principle”. Machine Learning, vol. 58,no. 2–3, 2005.

[4] J. Lin, E. Keogh, S. Lonardi and P. Patel. “FindingMotifs in Time Series”. In Proceedings of the Interna-tional Conference on Knowledge Discovery and DataMining. 2002.

[5] D. Cope. Computer and Musical Style. Oxford Uni-versity Press, 1991.

[6] P.-Y. Rolland. “Discovering Patterns in Musical Se-quences”. Journal of New Music Research, vol. 28,pp. 334–350, 1999.

[7] W. Dowling and D. Harwood. Music Cognition. Aca-demic Press, 1986.

[8] E. Cambouropoulos. Towards a General Computa-tional Theory of Musical Structure. Ph.D. thesis, Uni-versity of Edinburgh, 1998.

[9] D. Meredith, K. Lemstrom and G. Wiggins. “Al-gorithms for discovering repeated patterns in mul-tidimensional representations of polyphonic music”.Journal of New Music Research, vol. 31, no. 4, pp.321–345, 2002.

[10] N. Pasquier, Y. Bastide, R. Taouil and L. Lakhal.“Discovering frequent closed itemsets for associationrules”. In Proceedings of the International Conferenceon Database Theory. 1999.

[11] J. Pei, J. Han and R. Mao. “Closet: An efficientalgorithm for mining frequent closed itemsets”. InProceedings of the ACM-SIGMOD Workshop on Re-search Issues in Data Mining and Knowledge Discov-ery. 2000.

[12] R. Agrawal and R. Skirant. “Mining Sequential Pat-terns”. In Proceedings of the International Conferenceon Data Engineering. 1995.

[13] M. Zaki. “Efficient Algorithms for Mining ClosedItemsets and Their Lattice Structure”. IEEE Trans-actions on Knowledge and Data Engineering, vol. 17,no. 4, pp. 462–478, 2005.

[14] B. Ganter and R. Wille. Formal Concept Analysis:Mathematical Foundations. Springer-Verlag, 1999.

[15] J. Han, W. Gong and Y. Yin. “Mining Segment-Wise Periodic Patterns in Time-Related Databases”.In Proceedings of the International Conference onKnowledge Discovery and Data Mining. 1998.

[16] J. Han, G. Dong and Y. Yin. “Efficient Mining of Par-tial Periodic Patterns in Time Series Database”. InProceedings of the International Conference on DataEngineering. 1999.

[17] S. Ma and J. Hellerstein. “Mining partially periodicevent patterns with unknown periods”. In Proceedingsof the International Conference on Data Engineering.2001.

[18] W. W. J. Yang and P. Yu. “InfoMiner+: MiningPartial Periodic Patterns with Gap Penalties”. InProceedings of the IEEE International Computer onData Mining. 2002.


[19] G. Assayag, C. Rueda, M. Laurson, C. Agon andO. Delerue. “Computer Assisted Composition at Ir-cam: From PatchWork to OpenMusic”. ComputerMusic Journal, vol. 23, no. 3, pp. 59–72, 1999.

[20] T. Eerola and P. Toiviainen. “MIR In Matlab: TheMIDI Toolbox”. In Proceedings of the InternationalConference on Music Information Retrieval. 2004.

[21] N. Ruwet. “Methods of Analysis in Musicology”. Mu-sic Analysis, vol. 6, no. 1–2, pp. 4–39, 1987.

[22] F. Lerdahl and R. Jackendoff. A Generative Theoryof Tonal Music. The MIT. Press, 1983.

Documents

Motivic Pattern Extraction in Music, And Application …arima.inria.fr/006/pdf/arima00602.pdf · 16 Reviewed Article — ARIMA/SACJ, No. 36., 2006 Motivic Pattern Extraction in Music,