View
220
Download
1
Tags:
Embed Size (px)
Citation preview
The Effectiveness Study of Music Information Retrieval
Arbee L.P. ChenNational Tsing Hua University
2002 ACM International CIKM Conference
Outline
• Motivation• The Ultima Project
– The 1D-List approach
– The APS approach
• The Effectiveness Study– Estimating all relevant references
• Conclusion
Motivation
• Various approaches were proposed to provide efficient and effective content-based retrieval of music objects– Music representations
• pitch, rhythm, contour, chord
– Index structures• tree-based index, list-based index, n-gram index
– Query processing methods• exact match, partial match, approximate match
Motivation (Cont.)
• A platform is needed for the evaluation of various music information retrieval methods– Efficiency
• response time
– Effectiveness• recall-precision curve
• The Ultima project builds such a platform– Same data set and query set– Also serves as a testbed whenever new approaches
are proposed
The Ultima Project (Cont.)
Architecture• Mediator
• Query processing module
• Report module
• Summarization module
• Query generation module
• Data store
Data Store (MS Access)
SMF
Med
iato
r
Query Processing Module
Table
to the InternetSummarization Module
1D-List APS APM
Report Module
Query Generation Module
The Ultima Project (Cont.)
• Two approaches have been compared – 1D-List
– APS
Approach Representation Index structure
1D-List melody string list-based
APS sequence of music segments suffix tree-based
The 1D-List Approach
• The 1D-List approach– Music objects are coded as melody strings
• “so-mi-mi-fa-re-re-do-re-mi-fa-so-so-so”
– Melody strings are organized as linked lists– Both exact and approximate matching can be handl
ed
• Exact link, insertion link, dropout link, transposition link
The 1D-List Approach (Cont.)
1:7 1:11:41:21:5
2:1 1:31:6
2:9 2:71:91:8
2:82:22:5
2:32:6
2:4
2:10
2:11 2:12
1:10 1:11
1:12
1:13
do re mi fa so la si
1:7 1:21:5
2:1 1:31:6
2:9 1:91:8
2:22:5
2:62:10
2:11 2:12
do re mi
start end 1:7 1:21:5
2:1 1:31:6
2:9 1:91:8
2:22:5
2:62:10
2:11 2:12
do re mi
start end
(a) (b) (c)
The APS Approach
• The APS approach– Music objects are coded as sequences of
music segments• four segment types to model the music contour• pitch and duration are considered
– Index structures• one-dimensional and two-dimensional augmented suffix
tree
– Both exact and approximate matching can be handled
The APS Approach (Cont.)
• Representation
note number
beat
60
62
65
64
67
(B, 3, -3)
(A, 1, +1)
(D, 3, -3)
(B, 1, -2)
(C, 1, +2)
(C, 1, +2)
(C, 1, +1)
type A
type B
type C
type D
The APS Approach (Cont.)
1 4
AB C
B C
C $
$
2 5
3
(a)
root
A
C
A
(b)
root
A<1,1>
C<7,8> C<1,3>
A<7,8> A<3,4>
N1
N2
The suffix tree of the string S=“ABCAB”
(a) An example of suffix tree(b) A 1-D augmented suffix tree
1
A
$
A
B
$B2
A
B
$3
The APS Approach (Cont.)
• Similarity measure– Given a query sequence Q = (i1, j1, k1) (i2, j2, k2) ... (in, jn, kn),
and a candidate sequence from the database C = (i1, x1, y1) (i2, x2, y2) ... (in, xn, yn).
iip
n
i
ppitch
n
iiid
dduration
beatdurationpitchduration
pitchpitchdurationduration
ykMaxPitchifMaxPitch
if
nsimdis
xjnMaxDuratioifnMaxDuratio
if
nsimdis
wwww
simdiswsimdiswCQSIMDIS
,MIN and ,1
_)3(
,MIN and , 1
_)2(
1,0 and 1)1(where
__,_
1
2
1
The Effectiveness Study
• Traditional measures of effectiveness are precision and recall
• However, the number of relevant references are usually unknown– it is unrealistic for the user to make relevant
judgments to all music objects in the database
retrievedarethatreferencesnumber of
relevantarethatreferencesretrievednumber ofprecision
referencesrelevantnumber of
relevantarethatreferencesretrievednumber ofrecall
The Effectiveness Study (Cont.)
• How to estimate the number of relevant references NR?– ASx is the set of relevant objects from the top x ranked results
– RSx is the set of the top x ranked results retrieved by an approach
– Assumption 1: , ,• the number of the retrieved results is a function of the number of retrieve
d relevant objects
• Assumption 2: , where B is a positive integer
• Based on the two assumptions, NR can be derives as follows:
xx ASfRS DBx 1for 1 and 1 AS
1 BBRS xASx
)1)(BRS(log
))1B(DB(log AS
xB
Bx
NR
The Effectiveness Study (Cont.)
rank relevance recall precision
1 Y 0.1 1
2 Y 0.2 1
3 Y 0.3 1
4 0.3 0.75
5 Y 0.4 0.80
6 0.4 0.67
7 Y 0.5 0.71
8 0.5 0.63
9 Y 0.6 0.67
10 Y 0.7 0.70
11 0.7 0.64
12 0.7 0.58
13 Y 0.8 0.62
14 0.8 0.57
15 0.8 0.53
16 Y 0.9 0.56
17 0.9 0.53
18 0.9 0.50
19 Y 1 0.53
20 1 0.50
Rank relevance recall precision
1 Y 0.1 1
2 Y 0.2 1
3 Y 0.3 1
4 0.3 0.75
5 Y 0.4 0.80
6 0.4 0.67
7 Y 0.5 0.71
8 0.5 0.63
|RSx| = 8, |ASx| = 5, |DB| = 20
The Effectiveness Study (Cont.)
MethodFactor
APS
1D-List1-D AST (duration
)
1-D AST (pitch
)2-D AST
Number of music objects for generating queries
10 10
Is the query sample a refrain or an incipit?
refrain/incipit refrain/incipit
Length of query sample, denoted L
6/10 (segment) 8/12 (note)
Number of query samples per music object
4 4
Threshold setting of approximation for a query
sample
th_d = 0, 0.5, 1.0th_p = 0, 0.5, 1.0
K=0, 4, 7 (for L=8)K=0, 6, 11 (for L=12)
Total number of posing queries 120 120
Experiment setup
The Effectiveness Study (Cont.)
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Recall
Pre
cisi
on
1D-AST_D0
1D-AST_P0
2D-AST_PD0
1D-List_K0
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Recall
Pre
cisi
on1D-AST_D0.51D-AST_P0.5
2D-AST_PD0.51D-List_K4,6
• Experiment results
The Effectiveness Study (Cont.)
• Experiment results
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Recall
Pre
cisi
on
1D-AST_D_R
1D-AST_P_R
2D-AST_R
1D-List_R
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Recall
Pre
cisi
on
1D-AST_D_In
1D-AST_P_In
2D-AST_In
1D-List_In
The Effectiveness Study (Cont.)
• Experiment results– 1D-list achieves a high precision in the limited
range of recall, while a moderate precision for the APS family can be obtained
– Comparing the APS family, the precision in a descending order is: 1D-AST (pitch), 2D-AST, and 1D-AST (duration)
– In average, the effectiveness of “incipit” queries is better than “refrain” queries
Conclusion
• The Ultima project builds a platform for evaluating the performance of various approaches of music information retrieval
• A new measure for estimating the number of relevant references is proposed
• Future work– Design and implement the summarization module
as well as the query generation module– Extend the project for evaluating polyphonic musi
c retrieval methods