Phonetic String Matching:Lessons from Information Retrieval

1Intelligent Database Systems Lab

國立雲林科技大學National Yunlin University of Science and Technology

Phonetic String Matching:Lessons from Information Retrieval

Advisor : Dr. Hsu

Graduate : Chih-Ling Wang

Authors : Justin Zobel

Philip Dart

2003 IEEE

.

2

Intelligent Database Systems Lab

N.Y.U.S.T.

I. M.

Outline Motivation Objective Introduction Phonetic matching versus information retrieval Phonetic matching techniques Performance assessment Combination of evidence Conclusions Personal Opinion

3


N.Y.U.S.T.

I. M.

Motivation We explore the accuracy of the phonetic string

matching.

4


N.Y.U.S.T.

I. M.

Objective In this paper we propose a new phonetic matching

techniques and describe the results of a new comparative investigation of phonetic matching.

5


N.Y.U.S.T.

I. M.

Introduction Phonetic matching is used to identify strings that may

be of similar pronunciation, regardless of their actual spelling.

6


N.Y.U.S.T.

I. M.

Introduction(cont.) There are two pragmatic issues that must be addressed

in such a phonetic matching system. One is of speed – answers should be found reasonably quickly. The other pragmatic issue is accuracy.

The parallels between information retrieval and phonetic matching mean that They can be measured by the same kinds of techniques. Methods for improving information retrieval performance may

also apply to phonetic matching.

7


N.Y.U.S.T.

I. M.

Phonetic matching versus information retrieval In information retrieval, ranking is the process of

identifying which of a set of documents are most likely to be similar in content to a given query.

Phonetic matching is the process of identifying which of a set of strings are most likely to be similar in sound to a given query string.

8


N.Y.U.S.T.

I. M.Phonetic matching versus information retrieval(cont.) In both cases the matching process is: fundamentally

inexact, since human judgment is required to tell whether the process’s guess is correct

Similarity is relative, unable in isolation to determine whether a query and potential answer are matches.

It is difficult to give an accurate definition of relevance.

9


N.Y.U.S.T.

I. M.Phonetic matching versus information retrieval(cont.) We consider phonetic matching to be the process of

identifying strings that, after elimination of possible transmission or cognition errors, may sound the same. Transmission errors include, sound-alike mistakes in data

entry ;mishearing of a spoken name on a imperfect transmission medium.

Cognition errors include, mistaking a pronunciation for an expected word.

10


N.Y.U.S.T.

I. M.

Phonetic matching techniques

Soundex Soundex uses codes based on the sound of each letter

to translate a string into a canonical form of at most four characters, preserving the first letter.

Soundex makes the error of transforming dissimilar-sounding strings to the same code, and of transforming similar-sounding strings to different codes.

There is no ranking of matches: strings are either similar or not similar.

11


N.Y.U.S.T.

I. M.

Example: reynold(r005043)=>r543

renauld(r050043) =>r543

Phonetic matching techniques(cont.)

12


N.Y.U.S.T.

I. M.

Phonix Phonix is a Soundex variant. Letters are mapped to a set of codes using the same al

gorithm, but a slightly different set of codes is used, and prior to mapping about 160 letter-group transformations are used to standardise the string.

The sequence tjv is mapped to chv if it occurs at the start of a string, and x is transformed to ecs.

These transformations provide context for the phonetic coding and allow c and s to be distinguished.


13


N.Y.U.S.T.

I. M.

Example: reynold(r005043)=>r543

renauld(r050043) =>r543


14


N.Y.U.S.T.

I. M.

In our experiments we consider a variant of Phonix, here called Phonix+, in which truncation is not applied and a minimal edit distance is used to compare the resulting strings.


15


N.Y.U.S.T.

I. M.

Q-gram methods A q-gram of string s is any substring of s of some fi

xed length q.

Simply counting q-grams does not allow for length differences.


16


N.Y.U.S.T.

I. M.

Example:rhodes;rod

We have used this q-gram method with q=2

rhodes= rod=


,2 tsts GGGG

es

de

od

no

rh

512252 tsts GGGG

od

ro

17


N.Y.U.S.T.

I. M.

Agrep

Agrep is a utility that embodies a fast algorithm for identifying strings that contain a substring which is identical to a query but for at most k insertions, deletions, or replacements, where k is a predefined constant.

Agrep was not designed for the task of phonetic matching, but rather for fast searching of large files.


18


N.Y.U.S.T.

I. M.

Edit distances A simple edit distance, which counts the minimal

number of single-character insertions, deletions, and replacements needed to transform one string into another, could be used for phonetic matching since similar-sounding words are often spelled similarly.


19


N.Y.U.S.T.

I. M.

For two strings s and t of length i and j respectively, this edit distance can computed with the recurrence relation edit(i,j).

The function returns 0 if and are identical, and 1otherwise.


)tr(s ji , is jt

20


N.Y.U.S.T.

I. M.

Example: rhodes;rod

Phonetic matching techniques(cont.) 10,5,10,6,11,5min1,6

11,5,11,6,12,5min2,6

10,4,10,5,11,4min1,5

11,4,11,5,12,4min2,5

10,3,10,4,11,3min1,4

11,3,11,4,12,3min2,4

10,2,10,3,11,2min1,3

01,2,11,3,12,2min2,3

10,1,10,2,11,1min1,2

11,1,11,2,12,1min2,2

00,0,10,1,11,0min1,1

11,0,11,1,12,0min2,1

12,0,12,1,13,0min3,1

12,1,12,2,13,1min3,2

12,2,12,3,13,2min3,3

02,3,12,4,13,3min3,4

12,4,12,5,13,4min3,5

12,5,12,6,13,5min3,6

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit

editediteditedit 3 7 4

2 4 3

1

3 2 2

3 2 2

4 2 3

3 1 2

2 2 0

2 2 1

1

3 1

3 2

2

2 4 3

2 5 3

3 5 4

3

4

4

5

5

6

6

7

4

5

5

6

3 3

21


N.Y.U.S.T.

I. M.

Editex Editex is a phonetic distance measure that combines t

he properties of edit distances with the letter-grouping strategy used by Soundex and Phonix.

Editex also groups letters that can result in similar pronunciations, but doesn't require that the groups be disjoint and can thus reflect the correspondences between letters and possible similar pronunciation more accurately.


22


N.Y.U.S.T.

I. M.

Editex is defined by the edit distance recurrence relation with a redefined function returns 0 if and are identical, 1 if

and are both occur in the same group, and 2 otherwise. The function is identical to .If a is h or w and

a b then is 1.


)tr(s ji , is jt

is jt

b)d(a, )tr(s ji ,

b)d(a,

23


N.Y.U.S.T.

I. M.

Phonometric methods Our algorithms for phonometric matching consist of

two stages: First, the string of letters is converted into a string of phon

emes by a string-to-pronunciation conversion algorithm. The second stage is comparison of strings of phonemes.

The distance between pronunciations as represented by strings of phonemes can be measured more precisely than the distance between strings of letters.


24


N.Y.U.S.T.

I. M.

Tapering Tapering is a refinement to the edit distance

techniques based on a human-factors property: differences at the start of a pronunciation can be more significant than differences at the end.

A tapered edit distance of particular interest is one in which the maximum penalty for replacement or deletion at start of string just exceeds twice the minimum penalty for replacement or deletion at end of string.


25


N.Y.U.S.T.

I. M.

We can now compare the various approaches to phonetic matching.Results are shown in Table 1, which is of 11-point recall-precision.

For many of the techniques tested, only a few distinct ranks are possible, and some techniques only return two ranks, match and not-match.

Performance assessment

26


N.Y.U.S.T.

I. M.

The least effective methods such as Phonix and Soundex only return a small number of answers for most queries.

Phonix and Soundex are not only finding many wrong answers but not finding many right ones.

The “baseline” results are for a trivial phonetic matching method: find all strings with at most one character – an insertion, deletion, or replacement – different from the query.

Performance assessment(cont.)

27


N.Y.U.S.T.

I. M.

A particular problem of best agrep is the tiny number of correct answers returned – less than one per query – but we stress that agrep was not designed for phonetic matching.


28


N.Y.U.S.T.

I. M.

An interesting discovery is that even the most successful of the methods fetch rather different sets of answers, sometimes almost without overlap.

As for information retrieval, it seems, two methods can perform well without finding the same answers.


29


N.Y.U.S.T.

I. M.

Phonetic matching has strong parallels with information retrieval.

Matching techniques fetch a ranked list of matches in which each entry has weight attached to it; this weight is the likelihood that the entry is a good match.

Combining the ranked lists produced by different retrieval mechanisms can improve performance.

Combination of evidence

30


N.Y.U.S.T.

I. M.

The “(none)” lines are the results of running the methods individually.

The best performance of all is given by the combination of Phonix+ and the q-gram method, neither of which works particularly well alone.

Combination of evidence(cont.)

31


N.Y.U.S.T.

I. M.

More sophisticated techniques for combination could be used: Weighting the ranks from the different techniques. Combining more than two methods.

That combination of evidence is successful in this context.

Combination of evidence(cont.)

32


N.Y.U.S.T.

I. M.

Conclusions

Two of our proposals – the Ipadist and Editex methods – do indeed lead to improved performance, whereas the third – tapering – was not successful.

We showed that combination of evidence, which has been successfully applied to information retrieval, consistently improves performance.

Our new methods are substantially more effective than existing methods such as edit distances, and that combination of evidence is as valuable in this domain as it is in information retrieval.

33


N.Y.U.S.T.

I. M.

Personal Opinion

The concept in this paper may use in our research, but I haven’t have a clear idea to implement it.I need more time to think…think…

Documents

Phonetic String Matching:Lessons from Information Retrieval