From linear sequences toabstract structures:

Distributional information in infant-direct speech

Hao Wang & Toby MintzDepartment of Psychology

University of Southern California

This research was supported in part by a grant from the National Science

Foundation (BCS-0721328).1

Socal Workshop 2009 @ UCLA

Outline

• Introduction– Learning word categories (e.g., noun and verb) is a

crucial part of language acquisition– The role of distributional information– Frequent frames (FFs)

• Analyses 1 & 2, structures of FFs in child-directed speech

• Conclusion and implication

2

Speakers’ Implicit Knowledge of Categories

Upon hearing:I saw him slich.

Hypothesizing:They slich.He sliches.Johny was sliching.

3

The truff was in the bag.

He has two truffs.She wants a truff.Some of the truffs are here.

Distributional Information

• The contexts a word occurs– Words before and after the target word

• Example– the cat is on the mat– Affixes in rich morphology languages

• Cartwright & Brent, 1997; Chemla et al, 2009; Maratsos & Chalkley, 1980; Mintz, 2002, 2003; Redington et al, 1998

4

Frequent frames (Mintz, 2003)

• Two words co-occurring frequently with one word intervening

5

FRAMEFREQ.

you__it 433you__to 265you__the 257what__you 234to__it 220want__to 219

. . .the__is 79

. . .

• Frame you_itPeter Corpus (Bloom, 1970)

• 433 tokens, 93 types, 100% verbsput see do did want fix turned get

got turn throw closed

think leave take open . . .

Accuracy Results Averaged Over

All Six Corpora (Mintz, 2003)

6

0.20.30.40.50.60.70.80.9

1

Mea

n Tok

en Ac

cura

cy

Categorization Type

Frame-Based Categorization

Chance Categorization

Structure of Natural Languages

• In contemporary linguistics, sentences are analyzed as hierarchical structures

• Word categories are defined by their structural positions in the hierarchical structure

7

• But, FFs are defined over linear sequences

• How can they accurately capture abstract structural regularities?

Why FFs are so good at categorizing words?

• Is there anything special about the structures associated with FFs?

• FFs are manifestations of some hierarchically coherent and consistent patterns which largely constrained the possible word categories in the target position.

8

Analysis 1

• Corpora– Same six child-directed speech corpora from CHILDES

(MacWhinney, 2000) as in Mintz (2003) – Labeled with dependency structures (Sagae et al.,

2007)– Speech to children before age of 2;6

Eve (Brown, 1973), Peter (Bloom, Hood, & Lightbown, 1974; Bloom, Lightbown, & Hood, 1975), Naomi (Sachs, 1983), Nina (Suppes, 1974), Anne (Theakston, Lieven, Pine, & Rowland, 2001), and Aran (Theakston, et al., 2001).

9

Grammatical relations• A dependency structure consists of grammatical

relations (GRs) between words in a sentence• Similar to phrase structures, it’s a representation of

structural information.

Sagae et al., 200510

• Consistency of structures of FFs• Combination of GRs to represent structure

– W1-W3, W1-W2, W2-W3, W1-W2-W3

• Measures– For each FF, percentage of tokens accounted for by

the most frequent 4 GR patterns

• Control– Most frequent 45 unigrams (FUs)– E.g., the__

Method

W1 W2 W3 11

0%

20%

40%

60%

80%

100%

W1-W3 W1-W2 W2-W3 W1-W2-W3

0.92 0.91 0.88 0.85

0.64

Mean percentage of tokens accounted for bythe most frequent 4 GR patterns

FFs

FUs

Results

12

*

t(5)=26.97, p<.001

Frequent frames GR of W1* GR of W3* Token count

what__you

2 OBJ 2 SUBJ 287

4 OBJ 2 SUBJ 46

5 OBJ 2 SUBJ 20

3 POBJ 2 SUBJ 5

you__to

0 SUBJ 2 INF 260

0 SUBJ 0 JCT 26

-2 SUBJ 2 INF 1

0 SUBJ 0 INF 1

what__that

0 PRED 0 SUBJ 216

0 PRED 2 DET 14

3 OBJ 2 DET 4

2 OBJ 2 SUBJ 4

you__it

0 SUBJ 0 OBJ 195

0 SUBJ 2 SUBJ 6

-2 OBJ 0 OBJ 2

-2 OBJ 2 SUBJ 1*The word position and head position for GRs in this table are positions relative to the target word of a frame. W1’s word position is always -1, W3 is always 1.

Top 4 W1-W3 GR patterns

13

Analysis 1 Summary

• Frequent frames in child-directed speech select very consistent structures, which help accurately categorizing words

• Analysis 2, internal organizations of frequent frames

14

Analysis 2• Same corpora as Analysis 1• GRs between words in a frame and words outside

that frame (external links) and GRs between two words within a frame (internal links)

• For each FF type, the number of links per token was computed for each word position

Internal links

External links

Not counted

15

Links from/to W1

0

0.2

0.4

0.6

0.8

1

Internal links from W1

Links from W1 to W2

External links from W1

External links to W1

0.73

0.49

0.170.23

0.31 0.31

0.510.58

FFs

FUs

16

Conclusion & implications• Frequent frames, which are simple linear

relations between words, achieve accurate categorization by selecting structurally consistent and coherent environments.

• The third word (W3) helps FFs to focus on informative structures

• This relation between a linear order pattern and internal structures of languages may be a cue for children to bootstrap into syntax

17

• References– MacWhinney, B. (2000). The CHILDES Project: Tools for Analyzing Talk.

Mahwah, NJ: Lawrence Erlbaum Associates.

– Mintz, T. H. (2003). Frequent frames as a cue for grammatical categories in child directed speech. Cognition, 90(1), 91-117.

– Sagae, K., Lavie, A., & MacWhinney, B. (2005). Automatic measurement of syntactic development in child language. ACL Proceedings.

– Sagae, K., Davis, E., Lavie, A., MacWhinney, B. and Wintner, S. High-accuracy annotation and parsing of CHILDES transcripts. In, Proceedings of the ACL-2007 Workshop on Cognitive Aspects of Computational Language Acquisition.

Thank you!

18

Pure frequent frames?

19

Ana. 2 mean token coverage

    Eve Peter Nina Naomi Anne Aran

Frequent frames

W1-W3 0.96 0.87 0.94 0.93 0.92 0.89W1-W2 0.94 0.87 0.93 0.92 0.92 0.88W2-W3 0.92 0.85 0.91 0.88 0.90 0.82

W1-W2-W3 0.89 0.80 0.89 0.86 0.86 0.79Bigrams W1-W2 0.69 0.57 0.68 0.63 0.68 0.61

20

Ana. 2 FF external links

CorpusToke

n count

External links

to W1 to W2 to W3 from W1 from W2 from W3Eve 3601 0.19 0.54 0.50 0.15 0.33 0.39Peter 4541 0.28 0.71 0.44 0.25 0.30 0.52Nina 6709 0.19 0.46 0.71 0.15 0.32 0.40Naomi 1447 0.20 0.77 0.46 0.13 0.36 0.52Anne 4435 0.24 0.50 0.54 0.18 0.32 0.43Aran 5245 0.27 0.61 0.51 0.17 0.39 0.51Average 0.23 0.60 0.52 0.17 0.34 0.46

Table 3 Average number of links per token for frequent frames

21

FF internal links

CorpusToke

n count

Internal links

W1->W2 W1->W3 W2->W1 W2->W3 W3->W1 W3->W2Eve 3601 0.52 0.25 0.10 0.28 0.10 0.29Peter 4541 0.44 0.21 0.16 0.27 0.13 0.20Nina 6709 0.48 0.29 0.09 0.37 0.07 0.23Naomi 1447 0.60 0.17 0.13 0.21 0.07 0.24Anne 4435 0.41 0.29 0.17 0.34 0.12 0.17Aran 5245 0.50 0.20 0.16 0.24 0.10 0.21Average

0.49 0.24 0.14 0.29 0.10 0.22

22

Ana. 2 FU links

 CorpusToken count

External links Internal links

to W1 to W2 from W1 from W2 W1->W2 W2->W1

Eve 28076 0.52 0.51 0.51 0.62 0.32 0.18

Peter 35723 0.65 0.48 0.53 0.66 0.28 0.20

Nina 37055 0.66 0.58 0.49 0.64 0.32 0.15

Naomi 12409 0.59 0.50 0.51 0.63 0.30 0.19

Anne 38681 0.52 0.48 0.44 0.62 0.36 0.16

Aran 49302 0.52 0.55 0.54 0.60 0.30 0.22

Average 0.58 0.52 0.51 0.63 0.31 0.19

23