Is Automatic Speech Recognition Ready for Direct Use by Classroom Teachers?

PEPNet 2004 - PresentationPittsburgh, PA, Sheraton Station Square, April 24, 2004, 10:15 –

11:30 AM

Presenter/Author: Kathleen Eilers Crandall, Ph.D.Contributors: Donna E. Gustina, and Stephen S. Campbell

National Technical Institute for the DeafRochester Institute of Technology

The Glossograph

• Fay wrote about an experimental mechanical device used to transcribe human speech, and said,

• “… it is not unreasonable to hope that some instrument will yet be contrived …“

Fay, E.A. (1883). The glossograph. American Annals of the Deaf, 28, 67-69.

Sci-Fi or Reality?

"The pen was an archaic instrument, seldom used even for signatures...Apart from very short notes, it was usual to dictate everything into the speak-write…” (Nineteen eighty-four. Orwell, 1949)

Project

• Direct teacher use of Continuous Automatic Speech Recognition:– English Classroom/Lab

Funded by a grant from the Parsons Foundation of California

English Classroom/Lab Project

English Classroom/Lab Project

Purpose

Investigate direct use of ASR by classroom teacher to learn:

• Is acceptable recognition level attained?

• Under what conditions?– Style of speaking– Communication mode– Language complexity

Related Work

Use of ASR by an intermediary • Intermediary, a ‘captionist,’ re-speaks

professor’s words into a computer• Intermediary summarizes professor’s

words into a computer (‘interpreted speech’)

• Intermediary may use C-print (a shorthand typing system) in combination with ASR http://cprint.rit.edu/

Related Work

Use of ASR by the primary speaker

• iCommunicator™ http://www.myicommunicator.com/product_info.html

• Liberated Learning Environment http://www.liberatedlearning.com (St. Mary’s University, Halifax, Nova Scotia)

English Classroom/Lab Project

English Classroom/Lab Project

Teacher -- Students• Teacher -- Speaker

– Native speaker of American English– User of ASL as a second language – Trained the ASR equipment

• Students -- Readers – Young adult college students who are deaf or hard-of-

hearing– Reading and writing skills at the lowest quartile of

entering students– Enrolled in basic level English language reading and

writing courses

English Classroom/Lab Project

Evaluation Procedures

• ASR Software: – Dragon Naturally Speaking– IBM ViaVoice– Microsoft Office

• Speaking styles: – Spontaneous conversation– Dictation-like speech

• Communication modes:– Speaking– Simultaneously speaking and signing

English Classroom/Lab

Teacher stationControl systemSmart Board & LCD Projector

Student Stations

English Classroom/Lab Project

Accuracy Needs

• Vary by population and message predictability– New vs. Known information– Fluent readers vs.

Language learners– Reading for pleasure vs. Reading to master new

information

• CLOZE research and prediction of missing information

English Classroom/Lab Project

Results: ASR Software

75%

80%

85%

90%

95%

100%

Dragon ViaVoice XP

Conversation

Dictation

English Classroom/Lab Project

Results: Communication Mode

80%

82%

84%

86%

88%

90%

92%

94%

96%

98%

Simultaneous Commmunication Speech Only

Conversation

Dictation

English Classroom/Lab Project

Results: Language Complexity

82%

84%

86%

88%

90%

92%

94%

96%

98%

< 7th Grade > 7th Grade

Conversation

Dictation

English Classroom/Lab Project

Correcting Text

• Error correction– What to correct – When to correct– How to correct

Multitasking Demands

• Normal tasks for speaker/teacher– Formulating ideas relevant to topic– Attending to learning needs of students – Meeting lipreading and sign language needs

• Added tasks for speaker/teacher – Speaking to produce readable ASR text– Monitoring text– Making corrections

RecommendationsDiscussionQuestions

Grammatical Correctness

• Is ASR accuracy affected by the grammatical correctness of the user’s speech?

• Student written responses spoken as written: Accuracy – 93.8%

• Student written responses spoken after corrected: Accuracy - 94.3%

Style of Speaking

1. Style of speaking that more closely resembles dictation approaches a usable accuracy rate.

2. Lowering the complexity does not improve accuracy.

Conditions of Use

Direct use of ASR by a language teacher --Useful only under very controlled conditions.• Illustrating the generation of written

language • Demonstrating the use of notes and

outlines to produce written text• Translating selected sign language

utterances into English text during discussions

ASR: Classroom Use

Prepared Outline

Student’s Screen

Teacher’s Screen

Considerations• Training

– Critical to reach over 90% accuracy– Training with conversation

• Corrections– Familiarity with strategies – Dictate, Spell, Right click

• Equipment– Microphone headsets - design, comfort, and size– Demand on computer processor– Effect of optional settings

Tips for Better Accuracy

• Powerful computer • No other programs running• Consistent microphone placement• Environment• Training• Profile• Join user groups, such as ms-

speech@yahoogroups.com

Language Processing

Teaching/Learning Issues:• Does ASR promote the learning of reading

and writing for Deaf and Hard-of-Hearing students?

• How do students process this information?• Do students attend to multiple inputs?• Can teachers attend to this many tasks

effectively?

Presenter – Contact Information

Kathleen Eilers-crandall, Ph.D.Department of English

National Technical Institute for the DeafRochester Institute of Technology Lyndon Baines Johnson Building - 2264

Phone: (585) 475-5111

Fax: (585) 475-6500

Email: kecncp@rit.edu

Web: http://www.rit.edu/~kecncp