1 Spoken Language Support for Software Development Andrew Begel Advisor: Susan L. Graham Computer Science Division, EECS University of California, Berkeley

Programming By Voice

Spoken Language Support for Software DevelopmentAndrew BegelAdvisor: Susan L. GrahamComputer Science Division, EECSUniversity of California, Berkeley

#1Broader research goal in building tools for programmers to improve productivity and access. Study programmers first and build tools second. and then build analyses to support tools

People spend a lot of time developing software. Theres a continuing demand for more people.Make people more productive. b. Enlarge the pool of developers.1b. Better education and helping people with disabilities1a. To serve community requires understanding of PL, SE and HCI.

MotivationProgrammers conventionally use keyboardLong hours at keyboard leads to higher risk of RSICan a programmer code using speech?Can a computer understand what the developer says?

while (counter < limit) { }#2Keyboard is unambiguous. Computers lacked processing power to do the workpeople are the precious resource. we need to make the computers resolve the ambiguitiesfor us. not we for them. addresses ambiguity and RSI. is a playpen to explore ambiguity. Mankoff Ambiguity mediators, bring amibugity throughout system.Programming by VoiceMy GoalFind out how developers use code verbally. Use this to develop a naturally verbalizable input form.Build development environment that supports verbal authoring, navigation, modification.Extend conventional compiler analyses to support ambiguities generated by speech.Learn how developers can use voice-based programming, and iterate design.

while counter is lessthan limit do ...#3rewritre to work with previous slide.push on technology for ambiguity resolution and see what its capable of.say human-centered again. methodology slide

ChallengesSpeech is inherently ambiguous.Programming tools were not designed for ambiguity.Speech tools are poorly suited for programming tasks.Programmers are not used to verbal software development.#4Not the best way to do things, but too hard to change, so instead well modify the input to the programming language to make it easier for people.i made progress on each of these. none of these is done yet.

Talk OutlineIntroduction and MotivationProgramming by VoiceProgram Analyses for Ambiguous InputsSPEech EDitor Programming EnvironmentSPEED User StudyConclusion#5target 5 minutesHow do Programmers Speak Code?10 programmers read Java code out loud (Begel 05)Graduate students in Computer ScienceFive knew Java, five did notFive were native English speakers, five were notFive were educated in U.S.A., five were notRead pre-written code into tape recorderAs if speaking to a sophomore-level CS undergrad who knows Java, but does not know the program

Most programmers spoke the same way#6

for int i equals zero i less than ten i plus plusfor (int i = 0; i < 10; i++ ) { }How do Programmers Speak Code?#7Didnt have to verbalize parens, semicolons, etc.

Spoken Words Can Be Hard To Write Down

2

How Do Programmers Speak Code?#8Think about how to say this


2 2, two, to, too



2 2, two, to, too

print



2 2, two, to, too

print print, Print



2 2, two, to, too

print print, Print

drop stack process



2 2, two, to, too

print print, Print

drop stack process drop stack processdrop stackprocessdropstack processdropstackprocessHow Do Programmers Speak Code?#13Think about how to say this

Many Ways to Say the Same Thing

bar[i]



bar[i] bar sub i, bar of i, i from bar




.




. period, dot




. period, dot

}




. period, dot

} right brace, close the if, end method




. period, dot


printlnHow Do Programmers Speak Code?#20Think about how to say this



. period, dot


println print line, print lin, print l nHow Do Programmers Speak Code?#21Think about how to say this

One Utterance May Mean Many Things

object stack



object stack Object stack;object.stackobject(stack)object().stack()




array sub i plus plus




array sub i plus plus array[i]++array[i++]


People Have Trouble Saying Some Things

System.out.println



System.out.println system out print linesystem dot out print linesystem dot out dot print line




(int)foo




(int)foo cast foo to integer int foocast something to integer. that something is foo.


Sometimes They Describe the Code

And then theres a class.




Set all the fields of that object to null.




Set all the fields of that object to null.

All of these are just assignment operations.How Do Programmers Speak Code?#32Think about how to say this

Design TradeoffsCommandLanguage

Easy to analyze,but prescriptiveNaturalLanguage

Flexible,but ambiguousProgrammingby Voice#33Programming by Voice Related WorkHuman-CentricComputer-CentricMultipleTasksAuthoringOnlyArnold 00Snell 00Price 00 02Desilets 01 04Gray 03Begel 05#34My goal is to create a solution that is both complete and human-centriccomplete in that it enables authoring, navigation, and editing.Human centric in that the machine adapts to the programmer. Everyone knows this is important workbut no one has accomplished it yet. My workgoes the farthest

A More Natural Way to Codepublic class symbol implements serializablepublic class Symbol implements Serializable { }#35Didnt have to verbalize parens, semicolons, etc.

A More Natural Way to Codestatic hash map hash table gets new hash mappublic class Symbol implements Serializable { static HashMap hashtbl = new HashMap();}#36All punctuation has English equivalentsOpen Brace, End For LoopMost punctuation is optionalProvide verbalization for all abbreviationsRelaxed phrasing for better fit with English(int)foo cast foo to integerfoo = 6 set foo to 6foo[i]++ increment the ith element of array foo

A More Natural Way to Codeend the classpublic class Symbol implements Serializable { static HashMap hashtbl = new HashMap();

}#37All punctuation has English equivalentsOpen Brace, End For LoopMost punctuation is optionalProvide verbalization for all abbreviationsRelaxed phrasing for better fit with English(int)foo cast foo to integerfoo = 6 set foo to 6foo[i]++ increment the ith element of array foo

A More Natural Way to Codefor int i equals zero i less than ten i plus plusfor (int i = 0; i < 10; i++ ) { }#38Didnt have to verbalize parens, semicolons, etc.

Too Many Ambiguitiesfor (int i = 0; i < 10; i++ ) { }4 int eye equals 0 aye less then ten i plus plusKW or #?Spelling of ID?KW or ID?for int i equals zero i less than ten i plus plus#39A human could eyeball this.

Sometimes Its Non-Obviousfor (times = 8; file(2, load); times == one) {}for times equals 8 file 2 load times equals onefore *= 8; file.tooLode.times = won 4; times = ate(file).to(load).equals(1) #40Spoken JavaSemantically identical to JavaSyntactically easier to say than JavaMethodology generalizable to any computer language

All punctuation has English equivalentsOpen Brace, End For LoopMost punctuation is optionalProvide verbalization for all abbreviationsRelaxed phrasing for better fit with English(int)foo cast foo to integerfoo = 6 set foo to 6foo[i]++ increment the ith element of array foo#41SPEED: Speech EditorBuild an editor that supports naturally verbalized programsSPEED: SPEech EDitorBased on IBM ViaVoice, Eclipse IDE, Harmonia

Spoken Java Language for CompositionSpoken Command language for Navigation, Editing, Template instantiation, Refactorings, SearchAudible and visual feedbackSimilar to JavaSpeak (Smith 2000)#42Harmonia Analysis FrameworkFramework to support interactive editorsLanguage-based, programmer-oriented toolsIncremental analysesLexing (Wagner 97), GLR Parsing (Wagner 97, Begel 04), Static Semantics (Garrison 87, Begel, Jamison)C, Java, Titanium, Cool, Flex, BisonAlso, languages where indentation and CRs are significantInteractive Program Transformations (Boshernitsan)CodeLink (Toomim et. al. 04)Shorthand Editing

#43Compile ErrorsStructural Navigation,v Browsing, Search, Code FoldingSemantic Info Display, Indentation, Syntax Highlighting

Talk OutlineIntroduction and MotivationProgramming by VoiceProgram Analyses for Ambiguous Inputs SPEech EDitor Programming EnvironmentSPEED User StudyConclusion#44Spoken input requires new compiler analysesExisting analyses are not built for ambiguityTarget 20 minutes.Traditional Compiler AnalysesLexicalAnalysisFORIParsingSemanticAnalysis

for (i = 0; i < 10; i++ ) { }Programming languages are designed to be unambiguousFor LoopFORAssign ExprI=0iLocalVarint#45Ambiguity-Aware Analyses

for i equals zero ...Handles input stream, syntactic and semantic ambiguitiesLexicalAnalysisFORIAmbiguousParsingSemanticAmbiguityResolutionFor LoopFORAssign ExprI=0iLocalVarintfoureyeLocalVar?4EYEFOUREYEAssign Expr=0Ambig Stmt#46Insight here is to generalize GLR algorithm for lexical ambiguitiestotally changes the algorithm.Scan Input Stream

CommercialSpeechRecognizerHomophoneDictionary

LexicalAnalysis

#47Homophones Cause Ambiguitiesforiequalsfor i =4 i equalsforeeye ==fore ayeequalsfoureyeequalsforiequalsConcatenated words cause them too4forefoureyeaye===#48Capitalization, abbreviationAmbiguity-Aware Analyses

for i equals zero ...LexicalAnalysisFORIXGLRAmbiguousParsingSemanticAmbiguityResolutionFor LoopFORAssign ExprI=0iLocalVarintfoureyeLocalVar?4EYEFOUREYEAssign Expr=0Ambig Stmt#49XGLR Parsing [Begel 04]

Documents

1 Spoken Language Support for Software Development Andrew Begel Advisor: Susan L. Graham Computer Science Division, EECS University of California, Berkeley