Musical Program Auralisation1 The cAitlIn system Musical Program Auralisation & External Representations of Pascal

Musical Program Auralisation 1

The cAitlIn systemMusical Program Auralisation

& External Representations of Pascal

http://computing.unn.ac.uk/staff/cgpv1/caitlin

Dr Paul VickersSchool of Informatics, Northumbria University

Professor J.L. AltyDean of Faculty of Science, Loughborough

University


Introduction

Major features Background to the project Sound in HCI A case for music The cAitlIn system Demonstration Future work


Major features

With cAitlIn we have:– A musical equivalent for various syntactic

components of a programming language = musical grammar

– Extended the earcon technique (q.v.)– An audio program structure diagram

(execution path)


Major features– A MPAS usable by non-musicians– Shown that music can be used to

communicate information about program state and structure

– In turn, this info can assist bug location

Empirical evaluation Organising principles Opened new line of enquiry &

researchAuditory External Representations


Background

Interests in music, computing, & programming

Hunch that music could be used to assist in the programming process

Develop and test a musical framework for communication of program information


Sound in HCI

Increasing research focus since mid ‘80s. A belief that non-speech audio useful as

interaction medium A nascent field called auditory display

spawns techniques like:– Sonification– Audification– Auralisation


Examples

Auditory icons Earcons Data visualisation (sonification &

audification) Assistive technologies


Program auralisation

Mapping of program data and events to sound

Main examples– LogoMedia– Sonnet– ADSL– LISTEN

Largely untested


Why music?

Gives multiple audio streams in parallel

Structured and organised according to specific rules

Time-ordered. Visual techniques are spatial.

Can music’s grammars be used to carry real information?


cAitlIn system

Computer Audio Interface to Locate Incorrect Nonsense

Uses musical motifs to render Pascal language constructs at run time

Motif design is hierarchic as is organisation of constructs


Hierarchic motif design

Construct type

Selection Iteration

IFs CASEs

IF...ELSEIF... CASECASE...

ELSE

Determin-ate

FOR

Indeterm-inate

WHILE REPEAT

Abstract auralizationEntry/Body/Exit leitmotifs

Genericselection leitmotif

Genericiteration leitmotif

GenericIF leitmotif

GenericCASE leitmotif


Musical grammars


Musical grammars

We map existing grammar (e.g. Pascal syntax, English, etc.) onto a musical grammar

Transliteration or translation? We may get emergent properties

leading to a new musical grammar


cAitlIn musical grammar

Uses motif classes and variations to render Pascal constructs

Allows– abstraction– nesting (tonality & harmony preserved

across motifs) Uses grammatical rules (e.g. a tonal

music system)


cAitlIn musical grammar cont Renders information not directly

communicated by source code or other visualisation techniques

a new grammar, conveys extra meaning


Advantages

Over earcons– do not show state

Over Auditory Icons– symbolic mappings = poor system aural

ecology Syntactic design expect retention Displays both flow and state


Design of Tunes

Points of Interest– Construct Entry– Evaluation of condition– Execution of body– Exit from construct

Boolean True=major, False=minor Drones


Selection Tunes

Metaphorically asking & answering a question.

Tune rises on entry and descends on exit


Iteration Tunes

Major/Minor chord to show when condition evaluated

Heard at start of WHILE loops and at end of REPEAT loops

FOR loops increase/decrease the pitch each time round the loop


Variations

All selections use same basic tune, modified for individual constructs

All iterations use another basic tune, again modified


Auditory Parentheses

As () bracket a thought in a sentence, so the auditory parentheses serve to open and close constructs


Other Statements

All other statements are currently represented by a drum beat, one per statement.


Demonstration


Examples

Generic selection motif Generic iteration motif


IF (true)

a := 10 ;IF a = 10 THEN Writeln (‘A is 10’) ;


IF (false)

a := 0 ;IF a = 10 THEN Writeln (‘A is 10’) ;


IF…ELSE (true)

a := 10 ;IF a = 10 THEN Writeln (‘A is 10’)ELSE Writeln (‘A is not 10’) ;


IF…ELSE (false)

a := 1 ;IF a = 10 THEN Writeln (‘A is 10’)ELSE Writeln (‘A is not 10’) ;


CASE match

a := 3 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ;END ;


CASE no match

a := 100 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ;END ;


CASE…ELSE match

a := 3 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; ELSE Writeln (‘No match’) ;END ;


CASE…ELSE no match

a := 100 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; ELSE Writeln (‘No match’) ;END ;


WHILE 2 iters

cntr := 1;WHILE cntr <= 2 DO cntr := cntr + 1 ;


REPEAT 3 iterations

cntr := 1 ;REPEAT cntr := cntr + 1 ;UNTIL cntr > 3 ;


FOR…TO 6 iters

FOR cntr := 1 TO 6 DO Writeln (cntr) ;


FOR…DOWNTO 6 iters

FOR cntr := 6 DOWNTO 1 DO Writeln (cntr) ;


Sequence & Nesting

Clue to nesting is in the drone If you hear a continuous note then you

are inside a construct. Each construct has a drone. For nested constructs you hear multiple

drones simultaneously Drone pitch increases as nesting

deepens


Example programProgram ExemplarVAR counter : Integer ;BEGINcounter := 1 ;WHILE counter <= 2 DO BEGIN IF counter MOD 2 = 0 THEN Writeln (‘Counter is even’) ; counter := counter + 1 ; END ;END.


Scoreiteration open WHILE drone

True

counter := counter + 1;

selection open

FalseIF drone

selection close

WHILE condition

True

IF drone

exit IF

Writeln...sleighbell denotes

terminating conditionfor WHILE

False exit WHILE

iteration close

closed triangle

Auralisation score of aprogram. Notice that

the drones for eachconstruct appear

together on the stafflabelled 'Drones'.

Likewise, all percussiveevents for each

construct appear onthe 'Drums' staff.

Shifts between majorand minor are shown

by changes in keysignature.

1999 Paul VickersAll rights reserved

counter := 1

True

counter := counter + 1;


Score


Structure DiagramProgram Sales

Prepare file Process Sales

Sales Record

Valid Sale Invalid Sale

Test sale Read next

Close file

*

o o


Experiment 1

Listening test to assess comprehensibility of motifs

22 subjects, 60 auralisations Recognition rate better than by

chance and with very little training Hierarchic organisation allowed class

level identification & abstraction


Experiment 2

Debugging exercise 22 subjects, 8 programs, 4

auralised, 4 normal Results suggest technique is

useful, especially for complex programs and where program output gives few clues

Identified avenues for further research


Limitations

No comparison with visual task yet Cognitive latency of motifs? No longitudinal results


Future work

Follow-up studies Extend the grammar to include

assignment statements and procedure calls– How far? Bugs not manifested in program

flow are v. hard to detect and generally require variable inspection to locate, which is impractical for auralisation techniques


Future work

Port cAitlIn to MS-Windows and conduct a longitudinal study

Assistive technologies. How can MPA aid blind programmers?


Future work

Preparing EPSRC bid to investigate Auditory External Representations

(c.f. EPSRC CRUSADE project at COGS in Sussex - du Boulay et al)


Publications–Journals & Reports Vickers, P. and Alty, J. L. (2002). When Bugs Sing,

Interacting with Computers in press. Vickers, P. and Alty, J. L. (2002). Musical Program

Auralisation: A Structured Approach to Motif Design, Interacting with Computers in press.

Vickers, P. and Alty, J. L. (2002). Using Music to Communicate Computing Information, Interacting with Computers in press.

Vickers, P. (2001). Using the World-Wide Web as a Platform for an Interactive Experiment, Liverpool: Liverpool John Moores University, Technical Report CMS-21, ISBN 1-902560-050.


Publications–Conferences Vickers, P. and Alty, J. L. (2000). Musical Program

Auralisation: Empirical Studies, in Proc. ICAD 2000 Sixth International Conference on Auditory Display (Atlanta, GA, 2-5 April, 2000), International Community for Auditory Display, pp. 157-166 ISBN 0-9670904-1-5

Vickers, P. and Alty, J. L. (1998). Towards some Organising Principles for Musical Program Auralisation, in Proc. ICAD '98 Fifth International Conference on Auditory Display (Glasgow, Nov 1-4, 1998), British Computer Society ISBN 1 902505 05 0

Alty, J. L. and Vickers, P. (1997). The CAITLIN Auralization System: Hierarchical Leitmotif Design as a Clue to Program Comprehension, in Proc. ICAD '97 Fourth International Conference on Auditory Display (Palo Alto, November 3-5, 1997), Xerox PARC, Palo Alto, CA 94304, pp. 89-96


Publications–Conferences cont. Alty, J. L., Vickers, P. and Rigas, D. (1997). Using Music as

a Communication Medium, in Proc. Refereed Demonstrations, CHI97 Conference on Human Factors in Computing Systems (Atlanta, GA, March 22-27, 1997), ACM Press, pp. 30-31 ISBN 0-8979-926-2

Vickers, P. and Alty, J. L. (1996). CAITLIN: A Musical Program Auralisation Tool to Assist Novice Programmers with Debugging, in Proc. ICAD '96 Third International Conference on Auditory Display (Palo Alto, Nov 4-6, 1996), Xerox PARC, Palo Alto, CA 94304, pp. 17-24

Documents

Musical Program Auralisation1 The cAitlIn system Musical Program Auralisation & External Representations of Pascal