© 2016 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

Enhanced voice-control for macOS and applications

System Frameworks #WWDC16

Session 717

Beyond Dictation

Sal Soghoian Product Manager Automation Technologies

macOS

macOS

macOS

Dictation • Enhanced Dictation • Advanced Commands • User Commands

Dictation • Enhanced Dictation • Advanced Commands • User Commands

Dictation • Enhanced Dictation • Advanced Commands • User Commands

Dictation • Enhanced Dictation • Advanced Commands • User Commands

Dictation • Enhanced Dictation • Advanced Commands • User Commands

Dictation • Enhanced Dictation • Advanced Commands • User Commands

Easy transcription everywhere

Dictation

TextEdit

Messages

Mail

Safari

Transcription everywhereDictation

Features• Universal text input throughout the OS• Good for entering names, titles, short phrases, and URLs

Limitations• Requires an Internet connection• Not intended for extended dictation• No editing commands

Enhanced Dictation

Window-scoped dictation and editing

Enhanced Dictation

Consectetur Ligula

Nullam id dolor id nibhultricies vehicula ut id elit.Aenean lacinia bibendumnulla sed consectetur. Integerposuere erat a ante venenatisdapibus posuere velit aliquet

Window-scoped dictation and editing

Donec sed odio dui

Curabitur blandit tempus

Purus Euismod

EnhancedDictation

Window-scoped dictation and editing

Consectetur Ligula

Nullam id dolor id nibh ultricies vehicula ut id elit. Aenean lacinia bibendum nulla sed consectetur. Integer posuere erat a ante venenatis dapibus posuere velit aliquet

Donec sed odio dui

Curabitur blandit tempus

Purus Euismod

EnhancedDictation

Window-scoped dictation and editing

Consectetur Ligula

Nullam id dolor id nibh ultricies vehicula ut id elit. Aenean lacinia bibendum nulla sed consectetur. Integer posuere erat a ante venenatis dapibus posuere velit aliquet

Donec sed odio dui

Curabitur blandit tempus

Purus Euismod

Window-scoped dictation and editing

EnhancedDictation

Consectetur Ligula

Nullam id dolor id nibh ultricies vehicula ut id elit. Amet lacinia bibendum nulla sed consectetur. Integer posuere erat a ante venenatis dapibus posuere velit aliquet

Donec sed odio dui

Curabitur blandit tempus

Purus Euismod

Window-scoped dictation and editing

EnhancedDictation

Offline use

Offline useContinuous dictation

Offline useContinuous dictationLive feedback

Audio Output Control

Recognition Sound

Audio Output Control

Keyword Control

Recognition Sound

Audio Output Control

Keyword Control

Recognition Sound

Audio Output Control

Command Suite

Command Suite

Commands

Command Suite

Search Commands

Commands

Command

Command

Syntax Options

Selection Suite

Select <phrase> Select all Select word Select previous word Select next word Select sentence Select previous sentence Select next sentence Select paragraph Select previous paragraph Select next paragraph Select previous Select next Deselect that

Selection Suite

Navigation Suite

Go to beginning Go to end Go to beginning of selection Go to end of selection Go to beginning of word Go to end of word Go to beginning of sentence Go to end of sentence Go to beginning of paragraph Go to end of paragraph Move left Move right Move up Move down Go to next field Go to previous field Scroll up Scroll down Scroll to top Scroll to bottom Send that

Navigation Suite

Editing Suite

Replace <phrase> with <phrase> Undo that Redo that Cut that Copy that Paste that Delete that Capitalize that Capitalize <phrase> Lowercase that Lowercase <phrase> Uppercase that Uppercase <phrase> Press Return key Insert date Literal <phrase>

Editing Suite

Formatting Suite

Bold that Bold <phrase> Italicize that Italicize <phrase> Underline that Underline <phrase>

Formatting Suite

System Suite

System SuiteStart Dictation Stop Dictation Show commands Hide commands How do I <phrase>

Bold that Bold <phrase> Italicize that Italicize <phrase> Underline that Underline <phrase>

Formatting SuiteReplace <phrase> with <phrase> Undo that Redo that Cut that Copy that Paste that Delete that Capitalize that Capitalize <phrase> Lowercase that Lowercase <phrase> Uppercase that Uppercase <phrase> Press Return key Insert date Literal <phrase>

Editing SuiteGo to beginning Go to end Go to beginning of selection Go to end of selection Go to beginning of word Go to end of word Go to beginning of sentence Go to end of sentence Go to beginning of paragraph Go to end of paragraph Move left Move right Move up Move down Go to next field Go to previous field Scroll up Scroll down Scroll to top Scroll to bottom Send that

Navigation SuiteSelect <phrase> Select all Select word Select previous word Select next word Select sentence Select previous sentence Select next sentence Select paragraph Select previous paragraph Select next paragraph Select previous Select next Deselect that

Selection Suite

Start Dictation Stop Dictation Show commands Hide commands How do I <phrase>

System Suite

Dictating contentEnhanced Dictation

Features• Offline use• Continuous dictation• Live feedback• Text editing and dialog navigation commands

Limitations• Procedural (step-by-step)

Advanced Commands

Dictation and Application controls

Advanced Commands

Dictation and Application controls

AdvancedCommands

Dictation and Application controls

AdvancedCommands

Dictation and Application controls

Advanced Commands

Advanced Commands

Navigation Suite

Go to beginning Go to end Go to beginning of selection Go to end of selection Go to beginning of word Go to end of word Go to beginning of sentence Go to end of sentence Go to beginning of paragraph Go to end of paragraph Move left Move right Move up Move down Go to next field Go to previous field Scroll up Scroll down Scroll to top Scroll to bottom Send that

Navigation Suite

Go to beginning Go to end Go to beginning of selection Go to end of selection Go to beginning of word Go to end of word Go to beginning of sentence Go to end of sentence Go to beginning of paragraph Go to end of paragraph Move left Move right Move up Move down Go to next field Go to previous field Scroll up Scroll down Scroll to top Scroll to bottom Send that

Press Escape key Show numbers Choose <number> Hide numbers Find the text <phrase> Find that Open result Close Spotlight Previous result Next result Scroll preview down Scroll preview up Close Siri Search now Go back Go forward New tab Close tab Zoom in Zoom out

Navigation Suite

System Suite

Start Dictation Stop Dictation Show commands Hide commands How do I <phrase> Start Siri Search Spotlight for <phrase> Search for <phrase> Make this speakable Open Dictation preferences

System Suite

Application Suite*

Switch to <application name> Quit <application name> Hide <application name> Quit application Hide application Click <menu name> menu

Application Suite*

Document Suite*

New item Open document Save document Close window Minimize window Click <item name>

Document Suite*

Bold that Bold <phrase> Italicize that Italicize <phrase> Underline that Underline <phrase>

Formatting SuiteReplace <phrase> with <phrase> Undo that Redo that Cut that Copy that Paste that Delete that Capitalize that Capitalize <phrase> Lowercase that Lowercase <phrase> Uppercase that Uppercase <phrase> Press Return key Insert date Literal <phrase>

Editing SuiteGo to beginning Go to end Go to beginning of selection Go to end of selection Go to beginning of word Go to end of word Go to beginning of sentence Go to end of sentence Go to beginning of paragraph Go to end of paragraph Move left Move right Move up Move down Go to next field Go to previous field Scroll up Scroll down Scroll to top Scroll to bottom Send that

Navigation SuiteSelect <phrase> Select all Select word Select previous word Select next word Select sentence Select previous sentence Select next sentence Select paragraph Select previous paragraph Select next paragraph Select previous Select next Deselect that

Selection Suite

Start Dictation Stop Dictation Show commands Hide commands How do I <phrase>

System Suite

Go to beginning Go to end Go to beginning of selection Go to end of selection Go to beginning of word Go to end of word Go to beginning of sentence Go to end of sentence Go to beginning of paragraph Go to end of paragraph Move left Move right Move up Move down Go to next field Go to previous field Scroll up Scroll down Scroll to top Scroll to bottom Send that

Press Escape key Show numbers Choose <number> Hide numbers Find the text <phrase> Find that Open result Close Spotlight Previous result Next result Scroll preview down Scroll preview up Close Siri Search now Go back Go forward New tab Close tab Zoom in Zoom out

Navigation SuiteStart Dictation Stop Dictation Show commands Hide commands How do I <phrase> Start Siri Search Spotlight for <phrase> Search for <phrase> Make this speakable Open Dictation preferences

System SuiteSwitch to <application name> Quit <application name> Hide <application name> Quit application Hide application Click <menu name> menu

Application Suite

New item Open document Save document Close window Minimize window Click <item name>

Document Suite

Formatting SuiteEditing SuiteSelection Suite

Commander of the user interfaceAdvanced Commands

Features• Control of all accessible application interfaces

- Application and pop-up menus- Buttons and tabs

Limitations• Procedural (step-by-step)

- Simulates a mouse or keyboard• Not all applications offer robust accessibility support

User Commands

Dictation on fire. Get the marshmallows. 😎

User Commands

Dictation on fire. Get the marshmallows. 😎

User Commands Suite

New Command

User Commands Suite

New Command

Command PhraseUser Commands Suite

New Command

Command PhraseUser Commands Suite

New Command

Command Phrase

Application

User Commands Suite

Command ActionNew Command

Command Phrase

Application

User Commands Suite

Command ActionNew Command

Command Phrase

Application

User Commands Suite

New Command

Everything/anything is possibleUser Commands

Features• Task-oriented• Exposes the power of macOS automation to dictation• Incorporates all existing dictation abilities

- Standard, enhanced, and advanced• Hands-free, always-on, visual feedback, no Internet connection required

Limitations• Cumbersome to create

Everything/anything is possibleUser Commands

Features• Task-oriented• Exposes the power of macOS automation to dictation• Incorporates all existing dictation abilities

- Standard, enhanced, and advanced• Hands-free, always-on, visual feedback, no Internet connection required

Limitations• Cumbersome to create

DemoCustom Command Sets

Dictation commands for iWork, Photos, and the Finder

DemoCustom Command Sets

Dictation commands for iWork, Photos, and the Finder

What Are They Good For?

Dictation Commands

What Are They Good For?

Remaining in Context

Multi Step Tasks

Tasks Requiring Dexterity

Moving Data

Data Transformation

Tasks not available in UI

Things the user wants to do, but does not know how to do

What Are They Good For?

What Are Dictation Commands Good For?

What Are Dictation Commands Good For?

Remaining in contextMultistep tasksTasks requiring dexterityMoving dataData transformationTasks not available in the UIThings the user does not know how to do

How It Works

Like magic wrapped in bacon

AppleScript • JavaScript (JXA)

Script Library

AppleScript • JavaScript (JXA)

Script Library

Script Library

English.lproj

Resources

Contents

Localizable.strings

Script Library

Script Library

Script Library

AppleScriptObj-C JavaScript (JXA)

Quartz (ImageKit)

FoundationAppKitMapKitEventKitSharingKit

Apple EventsUNIX CLI

Script Library

AppleScriptObj-C JavaScript (JXA)

Make a new presentation

Document-New-Default.scpt

Document-New-Default.scpt

Library("DC-Keynote").newDefaultDocument()

Document-New-Default.scpt

Library("DC-Keynote").newDefaultDocument()

on newDefaultDocument() tell application "Keynote" activate if playing is true then tell document 1 to stop end if make new document end tell end newDefaultDocument

Document-New-Default.scpt

Library("DC-Keynote").newDefaultDocument()

on newDefaultDocument() tell application "Keynote" activate if playing is true then tell document 1 to stop end if make new document end tell end newDefaultDocument

Resources

dictationcommands.com

Resources

Summary

D I C T A T I O N C O M M A N D S

Summary

D I C T A T I O N C O M M A N D S

Summary

Dictation is no longer “an alternative text input method”

D I C T A T I O N C O M M A N D S

Summary

Dictation is no longer “an alternative text input method”Speech is no longer an “assistive technology”

D I C T A T I O N C O M M A N D S

Summary

Dictation is no longer “an alternative text input method”Speech is no longer an “assistive technology”Voice is a peer interface to touch, keys, and cursor

D I C T A T I O N C O M M A N D S

macOS

More Information

https://developer.apple.com/wwdc16/717

Labs

Dictation and Enhanced Voice-Control for macOS Lab Fort Mason Friday 9:00AM