26
WINDOWS 10 DEVELOPMENT Hey Cortana!

Code Camp - Presentation - Windows 10 - (Cortana)

Embed Size (px)

Citation preview

Page 1: Code Camp - Presentation - Windows 10 - (Cortana)

WINDOWS 10 DEVELOPMENT Hey Cortana!

Page 2: Code Camp - Presentation - Windows 10 - (Cortana)

WHO ARE WE? Edward Moemeka – Chief Architect, e-Builder Follow me on Twitter @moemeka Email me at [email protected] Connect with me on LinkedIn https://www.linkedin.com/in/edwardmoemeka

Obinna Igbokwe – Solution Architect, e-builder Email [email protected]

Page 4: Code Camp - Presentation - Windows 10 - (Cortana)

WHY CORTANA? The obvious reasons World’s most personal digital assistant helps you get things done throughout the day

Be the one helping! A whole new natural, way of interacting with your PC

Create the experience Available to millions of users across Windows 10 PCs, tablets, and phones

More money, more problems Integrated search across device, the Web and personal and professional clouds

Omni-channel integration can lead directional growth

Typical usage scenarios “App first” experiences

I’ve got a pre-existing app and I want to integrate voice I’ve got an app on the system and I want to surface it's content

“Voice first” experiences I want to create an audio interaction experience

Page 5: Code Camp - Presentation - Windows 10 - (Cortana)

MAKING YOUR PROGRAM SPEAK THE EASY WAYText to speech, commonly referred to as TTS, uses a speech synthesis engine (voice) to convert a text string into spoken words. The simplest form of this takes as input the text you actually want the engine to utter.

Page 6: Code Camp - Presentation - Windows 10 - (Cortana)

DEMO Text-to-speech (basics)

Page 7: Code Camp - Presentation - Windows 10 - (Cortana)

MAKING YOUR PROGRAM SPEAK THE HARD WAY

For added flexibility, Speech Synthesis Markup Language (SSML) can be used to describe the manner in which you want a speech synthesis engine to read text that is inputted into it.

Page 8: Code Camp - Presentation - Windows 10 - (Cortana)

DEMO Text-to-speech (cortana)

Page 9: Code Camp - Presentation - Windows 10 - (Cortana)

RECOGNIZING SPEECH Converts words spoken by the user into text – which ultimately can be used for form input, text dictation, to specify an action or command, and/or to accomplish tasks. Supports

pre-defined grammars for free-text dictation and web search

Custom grammars authored using Speech Recognition Grammar Specification (SRGS)

Page 10: Code Camp - Presentation - Windows 10 - (Cortana)

DEMO Voice Recognition

Page 11: Code Camp - Presentation - Windows 10 - (Cortana)

RECOMMENDATIONS ON SPEECH RECOGNITION Always provide a visual cue to indicate that speech recognition is supported and available to the user and whether the user needs to turn it on.

Provide ongoing recognition feedback to minimize any apparent lack of response while recognition is being performed.

Let users revise recognition text using keyboard input, disambiguation prompts, suggestions, or additional speech recognition.

Stop recognition if input is detected from a device other than speech recognition, such as touch or keyboard. This probably indicates that the user has moved onto another task, such as correcting the recognition text or interacting with other form fields.

Specify the length of time for which no speech input indicates that recognition is over. Do not automatically restart recognition after this period of time as it typically indicates the user has stopped engaging with your app.

Disable all continuous recognition UI and terminate the recognition session if a network connection is not available. Continuous recognition requires a network connection.

Page 12: Code Camp - Presentation - Windows 10 - (Cortana)

HELLO CORTANA Utilizes VCD files Requires an alias that uniquely identifies your app If an alias name collision happens, a prompt (within the Cortana interface) will be presented to the user which

gives them the option to select which app the would like to use to service the request Does not require apps to be running for it to function Allows for you to use Cortana to launch your UWP apps as though it is a command intrinsically built into the

system Can launch your app Can use your app to service the customer’s requests

Two categories Foreground apps

Apps launched in this manner are launched into the foreground, meaning that the app takes focus and Cortana is dismissed. Your voice input is passed into the app as a string parameter of the OnActivated method. Saved best for commands that require additional context or user input.

Background apps The user is given no visual indication that your app is servicing the request (the primary request from the user is handled by the Cortana

interface) Allows for providing lists, secondary actions (links), images, to the customer. Your app can be launched from one of those actions if need be

Page 13: Code Camp - Presentation - Windows 10 - (Cortana)

STEPS TO CREATING FOREGROUND INTEGRATION Create a VCD file. This is an XML document that defines all the spoken commands that the user can say to initiate actions or invoke commands when activating your app.

Register the command sets in the VCD file when the app is launched.

Handle the foreground activation of the of the app through the OnActivated handler. As part of the launch parameters you will have access to command that was triggered by Cortana and the text of the words that were uttered.

Page 14: Code Camp - Presentation - Windows 10 - (Cortana)

VCD FILES

Page 15: Code Camp - Presentation - Windows 10 - (Cortana)

LOADING PHRASELISTS DYNAMICALLY

Page 16: Code Camp - Presentation - Windows 10 - (Cortana)

HANDLING ACTIVATION FROM CORTANA

If the window content is not set to a UIElement by the time the OnActivated method completes it will not be initialized

Page 17: Code Camp - Presentation - Windows 10 - (Cortana)

DEMO Cortana foreground apps

Page 18: Code Camp - Presentation - Windows 10 - (Cortana)

STEPS TO CREATE BACKGROUND INTEGRATION Create a VCD file. This is an XML document that defines all the spoken commands that the user can say to initiate actions or invoke commands when activating your app.

Create an app service (Windows.ApplicationModel.AppService) that Cortana invokes in the background.

Register the command sets in the VCD file when the app is launched.

Handle the background activation of the of the app service and the execution of the voice command.

Display and/or speak the appropriate feedback to the voice command within Cortana.

Page 19: Code Camp - Presentation - Windows 10 - (Cortana)

RETRIEVING WHAT WAS SAID

Page 20: Code Camp - Presentation - Windows 10 - (Cortana)

RETRIEVING WHICH PHRASE WAS UTTERED

Page 21: Code Camp - Presentation - Windows 10 - (Cortana)

DEMO Cortana prompting confirmation

Page 22: Code Camp - Presentation - Windows 10 - (Cortana)

DEEP LINKING A "Go to <app>" link on various Cortana screens. A link embedded in a content tile on various Cortana screens. The app service programmatically launches the foreground app. Requires element added to the Extensions node of the Application element: <uap:Extension Category="windows.personalAssistantLaunch"/> As with any protocol contract, your app must override its OnActivated event and check for an ActivationKind of Protocol.

When your app is launched in this manner the resulting URL sent to it is "windows.personalassistantlaunch:?LaunchContext=<AppLaunchArgument>"

Page 23: Code Camp - Presentation - Windows 10 - (Cortana)

PROMPTING THE USER In certain scenarios it may be necessary to ask the user for a confirmation before proceeding with an action through the background app service. For this kind of situation, the method CreateResponseForPrompt on the class VoiceCommandResponse can be used

Page 24: Code Camp - Presentation - Windows 10 - (Cortana)

DEMO Cortana background apps

Page 25: Code Camp - Presentation - Windows 10 - (Cortana)

WHO ARE WE? Edward Moemeka – Chief Architect, e-Builder Follow me on Twitter @moemeka Email me at [email protected] Connect with me on LinkedIn https://www.linkedin.com/in/edwardmoemeka

Obinna Igbokwe – Solution Architect, e-builder Email [email protected]