Technical Director experience using machineEnhancing the Umbraco experience using machine learning...

Enhancing the UmbracoEnhancing the Umbracoexperience using machineexperience using machine

learninglearning

Steve TempleTechnical Director

About meAbout meTechnical Director and co­owner of Gibe

Umbraco MVPUmbraco Certified Master

Organiser of Umbraco Spark conference 

Will be using AI and Machine Learning interchangeablyApproaching Machine Learning as a developer

I am not a data scientistHave been playing with Azure ML since it came outFirst experiments were predicting horse racing results

UmbracoSpark Developer Conference

Bristol6th March 2020

Point of the talkPoint of the talk

No �nished out of the boxNo �nished out of the boxsolutionssolutions

Intention is to spark ideas and inspire packagesThere is a massive learning curve to some of these

But for some solutions the hard work has been done an it's up to usto figure out how to use it

Machine Learning toolsMachine Learning tools

On the shoulders of giantsOn the shoulders of giants 

Services available from amongst others:

Microsoft Azure, AWS,Microsoft Azure, AWS,Google, IBMGoogle, IBM

What can you useWhat can you useMachine Learning for?Machine Learning for?

ClassificationPrediction

Anomaly detection

How can we use these to solve problems inHow can we use these to solve problems inUmbraco sites?Umbraco sites?

Classi�cationClassi�cationCategorising & tagging mediaCategorising & tagging media

or contentor content

Problem to solveProblem to solveHow can we use AI to improveHow can we use AI to improve

Umbraco's media section?Umbraco's media section?

1. No standard location for media2. Doesn't scale to 10,000s of images 3. Hard to find the exact image you want

The Media TreeThe Media TreeDoes a tree make sense for Media?

Has some downsides:Needs more information on each image/video/file

Editors would have to tag/categorise all media including existingmedia

Could we searchCould we searchinstead?instead?

Tagging of mediaTagging of mediaUse Azure Vision API to autoUse Azure Vision API to auto

tag/categorise/detect featurestag/categorise/detect featuresin imagesin images

 DEMO

Great demoGreat demoBut rubbish IRLBut rubbish IRL

Custom Vision APICustom Vision APIWe can train our own modelWe can train our own modelwith our own tags which wewith our own tags which we

want to detectwant to detect 

DEMO

What other APIsWhat other APIscould we use?could we use?

Focal point?Video analysis?Image cropper?Visual search?

What about content?What about content?Auto tagging / subject identification

Automated translation

Optical Character Recognition (OCR)https://our.umbraco.com/packages/backoffice­extensions/translation­manager/

https://our.umbraco.com/packages/backoffice­extensions/ocr­content­app/

PredictionPredictionRecommending other contentRecommending other content

Problem to solveProblem to solveHow can we use AI toHow can we use AI to

recommend content on a site?recommend content on a site?

Azure Machine Learning StudioAzure Machine Learning Studio

Azure Machine Learning StudioAzure Machine Learning Studio

Azure Machine Learning ServiceAzure Machine Learning Service

Replaces Azure ML StudioCan use VS Code or Jupyter Notebooks

Azure DatabricksAzure Databricks

Very powerfulVery complicated!

Works with Azure ML Services

Solutions/ExamplesSolutions/Examples

https://gallery.azure.ai

RecommendationsRecommendationsSolutionSolution

Prebuilt solution template that deploys Azure resources pre­configured for performing recommendations

 Needs usage information and catalog information to train the

model 

https://github.com/Microsoft/Product­Recommendations

Recommendations solutionRecommendations solution

Needs to be supplied with 2 models to createrecommendations

UsageUsageModelsModels

How users are using the site

CatalogCatalogModelsModels

Product or content information

Con�guring and usingCon�guring and usingthe API the API 

REST API with Swagger UIREST API with Swagger UI1. Create a trained model2. Make requests to the API for recommendations

 DEMO

Suggesting media to editorsYou might also like / related content

Estimated time to readPersonalisation

...

How about?How about?

Anomaly DetectionAnomaly DetectionMaking sites more secure andMaking sites more secure and

easier for the userseasier for the users

Problem to solveProblem to solveCan we use AI to makeCan we use AI to make

user/member logins simpleruser/member logins simplerwhile being more securewhile being more secure

LoginLoginCould we serve different user orCould we serve different user or

member login experiencesmember login experiencesbased on probability of the userbased on probability of the user

being legitimatebeing legitimate

Factors we mightFactors we mightconsiderconsider

IP AddressCountryRegionTime

Browser / cookie recognisedPrevious logins

Recent failed attemptsPrivileges

Score the attemptScore the attempt> 98% 

Auto login?Remember me, etc

75% +Regular login

 < 75%Show RecaptchaRequire 2FA

This all sounds superThis all sounds supercomplexcomplex

https://giphy.com/embed/5dYeglPmPC5lL7xYhs

How to get startedHow to get startedSo much potentialSo much potentialGet a free azure account & experiment

There are multiple competitions/challenges all come with brilliantdata sets to get started with

Steep learning curve, but look at sample solutions and work fromthere 

ResourcesResourceshttps://studio.azureml.net/

https://azure.microsoft.com/en­gb/services/cognitive­services/

https://gallery.azure.ai/

https://github.com/Microsoft/Product­Recommendations

https://docs.microsoft.com/en­us/azure/machine­learning/studio/algorithm­choice

Thank youThank you