Mobile App Instrumentation Experimentation FTW

(mobile app) Instrumentation & Experimentation FTWDavid TesarSenior Technical Evangelist, Microsoft@dtzar

Representing:OneDrive app TeamOutlook app iOS TeamNext Lock Screen Team

OneDrive mobile – circa 2013

Feedback without the information to act

Instrumentation & Experimentation

1. Direct App Feedback2. Automated Builds + Instrumentation3. Scaling Feedback Channels4. Telemetry5. A/B Testing

FTW?

Direct App Feedback (OneDrive)Parature Console

Direct App Feedback (Next)

1.Friday 1 am – Deployment to Play Store2.Friday 1:04 am – 1 email for feedback3.<sleep>4.Friday 9 am – 200 emails5.Saturday – 2,000 emails

Direct App Feedback (Next)

Direct App Feedback (Outlook)

Powered by Helpshift

Direct App Feedback – FTW

Results• Higher ratings!• Better prioritization of features• Solve some bugs where there are no

crashes• Actionable New bugs and features

discovered Lessons• Have a way to organize feedback at start• Centralized feedback across platforms

useful

Automated builds to phones + instrumentation

Power of Instrumentation (Outlook)

Power of Instrumentation (Outlook)

Automated builds to phones + instrumentation - FTW

Results• Found many more bugs in dev• Saved hours every week getting builds to

phones• Cross team feature testing and feedbackLessons• Be super responsive on daily builds• Have as many people possible using these

builds• Balance frequency of updates

Scaling Feedback (OneDrive)

Feature Flags Production

Test Flight

Hockey App

Dev Machine

Scaling Feedback (Next)

Radek – screenshot of Google+ Alpha->Beta channels and/or (ideally) Azure Mobile Engagement instrumentation with being able to slice/dice crash data info

Scaling Feedback (Next)

Capptain Azure Mobile Engagement

Scaling Feedback - FTW

Results• Faster deployments/agility• Increase in quality, reduced bugs in

productionLessons• Get all crash reports• Capture log information in the crash report• Don’t release builds on Friday• Develop a beta community

Telemetry (Outlook)

Internal MSFT tool

Telemetry (OneDrive)

App Insights / HockeyApp + PowerBI

Telemetry (Next)

Radek – AzureME screenshot of usage

Azure Mobile Engagement

Telemetry - FTW

Results• Know if features are useful and successful• Reduction in technical debt• Prioritize features/platforms to no longer support• Quicker determination of service or app problemLessons• Capture all unexpected error scenarios• Instrument any major changes to see impact• Have a single view of app and service

A/B Testing (OneDrive)

A/B Testing (OneDrive)

A/B Testing* (Next)

Radek – screenshot of Google+ Alpha->Beta channels and/or (ideally) Azure Mobile Engagement instrumentation with being able to slice/dice crash data info

*not scientifically

A/B Testing - FTW

Results• > 50% increase in clicks to learn

more about purchase of more storage link• Shipped the i button to 100% of our users• Next: Fast feedback for ideas, low costLessons• Data trumps opinion!

OneDrive – Outlook – Next Apps

For The Win

Resources

Learning• DevOps Dimension Show on @CH9

Special episode on mobile DevOpshttp://aka.ms/DoDmobile

Products Mentioned• MSFT: http://hockeyapp.net• MSFT: http://parature.com• MSFT: http://aka.ms/azme (Azure Mobile Engagement)• MSFT: http://aka.ms/VSTSinfo (Visual Studio Team Services)• 3rd party: http://helpshift.com• 3rd Party: http://optimizely.com• 3rd Party: http://uservoice.com

Special thanks to:@RayyanJaber@ogkent@krakookerad