Michael KoesterSr. Product Marketing Manager, Visual Studio Online, ALM & DevOps

Part of the Visual Studio product team and a developer technologies specialist

Based in Frankfurt, Germany (it’s the airport …)

Worked for Microsoft for 12 years in a number of different roles:

Program Management, Product Management, Sales, Marketing, Consulting …

Why does this matter?Agile and DevOps practices are changing the way teams develop software

An organization’s ability to deliver quickly and reliably is a key competitive differentiator

Being responsive means understanding what’s going on in a deployed application

The engineering system is central to developer productivity and job satisfaction

The end-to-end tools, processes & culture enabling a software team to

turn a customer scenario into deployed product code and improve it

through a continuous learning system

ToolsProcessCulture

Practicing Build, Measure, Learn

Adopting the best of OSS

Focus on quality

Change one thing – change everything!

SchedulingWe did ask for feedback after each milestone – planning, Beta, RTM

We did find bugs with this process and fix them - no problems there

But we couldn’t react to anything customers using the product were telling us

For the most part, we would tell everybody “sorry”… and push things to the next release

Planning

Beta? RTM?

Schedule

Code Test & Stabilize Code Test & Stabilize

2,5 years

Our planning process has changed - from long upfront planning

cycles to shorter build, measure, learn cycles …

… with continuous alignment to our long term vision

W O R K S F O R

E V E R Y O N E

A G I L E A T S C A L E

M O D E R N C O D I N G

A C T I O N A B L E

I N S I G H T S

M A N A G I N G

T E C H N I C A L D E B T

F I R S T P A R T Y I S

T H I R D P A R T Y

C O N T I N U O U S

D E L I V E R Y

Enterprise Agile

Visual Studio 2015

Our engineering system is hosted on Visual Studio Online:

The same Cloud service we offer to our customers

All new code is deployed to SU0 (our own branch)

first - before our customers get it

VSO SU1

Chicago

VSO SU0

San Antonio

VSO SU5

Amsterdam

Shared Platform Services

San Antonio

All code is deployed, but feature flags control exposure

Flags provide runtime control down to individual user

Users can be added or removed with no redeployment

Enables “dark” launch

Mechanism for progressive experimentation & refinement

Too many actions and distractions can

prevent customers from taking the next

natural step of creating a new project

Previous project creation experience

Focused project creation

experience - integrated

into the IDE flow

IDE

Web

% of accounts creating projects the day of account creationJanuary 22 to February 11 (all sources)

A second experiment,

running in parallel,

invalidated results

First project creation

jumps from 15% to 50%

First project creation

from IDE jumps from 3%

to 20%

% of accounts creating projects the day of account creation January 22 to February 5 (IDE only)

Customer IntelligenceBusiness IntelligenceOperational Intelligence

ActivityLogging

TracesKPI

MetricsJob

HistoryCustomer

IntelligenceSynthetic

Perf Counters

DependentServices

Customer IntelligenceBusiness IntelligenceOperational Intelligence

ActivityLogging

TracesKPI

MetricsJob

HistoryCustomer

IntelligenceSynthetic

Perf Counters

DependentServices

Customer IntelligenceBusiness IntelligenceOperational Intelligence

ActivityLogging

TracesKPI

MetricsJob

HistoryCustomer

IntelligenceSynthetic

Perf Counters

DependentServices

Customer IntelligenceBusiness IntelligenceOperational Intelligence

ActivityLogging

TracesKPI

MetricsJob

HistoryCustomer

IntelligenceSynthetic

Perf Counters

DependentServices

Customer IntelligenceBusiness IntelligenceOperational Intelligence

ActivityLogging

TracesKPI

MetricsJob

HistoryCustomer

IntelligenceSynthetic

Perf Counters

DependentServices

Customer IntelligenceBusiness IntelligenceOperational Intelligence

ActivityLogging

TracesKPI

MetricsJob

HistoryCustomer

IntelligenceSynthetic

Perf Counters

DependentServices

Front-doors, conversion funnel, engagement, top customers

Input metrics vs. output metrics

Watch out for vanity metrics!

Application Insights

Visual Studio 2015

We are bringing the benefits of open source and services culture

to our internal development workflow …

… while maintaining the rigors of enterprise assets

A transition of this magnitude will take time, but

we’re already well on our way!

… source across the company is available to anyone

… any developer can offer improvements to anything in the company

… IP the company has built up over the years is made of re-usable components

… anybody can find and potentially re-use components from anywhere else

… developers are rewarded for creating popular components

… minimal lag from a developer change to when the rest of the company sees it

… developers can move to another team and already know how to work

… products built this way meet the rigors of quality and support our customers expect

Update 1

Update 2

Update N

One code base with multiple delivery streams

Single master branch, multiple release branches

Shared abstraction layer (VSSF)

Be able to understand what it does

Can easily pick it up and use

Fork/Follow/Favorite

Download

Answer questions, write blog posts

Fix/add documentation

Log bugs

Fix typos Actually contribute code patches that fix bugs

Contribute entirely new features

Maintain platforms

Translate

Stick around - influence the direction of the project

Become a core committer (get write access)

Accept/validate code contributions

Nurture new people

Own

Contribute code

Contribute time

Use

Our move to GitMigrated a team of 250+ engineers from TFVC to Git

Left history behind – no complaints so far!

You can never do enough Git training, hands-on workshops is the way to go

Git scales reasonably well with file size

Using a one-branch model with good results (made feasible by significant investments in speeding up our build time)

Pull requests have formalized our code review process

GitStarts with the characteristics and workflows for sharing

Brings the culture of sharing, reinforces componentization mindset

A modern source control system, a popular tool in the industry

Investing to build an enterprise class & scale cloud Git service

Quality in the Cloud cadence optimizes

code velocity and release velocity

Change in the engineering roles was needed to enable this shift

This doesn’t mean our quality bar to customer is lowered - quite the opposite!

Ship frequently with minimum frictionDream of a system that can do hands-free push of a hotfix to production in 15 minutes

Fast and predictable Inner LoopInner Loop = Build >Deploy >Test cycle which runs reliably hundreds of times a day

Quality approach designed for Cloud cadence

Continue to meet a rigorous on-premises quality bar

PrinciplesTests should be written at the lowest level possible

Write once, run anywhere - includes production system

Product is designed for testability

Test code is product code, only reliable tests survive

Testing infrastructure is a shared service

New Engineer role combines responsibilities from dev and testEveryone has a new role, everyone is learning new skills

Every engineer and team has E2E accountabilityNo more handoffs for testing, stronger emphasis on peer reviews

Big cultural shift across the company

IntelliTest

Visual Studio 2015

Code Test & Stabilize Code Test & Stabilize

Beta RTM

Planning

Code

Complete

Cu

stom

er

Jou

rney

Discover Value

Acquire Product

Learn

Use Features

Pay

Optimize Value

Program Manager… understands customer needs

… designs great features

Customer

Cu

stom

er

Jou

rney

Discover Value

Acquire Product

Learn

Use Features

Pay

Optimize Value

Program Manager… understands customer needs

… designs great features

… attracts customers into the funnel

… describes the product value

… documents product usage

… forms business models

… analyzes and obsesses over customer data

Customer

Live Site Health/DebtTime to Detect, Time To Mitigate

Incident prevention items

Aging live site problems

Customer support metrics (SLA, MPI, top drivers)

Engineering Health/DebtBug cap per engineer

Aging bugs in important categories

Pass rate & coverage

Velocity Time to build

Time to self test

Time to deploy

Time to learn (Telemetry pipe)

We are modernizing the process for documentation, engineering

and community to easily create developer content

Documentation writing is coming to the Cloud Cadence

Building a system and culture that treats “Doc as Code”

M2 Feature DevelopmentPlanning M1 Feature Development

Result:

Specifications1 Result:

Feature Complete2

DraftsFirst draft of

documentation

ReviewDraft doc sent

out for review

ReviewChanges are

incorporatedPreviewWriter publishes to

sandbox for preview

StagingWriter promotes

content to staging

Go-LiveWriter promotes

content to live

Schedule

Documentation process

Pub

lished

It takes a full sprint or release

until docs come together and

work is only visible to a small

set of users and stakeholders

Refining

Docs continue to get

refined and updated

Each update is just a

small increment

Review

Automated review mail

is sent to the team and

stakeholders

Docs are automatically

published to an internal

staging website

Week

Planning1

Drafts

Specs created, initial

docs authored by PMs

API docs authored by

PM/Engineers before

any code is written

Week

Coding2/3

Changes

Anyone wanting to add

a doc change simply

does a pull request

Merging

Content owner/PM

gets a notification

and merges

PublishedContent updated to live branch, automatically

pushed to live public website

Sprint

3 Weeks

Microsoft is creating ‘One Engineering System (1ES)’ for the whole

company designed for the Cloud Cadence

Visual Studio Online is the foundation for this system

Our team is building many of its components and leading the change

Summary

Internal shared source experience

An agile work management system

Source code management strategy using Git

New quality approach

Distributed cloud based build system

A componentization system

Live site management tools

Telemetry

We are building all this …

… in