Disruptive Innovation: how do you use these theories to manage your IT?

Disruptive Innovation: Past, Present, Future (how to use these theories to manage your IT) February, 2016

Mark Madsen - @markmadsen - http://ThirdNature.net

Innovation: The Cargo Cult of Management Consulting

The go-to innovation company is Google

You can’t get fired for doing what everyone else did (but you can get fired for not getting the results they did)

If you do what google did you could:

Make a data center out of shipping containers.

▪ That didn’t work out.

Build your own servers.

▪ Made out of “razor blades and hate”

▪ Start fires in the data center

Build your own environmental cooling data center

▪ Generating fog and rain inside the data center

As Dan Luu noted, at least copy current engineering practices rather than things done in 1999.

By the way, are you using MapReduce?

Saying there’s a process you can follow to be innovative is like saying there’s a recipe that will make you a chef.

You keep using that word.

I do not think it means

what you think it means.

Innovation?

Innovation is not “add new features”

“Better experiences, not more features.” Roland Rust

“When technology

delivers basic needs,

user experience

dominates”

Don Norman

Value is not in the product, it’s in the practice

Innovation is not a characteristic of things

Innovation is change. Change is often not appreciated.

Paradox: Innovation becomes best practice

Innovation isn’t reproducible. Only the conditions that permit it are

HOW DOES THE MARKET WORK AND WHAT IS HAPPENING TO OUR TECHNOLOGIES?

Commoditization of Computing Technology is the Driver

“There is no reason anyone

would want a computer in

their home.”

Ken Olson, CEO of DEC, 1977

“…by 2008 we will be producing

one billion transistors for every

man, woman and child on earth”

Semiconductor Industry Association, 2007

How significant is the computing improvement?

1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000

Data: Ray Kurzweil, 2001

10,000 X improvement

DW architecture and

methods start here in

the mid 80s Term “BI” coined

Mechanical Relay Vacuum tube Transistor Integrated circuit

Don’t worry about performance, just buy more hardware

10,000 X performance

improvement, soon 100K

There are always limits

“If the automobile had followed

the same development as the

computer, a Rolls-Royce would

today cost $100, get a million

miles per gallon, and explode

once a year killing everyone

inside.”

Robert Cringely

Anything

Reality

RIP Moore’s Law. Data is growing faster than compute. This forces an architectural shift.

We’ve reached another generational technology shift

1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000

Mechanical Relay Vacuum tube Transistor Integrated circuit

Data: Ray Kurzweil, 2001

Multicore and networked

parallelism is the next wave

What’s different?

Parallelism

We’re not getting more CPU power, but more CPUs.

There are too many CPUs relative to other resources, creating an imbalance in hardware platforms.

Most software is designed for a single worker, not high degrees of parallelism and won’t scale well.

Reality: you must assume distributed architecture

Why by default? Because the upgrade between single node and distributed is a major change to designs. It carries new component linkages and complexities. A new ecosystem.

The future holds cloud provisioning, software-defined environments and a lot less single-server provisioning.

Copyright Third Nature, Inc.

(a) Scaling up with a larger server (b) Scaling out with many small servers (aka “++ungood)” (aka “the future”)

The future is already here, it just isn’t evenly distributed yet.

An important cloud computing benefit

Scalability is free (if you have the right software)

If your task requires 10 units of work, you can decide when you want the results:

10 servers, 1 unit of time

Cost is the same. Not true of the conventional IT model

1 server, 10 units of time

We are in a transitional phase in IT architecture

Then State of Practice Now, forward

Architecture Timeshare Client/server Cloud

Data Core TXs All TXs, some events, docs

All data

Rate of change Slow Rapid Continuous

Uses Few Many Everything

Latency Daily+++ < daily to minutes

Immediate

Data platform Uniprocessor SMP, cluster Shared nothing

Majority use of computing over time

1930s-1950s: Calculate

1960s-1980s: Automate

1990s-2010s: Informate

2010s+: Analyze and Actuate

Computing technology has become a tool of observation

lexity

Evolution of data

50s-60s: data as product

70s-80s: data as byproduct

90s-00s: data as asset

2010s +: data as substrate

The real data revolution is in business structure and processes and how they use information.

Not hype: another round of infrastructure change

Mainframe c/s cloud

Batch online event driven

Infrastructure takes a long time.

Value is driven by new capabilities used to do new things, less by doing old things better or cheaper

Disruption Time

The Internet forced a new architectural evolution.

IT has had a hard time keeping up, and new entrants in many markets are taking advantage of the new architecture to change how IT work is done.

Any time you have a backlog of resisted innovations, the pressure will eventually force wholesale change.

UNDERSTANDING INNOVATION AND COMMODITIZATION PROCESSES*

This is how things change.

An amalgam of Everett Rogers, Yochai Benkler, Geoffrey Moore, Clayton Christensen, Stephen Gould, Eric von Hippel and others.

aka I like S curves

*the very short version

COMMODITIZATION

Four phases of technology commoditization

Early adopters

show results

Market

growth

Innovation

Development

Maturation

Saturation

Early mainstream

starts to pay attention

Mainstream buy-in

This axis could be considered market penetration, adoption, product maturity

Invention /

discovery

Characteristics of software as it evolves

Innovation

Unique

Custom built

High value

High cost

Differentiator

Not well understood

High rate of change

Few vendors

Oddity known

Focus on integration not

building, customizable

products

High value

Lowered costs or

cost/speed/fit tradeoffs

Point of competition

Better understood

Slowed rate of change

Growing then shrinking

vendor count

Maturation

Ubiquitous

Configurable product

High to low value*

Low cost

Barrier to entry

Purpose and limitations well

understood

Negligible rate of change

Few, large vendors

Saturation

Compete on

differentiating

Vendor strategies (in general) vary by phase

Maturation

Compete on

product &

features

Compete on

process

Saturation Innovation

Market

growth

Should you be a first mover or fast follower?

Little product

substitution is

possible here.

Few competitive

bids or RFPs.

Maturation

Uncertain

tradeoffs here.

Competitive

bids for unlike

products. Early

it’s less “what

feature” and

more “how to

accomplish my

task”, later it’s

the opposite.

Predictable

cost and

feature

comparison

until practices

change. That

change can

take a long

time to occur.

Market

growth

Few product

choices

The vendor landscape changes over time

Many, expanding

product choices

Many, contracting

product choices

Relatively few

product choices

Market

growth A particularly

dangerous time

to pick vendors

We see this pattern in evolutionary processes

etitio

ts Evolution in most complex

systems goes through periods

of rapid change followed by

periods of general stability,

referred to as “punctuated

equilibrium”

Invertebrates

Vertebrates

Bacteria

Insects

Innovation – Adaptive radiation – Selection – Convergence

3.5b Bacteria (Cell)

2.5b Sponge (Body)

0.7b Clams (Nerves)

0.5b Trilobites (Brains)

0.1b Mammals

Timing

The same model can be applied to technology

etitio

Innovation – Adaptive radiation – Selection – Convergence

With a long view a pattern emerges

Evolution in most complex systems goes through periods of rapid change followed by periods of general stability, referred to as “punctuated equilibrium”

New technologies take the place of old, establishing new ecosystems which are in turn disrupted by newer technologies and ecosystems.

Chaotic Stable Chaotic Stable Chaotic

Activities, products and practices evolve over time

Source: Simon Wardley

Technology doesn’t just fulfill a need. It generates new needs and new problems. Business practices and technology co-evolve.

As practices evolve based on new capabilities…

A new level of complexity develops over top of the older, now better understood processes, leading to new needs.

Evolution and the Salaman-Story Paradox

”Survival requires efficient exploration of current competencies and ‘coherence, coordination and stability’; whereas innovation requires discovery and development of new competencies and this requires the loosening and replacement of these erstwhile virtues”

As a technology moves from emerging to commodity the nature of acquiring, using and managing it should change

Generate

options

Innovation

Novel practice

Maximize value

Maturation

Standardize /

minimize choice

Acquisition

Best practice

Minimize costs

e.g. BI which went from many tools to a few vendors, now being

disrupted by new technologies and capabilities

Constrain

choices

Adaptation

Good practice

Optimize

In terms of technology, we are in a tough position because the ecosystem is in a disjoint state

Maturation Saturation Innovation

Big data and

analytics is here

BI / DW is here

ADOPTION: ENOUGH ABOUT THE ADOPTEES, WHAT ABOUT ADOPTERS?

The Enterprise IT Adoption Cycle

Wardley IT adoption reality

The Incredible Rate of Technology Change

Big data?

The Incredible Rate of Technology Change

We told you about

it in 2004…

Adoption

Some Innovation Adoption Theory

End of Life New innovation Time

Adoption

End of Life New innovation

Adopter Categories

Innovators Late

Majority

Adopters

Laggards

Ability to adopt is governed by people & organizations

Innovators Late

Majority

Adopters

Late adopters

People here tend to view a technology as a means to capitalize on future opportunities*

e.g. big new projects, process change

People here tend to view technology as a means to resolve present problems.

e.g. more focused projects, process improvement

*adopter status is based on the person/org and a given technology, it’s not a blanket statement

Slowing it down: innovation is gated by ability to adopt

No technology stands

entirely alone – these

dependencies slow

adoption, stretching

the maturation phase.

This ecosystem effect

is what creates

technology regimes

that can last decades.

Younger companies

have a relative

advantage when it

comes to absorbing

new infrastructure.

Cumulative

Adoption

Market Adoption

Hard work

Tipping point

Product

Maturity

Some Ideas Aren’t That Good

End of Life Time New innovation

Some ideas aren’t that

good, like object

databases in the 1990s

These Curves Can Explain a Lot

Product

Maturity

Analyst revenue

predictions

Executive interest “Gartner Gap”

The “experts” often have a foreshortened view

“Open source is not worth paying attention to.”

A Gartner analyst talking about the database and analytics market, January, 2006.

Where the analysts are on the

adoption curve

Crossing the Chasm (1991)

Geoffrey Moore’s Ideas

Built on Rogers’ ideas, extended them to tech marketing and product management. The original focus was on the development of technology (gray).

Just say no

Stick with the proven

Stick with the herd

Stay ahead of

the herd

Just try it

Core BI / DW technology is mainstream-stable

The data management market has many segments, some new, some mature, some being rejuvenated.

Platforms (this

should scare

everyone)

Databases* Reporting

ETL and DI

Analytics

Product evolution in software markets

4 5 6 Image: Geoffrey Moore, Dealing With Darwin”

INNOVATION

Innovation and Commoditization

This section isn’t really a summary

Image: Harvard Business Review, “Skate to Where the Money Will Be”

Theory of Disruptive Innovation

i.e. you don’t pay attention and do

what you always did and the other guy

eats your market from below

Disproving Christensen

a) 9% of the cases fit the model

b) Disruptive innovation <> success; banks disruptively innovated debt products and we know how that turned out

c) The model fails to predict failure too:

In 2007, Christensen told Business Week that “the prediction of the theory would be that Apple won’t succeed with the iPhone,” adding, “History speaks pretty loudly on that.” In its first five years, the iPhone generated a hundred and fifty billion dollars of revenue. In the preface to the 2011 edition of “The Innovator’s Dilemma,” Christensen reports that, since the book’s publication, in 1997, “the theory of disruption continues to yield predictions that are quite accurate.”

Types of innovation

Incremental or “sustaining”

▪ Incremental is based on existing concepts, framing; smaller changes within the same general framework

Disruptive

▪ Based on new concepts, science, principles; requires new knowledge, skills; over time has significant consequences to market

Architectural – the third path

▪ Changes how the parts are related. It devalues advantage of experience, knowledge, usefulness of prior knowledge, but doesn’t affect the existing knowledge. (Christensen missed this one)

Adoption and decline – everything gets old

For most businesses, nearly 80% of IT budget is dedicated to basic infrastructure

…and more than 60% of IT labor cost goes to keep things running, i.e. basic operations and support.

Strategic

Commodity

It Wasn’t Always This Way

As technologies mature and spread to competitors, they cease to be differentiators. Unfortunately, this is what packaged software vendors do to your “best practice.”

Commodity Commodity

The old advantages becomes the new focus of cost reduction.

For example, your data warehouse.

Strategic Strategic

Adoption and decline

Rarely does anyone talk about the core problem: preexisting conditions

You have something new. How does it affect the old?

▪ Replaces it?

▪ Adds something new?

▪ Overlaps it, forcing you to make hard decisions about what parts to keep, change, throw away?

The heart of this problem is the process of architecture: integrating changes to systems over time. The integration is not purely technical, it’s practices of use, operation, deployment.

Most data tech is a commodity, a cost of doing business

Adopting new things: there’s a problem with your budget

How IT strategies evolved with commoditization

Equipment

Expensive: outsource to reduce equipment cost

Affordable: insource for control, innovation

Dirt cheap: outsource to reduce labor cost

The cost flip in the business intelligence world

Cost factors traded positions 1990 - 2010

Equipment

Software

For small to mid-sized organizations it’s very affordable

TCO and BI

What can you control?

▪ Labor effort is almost identical across BI products.

▪ Hardware use by BI tools is similar across products.

▪ You can negotiate the software costs.

3 Year BI TCO Cost Categories

Source: Third Nature Open Source cost study

BI Market: Cost is normally driven out by commodities, not increased

This is an old problem

BI tools are better, but the model being applied in most organizations is not different from the past.

November 2010 Mark Madsen

If BI is a commodity, why does it cost so much?

Processes Applications Data Integration Storage EDM / BRM Delivery Consumers

Purchasing

Distribution

Manufacturing

Sales &

Service

ERP Data warehouse

Stream db / cache

Content store

Identify

Analyze

Debt<10% of Income Debt=0%

Credit

YesYes

Income>$40K

Predict

Batch ETL

CRM ESB

Monitor

Explore

Data mart

Low-lat ETL

Prescribe

Data services

Transaction services

Manual feedback

Automated feedback

Lessons to take from this

1. Business intelligence is still expensive for many organizations, with the largest proportion of cost being labor.

2. Business intelligences is not a technology problem, or the failure rate and costs wouldn’t be so high.

3. BI tools being a commodity does not make BI a commodity.

4. Architecture has an outsized impact on your ability to adopt and adapt.

5. What you remove is as important as what you add.

WHAT CAN YOU DO KNOWING HOW THE MARKET EVOLVES?

Questions to ask

Why innovate?

▪ Usual answer: profit

▪ Proper answer: solve a problem

Innovation for what?

▪ A product or service you are selling to customers

▪ Internal products and services, how you run your business or department

Reinforcing relationships resist change, despite radical technology and practice shifts

Note how only one third is tech

ArchitecturalRegime

Methodology Technology

Organization

Organization defines where the work is done and the roles.

Technology defines what work can be done in a given area. Methodology

defines how work is done and what that work is.

Designing for data: monolithic vendor technology-based classifications of the ecosystem won’t help

These types of eye charts provide a categorization of what’s available, not what you need. They ignore the contexts of use that are most important.

It’s tough making it decisions in a turbulent market

If you’re here you probably

don’t want to be making

long term technology or

vendor commitments.

Today: repeating the experience of the 80s & 90s

This is the turbulent

phase of the market

as it goes through

rapid development,

then product and

service changes.

The Internet combined with commodity computing is forcing a new

architectural evolution, already well underway.

Rule of thumb: when a product is in phase…

Market

growth

Build Integrate Buy

Methods change too, one size doesn’t fit all

Agile &

exploratory

methods

6 Sigma &

efficiency

methods

How procurement decisions are made

Deliberation ▪ Actions are consciously

chosen. Don’t attribute to malice what you can attribute to stupidity, and don’t attribute to stupidity what you can attribute to laziness.

Rationality ▪ People make logical

decisions. Sure they do.

Order ▪ System are understandable

and the results of actions predictable.

90% of

EVERYTHING

is crap

“Sturgeon’s Revelation”

“Choose Boring Technology”

You only get so many chances to make big changes at a company. Don’t waste them.

You can spend X time focusing on the goal and worry less about the known tech, or you can spend X time learning the new tech and less time focusing on the goal.

The important thing is not the choice of tech, it’s knowing when the time is right to make a new tech choice.

Beware unintended consequences

Unintended consequences

In other words… Software is like puppies. Getting a puppy is easy, raising one is hard.

“The short term benefits of using a new [type of] database exceed the long term cost of operating it.”

Dan Mckinley

Where does innovation come from?

“It has long been assumed that product innovations are typically developed by product manufacturers. …it now appears that this basic assumption is often wrong.”

Eric von Hippel

How to find “innovative” solutions

N.W.A. Answer: steal them from somewhere else.

Being innovative and culture

Myth of process – there is no “process for innovation”, only principles and exceptions

e.g. “It’s best to work in small teams, keep them crowded and foster serendipitious connections.” – Eric Schmidt

It depends on creative problem solving, and solving problems people care about.

Removing deviance removes change, so you have to be careful about best practices.

No silver bullet

It’s culture-dependent, and creative and messy and idiosyncratic and slow and hard, no process, just survival bias and heuristics and principles.

“The future, according to some scientists, will be exactly like the past, only far more expensive.” ~ John Sladek

Disruptive Innovation: how do you use these theories to manage your IT?

Technology

Edelweiss DISRUPTIVE India Conference 2016 INNOVATIONSbudget.edelweissfin.com › report › India_Post... · Pantaloon growth plan: FY14-Manage the transition, FY15–Lay the foundation,

Investing in innovative solutions to manage cascading ... · communities are exposed and vulnerable to both. For example, cyclone and flood shelters are ... disruptive technologies

Disruptive docks

Disruptive Technologies: An Analysis of Disruptive Innovations

From Disruptive Technology to Disruptive Business Model Innovation

Reducing Disruptive Student Verbalizations through the Use of … · 2019-04-30 · 2 reinforcement. A third strategy that teachers can use to prevent and manage disruptive behavior

IDENTIFYING DISRUPTIVE TECHNOLOGIES FACING THE · PDF fileIDENTIFYING DISRUPTIVE TECHNOLOGIES FACING THE ... Disruptive Technology, Disruptive Innovation, Military Disruptive Technology,

Disruptive Collaboration

Disruptive Docks

What Is Disruptive Innovation? - United Technologies is disruptive innovation.pdf · DISRUPTIVE INNOVATION What Is Disruptive Innovation? by Clayton M. Christensen, Michael E. Raynor,

Disruptive technology

Disruptive change

Disruptive Education

Best Practices to Manage Disruptive Technologies - PMI New … · 2019-02-27 · Best Practices to Manage Disruptive Technologies. Best Practices to Manage Disruptive Technologies

DISRUPTIVE INNOVATION What Is Disruptive Innovation?

Disruptive Technology: Using Robots as a Service Disruptive Technology: Using Robots as a Service Mike Oitzman, CEO/Founder, Market Spec Group ... Strategy Budget Manage Robotic Workforce

Dealing with Disruptive Patients - Connecticut Academy of ... · Dealing with Disruptive Patients What is Disruptive behavior Types of Disruptive Patients Why are they upset? Patient

Disruptive perception

DISRUPTIVE TECHNOLOGIES - Vibrant Gujaratvibrantgujarat.com/.../50-Disruptive-Technologies.pdf · DISRUPTIVE TECHNOLOGIES A Disruptive Technology is a technology or innovation, which

PROGRAM CATALOG 2014 - 2015 · competencies that are necessary to create and manage business. Entrepreneurial activities involve disruptive innovation and discovery-oriented creativity