Machine Data Intelligence (MDI) Capability Maturity Model · • Workflow and processes are in place to enable consistent and uniform monitoring of metrics. • There is a centralized

Machine Data Intelligence (MDI) Capability Maturity Model

A multi-level framework for achieving advanced machine data intelligence at scale

2 • Machine Data Intelligence (MDI) Capability Maturity Model

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

MDI Capability Maturity Model Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

What’s Your Business’s MDI Capability Maturity Stage? . . . . . . . . . . . . . . . . . 6

Level 1: Basic Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Level 2: Advanced Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Level 3: Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

Level 4: Innovation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

Level 5: Monetization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25

Contents


OverviewIt is estimated that within the next ten years there will be a trillion connected computers

and devices on the planet. You have many, many of these machines in your own enterprise

— some you are aware of, probably some that you aren’t. These machines range from

traditional servers in data centers and cloud infrastructure to modern day IoT sensors and

devices. They power everything from conventional computing to smart medical devices and

autonomous vehicles.

These machines produce an unfathomable amount of data — technically “telemetry” or

measurement data — that is growing so exponentially, there are no reliable estimates on

just how much is being produced. Now imagine the wealth of insights and potential business

value contained in that data if it could be harnessed — at scale and in real-time. It is data

so dense and rich with potential value, it dwarfs traditional business intelligence as the next

frontier of competitive advantage.


It’s called “machine data intelligence (MDI).” Every aspect of our lives and our world is

becoming rapidly digitized and computerized. We’ve deployed machines to do things for

us but what can they now tell us as a result? As it turns out, quite a lot. Machine data can

be used certainly for monitoring of availability and performance of IT infrastructure, but

also for optimization of devices and operations, predictive analytics and maintenance, and

innovating entirely new products and services that can be monetized.

So how do you get started? Where do you start? This eBook was developed to help

enterprises develop a strategy and a game plan to implement an MDI program and begin

to reap its benefits. It is based on a capability maturity model not unlike maturity models

for software development. It consists of five easy-to-understand levels of maturity and

practical suggestions on how to move from one level of maturity to the next. Like any good

framework, you can plot where your enterprise fits today, determine where you’d like to be,

and then develop a plan to close the gap. You may find yourself at the earliest stages of this

model. Don’t despair. You are not alone, but the competitive landscape is moving quickly.

The Circonus MDI Capability Maturity Model is a multi-level framework that helps you to

identify the current situation of your enterprise and next steps to take, including:

• What should be improved at the stage you’re currently in

• Capabilities needed to master a level

• What your MDI capability maturity should look like upon completion of a level

• The must-haves for moving to the next level


MDI Capability Maturity Model Chart

M O N E T I Z A T I O N

I N N O V A T I O N

O P T I M I Z A T I O N

A D V A N C E D M O N I T O R I N G

B A S I C M O N I T O R I N GL E V E L 1

L E V E L 2

L E V E L 3

L E V E L 4

L E V E L 5

MDI C A P A B I L I T Y M A T U R I T Y M O D E L C H A R T


What’s Your Business’s MDI Capability Maturity Stage?

Level 1: Basic Monitoring

In level 1, there is no overarching MDI strategy for the enterprise. It is likely that a complete

inventory of all machines and associated metric streams does not exist. There is some

monitoring of IT infrastructure, but there is a lack of standards and consistent processes.

No attempt is made to harness machine data for more advanced uses like predictive

maintenance. MDI is not a strategic objective for the company and does not have visibility

in the C-suite.

Level 2: Advanced Monitoring

In level 2, enterprises have invested heavily in monitoring tooling, but they are using multiple

tools and platforms — resulting in disparate data sources, no single source of truth, and the

inability to aggregate data easily across the enterprise. There is no centralized platform for

the collection, storage, and subsequent analysis of machine data. Awareness of, and interest

in, machine data intelligence is rising but there is not yet a corporate objective to build a

comprehensive MDI strategy.

Level 3: Optimization

In level 3, the enterprise has consolidated and rationalized its monitoring and data collection

capabilities across the enterprise and has built a solid foundation on which to begin deriving

additional value from machine data. At this stage, the enterprise is using machine data to

optimize device performance, implement predictive analytics and maintenance programs,

and optimize company operations. The enterprise begins to move from basic monitoring

into optimization of product and service delivery, and MDI is becoming a topic of discussion

in the C-suite.


Level 4: Innovation

In level 4, the enterprise begins to move into true intelligence mode. It’s optimizing

operations and product and service delivery and, as a result, realizing significant efficiencies

and impact to its bottom line. At this stage there is a bedrock foundation of rich data, and

multiple business units are using a sophisticated data science platform to innovate new

products and services. Entirely new potential revenue streams such as packaged information

products are beginning to be identified. A company champion emerges and work begins on

an overarching MDI strategy.

Level 5: Monetization

In level 5, there is a cohesive and comprehensive MDI strategy in place for the entire

enterprise. It is a strategic imperative of the company with executive sponsorship at the CEO

level. At this level, all machines and metric streams have not only been identified but also

mapped to strategic corporate initiatives. All business units are leveraging machine data to

drive innovation in products and services, and there are frequently new ideas for monetizing

machine data and creating entirely new revenue streams. MDI has become a strategic

competitive advantage and is driving both top line revenue as well as bottom line profitability.


Level 1: Basic MonitoringAt this level, some monitoring is likely to be occurring but conducted in an inconsistent and

patchwork way, typically according to the preference of individual staff and developers.

There is a lack of a defined and communicated organizational mandate outlining what and

how to monitor across the enterprise’s IT organization, and what the results of a monitoring

effort should be. Consequently the business is unable to monitor their systems and services

in any uniform manner which prevents them from surfacing value from those efforts.


I N N O V A T I O N




L E V E L 2

L E V E L 3

L E V E L 4

L E V E L 5


What this level looks like

• No defined standards on what and how to monitor across the enterprise’s IT organization.

• No inventory of systems, services, or custom applications.

• Monitoring is done inconsistently and according to individual staff and developer preference.

• Disparate tools are being leveraged by different functions within the organization.

• Systems and services are not being monitored in a uniform manner, providing little to

no value.

Capabilities required

• Technical expertise to create an inventory plan and permissions to execute upon that plan.

• Technical expertise to add monitoring for existing and future infrastructure and core services.

• Creation or identification of tooling or libraries through which to solicit or submit

application monitoring metrics.


Checklist to move to next level

Ú Identify a platform through which to consolidate your monitoring and subsequent

analytics efforts.

Ú Create a comprehensive inventory of existing infrastructure, services, and applications.

Ú Create a plan through which the business identifies and defines the specific metrics and

KPIs it expects to collect from its infrastructure, services, and applications.

Ú Establish observability standards for all physical, virtual, cloud, and container

infrastructure, along with any services and custom applications.

The completion of this level’s requirements leaves the business with a well-defined “monitoring

plan” of the assets to monitor, why to monitor a specific asset, and how specifically to monitor it.


Level 2: Advanced MonitoringWhile much of the effort from the previous level was done manually, the advanced

monitoring stage is where everything is automatically and uniformly monitored. There

is a single, consolidated monitoring platform in place for the purposes of infrastructure,

service, and application monitoring. A defined and communicated set of standards that

outline the monitoring of infrastructure, services, and applications is in place. There

is a comprehensive inventory of infrastructure, services, and applications and all are

monitored according to standards.


I N N O V A T I O N




L E V E L 2

L E V E L 3

L E V E L 4

L E V E L 5



• Business has a monitoring plan with a well-defined set of standards on the monitoring of

infrastructure, services, and applications.

• There is a comprehensive inventory of infrastructure, services, and applications.

• There is a centralized platform for the collection, storage, and subsequent analysis of metrics.

• Monitoring is consolidated into the centralized monitoring platform and disparate,

legacy monitoring tools are removed.

• Inventoried assets are monitored in accordance with the monitoring plan.


• A centralized, performant platform through which to consolidate the business’s

monitoring efforts. Specifically a platform that can monitor physical/virtual servers, cloud

services, containers, commonly encountered services, and custom application metrics.

• Fully automated monitoring for all provisioned infrastructure and services.

• Technical expertise to incrementally retrofit existing applications to emit metrics through

agreed upon tooling or libraries.

• A DevOps function to ensure that all freshly provisioned infrastructure and services are

monitored by default, while also providing enforcement of agreed upon monitoring

standards for custom applications.



Ú Conduct staff training on the usage of the identified centralized monitoring platform

and overall capabilities.

Ú Create a set of standards and guidelines for DevOps within the business, and create a

plan through which freshly provisioned infrastructure, services, and applications are

automatically monitored.

Ú Enforce standards and guidelines for software development within the business to

ensure consistent and comparable metrics.

The conclusion of this level effectively guarantees comprehensive, uniform, and cost-effective

monitoring even in the event of staff churn, hypergrowth, etc.


Level 3: OptimizationAt the optimization level, businesses begin to use and leverage the data that's been collected

to identify new opportunities for improvement. Workflows are created to push the resulting

intelligence to the necessary roles within the business to ensure proper budgeting and

execution. The key to capitalizing on optimization opportunities is the ability to capture

all the data at the speed and frequency it is being generated and the ability to retain that

resolution of data without limit. The greater the density and richness of the data, the greater

the accuracy, precision, confidence, predictive qualities, and insights.


I N N O V A T I O N




L E V E L 2

L E V E L 3

L E V E L 4

L E V E L 5



• Workflow and processes are in place to enable consistent and uniform monitoring of metrics.

• There is a centralized platform for the collection, storage, and subsequent analysis of metrics.

• High volume, high frequency metrics are collected across a wide spectrum of metrics

streams or sources.

• Analysis of the collected metrics is conducted to inform efficiency and

operational improvements.

• Workflows that ensure the resulting intelligence are shared within the organization for

proper budgeting and execution.


• Staff with the necessary skills and time to extract actionable insights from the

centralized intelligence platform.

• Processes and workflows in place to facilitate the identification of opportunities for

operational and efficiency improvements.

• Processes and workflows in place to communicate findings up to managers

and executives.

• Additional platform capabilities, including real-time alerting, anomaly and fault

detection, outlier detection, and trend analysis.

• SLO/SLA monitoring and altering.



Ú Analyze collected metrics and draw conclusions about existing workflows and processes.

Ú Facilitate platform training for engineers and data science teams to broadly leverage

the data moving forward.

Ú Develop processes and workflows to facilitate the identification of opportunities for

operational and efficiency improvements:

ȩ Create reports that show what has happened previously, and what’s happening now.

ȩ Quantify the impact of software improvements, as well as outages, etc.

ȩ Generate predictive analytics.

Ú Develop processes and workflows to communicate findings to managers and executives.

Ú Identify executive sponsorship and a company champion.

Completion of this level results in a business now able to effectively leverage the collected

data. Concrete opportunities for improvement are identified and workflows are created to

push this information around the organization to those who need it.


Real-World ExampleA global oilfield services provider uses machine data intelligence to

drive better decision-making and optimize well production.

A global oilfield services provider is collecting high frequency analytics from

sensors inside wells to measure hydraulic pressure and determine if in-fill wells

(child wells) will compromise the productivity of parent wells. This allows the

service provider to manage well interference while maximizing the density of well

spacing and minimizing the risk of damage from frac hits.

Frac hits occur when the hydraulic pressure applied in a child well impacts the

operations of the parent well. They can be very expensive in terms of damages

to production tubing, casing, and well heads, but they can also result in a

substantial reduction in parent well production, causing economic losses in the

millions of dollars.

Ultimately, machine data intelligence gives the organization extreme clarity into

its well operations so that it can improve profitability and reduce costs.


Level 4: InnovationAt the innovation stage, enterprises move into greater intelligence related activities by

employing advanced analytics and data science. Tapping into the full value of machine data

can be transformational, delivering real and measurable results. The key to unlocking this

enormous potential is the ability to harness and make sense of the wealth of machine data

already being generated in the enterprise. The ability to gather and analyze vast amounts

of machine generated data, including data from IT infrastructure, sensors, systems, and

connected devices, achieves new levels of insight that drive smarter operations, better

decision-making, and new business opportunities.


I N N O V A T I O N




L E V E L 2

L E V E L 3

L E V E L 4

L E V E L 5



• All data science teams in all departments and functions have access to data assets and

advanced analytics capabilities.

• The company’s inventory of data assets is mapped to strategic company objectives to

identify potential opportunities.

• Opportunities are ranked by completeness and density of data assets and highest

priority objectives.

• Ideation techniques are employed to foster creative thinking.

• Time and money budgets are allocated appropriately.


• Performant analytics query language with high fidelity data science functions.

• High frequency, real-time streaming analytics.

• Anomaly and fault detection.

• Outlier detection and trend analysis.

• Historical analytics.



Ú Executive sponsorship and buy-in at the C-suite level.

Completion of this level results in a business that is in command of all of its machine

generated data and quickly moving to the forefront among its industry peers. The business

is now not just finding new sources of competitive advantage but completely reinventing the

ways services and products are delivered and has become a disruptor in its industry.


Real World ExampleAn energy tech company uses machine data intelligence to help

buildings become more efficient and reduce operational costs.

An energy tech company uses high frequency telemetry data pulled off of

thousands of sensors within a building to improve efficiency and lower costs.

Data collected and analyzed include electrical usage, temperature, and human

movement by floor, as well as power utilization by outlet and appliance.

Insights as to when energy can be dialed back or when it’s cost-efficient to

replace appliances help “smart buildings” cut costs, reduce maintenance time,

and improve overall functioning of the building. This type of rich information is

also playing a major role in risk mitigation and driving entirely new approaches

to underwriting in the insurance industry.


Level 5: MonetizationAt this level, a business now has the requisite data stored within a platform, methods for

accessing and leveraging that data, staff to perform the necessary work, and processes

and workflows in place to communicate conclusions to the necessary staff within the

business. Businesses at this level are able to collect and ingest incredibly high volume and

high frequency data. They can retain, find, and quickly retrieve data, as well as execute

sophisticated and complex real-time, historical, and predictive analytics against that

data. Machine data can be mined at will, without compromise or constraints — unlocking

unprecedented insights and value creation opportunities.


I N N O V A T I O N




L E V E L 2

L E V E L 3

L E V E L 4

L E V E L 5



• Businesses reap the rewards of having a centralized intelligence platform with full

visibility into existing and future operations.

• Infrastructure, services, and software are constantly monitored and improved as needed.

• Processes and workflows are developed and implemented to inform leadership about

opportunities being identified throughout the company.

• Machine data intelligence has become a source of sustained competitive advantage.


• Staff with the necessary skills and time to extract actionable insights from the

intelligence platform.

• Processes and workflows in place to facilitate the identification of opportunities for

operational and efficiency improvements.

• Processes and workflows in place to communicate findings up to managers and executives.

• Performant analytics query language with high fidelity data science functions.

• High frequency, real-time streaming analytics.

• Anomaly and fault detection, outlier detection and trend analysis, and historical analytics.

This final step is not meant to be the end of the journey. Once a business achieves this level

of sophistication, it must ensure all accomplishments in prior steps remain in place, while

continuing to maintain a constant focus on data-driven improvement of processes and

workflows throughout the business.


Real-World Example A professional sports league uses machine data intelligence to

automate monitoring, optimize viewing performance, and

generate revenue.

This organization has completely automated the monitoring of its entire

infrastructure, including both its cloud and physical environments. New

infrastructure is automatically monitored as it is spun up, and the league is

collecting detailed metrics at the application layer.

This allows the league to monitor the service levels across all of its APIs. The

ability to keep track and meet the SLAs of all their clients, who depend on

the ability to provide optimal viewing experiences for their over 100 million

consumers, has a substantial direct effect on the organization’s bottom line.


SummaryThe possibilities of machine data intelligence, across all sectors including healthcare,

insurance, utilities, ad tech, manufacturing, and many others, are boundless — limited

only by our creativity and imagination. As the number of things we want to monitor and

measure grows, and sensors proliferate our world, we can only imagine what we will find

and what we can create. Ultimately, however, we know that being smarter means making

better decisions. It means that our decisions are driven by data that we can rely on for its

accuracy and timeliness. It means that we have new knowledge that can drive competitive

advantage and change the trajectory of our products, our operations, our IT infrastructure,

and our business performance.

This is the “Internet of Everything” economy. The companies that learn to harness machine

data to optimize operations, innovate new products and services, and create entirely new

revenue streams will be the clear winners.


Circonus is the machine data intelligence expert, providing the only machine data

intelligence platform capable of handling billions of metric streams in real time to drive

unprecedented business insight and value. Led by experts in large-scale distributed systems

and data science, Circonus is pioneering the way that machine data at scale is leveraged

throughout the enterprise, from operational analytics to IoT applications.

To learn more, visit http://www.circonus.com

Contact us today to learn how your organization can realize the full power

of machine data intelligence .

CONTACT US LONG-ARROW-RIGHT

http://www.circonus.com

https://www.circonus.com/contact-us/

Documents

Machine Data Intelligence (MDI) Capability Maturity Model · • Workflow and processes are in place to enable consistent and uniform monitoring of metrics. • There is a centralized