Big Data Impacts on Hybrid Infrastructure and Management

Shamus McGillicuddy

Senior Analyst

[email protected]

@shamusEMA

Big Data Impacts on Hybrid Infrastructure and Management

April 18, 2023

mailto:[email protected]

Today’s Presenters

Slide 2

Shamus McGillicuddy, Senior Analyst, EMA Shamus has more than nine years of experience in the IT industry, primarily as a journalist covering the network infrastructure market. At Enterprise Management Associates (EMA), he is the senior analyst for the network management practice.

Prior to joining EMA, Shamus was the news director for TechTarget's networking publications. He led the news team's coverage of all networking topics, from the infrastructure layer to the management layer. He has published hundreds of articles about the technology and competitive positioning of networking products and vendors. He was a founding editor of TechTarget's website SearchSDN.com, a leading resource for technical information and news on the software-defined networking industry.

Slide 3

Logistics for Today’s Webinar

An archived version of the event recording will be available at www.enterprisemanagement.com

• Log questions in the Q&A panel located on the lower right corner of your screen

• Questions will be addressed during the Q&A session of the event

Questions

Event recording

http://www.enterprisemanagement.com/

Shamus McGillicuddy

Senior Analyst

[email protected]

@shamusEMA

Big Data Impacts on Hybrid Infrastructure and Management

April 18, 2023

mailto:[email protected]

Research Sponsor

Slide 5 © 2015 Enterprise Management Associates, Inc.

Research Goals

• Investigate the impacts that big data collection and analysis have on infrastructure performance and behavior.• Determine whether these infrastructure

impacts affect engineering and operational practices

• Examine how IT organizations make big data work for them• Identify the infrastructure monitoring data IT

organizations export to big data environments• Determine how analysis of that data supports

IT planning, monitoring and troubleshooting • Identify the key organizational benefits

experienced by successful users of these techniques


Research Demographics

• 156 participants• Mix of IT professionals and IT directors/executives

• 43% executives and 57% technical staff

• Technology savvy professionals• Strong IT/IS/Network focus: 69%• The rest are telecommunications, engineering/R&D, technical customer

support, quality assurance and general management

• North American focus (97%)• Broad distribution of company size

27% small enterprise (1-999 employees) 35% medium enterprise (1,000-4,999) 38% large enterprise (5,000 and up)

• Two statically significant vertical industries 16% Finance/Banking/Insurance 15% Manufacturing (excluding computer hardware)


Big Data Maturity Perspective

• Research participants identified the number of big data projects they have in production• Reveals levels of experience and maturity organizations have

with big data• Does an organization's approach to big data analytics for IT

change as they gain experience?

• Classifying organizational experience by number of big data projects• 29% Beginners: 1 or 2 big data projects in production• 38% Intermediates: 3 to 5 projects• 33% Advanced: 6 or more projects


Big Data Impacts on Infrastructure


Increased network traffic load due to big data collection

Increased network traffic load for backup of big data repositories

Increased compute demands for big data processing/analytics

Increased storage needs for big data stores

Increased use of cloud services for big data storage

Increased use of cloud services for big data analytics

No changes observed

45%

46%

42%

58%

42%

30%

13%

Have you witnessed any changes in the behavior or performance of your orgs IT infrastructure due to any types of big data projects?

Sample Size = 156, Valid Cases = 156

Big Data Impacts on Infrastructure

• Beginner big data users feel fewer infrastructure impacts• 35% of beginners report no infrastructure impacts• But that won’t last

94% of advanced firms observe infrastructure impacts

• Beginners typically only experience increased storage demands (46%)• Other tech domains impacted only 20% to 26% of the time

• Advanced firms feel infrastructure impacts everywhere• 63% increased storage demands• 61% increased network traffic from big data backups• 55% increased network traffic from big data collection• 55% increased use of cloud services for big data storage• 53% increased compute demands for big data analytics


Big Data Impacts on Infrastructure: Vertical insights

• Manufacturers are leaning on the cloud for big data storage and analytics• 54% see increased use of cloud-based big data storage (vs. 42% of

general sample)• 46% see increased use of cloud-based big data analytics (vs. 30% of

general sample)• Manufacturers demonstrate an affinity for the cloud throughout this

research

• Finance uses cloud for big data storage (44%) but not for analytics (8%)


Big Data Impacts on Infrastructure Planning and Engineering


Datacenter systems managers

Database managers

Cloud managers

Security managers

Application developers

Storage managers

Network Managers

Other

None of the above

46%

52%

47%

47%

41%

45%

53%

1%

6%

Which teams have seen impact to infrastructure planning and design practices due to the presence of big data projects?


Big Data Impacts on Infrastructure Planning and Engineering


Datacenter systems managers

Database managers

Cloud managers

Security managers

Application developers

Storage managers

Network Managers

Other

None of the above

35%

39%

24%

39%

26%

43%

46%

0%

13%

46%

54%

47%

39%

44%

37%

47%

2%

3%

55%

61%

67%

65%

51%

55%

67%

0%

4%

Which teams have seen impact to infrastructure planning and design practices due to the presence of big data projects?

Advanced

Intermediate

Beginner


Big Data Impacts on Infrastructure Operations


Network ops

Security ops

Datacenter systems ops

Application support

Cloud ops

IT (cross-domain) ops

Other

None of the above

49%

49%

52%

37%

43%

49%

0%

4%

Which teams have seen significant impact to their daily ops practices due to big data projects?


All aspects of operations equally affected, except app support and cloud ops

Big Data Impacts on Infrastructure Operations


Network ops

Security ops

Datacenter systems ops

Application support

Cloud ops

IT (cross-domain) ops

Other

None of the above

46%

35%

37%

22%

35%

48%

0%

9%

42%

51%

49%

32%

36%

46%

0%

2%

61%

61%

69%

55%

59%

55%

0%

4%

Which teams have seen significant impact to their daily ops practices due to big data projects?

Advanced

Intermediate

Beginner


Advanced big data firms have ops issues across the board.Beginners concentrate on network and cross-domain operations.

Big Data Analytics for IT


Yes, currently using

No, but planning to use in next 12 months

No, but considering at some point

No use or plans

71%

18%

11%

0%

Are you currently using or planning to use big data systems and technologies to collect and/or analyze

IT infrastructure monitoring data?

Research participants: EMA sought enterprises that were considering, planning or currently applying big data analytics to IT infrastructure monitoring data.

Organizations with no plans to do so were excluded.

IT Monitoring Data Exported to Big Data Repositories


Time series performance data

Log entries

Flow records (NetFlow, sFlow, IPFIX)

Raw network packets

Interpreted packet flow/wire data

Application performance data

Transaction records

Cloud provider API metrics

Other

46%

41%

43%

30%

39%

59%

52%

50%

1%

Which types of IT data are you collecting and adding to a big data repository?


Exporting IT Monitoring Data to Big Data Repositories



Log entries


Raw network packets



Transaction records


Other

28%

28%

28%

10%

28%

69%

55%

34%

3%

47%

36%

45%

30%

32%

53%

47%

49%

0%

57%

54%

50%

41%

54%

59%

57%

61%

0%

Advanced

Intermediate

Beginner


Other IT monitoring data types becomes more popular as organizations gain big data experience.

Advanced users are exporting everything.

Advanced users find other data types more valuable; APM and transaction records lose relative importance



Log entries


Raw network packets



Transaction records


Other

24%

14%

21%

3%

10%

55%

45%

31%

3%

23%

13%

32%

15%

23%

43%

32%

32%

0%

37%

22%

22%

17%

22%

39%

35%

39%

0%

Which types of IT data are the three most important for big data analytics for IT?

Advanced

Intermediate

Beginner


What IT Management Practices are Supported by Big Data Analytics?


Internal infrastructure planning

Hybrid/Cloud infrastructure planning

Ops monitoring

Infrastructure security

Troubleshooting/diagnostics

Service quality analysis/improvement

Business activity insights

None of the above

50%

44%

56%

53%

37%

45%

35%

4%


The top 3:• Operations monitoring• Infrastructure security• Internal Infrastructure planning

What IT Management Practices are Supported by Big Data Analytics?

• A deeper look into these supported management practices• Troubleshooting becomes more important as

users gain experience with big data Nearly half of advanced users apply big data

analytics to troubleshooting and diagnostics

• Service quality analysis/improvement becomes a majority use case for advanced users (55%)

• The discovery of business activity insights (33%) is the the least supported management practice

Key enabler of IT-business alignment No variation with big data experience


Three Constituencies Benefit from Big Data Analytics for IT


IT Infrastructure Engineering

IT Infrastructure Ops

Architecture

DevOps

Service portfolio planning

Service Delivery

Other cross-domain service mgmt

Configuration mgmt

Change mgmt

Asset mgmt /Financial planning

Capacity planning

Vendor mgmt

User Experience Mgmt

On-line Ops

Executive IT

Other

37%

49%

10%

9%

9%

9%

6%

14%

9%

17%

19%

8%

15%

11%

39%

1%

Which functional groups/practices will most benefit from leveraging the results of big data analytics for IT?


How Do IT organizations Use Big Data Analytics

• The research explores use cases across three key practice areas• Planning and Engineering• Technical performance monitoring• Trouble triage and diagnostics

• Which types of IT monitoring data are valuable to each of these analytical tasks?


Planning & Engineering Tasks: Capacity Planning is THE Use Case for Big Data Analytics for IT


Network capacity planning

Server capacity planning

Storage capacity planning

DevOps

External Cloud capacity planning

SLA/SLO planning

Other

None of the above

57%

66%

70%

29%

41%

23%

0%

6%

For which types of planning & engineering tasks does your org use or plan to use big data analytics for IT?


Planning & Engineering Tasks Supported by Big Data

• Advanced users are more innovative with:• External cloud capacity planning (54% of advanced users versus 41%

of general research pool)• SLA/SLO planning (38% of advanced firms versus 23% of general pool)

• Manufacturers also more active around external cloud capacity planning (54%)

• Finance lags in network capacity planning (36%), external cloud (20%) and SLA/SLO planning (4%)


Most Important Data Types for Planning & Engineering Tasks


Events


Log entries


Raw network packets



Transaction records


Other

26%

44%

35%

34%

32%

36%

51%

51%

48%

0%

Which types of data are most important for planning & engineering via big data analytics for IT?


Application performance data, transaction records, & cloud provider API metrics are top three.

Most Important Data Types for Planning & Engineering Tasks

Other data types gain importance for advanced big data users


Flow records (39% of advanced users

versus 34% of general pool)

Interpreted packet flow (43% of advanced versus

36% of general)

Raw network packets (41% of advanced versus

32% of general)

Technical Performance Monitoring Tasks Supported by Big Data


Cross-domain service performance and availability

Network availability and performance

Systems availability and performance

Storage availability and performance

Service transaction response time

Configuration/change mgmt effectiveness

Infrastructure optimization

Application performance optimization

Security-related issues

Other

None of the above

25%

53%

52%

49%

36%

32%

45%

36%

27%

0%

3%

For which technical performance issues does your org use or plan to use big data analytics for IT?


Technical Performance Monitoring Tasks Supported by Big Data

• Advanced big data users• Especially focused on systems availability and performance monitoring

(68%)• More focused (44%) on application performance optimization• Other advanced use case

Service transaction response time monitoring: 42% Configuration/change management effectiveness: 44%

• Executive IT• Bigger focus on infrastructure optimization (54% versus 39% of IT staff)• Like advanced users, exploring configuration/change management

effectiveness (40% versus 26% of staff)


Most Important Data Types for Technical Performance Monitoring


Events


Log entries


Raw network packets



Transaction records


Other

32%

47%

48%

48%

33%

37%

63%

36%

25%

0%

Which types of data are most important for technical performance monitoring via big data analytics for IT?


Triage & Diagnostics Tasks Supported by Big Data: No Dominant Use Case


Isolate problem to application or infrastructure

Triage across application tiers and/or middleware

Isolate the relevant service provider

Triage across virtualized systems

Isolate infrastructure issues in the network

Isolate infrastructure issues internal to systems

Isolate infrastructure issues within the DB tier

Isolate infrastructure issues in storage

Isolate non-mobile end device issues

Isolate mobile-specific end-device issues

Isolate security-related issues

Other

None of the above

30%

37%

37%

41%

35%

37%

29%

43%

25%

21%

27%

0%

7%

For which types of technical triage and diagnostics tasks does your org use or plan to use big data analytics for IT?


Triage & Diagnostics Tasks Supported by Big Data: Advanced Users Identify Key Use Cases


Isolate problem to application or infrastruc-ture

Triage across application tiers and/or middleware

Isolate the relevant service provider

Triage across virtualized systems

Isolate infrastructure issues in the network

Isolate infrastructure issues internal to sys-tems

Isolate infrastructure issues within the DB tier

Isolate infrastructure issues in storage

Isolate non-mobile end device issues

Isolate mobile-specific end-device issues

Isolate security-related issues

Other

None of the above

34%

27%

22%

32%

24%

37%

20%

44%

22%

22%

44%

0%

12%

25%

41%

36%

46%

32%

31%

32%

42%

20%

24%

20%

0%

5%

32%

42%

52%

44%

46%

44%

34%

44%

34%

18%

22%

0%

4%

Advanced

Intermediate

Beginner


Most Important Data Types for Triage and Diagnostics


Events


Log entries


Raw network packets



Transaction records


Other

28%

42%

38%

35%

28%

35%

56%

41%

47%

0%

Which types of data are most important for triage and diagnostics via big data analytics for IT?


What Benefits Do Enterprises Experience From Big Data Analytics for IT


Faster time to repair problems

Proactive ability to prevent problems

More efficient use of infrastructure capacity

Better correlation between change and performance

Better alignment with the business

Improved OpEx efficiencies within IT

Improved compliance with industry requirements

Fast identification of security threats

Faster time to deliver new IT services

Less overhead writing/maintaining rules and thresholds

Surprises / unexpected insights

Other

None so far, it's still too soon to tell

31%

45%

46%

36%

41%

45%

29%

36%

35%

23%

9%

0%

4%


Summary

• Big data most often impacts storage capacity and network performance• All infrastructure planning & operations practices are affected, especially as

firms gain more big data experience

• IT monitoring data imported into big data environments• Application performance data, transaction records, cloud provider API

metrics are most popular

• Use cases for big data analytics for IT• Storage, systems and network capacity planning are tops• Availability and performance monitoring of these 3 technologies also

important• No dominant troubleshooting use case, but storage problem isolation is

prominent

• Key benefits:• Efficient use of infrastructure and OpEx, proactive problem prevention


Download the report for more insights

• The report identifies:• The backend databases these

enterprises use• The analytical tools they apply to these

big data environments

• More insights by vertical industries, size of company, and role

• More Analysis of how big data impacts and practices vary by big data maturity levels

• The technical and cultural issues that impact big data success


Learn more & access report at www.enterprisemanagement.com


Data & Analytics

Big Data Impacts on Hybrid Infrastructure and Management