Embed Size (px)
Citation preview
Do You Hadoop?A Survey of Big Data Practitioners
February 12, 2014
Bradley Graham
2
Big Data Research from Sand Hill Group
• Mindset over Data Set: A Big Data Prescription for Setting the Market Pace– Presents powerful learnings of some of the most
successful implementers of enterprise Big Data – Provides prescriptive executive-level guidance for
adopting and using Big Data– Use as a planning guide or benchmarking tool– Purchase at http://bit.ly/SH_BD_14_S
• Do You Hadoop? A Survey of Big Data Practitioners– Clarifies Big Data (Hadoop-based) initiative status– Identifies pain points and barriers to adoption– Illuminates usage changes over the next 12-18 months– Use as a benchmarking tool– Download at http://bit.ly/SH_H2014
3
Technology/Analytics professional
50.4%
Business sponsor/user
23.0%
Consultant14.1%
Academic3.7%
Other8.9%
Participant Roles
Consumer Services11.9%
Education5.9%
Financial Services
8.1%Government
1.5%Healthcare
2.2%
Industrials22.2%
Technology32.6%
Telecommunications2.2%
Other13.3%
Industries
Broad Cross-Sectional View of the User Base
• Startups and large established companies are leading the charge
• Technology industry use is strongly correlated to startup companies
• Companies serving large and/or diverse customer groups are a natural fit for Big Data (e.g., retail, media and entertainment, and financial services)
• Business sponsor and user participation renders a more complete picture of Big Data’s value and impact
Small33.3%
Medium19.3%
Large47.4%
Company Size(Number of Employees)
4
Still Early Days
• Solid progress is being made
• Majority of the companies have identified a business problem to address
• Use of multiple analytics suggest compelling value has been realized
44.4%
16.3%11.1%
8.1% 9.6% 10.4%
0%
10%
20%
30%
40%
50%
Exploring andeducating
Conducting POC Developing firstsolution
Piloting firstsolution
First solutiondeployed
Supportingmultiple analytics
Perc
ent o
f Tot
al P
opul
atio
n
Hadoop Initiatives Status(All Companies)
5
Beware of Small and Agile Competitors
• The median phase by company size is:– Small: Exploring and educating– Medium: Conducting POC– Large: Conducting POC
• Data-centric startups enabled small companies to surpass medium-size companies in the advanced stages
45.0%
18.2% 26.7% 36.4%
7.7%
35.7%
20.0%
18.2%13.3%
27.3%
23.1%
14.3%
35.0%
63.6% 60.0%36.4%
69.2%50.0%
0%
20%
40%
60%
80%
100%
Exploring andeducating
Conducting POC Developing firstsolution
Piloting first solution First solutiondeployed
Supporting multipleanalytics
Hadoop Initiative Status(by Company Size)
Small Medium Large
Perc
ent o
f Cat
egor
y
6
Satisfaction is Driving Continued Investment
• Higher overall satisfaction among business sponsors and users relates to:– Profound insights– More effective actions
• Technology professionals’ higher than expected satisfaction is likely attributed to:– Success with the technology– Producing results that satisfied the business
stakeholders
• Challenges associated with Big Data are nevertheless impacting satisfaction– 3x more were less than satisfied (35.6%) vs.
more than satisfied (11.1%)
35.3% 29.0%
48.5% 61.3%
16.2% 9.7%
0%
20%
40%
60%
80%
100%
Technology/Analyticsprofessional
Business sponsor/user
Hadoop/Big Data Initiative Satisfaction (By Role)
Less Than Meets Better Than
7
• Leading current uses are:– Foundational– Support or augment the existing
solution portfolio (e.g., DW/BI and small data analytics)
– Support Big Data experimentation
Most Commonly Reported #1 Uses of Hadoop(Current vs. Future)
#1 Current Uses(as of October 2013)
#1 Future Uses(in 12 – 18 months)
ChangeFrom
Current
Data Preparation (25.2%) Advanced Analytics (24.4%) —
Business Intelligence (17.8%) Data Preparation (17.8%) -7.4%
Basic Analytics (17.0%) Business Intelligence (14.1%)Archive More Data (14.1%)
-3.7%—
Top Current Uses(as of October 2013)
Top Future Uses(in 12 – 18 months)
ChangeFrom
Current
Basic Analytics (58.5%) Advanced Analytics (61.5%) —
Business Intelligence (48.1%) Business Intelligence (45.9%) -2.2%
Data Preparation (45.9%) Data Preparation (40.7%) -5.2%—
Top Uses of Hadoop(Current vs. Future)
Mastering the Basics and Moving on to Advanced Applications
• Leading uses in 12-18 months emphasize:– New data types (streaming, geographic,
syndicated, etc. data)– Advanced analytics (e.g., risk,
propensity and optimizations)
8
52.6%60.7%
45.9%
22.2%28.9%
16.3% 19.3%14.1%
0%
20%
40%
60%
80%
Operational Log Online Geographic Partner 3rd party Files(Documentsand Media)
Streaming
Perc
ent o
f Tot
al P
opul
atio
n
Data Types in the Hadoop Environment
The Data Does Indeed Tell the Story
• Declining storage costs encourage a store everything approach
• Most prevalent data types parallel the focus of current usage
• Less frequently hosted data types hint at future Big Data applications
9
Big Data and Hadoop are Complicated
• Resources remain the dominant issue for the foreseeable future– Internal skills and experience gap– Limited ability to repurpose existing resources– Competition for “journey talent” (i.e., those successfully navigating the process at least once)
• Other frustrations are the technology challenges and level of effort related to:– Implementing, maintaining and provisioning the environment– Designing, building and maintaining solutions
• Performance, interoperability and other current second tier issues may prove to be larger than expected issues down the road if left unaddressed
Most Commonly Reported#1 Hadoop-related Challenges Top Hadoop-related Challenges
Knowledge and experience (46.7%) Knowledge and experience (65.2%)
Skills availability (20.7%) Skills availability (52.6%)
Development effort (6.7%) Development effort (40.7%)
Challenges Associated with Hadoop/Big Data
10
Success in Numbers
• Navigating the Hadoop/Big Data complexities requires a trusted partner ecosystem– It's far too complex at this point to go it alone
• Augment and, through a collaborative working model, edify internal resources
• Gain access to value-added products and services that simplify:– Infrastructure implementation and provisioning– Solution development and use– Data access and management
Effective partnering can address critical and second tier issues while reducing time to value
