Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon

Making Big Data Work for Your Organization

Alan Simon Zunesis Senior Fellow, Analytics & Data Management December 3, 2015

© 2015 Alan Simon. All Rights Reserved

Introductions

2

Select experience • Arizona Attorney General’s Office • Pennsylvania Department of Transportation (PennDOT) • Pennsylvania Department of Health • Wisconsin Department of Administration • Santa Clara County (CA) • Washington (State) Department of Early Learning • Arizona State University • USAF • Department of Defense

3

Check out this mission statement…

• “Our investigators need instantaneous access to a broad range of data at their fingertips…not only all direct person-to-person and person-to-business relationships, but also relationships several levels removed. The same direct and indirect relationships to assets also need to be immediately available to our investigators…”

4

5

Nope! • Arizona Attorney General’s Office, 1979-1980

6

What do we want from analytics and big data?

Data-Driven Insights

Insight-Driven “Better” Decisions

7

Proposition:

8

Big data success isn’t “automatic”!!! 9

However…

7 KEYS TO BIG DATA SUCCESS

10

“The art of the now possible” • Put deferred solutions back on the table

and also

• Reengineer and rebuild old, overly complicated solutions

11

1

2001-2002: The Anthrax Attacks

12

Bioterrorism fears go viral !

• Anthrax • Smallpox • Plague • Viral hemorrhagic fever • …?????

13

Worries about timeliness of response • Many bioterrorism agents show symptoms…but cannot be

confirmed for several more days •  In meantime outbreaks could get much worse •  The “solution” – a concept known as

Syndromic Surveillance

14

Interdiction

Patterns

Reports

Hypotheses

Analysis

Circa 2001-2002 solution:

15

But if I could figure out time travel…

16

“Big data” doesn’t just mean Hadoop!

• Hadoop is certainly important but also… • HANA • MongoDB • Columnar and other specialized databases • Pretty much any “post-relational” data management technology!

• Also: “big” data is relative! 17

2

Hadoop is evolving at light speed!

18

3

https://blog.cloudera.com/blog/2015/09/kudu-new-apache-hadoop-storage-for-fast-analytics-on-fast-data/ http://www.infoworld.com/article/2986675/hadoop/cloudera-kudu-hdfs-hbase-in-one.html

Evolution often yields confusion…

19

http://sdsblog.com/2015/10/12/kudu/

Traditional BI and DW not out of the picture…at least for now…

20

4

Hadoop data staging area AND ANALYTICS SANDBOX

Relational

EDW

Sqoop Flume …

Sqoop

Sqoop Flume …

Hadoop as “supersized data staging area” in front of EDW

Hadoop EDW: staging + user-accessible data

Hadoop/data lake as next-generation EDW

Sqoop Flume …

Sqoop Flume …

Prescriptive analytics are critically important!

23

5

Category Purpose

Descriptive analytics

Tell me what happened, and why Tell me what is happening right now, and why

Predictive analytics

Tell me what is likely to happen, and why

Discovery analytics

Tell me something important…even without me asking specific questions!

Prescriptive analytics

Tell me what my options are Tell me what I should do

The Analytics Continuum

The prescriptive analytics framework Detect Events

Categorize and Process Events

Apply Analytical Models

Form Hypotheses

Take Initial Actions

Update and Correlate Data

Prove or Disprove Hypotheses

Take Prescribed Actions

Analytics BPM +

25

Why?

26

The hard work doesn’t go away with big data

• You still need: • Master data management

• Data standardization

• Data governance

• Data quality management

• …

27

6

Your big data strategy and architecture must also include your social media listening and engagement

28

7

Everyone is Connected! •  Constituents •  Service Consumers •  Residents •  Voters •  Veterans

This is the world we now live in… Your Citizens are Online

Communications Manager Public Info. Office Public Services

iWeSocial IQ Social Media Listening Platform iWeSocial

Research Analyst

Public Web:

iWeSocial Insights (monthly readout)

Government Organizations

Focus: Services, Security, Sentiment

Social Listening for State and Local Government Technology + Human Analysis = Actionable Insight

Social Listening Use Case: McCarran International Airport

Business Return: •  Discover what’s

being said about the services you provide: ü  Sentiment ü  Problem areas ü  What's

working

•  Crisis Management – listen for spikes in conversation in real-time

•  Increase engagement by better understanding the conversation

HOW DO YOU GET STARTED?

32

Good news: the classic approach still works!

33

Assessment

Strategy

Business Architecture

Technology Architecture

Roadmap

During assessment, focus on:

• Both opportunities and “points of pain” • What you cannot do today • What is challenging and cumbersome to do today • What has been “put into mothballs” • “Blue skies and green fields”

34

Big data and analytics assessment

35

Assessment • Analytics continuum • Today’s technology • In-progress initiatives

• Hypotheses vs. facts • Pain points • Known opportunities

• Deferred initiatives • Analytics appetite • Competition

• Typically 3 – 5 weeks • Grounded in your organization’s reality

Followed by…

36

Strategy • Commitment to analytics • Change management • Pace of change

• Migration strategy • Leadership (RACI) • Resourcing models

Business Architecture

• “Day in the Life” scenarios • Use cases and process flows • Roles and responsibilities

Technology Architecture

• DW/BI role • Big data role • Platforms

• Tools • Migration plans • Integration architecture

• Support • Data flows • Control flows

Roadmap • Risk mitigation • Program leadership • Sponsorship

• Defining success • CSFs • User adoption

• Phases • Stretch goals • Contingency plans

• Typically 8-12 weeks • Complete blueprint

We’d love to hear from you!

Zunesis

Email: info@Zunesis.com

Website: www.zunesis.com

Headquarters 8375 S. Willow Street 5th Floor Lone Tree, CO 80124 720-221-5200

37

Las Vegas Office 6280 S. Valley View Blvd. Suite 604 Las Vegas, NV 89118 702-837-5300

• My Wednesday analytics blog: https://humpdayanalytics.wordpress.com/

• Cross-posted on LinkedIn

•  Follow me on Twitter: @HumpDayAnalytic

• My LinkedIn/Lynda.com courses:http://www.lynda.com/Alan-Simon/3981678-1.html

For more information…

38

39