Tdwi solution spotlight presentation slides

Helping Organizations Transform Data into Business Value with the Power of

Intelligence.

2

TDWI Events• Learn More About Upcoming Seminars & Conferences at

www.tdwi.org/events

• CBIP Certification• Exhibit hall featuring vendors showcasing the latest in

technology offerings• Valuable networking opportunities

Conferences• July 18-20 , Boston

• October 2-7, San Diego

• December 4-9, Austin

Seminars

• July 25-28 , Salt Lake City

• September 19-22, Washington DC

• September 26-29, Minneapolis

• October 17-20, New York

• Upside – The content hub where new BI & analytics articles are published daily.

• eLearning – Learn everything from data viz to predictive analytics anytime, anywhere online.

• Seminars – Three-day in-depth learning opportunities in a city near you.

• Conferences – A week long opportunity to network & learn from seasoned instructors.

• Onsite Education – TDWI Instructors providing hands-on training in your offices.

• More information: tdwi.org/education

• TDWI Premium Membership helps you:– Stay current on trends & best practices.– Validate your data direction & approaches.– Network with peers in the industry.– Gain access to a trusted, independent source

for actionable insights. – Save money on TDWI events.

• More information: tdwi.org/membership

Data Warehouse Modernization

A Response to Big Data, Advanced Analytics, and Other Business Opportunities

Philip RussomSenior Research Director for Data Management, TDWI

Agenda

PLEASE TWEET@pRussom, #TDWI, #DataWarehouse, #Modernization, #Analytics, #RealTime

• A Time of Great Change– The evolution of data, its

management, and its use• What is data warehouse

modernization?– What is its state?– Why is it important?

• Benefits and Barriers• Best Practices

– Organizing DW Mod projects– Modernization strategies

• Trends in DW Mod– DWEs & evolving architecture– Role of Hadoop

• Top 12 Priorities for DW Mod• How to get the TDWI report

this presentation is based on• Q&A

6

TRENDS DRIVING


• Data is evolving• Data management is evolving• Business use & leverage of data is evolving

Data is Evolving

• Exploding data volumes– Demands speed & scale

from DM platforms & solutions

• Big data is more than big– It’s new, diverse, loaded with opportunity

• Structural diversity– Coming from new sources, feeding new targets– Data types and structures that are new to you

• Generated more frequently– Demands use of event processing and real-time tech– Demands monitoring and reaction from business

Data Management is Evolving

• Emerging practices– Data exploration, discovery analytics– Data lakes, data hubs

• New data platforms– Columnar, appliances, Hadoop, cloud

• New tools and methods– Data prep, self-service data access– Event processing, true real time– Early ingestion, in-line analytics

Biz Use & Leverage of Data is Evolving

• More business value and organizational advantage– Decisions based on more and better facts– More complete views of customers– Operations move faster, based on fresher data– More competition based on analytics, with massive data– More analytics, in general, in more advanced forms

• Greater governance for biz compliance & data standards

“DW Modernization” takes many forms…• Additions to existing data warehouse environment (or ecosystem)

– New data subjects, sources, tables, dimensions, etc.– More server instances, nodes, bigger storage

• Upgrades– Newer versions of current

database or integration software– Bigger and faster hardware

• More standalone platforms & tools– Complement DW wo/replacing it– Tools for analytics, real time,

new data types, new interfaces– New appliances,

columnar databases, Hadoop, NoSQL, etc.

• Architectural Adjustments– Logical DW design across

multiple platforms– Extending data integration (DI)

• Rip and Replace– Decommission current DW platform or misc tools; migrate to others

11

ACCORDING TO TDWI SURVEY RESULTS

State of Data Warehouse Modernization

• Modernizing a DW is extremely important (58%) or moderately important (33%)

• DWs are evolving dramatically (22%) or evolving moderately (54%)

• DWs are fully (7%) or mostly up-to-date (41%)• DWs are somewhat (38%) or far behind (12%)• DWs are still very relevant (49%) or

somewhat relevant (39%) to what biz wants

12

SOURCE: TDWI 2016 DW Modernization Report, extracted from Figures 2, 4, 5, 6.


is mostly a problem

Data Warehouse Modernizationis mostly an opportunity

SOURCE: TDWI 2016 DW Modernization Figure 7.

• “One has to keep up with the volumes and variety of data that can enhance your analytical results for better decision making and customer service.”

– Data architect, Financial services, Africa• “The business landscape is constantly changing, and it’s evolving the

data requirements. If you do not change with the times, you will become obsolete.”

– Enterprise architect, Petroleum, Canada• “In the past, week-old data might have sufficed; but today we need

near real-time data.”– BI Manager, State/local government, USA

• [We need DW mod.] “to achieve low TCO, integrate with digital channels, support fast business decisions, allow complex analytics.”

– CTO team member, Financial services, Asia• [Our] “current solution was built five years ago on twenty-year-old

technology and patterns. Latency, performance, and scope all lag far behind today’s needs.”

– Data architect, Insurance, USA

SOURCE: TDWI 2016 DW Modernization Report, extracted from Figure 3

IN USERS’ OWN WORDS

Why is DW Modernization Important?

14

Biz Benefits of DW Modernization

0% 10% 20% 30% 40% 50% 60%

Complete views of customers and other important entities

Address new business requirements

Competitive advantages

Agile delivery of solutions, for nimble business responses

Operational efficiency of business

Fast and frequent report/analysis cycles, near real time

Business decision making, both strategic and operational

Analytics, including visualization and exploration

SOURCE: TDWI 2016 DW Modernization Report, top half of Figure 8

What are the top business tasks that would benefit from data warehouse modernization?

15

Challenges to DW Modernization

• Overcoming bad practices in data management– Poor stewardship/governance, data quality, metadata

• Personnel problems– Inadequate staffing, skills, experience

• Paying the price of modernization– Cost of implementation, hardware/software upgrades

• Complexity of architecture– Designing & managing a multi-platform systems environ

• Limitations of existing systems– Current environment won’t scale up to big data, ingest data fast enough

• Outmoded development environment and practices– Need tools that foster speed for agility, plus reuse for productivity

SOURCE: TDWI 2016 DW Modernization Report, extracted from Figure 9

16

TO SURVIVE THE CHALLENGES,

Recognize Standard DW Modernization ScenariosAND STAFF OR SCHEDULE THEM ACCORDINGLY.

• System modernization (53%)– upgrades and patches for

hardware/software servers or tools• Arbitrary modernization (47%)

– based on business needs of a specific project, or urgent request for info/analysis

• Non-data modernizations (44%)– modernizing reporting, analytics, data

integration• Optimization modernization (42%)

– performance tuning and similar tweaks• Continuous modernization (37%)

– quarterly updates, complete views, etc.• Disruptive modernization (21%)

– rip and replace platforms, tools, datasets

SOURCE: TDWI 2016 DW Modernization Report, Figure 10

DEFINITION

Data Warehouse Environments (DWE)

• Many enterprise data warehouses (EDWs) are evolving into multi-platform data warehouse environments (DWEs).

• Users continue to add additional standalone data platforms to their warehouse tool and platform portfolio.– New platforms = relational DBMSs based on columns,

appliances, clouds; real-time complex event processing; Hadoop• The new platforms don’t replace the core warehouse, because it is

still the best platform for the data that goes into standards reports, dashboards, performance management, and OLAP.

• Instead, the new platforms complement the DW, because they are optimized for workloads that manage, process, and analyze new forms of big data, non-structured data, and real-time data.

BEST PRACTICES

Modernization Strategies• Most common strategy – DW Augmentation (42%)

– Add more data platforms to DWE, to complement existing core DW• For only 15%, replacing DW’s primary data platform has been a strategy• 24% modernize on per case basis; 14% don’t have a strategy

19


TRENDS in DW MODERNIZATION

Evolving DW Platform Architectures• Single-monolith DW architectures aren’t that common and are slipping away• Simple DWE (a few platforms) is now the norm for DW systems architecture• Future: we’re trending strongly toward complex DWEs (many platforms)

20


Seventh Inning Stretch…


Role of Hadoop in DW Modernization• Hadoop in DWEs is still rare today (15%), but will increase 5x (78%) in 3 years• Hadoop usually complements a primary DW platform – 17% today, 36% in 3yrs• Hadoop rarely replaces a DW – 1% today, 6% in three years

22


• Hadoop offloads DW and extends DWE– Captures and manages big data at scale– Data landing & staging on steroids– Repository for detailed source data– Processing for analytics & data integration– Advanced forms of algorithmic analytics

(mining, graph, predictive)– ELT push-down processing– Manages multi- and unstructured data– Inexpensive compared to capacity based

pricing on average relational DW or DBMS


How can Hadoop Modernize a DW?

Top 12 Priorities for DW ModernizationThese are recommendations, requirements, or rules that can guide you.1. Embrace change.2. Make realignment with business goals your top priority.3. Make DW capacity a high priority on the technology side.4. Make analytics a priority, too.5. Don’t forget the related systems that also need modernization.6. Don’t be seduced by new, shiny objects.7. Assume that you’ll need multiple

manifestations of modernization.8. Know the tools and techniques of modern DWEs.9. Adjust the large-scale architecture of your DWE.10. Reevaluate your DW platform.11. Consider Hadoop for various roles in the DWE.12. Develop plans and recurring cycles for DW modernization.

24

Download a free copy of the report that this presentation is based on

• Download the report in a PDF file at:

tdwi.org/bpreports

• Feel free to distribute the PDF file of any TDWI Best Practices Report

25

The Modern Data WarehouseSAP Solutions for a New Era

June 2016

Michael BreenPlatform ArchitectCustomer Innovation & Enterprise Platform Group, SAP

27© 2015 SAP SE or an SAP affiliate company. All rights reserved.

75%of global workforce

will beMillennials

We are entering into a new era of unprecedented change across a multitude of dimensions

5 billionpeople worldwide

will becomemiddle class

50%of the world’s population

will live underwater shortage

1.3 billionpeople on business & social networks today

50 billion connected devices and

“internet of things” by 2030

Rising Customer Expectations A Dramatically Changing Workforce Pressure on Resources

Network Effect/Explosion in Structured and Unstructured data

27

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 28Public

Data LakeData volumes will continue to grow to 6 billion petabytes, including unstructured data such as social networking data and low level IoT data. Mining the value from this data is essential

CloudCloud spending will surge by 25%, reaching over $100 billion. There will be a doubling of cloud data centers.

Internet of Things30 billion devices, sensors in 2020 –driving $8.9 Trillion in revenue. The need for real-time processing and analytics will explode

Mobile

CRM Data

Planning

Opportunities

Transactions

Customer

Sales Order

Things

Instant Messages

Demand

Inventory

Big Data

Sales Order

Things

Mobile

Demand

Big Data

CRM Data

CustomerPlanning Transactions

Key Trends

28


40% executives worry that their organizations will not keep pace with technology change and lose their competitive edge.

– McKinsey study, 2013”“

Complexity built up over decades limits the ability to innovate; radical simplification is needed to unlock the potential.

Drive business innovation

Keep the lights on

IT EnvironmentCollapse redundant infrastructure layers

User ExperienceEngage front line employees/customers

28%

72%ConsumptionFor immediate business impact

Forrester IT Survey, 2013

Drive business innovation

Keep the lights on

29


SAP’s Data Warehouse enables a revolutionary approach streamlines and simplifies data warehousing

Providing greater speed and scale along with agility for development and efficiency that reduces data movement and data preparation. SAP’s complete architecture offers:

A-z

Flexible Architecture Rapid DeploymentPre-packaged or Customize

30


e

Customized Data Warehouse

• Controlled schemas, often prepopulated with structure

• Lifecycle management of schemas

• High level languages and less programming

• More prebuilt tools to purpose

SAP HANA platform

Processing Engine

Application Function Lib. & Data Models

Integration Services

SAP HANA PLATFORMReal-time transactions + end-to-end analytics

Extended Application Services

HANA Smart Data Streaming

HANA Dynamic Tiering

• Usually depends on SQL tools and low-level programming

• Fewer controls on schema updates• Easier to change

- An integrated architecture that reduces data redundancy while keeping all information at hand

- Utilizes state-of-the-art in-memory techniques that furnish answers in-context, in real time

- Makes more data available at the right time to the right person at the right place in the business process

SAP Provides The Best of Both Approaches!

More

Deg

rees

of F

reed

omLess Time to Implement

SAP Gives You The Power of Both Custom and Packaged

Pre-Packaged Data Warehouse

HANA Advanced Analytics

31


Traditional Data Warehouses Just Copy Data AndCreate More And More Copies In Indexes

CO

PY

Business Data:ERP, CRM, SCM

Reference/Supplier Data

Data FabricData Remains in Place!

Hadoop /Social Media

Data Bloat slows the database & becomes hard to manage

Historical Data

Hadoop /Data Lake

ReferenceData

HANA Keeps Critical Data in Memory withoutCopies or Support Indexes

Business Data:ERP, CRM, SCM

Streams & Context

Computations & Management are Streamlined without bloating database

Cop

y sc

hedu

les

dela

y da

ta

Cub

es a

nd In

dexe

sta

ke ti

me

to b

uild

HANA Flexible Architecture Example: Data Fabric

Real TimeSmart Data Access

32


YARN

HDFS

Enable Precision DecisionsWith Contextual Insights In Enterprise Systems

Other Apps

Files Files Files

HANA-Spark Adapter for improved performance between distributed systems

Gain business coherence with business data and big data

Compiled queries enable applications & data analysis to work more efficiently across nodes

Familiar OLAP experience on Hadoop to derive business insights from big data such as drill-down into HDFS data

Compiled Queries

Spark Adapter

Drill Downs

SAP HANA in-memory platform

Vora

Spark

Vora

SparkIn-Memory

StoreApplication Services

Database Services

Integration Services

Processing Services

SAP HANA Platform

Vora

SparkHANA-Spark Adaptor

33


Customer value delivered by SAP Data Warehouse

Internet of Things

Eliminate or reduce data movement

Fewer copies of data

Enterprise Wide Analytics

Simplified Architecture

Real-time Analytics

Data Lake

Access data across your enterprise

Unmatched federation of data without centralizing

In-memory performance gives answers in seconds, not hours

Reduced latency means current data is addressed not old data

Petabytes of historical data storage

Advanced analytics for mining non-traditional data

Extensive Hadoop and no-SQL support

Data management and analytics from device to enterprise

Streaming analytics

34


Winning Combinations

SAP HANA* and Intel® Xeon® processors help customers get the most from their growing data

*See the latest SAP HANA* certified OEMs and appliances: http://global.sap.com/community/ebook/2014-09-02-hana-hardware/enEN/index.htmlSoftware and workloads used in performance tests may have been optimized for performance only on Intel microprocessors.

Optimized for Flexibility

Deploy SAP HANA

On premises On demand/hybrid cloud

Built for Each Other

More transactions per minute

Collaborative Partnership

Using your platform of choice from 15 industry leading OEMs* & CSPson the Intel Xeon processor E7 v3 family1, 2

+

35


From 50m to 5m for failed tests…


“We have built a highly innovative and scalable platform for the future. We really see this solution as a game-changer for the automotive industry.” Dirk Zeller, Head of IT Consulting, Mercedes-AMG GmbH

… led to 15% increase in overall test cycle capacity

In the future, Reinhard Breyer, CIO of Mercedes-AMG GmbH, explained that, “This breakthrough innovation is just the start. Ultimately we want to monitor engine performance in customer vehicles.”


500Metrics analyzed to identify outliers

100%Accuracy

97%Confidence that a signal is a true positive

6 weeksProject duration

Anticipates consumer behavior


eBay Early Signal Detection System Powered by Predictive Analytics on SAP®

HANA

CompanyeBay

HeadquartersSan Jose, California

IndustryProfessional Service (Internet)

ServicesOnline Marketplace

Employees

31,500 (2012)

Revenue or BudgetUS$ 14.1 billion (2012)

Web Site

eBay.com

Business Challenges Increase ability to separate signal from noise to identify key changes to

the health of eBay’s marketplace Improve predictability and forecast confidence of eBay’s virtual economy Increase insights into deviations and their causes

Technical Challenges Detect critical signals from 100 PBs of data in eBay EDW Highly manual process because one model does not fit all the metrics

hence requires analyst intervention

Key benefits Automated signal detection system powered by predictive analytics on

SAP HANA selects best model for metrics automatically; increases accuracy of forecasts

Reliable and scalable system provides real-time insights allowing data analysts to focus on strategic tasks

Decision tree logic and flexibility to adjust scenarios allows eBay to adapt best model for their data

“HANA is valuable in the sense that it accelerates that speed to insight. HANA, with in-memory capability, with multicore, fast, lots of data, all of that coming together is how I think analytics is going to work broadly in the future.”

David Schwarzbach, VP&CFO eBay North America at eBay Inc.

“HANA system will free up all the bandwidth right now involved in figuring out what is going. The user just has to feed in their metric, doesn’t have to really worry about which algorithm is the best and be able to use the system because it is inherently intelligent and configurable.”

Gagandeep Bawa, Manager, North America FP&A at eBay Inc.

Determine with 100% Accuracythat a signal is positive at 97% confidence

Automated Early Signal Detectionsystem powered by SAP HANA


+ $10M Revenue

Inventory scheduling andre-allocation in real-time

- 3% Cost

Sense deviations in tire temperature and pressure

+ $17M Revenue

Help brands harness word-of-mouth from social media

- 50% Inventory

Enable greater supply chain control to improve inventory

- €500k Capital

Drive profitable decisions with real-time analysis

+ $1.1M Revenue

Faster, earlier intervention to reduce student drop-outs

100% Accuracy

Monitor marketplace health through automated signal detection

$10-25savings per win-back

Measure the value of marketing campaigns: promotions, customer loyalty, adoption rates

~2 secondExecution

Gain competitive advantage with predictive analysis

99% fasterETL Load time

Offer rapid social media analysis to track consumers and influencers

5M People

Implement austerity guidelines to achieve cost savings of ~25%, and provide better care

€ 1.2Mpotential savings

Improve out-of-stock and loss prevention through real-time analysis

- 5% total cost

Use real-time info to operate call centers: greater productivity, first-call resolution rate, and lower cost

© 2015 SAP SE or an SAP affiliate company. All rights reserved.

Thank you

© 2014 SAP AG or an SAP affiliate company. All rights reserved.

http://hana.sap.com/dwl

Michael [email protected]

Data & Analytics

Tdwi solution spotlight presentation slides