Data Warehouse Modernization - Amazon S3...Data Warehouse Modernization in the Age of Big Data Analytics Philip Russom, Ph.D. Senior Research Director for Data Mgt, The Data Warehousing

#TDPARTNERS16 GEORGIA WORLD CONGRESS CENTER

Data Warehouse Modernizationin the Age of Big Data Analytics

Philip Russom, Ph.D.Senior Research Director for Data Mgt,The Data Warehousing Institute (TDWI)

• A Time of Great Change• The evolution of data, its management, and its use

• What is data warehouse modernization?• What is its state?• Why is it important?

• Benefits and Barriers• Best Practices

• Organizing DW Mod projects• Modernization strategies

• Trends in DW Mod• DWEs & evolving architecture• Role of Hadoop

• Top 12 Priorities for DW Mod• How to get the TDWI report this presentation is based on• Q&A

Today’s Agenda

#TDPARTNERS16 – @prussom from #TDWI on #DataWarehouse #Modernization

TRENDS DRIVING

Data Warehouse Modernization

• Data is evolving• Data management is evolving• Business use & leverage of data is evolving


“DW Modernization” takes many formsAdditions to existing data warehouse environment (or ecosystem)

• New data subjects, sources, tables, dimensions, etc.• More server instances, nodes, bigger storage

More standalone data platforms and tools• Complement DW without replacing it• Tools for analytics, real time,

new data types, new interfaces• Appliances, columnar, Hadoop

Architectural Adjustments• Logical DW across multiple platforms• Extending data integration (DI)

Upgrades• Newer versions of current database• Bigger and faster hardware

Rip and Replace• Decommission current DW platform or misc tools; migrate to others


ACCORDING TO TDWI SURVEY RESULTSThe State of DW Modernization

• Modernizing a DW is extremely important (58%) or moderately important (33%)

• DWs are evolving dramatically (22%) or evolving moderately (54%)

• DWs are still very relevant (49%) or somewhat relevant (39%) to what biz wants

• DWs are fully (7%) or mostly up-to-date (41%)• DWs are somewhat (38%) or far behind (12%)

SOURCE: TDWI 2016 Report on DW Modernization, extracted from Figures 2, 4, 5, 6, 7.

6

Data Warehouse

Modernizationis mostly a

problem

Data Warehouse Modernizationis mostly an opportunity

SOURCE: TDWI 2016 Report on DW Modernization, Figure 7.

• “One has to keep up with the volumes and variety of data that can enhance your analytical results for better decision making and customer service.”

• Data architect, Financial services, Africa• “The business landscape is constantly changing, and it’s evolving

the data requirements. If you do not change with the times, you will become obsolete.”

• Enterprise architect, Petroleum, Canada• “In the past, week-old data might have sufficed; but today we

need near real-time data.”• BI Manager, State/local government, USA

• [We need DW mod.] “to achieve low TCO, integrate with digital channels, support fast business decisions, allow complex analytics.”

• CTO team member, Financial services, Asia• [Our] “current solution was built five years ago on twenty-year-old

technology and patterns. Latency, performance, and scope all lag far behind today’s needs.”

• Data architect, Insurance, USA

SOURCE: TDWI 2016 DW Modernization Report, extracted from Figure 3.

IN USERS’ OWN WORDS

Why is DW Modernization Important?

Biz Benefits of DW Modernization

0% 10% 20% 30% 40% 50% 60%

Complete views of customers and other important entities

Address new business requirements

Competitive advantages

Agile delivery of solutions, for nimble business responses

Operational efficiency of business

Fast and frequent report/analysis cycles, near real time

Business decision making, both strategic and operational

Analytics, including visualization and exploration

SOURCE: TDWI 2016 Report on DW Modernization, top half of Figure 8.

What are the top business tasks that would benefit from data warehouse modernization?

Challenges to DW ModOvercoming bad practices in data management

• Poor stewardship/governance, data quality, metadataPersonnel problems

• Inadequate staffing, skills, experiencePaying the price of modernization

• Cost of implementation, hardware/software upgradesComplexity of architecture

• Designing & managing a multi-platform systems environLimitations of existing systems

• Current environment won’t scale up to big data, ingest data fast enough

Outmoded development environment and practices• Need tools that foster speed for agility, plus reuse for productivity

SOURCE: TDWI 2016 Report on DW Modernization, extracted from Figure 9.

TO SURVIVE THE CHALLENGES,

Recognize Standard DW Modernization ScenariosAND STAFF OR SCHEDULE THEM ACCORDINGLY.

System modernization (53%)• upgrades and patches for

hardware/software servers or toolsArbitrary modernization (47%)

• based on business needs of a specific project, or urgent request for info/analysis

Non‐data modernizations (44%)• modernizing reporting, analytics,

data integration, business processesOptimization modernization (42%)

• performance tuning and similar tweaksContinuous modernization (37%)

• quarterly updates, complete views, etc.Disruptive modernization (21%)

• rip and replace platforms, tools, datasetsSOURCE: TDWI 2016 Report on DW Modernization, Figure 10.

DEFINITIONData Warehouse Environments (DWE)

• Many enterprise data warehouses (EDWs) are evolving into multi‐platform data warehouse environments (DWEs).

• Users continue to add additional standalone data platforms to their warehouse tool and platform portfolio.

• New platforms = relational DBMSs based on columns, appliances, clouds; real‐time complex event processing; Hadoop

• The new platforms don’t replace the core warehouse, because it is still the best platform for the data that goes into standards reports, dashboards, performance management, and OLAP.

• Instead, the new platforms complement the DW, because they are optimized for workloads that manage, process, and analyze new forms of big data, non‐structured data, and real‐time data.


Modernization Strategies• Most common strategy – DW Augmentation

• Add more data platforms to DWE, to complement existing core DW• For 15%, replacing the DW’s primary data platform has been a strategy• 24% modernize on per case basis; 14% don’t have a strategy


Rip & Replace DW Platforms• A third have no plans to replace their DW platform

• Apparently, many platforms work fine or were recently modernized/replaced• 9% were replaced in recent past (versus 15% on previous slide)• Almost half anticipate replacing their DW platform in the future


Evolving DW Platform Arch’s• Single‐monolith DW architectures aren’t that common and are slipping away• Simple DWE (a few platforms) is now the norm for DW systems architecture• Future: we’re trending strongly toward complex DWEs (many platforms)


Role of Hadoop in DW Mod• Hadoop in DWEs is still rare today (15%), but will increase 5x (78%) in 3 years• Hadoop usually complements a primary DW platform – 17% today, 36% in 3yrs• Hadoop rarely replaces a DW – 1% today, 6% in three years


How can Hadoop Modernize a DW?Hadoop offloads DW and extends DWE

• Captures and manages big data at scale• Data landing & staging on steroids• Repository for detailed source data• Processing for analytics & data integration• Advanced forms of algorithmic analytics

(mining, graph, predictive)• ELT push-down processing• Manages multi- and unstructured data• Inexpensive compared to cost of capacity

on average relational DW or DBMS


Top 12 Priorities for DW ModernizationThese are recommendations, requirements, or rules that can guide you.1. Embrace change.2. Make realignment with business goals your top priority.3. Make DW capacity a high priority on the technology side.4. Make analytics a priority, too.5. Don’t forget the related systems that also need mods.6. Don’t be seduced by new, shiny objects.7. Assume that you’ll need multiple modernizations.8. Know the tools and techniques of modern DWEs.9. Adjust the large‐scale architecture of your DWE.10. Reevaluate your DW platform.11. Consider Hadoop for various roles in the DWE.12. Develop plans and recurring cycles for DW modernization.

Download a Free Copy

Read the report that this presentation is based onDownload the report in a PDF:

tdwi.org/bpreportsFeel free to distribute the PDF of any TDWI Best Practices Report


Thank You

Questions/CommentsEmail:

Follow MeTwitter @

Rate This Session # with the PARTNERS Mobile App

Remember To Share Your Virtual Passes

[email protected]

prussom

182

Philip Russom - TDWI


Documents

Data Warehouse Modernization - Amazon S3...Data Warehouse Modernization in the Age of Big Data Analytics Philip Russom, Ph.D. Senior Research Director for Data Mgt, The Data Warehousing