26
Qlik Data Integration Data Revolution Tour 2019 Ian Crosland

Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

  • Upload
    others

  • View
    13

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

Qlik Data IntegrationData Revolution Tour 2019

Ian Crosland

Page 2: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

2

Qlik’s 3rd Generation BI Vision

Democratization

of Data

All data, governed, and

universally accessible

Augmented

Intelligence

Raising Data Literacy

through technology

Embedded

Everywhere

From the edge

to the C-Suite

Enterprise-Class Technology

Hybrid, multi-cloud, and edge continuum

Page 3: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

3

Qlik’s 3rd Generation BI Vision

Democratization

of Data

All data, governed, and

universally accessibleQlik Data Catalyst®

Attunity®

A Division of Qlik

Page 4: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

4

Qlik’s 3rd Generation BI Vision

Democratization

of Data

All data, governed, and

universally accessibleQlik Data Catalyst®

Attunity®

A Division of Qlik

• From raw to analytics-ready, faster

• Accessible and actionable

• Reuse & collaborate, securely

Qlik Sense®

Page 5: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

5

ANALYTICS

Qlik Data and Analytics Supply Chain

Speed and efficiency from raw data to shared insights

Page 6: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

6

DataOps CatalogProfiling, Lineage,

Governance, Collaboration,

and Provisioning

Change Data CaptureReal-time streaming, replication,

Efficient Cloud Delivery

Data Warehouse AutomationETL generation, Self-Service Marts,

Cloud Optimization Data PreparationEnrichment, transformation,

protection of derivative

datasets

Data Lake PipelinesOrchestration and automation

Analytics-Ready Datasets

Qlik Data Integration Platform

• DataOps for Analytics

Qlik Data

Catalyst

Attunity(Qlik)

Page 7: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

7

Shifting Data Architectures

$205B by 2020Cloud & Multi-cloudOn Premises

2X DATA Every 2 YearsCloud Warehouses

+ Data LakesData Warehouse

82%Adopting Real-TimeStreamingBatch

Sources: WW Public IT Cloud Services Revenue in 2020, IDC. The Digital Universe of Opportunities, IDC. Cloudview Survey, IDC.

Page 8: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

8

Growing Demands

Automated, Agile

Data Delivery

Governed,

Enterprise-Ready

Data

Real-Time Data

SPEED SKILLS TRUST

Page 9: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

9

CODE DATAINFRASTRUCTURETOOLS

PEOPLE TECHNOLOGYPROCESS

• Emerging discipline to build and

manage efficient data pipelines

• Applies principles of DevOps to

data integration

• Improves collaboration between

data managers and consumers

Source: Gartner Innovation Insight for DataOps, December 2018

Long cycles

Batch processing

Brittle ETL scripts

DATA MGRSDATA USERS

Fast cycles

Real-time data

Resilience

DataOps: Accelerating Democratization of Data

Page 10: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

10

DataOps for Analytics

17

AUTOMATE

REFINEMENTCONTINUOUS

DELIVERY

CATALOG

& GOVERN

1 Real Time Data for

Faster, Better Insights 2 Agile Data Delivery 3 Trusted, Enterprise

Ready Data

Qlik Data Catalyst®Attunity Replicate® Attunity Compose®

Page 11: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

11

Continuous Delivery & Refinement

MAINFRAME

SAP

Other…

Databases

PaaS DB

DatabaseReplication

Data Warehouse Automation

SAAS

APPS

FILES

DATA WAREHOUSE

RDBMS

StreamingData PipelineAutomation

Design & Manage

Generate Deliver Refine

change stream

to cloud, lakes

for analyticuse

Data Lake Automation

Other…

Data Warehouses

AzureSQL DW

Redshift

Other…

Cloud & Data Lakes

Page 12: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

12

Database Replication

Enterprise-wide Data

Replication &

Distribution for

Multiple Analytics Use

Cases

Mainframe to Oracle

Data Consolidation for

Customer 360 Initiative

Oracle Transaction

Streaming and for

Multi-Channel

Customer

Engagement

Features

• Agentless, real-time data capture and

delivery

• Automated target table creation,

instantiation and mappings

• Supports RDBMS, file systems and

streaming targets, on-premise & Cloud

Benefits

• Ensures consistency and ease of use

across all major platforms (sources and

targets)

• No impact to production systems

• Easy to manage, automated, low TCO

Page 13: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

13

Data Warehouse Automation

SQL Server Data

Warehouse

Automation Cut ETL

scripting time 96%,

accelerated change

process 6x

High-Scale, Multi-

sourced Data

Consolidation into

Azure Data Lake and

Azure SQL DW

Real-Time

Consolidation from

SAP, Oracle to

Snowflake for DW

Modernization

Features

• Automates Data Model/Data

Warehouse design, build, maintenance

and documentation

• Automated table creation, instantiation

and mappings

• Continuous, real-time data replication

Benefits

• Reduce risk, save time and money

– no scripting or coding required

• DW now built in hours, changed in

minutes

• Future-proof for new requirements and

new platforms

Page 14: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

14

Data Lake Automation

Mainframe Data

Streaming and

Refinement for

Microservices

Enablement

Data Lake

Consolidation for

Marketing/Sales

Analytics and

Engagement

Multi-sourced Data

Integration and

Refinement for

Operational Analytics

Features

• Automates creation of tables, organizes

data structures and tracks lineage to

deliver a Managed Data Lake

• Instantiates data, maps source and

target, keeps schemas in sync

• Continuous, real-time data replication

Benefits

• Quickly and easily create high-scale

data pipelines

• Eliminate risky, expensive and complex

custom coding

• Close the “last mile” by provisioning

analytics-ready data in real-time

Page 15: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

15

Catalog and GovernQlik Data Catalyst

MainframeFilesRDBMS NoSQLBig Data Cloud

BI/Reporting

Data Science

Applications

Cloud Platforms

Business

Processes

• Available

• Searchable

• Reliable

• Relatable

• Secure

• Role Based

Qlik Data Catalyst

Page 16: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

16

Qlik Data Catalyst Benefits

DATA ENGINEER

Brings new assets into the

data catalog

• Quickly add new datasets

• Understand the exact

content and condition of

onboarded data

• Easily filter, transform or

combine incoming data

DATA STEWARD

Makes sure naming conventions

and security standards are enforced

• Improve consumer‘s ability to

find the right dataset

• Systematically enforce and

monitor data policies and usage

• Ensure sensitive or personal

data is automatically masked

DATA CONSUMER

Find and utilize the data they

need to gain new insights

• Eliminate reliance on data

experts or IT

• Reuse and build on previously

created assets

• Easily export, share or

automatically publish data sets

Page 17: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

17

Qlik Data Catalyst

Features

• Smart, Integrated Data Catalog of all

technical, operational & business metadata

• Data access and obfuscation capabilities to

guarantee that data is always protected

and secure

• Shop for Data with role-based security

• Quickly publish data to Qlik Sense, other BI

tools, and cloud storage platforms

Benefits

• Improved data literacy as users can find,

understand and generate insights faster

• Removes potential failure points, tightens

security, and reduces risk

• Removes the IT bottleneck through a

governed, self-service approach

• Reduced data onboarding from 8 weeks to as little as 2 hours

• Cost to onboard data 90% less vs. conventional ETL

• Cost of Environment 30x cheaper than least expensive RDBMS alternative

• Able to utilize cloud storage by carefully tracking and labeling PII data

Page 18: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

18

Qlik Data Catalyst now supports QVD filesSee, Profile, Shop For, and Share QVDs

Manage and Re-use Your QVD Investments

• Full, searchable catalog of QVDs created in Qlik Sense and QlikView

• Full profile, data tagging of data elements in QVDs with ability to secure and obfuscate data when sending out of the Qlik environment

• Ability to prepare and integrate QVDs with non-Qlik assets

• Shopping and Publication of QVDs to Qlik, data science tools, and competitive BI tools

Publish

Data Science Tools

Within Qlik

Competitors

Page 19: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

19

DataOps for Analytics

17

AUTOMATE

REFINEMENTCONTINUOUS

DELIVERY

CATALOG

& GOVERN

1 Real Time Data for

Faster, Better Insights 2 Agile Data Delivery 3 Trusted, Enterprise

Ready Data

Qlik Data Catalyst®Attunity Replicate® Attunity Compose®

Page 20: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

20

By 2021, organizations that offer

a curated catalog of internal and

external data to diverse users

will realize twice the business

value from their data and

analytics investments than

those that do not.

Source: Gartner - Augmented Data Catalogs: Now an Enterprise Must-Have for Data and Analytics Leaders - September 2019

2X

Page 21: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

21

DBaaS

STORAGE

HADOOP

STREAMING

DWaaS

OTHER DWaaS

SPARK

Multi-Cloud & Partner Alignment

All

ADLS, BLOB

HDInsight

Event Hubs

Azure SQL DW

Snowflake

Databricks

DB All

GCS

DataProc

Cloud Pub/Sub

BigQuery

RDS (All)

S3

EMR

Kinesis

Redshift

Snowflake

Databricks

* Planned

for 2020

*

Snowflake*

This document and Qlik‘s strategy and possible future developments are subject to change and may be changed by Qlik at any time for any reason without notice. This document is provided without a warranty of any kind. The document may not be copied, distributed, or otherwise shared with any third party.

Page 22: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

22

Production

Systems:

Fast Growing

Platforms: (SPARK)(KAFKA)

Major Cloud

Platforms:

Hadoop

Evolution:

Comprehensive Partner Ecosystem

Page 23: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

23

What’s Next – Roadmap

• Continue to lead in universal access – new sources, targets

- CDC for Salesforce and MongoDB now available! Additional NoSQL and SaaS apps coming soon.

• Dynamic, hybrid and multi-cloud scaling via Kubernetes

• Continued integration of Attunity with Qlik Data Catalyst and the rest of the Qlik portfolio

This document and Qlik‘s strategy and possible future developments are subject to change and may be changed by Qlik at any time for any reason without notice. This document is provided without a warranty of any kind. The document may not be copied, distributed, or otherwise shared with any third party.

Page 24: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

24

Qlik Data Integration – Guiding Principles

Independent Real-TimeUniversal

Agile &

AutomatedSelf-ServiceGoverned

Page 25: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization

25

Next Step

Read the Attunity and

Gartner report:

“Enabling DataOps

for Analytics”

@ www.qlik.com

Page 26: Qlik Data IntegrationChange Data Capture Real-time streaming, replication, Efficient Cloud Delivery Data Warehouse Automation ETL generation, Self-Service Marts, Cloud Optimization