Embed Size (px)
Citation preview
June 15, 2016
)How to Operationalise Real-Time Hadoop in the Cloud
WEBINAR:
Welcome! Meet Today’s Speakers
•Ted Orme• VP Technology
EMEA - Attunity
•Ian Archibald
• Pre Sales Director EMEA - Attunity
Reminder • Attendees are on mute• Q & A at the end via chat• Webinar is recorded for on-demand review
•David Tishgart• Product Marketing
Director, Cloud Solutions - Cloudera
•Hans Wieser• Advanced
Analytics Partner Lead - Microsoft
•Introductions •Attunity - How to ingest the most valuable enterprise data into Hadoop
•Cloudera - Real-life use cases •Microsoft - The scalable flexibility of Azure•Attunity Replicate Demo•Live Q&A session
Agenda
•Accelerate data delivery across enterprise and cloud•Empower rapid utilisation of data by the business•Continually optimise with intelligent insight
Attunity Corporate Overview
Over 2000 Customers in 65 CountriesFinancial Services Manufacturing / Industrials GovernmentHealth Care
Technology / Telecommunications Other IndustriesEnterprise Data ManagementOn Premise | Cloud | Across Platforms
Global Organisation
USAEMEA
APAC
Attunity Platform for Enterprise Data Management
• Accelerate data delivery
• Empower rapid utilisation of data
• Continuously improve the management of data
Attunity Replicate Attunity Compose Attunity Visibility
Universal Data Availability Data Warehouse Automation
Metrics Driven Data Management
Integrate new platforms
Automate ETL/EDW
Optimiseperformance and cost
On Premises / Cloud
Hadoop FilesRDBMS EDW SAP Mainframe
Attunity ReplicateMove the data that moves your business
Demand•Hadoop has moved out of test•Enterprise Use Case
• Closer to Production• Business impact
Enterprise + Real-timeVS
Sqoop + Batch
Attunity - Replicate• High performance connectivity to Hadoop
though native APIs for data ingest and publication
• Automated schema generation in Hcatalog• Drag & drop configuration with Click-2-Replicate
design• High-speed data load options:
• Full reload with overwrite• Insert only appends• Change Data Capture(CDC)
• In-memory data filtering and transformation • Monitoring dashboard with web-based metrics,
alerts and logfile management
Operationalising Hadoop
Hadoop
Demand•Easy Ingest + CDC
•Real-time processing•Real-time monitoring•Real-time Hadoop
•Scalable to 1000’s Applications•One Publisher – Multiple Consumers
Attunity - Replicate•Direct integration using Kafka APIs
•In-memory optimised data streaming
•Support for multi-topic and multi-partitioned data publication
•Full Load and CDC•Integrated management and monitoring via GUI
Kafka and Real-time Streaming
T1/P0
T2/P1
T3/P0
Broker 1
Attunity Replicate for Kafka - Architecture
M0 M1 M2 M3 M4 M5 M6 M7 M8
M0 M1 M2 M3 M4 M5
M0 M1 M2 M3 M4 M5 M6 M7
T1/P1
T2/P0
Broker 2
M0 M1 M2 M3 M4
M0 M1 M2 M3 M4 M5 M6
Facilitate data availability in the Cloud for BI applications BI & Analytics in CloudDatabase, Data Warehouse
Ingest data for Big Data Analytics using Hadoop in the CloudHadoop/Big Data Hybrid, All in Cloud
Simplify and accelerate database & application migration (“Lift & Shift”)
Cloud Data Migration
• Prioritise EDW archiving, off-load cold/hot data to the CloudEDW Optimisation and Migration
Hadoop in the Cloud
Attunity Replicate for Enterprise and Cloud
• Centralised control.On Prem and Cloud.• On-Prem to On-Prem• On-Prem to Cloud• Cloud to On-Prem• Cloud to Cloud
• Wider breadth of sources and targets
• Optimized, Hi-Speed data transfer • Data Compression• Parallel/Concurrent data transfer• Configurable batch sizesTargetsSources
On Premises
Cloud Platform
HadoopRDBMS
Data Warehouse
Hadoop
RDBMS
Data Warehouse
Cloudera - Real life use cases
Hadoop is driving board level initiativesDRIVE CUSTOMER
INSIGHTSIMPROVE PRODUCTS & SERVICES EFFICIENCY
LOWER BUSINESS RISKS
Modernize Data Architecture
Drivers for Hadoop in the Cloud
Where data lives
Flexible resourcing
Enterprise Acceptance
Key requirements:• Portability• Security without
compromise• Ecosystem support
The new analytics paradigm
Understand why it
happened
Change what
happens next
Determine what
happened
Make it happen
consistently
Common Use Cases for Cloudera on AzurePerpetually “on” clusters in the cloud
Common lift-and-shift cluster requirements: • High availability and disaster recovery• Cluster operational management• Cluster auto-scaling• Resource management• Security
Examples of lift-and-shift-use cases in the cloud: • HBase clusters• Kafka clusters• BI analytics• Large, multi-user clusters
TRAVEL» CUSTOMER EXPERIENCE» INTERNET OF THINGS» ADVANCED ANALYTICS
Preventative Maintenance
• To improve traveler satisfaction and safety, a European needed to reduce downtime for critical operational machines
• Cloudera Enterprise on Azure captures and correlates sensor data with transactional data to proactively assess the health of its machines and deliver necessary fixes to prevent failure
HUMAN RESOURCES» IMPROVED PRODUCTIVITY» COST REDUCTION» MACHINE LEARNING
Recruiting & Job Matching
• In a competitive market, Adecco S.A. wanted to improve the accuracy of its job placement technology
• Cloudera Enterprise on Azure powers unrivaled search and match solution to more quickly connect qualified candidates to job vacancies
• 30% reduction in time to fill vacancies• 20% reduction in job board spend within first
three months of go-live date
Microsoft - The flexibility & security of Azure
Cloud is becoming integral to business transformation
22
“71% of strategic buyers cite scalability, cost, and business agility as the most important drivers for using cloud services.“
– Gigaom Research
Leverage economies of scale and expertise
Reshape how you engage with customers
Drive new and more rapid sources of innovation
23
Privacy & ControlYou control use and access to your data.
SecurityThe confidentiality, integrity, and availability of your data is protected.
ComplianceDesigned to help meet your compliance needs.
TransparencyYou have visibility into where your data is located and how it is managed.
Our trusted cloud principlesCommitment to principles worthy of your organization’s trust
Continual evaluation, benchmarking, adoption, test & auditCompliance strategy helps customers address business objectives, industry standards, and regulations, including ongoing evaluation and adoption of emerging standards and practices.
Independent verificationRegular verification on a regular basis by third-party audit firms.
Access to audit reportsMicrosoft shares audit report findings and compliance packages with customers to help them assess Microsoft services against their own legal and regulatory requirements.
Best practicesPrescriptive guidance on securing data, apps, and infrastructure in Azure makes it easier for customers to achieve compliance.
Compliance certificationsMicrosoft maintains a team of experts focused on ensuring Microsoft meets its own compliance obligations, which helps customers meet their own compliance requirements.
The Microsoft approach to complianceTo help you assess Microsoft services against your own legal and regulatory requirements
24
Private CloudConsolidate
datacenter operationsMICROSOFT SOLUTIONS
Windows ServerSystem Center
Windows Azure Pack
Migrate to the cloud at your own pace
Microsoft Confidential 25
Consistent platform and tools | Single management console
Public Cloud “Europe”
Achieve scale, agility and lower cost
MICROSOFT SOLUTIONSMicrosoft Azure
Office 365Microsoft Dynamics CRM Online
Public Cloud “Germany”
Achieve scale, agility and lower cost with
German Data Trustee Model
MICROSOFT SOLUTIONSMicrosoft Azure Germany
Office 365 GermanyMicrosoft Dynamics CRM Online
Germany
Hybrid CloudMigrate less sensitive
dataMICROSOFT SOLUTIONS
Risk Assessment and Data Governance services
Why a German cloud?
26
1Gartner, Market Trends: Cloud-Based Security Services Market, Worldwide, 2014, October 2013
Data privacy regulations in the European Union (EU) are among the strictest and strongest in the world.
Regional regulation
German privacy regulations are outlined and enforced through Federal Act and individual state laws.
Local regulation
Customers want to know where their customer data resides, who has access to it, and which country's laws govern that access.
Data residency
According to Gartner1, privacy requirements have severely impacted the deployment of all forms of cloud-based services in the region
27
Complete cloud
Hybrid options
Commitment to compliance
Commitment to innovation
Microsoft Cloud differentiators
Replicate Product Architecture and Demo
Attunity Replicate Architecture
Transfer
TransformFilterBatch
CDC Incremental
In-Memory
File Channel
Batch
Hadoop
Files
RDBMS
Data Warehouse
Mainframe
Cloud
On-prem
Cloud
On-prem
Hadoop
Files
RDBMS
Data Warehouse
Kafka
Persistent Store
Heterogeneous – Broad support for sources and targets
RDBMS
OracleSQL ServerDB2 LUWDB2 iSeriesDB2 z/OSMySQLSybase ASEInformix
Data Warehouse
ExadataTeradataNetezzaVerticaActian VectorActian Matrix
HortonworksClouderaMapRPivotal
Hadoop
IMS/DBSQL M/PEnscribeRMSVSAM
Legacy
AWS RDSSalesforce
Cloud
RDBMS
OracleSQL ServerDB2 LUWMySQLPostgreSQLSybase ASEInformix
Data Warehouse
ExadataTeradataNetezzaVerticaPivotal DB (Greenplum)Pivotal HAWQActian VectorActian MatrixSybase IQ
HortonworksClouderaMapRPivotal
Hadoop
MongoDB
NoSQL
AWS RDS/Redshift/EC2Google Cloud SQLGoogle Cloud DataprocAzure SQL Data WarehouseAzure SQL Database
Cloud
Effective: 12/10/2015
Kafka
Message Broker
targets
sources
