Upload
jayush-luniya
View
121
Download
1
Embed Size (px)
Citation preview
2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Speakers
Apache Ambari PMC
Staff Engineer @ Hortonworks
Apache Ambari PMC
Principal Engineer @ Hortonworks
Jaimin Jetly Jayush Luniya
4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
Ambari Overview
v.Next Architecture
Management Packs
Multi Everything
Upgrades
UI Enhancements
Other Improvements
7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache Ambari
Ambari ServerProvision | Manage | Monitor
Ambari REST API
Ambari Stacks
Apache Ambari is a 100% open sourceplatform for provisioning, managing andmonitoring Apache Hadoop clusters
Other Operations Tools
Am
bar
i Age
nts
10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
LimitationsOne-One Relationships
Monolithic Single monolithic stack
Entire cluster upgrade
Ambari upgrades for refreshing stack definitions
Ambari Cluster
Cluster Stack
StackService Version
ClusterService
Instance
HostComponent
Instance
AmbariHosting Platform
12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Vision
Refactor at service level
Replace monolithic stacks
Decouple Ambari core release
Independent release cycles
Modular upgrades
3rd party onboarding platform
Multiple Clusters
Multiple Stacks
Multiple Services
Multiple Service Versions
Multiple Host Components
Multiple Hosting Platforms
Modular Multi Everything
13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
New Concepts
Deployment artifact
Individual ownership of services
Independent release cycles
Examples:o HDFS-3.0.0-packlet
o SPARK-2.0.0-packlet
Distribution artifact
Composed of multiple packlets
Modular release & upgrades
Examples:o HDP-3.0.0-mpack
o HDF-3.1.0-mpack
Packlets Management Packs (Mpacks)
Mpack Repository
Catalog of management packs
Compatibility between mpacks
Examples:o Hortonworks Mpack Repository
14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
New Concepts
Runtime concept
Logical grouping of services
Examples:o Core Service Group
o Stream Service Group
Defined in management packs
Prescribes how to deploy cluster
Examples:o Shared v/s Dedicated Zookeeper
Service Groups Deployment Models
15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Cluster2
Cluster1
Ambari Hosting Platform
v.Next Architecture
ZK1 HDFS YARN HBASE
Core SG
NIFIZK2 KAFKA
Stream SG
Mpack Repository
HDP3.0.0
HDF3.1.0HDF3.2.0
Ambari
BROKER-1
BROKER-2
BROKER-3
HBASE2
Yapp SG
YarnApp1.0.0
Yarn Hosting Platform
HMaster
RServer-1
RServer-2
16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Cluster2
Cluster1
Ambari Hosting Platform
v.Next Architecture
ZK1 HDFS YARN HBASE
Core SG
NIFIZK2 KAFKA
Stream SG
Mpack Repository
HDP3.0.0
HDF3.1.0HDF3.2.0
Ambari
BROKER-1
BROKER-2
BROKER-3
HBASE2
Yapp SG
YarnApp1.0.0
Yarn Hosting Platform
HMaster
RServer-1
RServer-2
18 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Management Packs
Decouple Ambari Core release
Replacement for monolithic stacks
Composition of verified combination of packlets
Support dynamic installation
Mpacks released and upgraded independently
Release Management
19 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Management Packs
Support multiple mpack repositories
Search and discover management packs
Compatibility between management packs
Recommendations for mpack bundles
Add-on Mpacks First class support for 3rd party services
Mpacks provide a third party onboarding platform
Independently upgradable services
Mpack Repositories
21 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Multi Everything
Logical grouping of services
Service Groups
Cluster v/s Service Group
Cluster – Uniquely identifies collection of physical hosts
Service Group – Logical grouping of correlated services
Mpack v/s Service Group
Mpack – Delivery concept
Service Group – Runtime concept
Inheritance to recommend service dependencies
Service Group Inheritance
22 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Multi Everything
Defined in management packs
Enriched user deployment experience
Prescriptions for deploying a clustero Service groups to create
o Service dependencies
Examples:
Deployment Models
Core SG
Stream SG
ZK HDFS
KAFKA
Core SG
Stream SG
ZK1 HDFS
ZK2 KAFKASAM SAM
Shared Zookeeper Dedicated Zookeeper
23 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Multi Everything
Talk: Apache Hadoop Yarn: Present and Future - Vinod Vavilapalli
Long Lived Yarn Services - YARN-896
First class support for Yarn Services - YARN-4692 (Yarn Assemblies)
Multiple Hosting Platformso Ambari Hosting Platform
o Yarn Hosting Platform
Ambari should enable first class support for Yarn services
Mpacks provide delivery platform for Yarn Services & Yarn Assemblies
Yarn Hosted Services
25 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Upgrades
Modular upgrades
Ability to upgrade subset of services in a cluster
Patch services without affecting other parts of the cluster
Performance improvements
Goals
Mpack Upgrades Upgrade packs defined at mpack level
Upgrade subset of services in a cluster that belong to a mpack
Download new version of a mpack and kickoff an upgrade
26 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Upgrades
Upgrade mechanism for hotfix releases
Selectively patch components without restarting other services
Purely Version Definition File (VDF) driven; no configuration changes
Does not require a new management pack; uses same upgrade pack
Applied patches can be reverted; no finalization step
Patch Upgrades
Service Upgrades
VDF driven if no configurations changes needed
Mpack driven for more complicated service upgrades
No revert capability (similar to current cluster upgrades)
28 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
UI Enhancements
Cosmetic changeso New style for all UI elements
o New design for all pages
Consistent experienceo New bootstrap skin
Utilize available screen spaceo Expandable Navigation bar
New Look & Feel
33 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
UI Enhancements
Ambari REST API explorer
Swagger Integration
Interactive console to execute REST APIs
REST API documentation
36 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Other Improvements
Topic: Tuning Apache Ambari Performance for Big Data at Scale
Speakers: Alejandro, Aravindan
Performance: Ambari Scale Testing
Upgrade: Host Ordered Upgrades
AMS: Anomaly Detection
AMS: Extensible - Support other storage choices
AMS: Scalability
Talk
38 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
mpacksmodular packlets
service groups multi
yarn hostedadd-ons
performance
39 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache JIRAs
AMBARI-14714 – Multi Everything ArchitectureUmbrella Apache EPIC
AMBARI-19621 – Mpack Based Operations Model
AMBARI-20463 – Multi Service Instances
AMBARI-20465 – Multi Host Component Instances
AMBARI-17353 – Yarn Hosted Services
AMBARI-12556 – Patch Upgrades
AMBARI-20466 – Multi Cluster
AMBARI-20435 - Swagger Integration
Other Apache EPICs
Release Timelines Features will be incrementally delivered in Apache Ambari 3.x releases.