Embed Size (px)
Citation preview
2
1001 Deployment Scenarios
▪ Many different deployment scenarios • Yarn • Mesos • Docker/Kubernetes • Standalone • Etc.
3
Different Usage Patterns
▪ Few long running vs. many short running jobs • Overhead of starting a Flink cluster
▪ Job isolation vs. sharing resources • Allowing to define per job credentials & secrets • Efficient resource utilization by sharing
4
Job & Session Mode
▪ Job mode • Dedicated cluster for a single job
▪ Session mode • Shared cluster for multiple jobs • Resources can be shared across jobs
5
Flink’s Current State
6
As-Is State (Standalone)
7
Standalone Flink Cluster
Client (2) Submit Job JobManager
TaskManager
(3) Deploy Tasks
(1) RegisterTaskManager
TaskManager
As-Is State (YARN)
8
YARNResourceManager
YARN Cluster
Client
(1) Submit YARN App.(FLINK)
Application Master
JobManager
TaskManager
TaskManager
TaskManager
(2) Spawn Application Master
(4) Start TaskManagers
(8) Deploy Tasks
(3) Poll status
(6) All TaskManager started
(5) Register
(7) Submit Job
Problems
▪ No clear separation of concerns ▪ No dynamic resource allocation ▪ No heterogeneous resources ▪ Not well suited for containerized
execution
9
Flink’s New Distributed Architecture
10
Flink Improvement Proposal 6
▪ Introduce generic building blocks
▪ Compose blocks for different scenarios
▪ Mainly driven by:
11Flip-6 design document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077
The Building Blocks
12
• ClusterManager-specific • May live across jobs • Manages available Containers/TaskManagers • Used to acquire / release resources
ResourceManager
TaskManagerJobManager
• Registers at ResourceManager • Gets tasks from one or more
JobManagers
• Single job only, started per job • Thinks in terms of "task slots" • Deploys and monitors job/task execution
Dispatcher
• Lives across jobs • Touch-point for job submissions • Spawns JobManagers • May spawn ResourceManager
The Building Blocks
13
ResourceManager
(3) Request slotsTaskManager
JobManager
(4) Start TaskManager
(5) Register
(7) Deploy Tasks
Dispatcher
Client
(1) Submit Job
(2) Start JobManager
(6) Offer slots
Building Flink-on-YARN
14
YARNResourceManager
YARN Cluster
YARN Cluster Client
(1) Submit YARN App.(JobGraph / JARs)
Application MasterFlink-YARN
ResourceManager
JobManager TaskManager
TaskManager
TaskManager
(2) Spawn Application Master
(4) StartTaskManagers
(6) Deploy Tasks
(5) Register(3) Request slots
Differences to old YARN mode
▪ JARs in classpath of all components
▪ Dynamic resources allocation
▪ No two phase job submission
15
Building Flink-on-Mesos
16
Mesos Master
Mesos Cluster
Mesos Cluster Client
(1) HTTP POST JobGraph/Jars
Flink Master ProcessFlink Mesos
ResourceManager
JobManager TaskManager
TaskManager
TaskManager
(3) Start Process (and supervise)
(5) StartTaskManagers
(7) DeployTasks
(6) Register(4) Request slots
Flink Mesos Dispatcher
(2) Allocate containerfor Flink master
Master Container
Flink Master Process
Building Flink-on-Docker/K8S
17
Flink-ContainerResourceManager
JobManager
Program Runner
(2) Run & Start
Worker Container
TaskManager
Worker Container
TaskManager
Worker Container
TaskManager
(3) Register
(1) Container framework starts Master & Worker Containers
(4) Deploy Tasks
Containerized Execution
▪ Single dedicated Resource- and JobManager container and multiple TaskManager containers
▪ Generalization • Start N containers • Use leader election to determine JobManager role; remainder
TaskManager role
▪ Enabling auto-scaling groups by rescaling job to fill all available slots
18
Multi Job Sessions
19
Building Standalone
20Standalone Cluster
Flink Cluster Client
(2) Submit JobGraph/Jars
Flink Master ProcessStandalone
ResourceManager
TaskManager
TaskManager
TaskManager
(5) Deploy Tasks
(1) Register(4) Request slots
JobManager JobManager
Dispatcher
(3) Start JobManager
Standby Master Process Standby Master Process
YARN Session
ApplicationMasterFlink-YARN
ResourceManager(5) Request
slots
JobManager (A)
JobManager (B)
Dispatcher
(4) StartJobMngr
YARNResourceManager
YARN Cluster
Client
(1) Submit YARN App.(FLINK – session)
TaskManager
TaskManager
TaskManager
(2) Spawn Application Master
(6) StartTaskManagers
(8, 12) Deploy Tasks
(7) Register(3) Submit Job A (11) Request
slots
(10) StartJobMngr
(9) SubmitJob B
21
Multi Job Sessions
▪ Dispatcher spawns for each job a dedicated JobManager
▪ Jobs run under session user credentials
▪ ResourceManager holds on to resources • Reuse of allocated resources • Quicker response for successive jobs
22
Miscellaneous
▪ Resource profiles • Specify CPU & memory requirements for individual
operators • ResourceManager allocates containers according to
resource profiles
▪ New RPC abstraction similar to Akka’s typed actors • Properly defined interface eases development • No longer locked in on Akka
23
Conclusion
24
Conclusion
▪ Different cluster environments have different deployment paradigms
▪ Support for “Job” as well as “Session” mode in various environments necessary
▪ Flip-6 architecture provides necessary flexibility to achieve both
25
26
Thank you!@stsffap @ApacheFlink @dataArtisans
We are hiring!
data-artisans.com/careers
