View
690
Download
0
Category
Preview:
Citation preview
© Copyright 2016 EMC Corporation. All rights reserved.© Copyright 2016 EMC Corporation. All rights reserved. 1
EXTENDING MESOS FOR STORAGE AS A GLOBAL RESOURCESTEVE WONGDEVELOPER ADVOCATE – EMC{code}
GREG MANNSOFTWARE ENGINEER – Mesosphere
© Copyright 2016 EMC Corporation. All rights reserved. 2
GOT CONTAINERS AT SCALE?YOU NEED A SCHEDULER…
© Copyright 2016 EMC Corporation. All rights reserved. 3
ASSUME 1000’S OF SERVICES
• How do you place an assortment of containers onto your servers efficiently?
© Copyright 2016 EMC Corporation. All rights reserved. 4
WHAT’S A SCHEDULER?• Manages compute nodes
– Takes reports from each node regarding their health and available capacity
• Manages incoming work requests– Accepts job requests– Based on requirements, decides which node
should fulfill each request– May attempt to optimize job placement to
maximize locality, utilization, etc.
on demand
scheduled
batch
© Copyright 2016 EMC Corporation. All rights reserved. 5
When you build a cluster, there’s no getting around it:you are building a distributed system
• Difficult to design, build and operate• Introduces complexity
Mesos takes on some of the "heavy lifting" of deploying and operating a distributed system
MESOS AS A CLUSTER MANAGER
© Copyright 2016 EMC Corporation. All rights reserved. 6
A distributed systems kernel
• Abstracts many independent nodes into a single pool• Negotiates among multiple schedulers
– Competing schedulers can suffer from resource starvation– Mesos offers resources to frameworks to ensure fairness
WHAT IS MESOS?
© Copyright 2016 EMC Corporation. All rights reserved. 7
WHAT IS MESOS?
Marathon
© Copyright 2016 EMC Corporation. All rights reserved. 8
• Where it started• Recent
enhancements• Roadmap
MESOS AND STORAGE
© Copyright 2016 EMC Corporation. All rights reserved. 9
CLUSTER NODES OFFER LOCAL RESOURCES
© Copyright 2016 EMC Corporation. All rights reserved. 10
MESOS MASTER PRESENTS OFFERS TO FRAMEWORKS
Marathon
© Copyright 2016 EMC Corporation. All rights reserved. 11
• Where it started• Recent
enhancements• Roadmap
MESOS AND STORAGE
© Copyright 2016 EMC Corporation. All rights reserved. 12
RESERVE CLUSTER NODE HOSTED VOLUME
© Copyright 2016 EMC Corporation. All rights reserved. 13
External volume mounts were added in 2015• Provides
Availability + Scale
MOUNT A VOLUME FROM EXTERNAL STORAGE
© Copyright 2016 EMC Corporation. All rights reserved. 14
• Where it started• Recent
enhancements• Roadmap
MESOS AND STORAGE
© Copyright 2016 EMC Corporation. All rights reserved. 15
NOT ALL RESOURCES ARE LOCAL
© Copyright 2016 EMC Corporation. All rights reserved. 16
AGENT FAILURE
• Agents are the source of truth for resource information
© Copyright 2016 EMC Corporation. All rights reserved. 17
AGENT FAILURE
• When an agent fails this information is lost, but so are the resources!
© Copyright 2016 EMC Corporation. All rights reserved. 18
AGENT FAILURE
• When a failed agent was using a global resource, the resource is still available!
© Copyright 2016 EMC Corporation. All rights reserved. 19
AGENT FAILURE
• The state of this resource must be stored off the agent
© Copyright 2016 EMC Corporation. All rights reserved. 20
NEED A NEW ABSTRACTION• We need a third
entity to track the use of global resources
© Copyright 2016 EMC Corporation. All rights reserved. 21
• Option #1: Integrate global resources into the Mesos offer cycle
HOW TO INTEGRATE GLOBAL RESOURCES?
© Copyright 2016 EMC Corporation. All rights reserved. 22
• Option #2: Handle global resources “out-of-band”
HOW TO INTEGRATE GLOBAL RESOURCES?
© Copyright 2016 EMC Corporation. All rights reserved. 23
Mesos was originally designed to manage compute node resources• Storage in this context meant DAS
A JIRA proposal exists to include global resources in the Mesos offer model, but this may not be necessary.• Feedback is welcome!
HOW TO INTEGRATE GLOBAL RESOURCES?
© Copyright 2016 EMC Corporation. All rights reserved. 24
HOW DO YOU TRACK WHAT STORAGE YOU HAVE?… AND WHERE IT’S IN USE?
© Copyright 2016 EMC Corporation. All rights reserved.
Create
Utilize
Retire
MESOS INTERFACES WITH STORAGE
Lifecycle management with governance
Mount managementAssignment to applications
© Copyright 2016 EMC Corporation. All rights reserved. 26
A LARGE SCALE INSTALLATION NEEDS TO ALLOW FOR MULTIPLE STORAGE PROVIDERS
What does this remind you of?
© Copyright 2016 EMC Corporation. All rights reserved. 27
MESOS = 2 LEVEL CONTAINER SCHEDULER
© Copyright 2016 EMC Corporation. All rights reserved. 28
TWO LAYER EXTERNAL STORAGE SCHEDULING
© Copyright 2016 EMC Corporation. All rights reserved. 29
STORAGE SCHEDULER = DELIVER SCALE OUTScale out is not just about adding capacity. A Storage Scheduler will let you:• Add, remove, replace storage provider platforms
– in or out of public cloud• Add, remove, replace schedulers• This is about giving you flexibility and
shifting the burden of keeping up to vendors
© Copyright 2016 EMC Corporation. All rights reserved. 30
POLLY AS A MESOS FRAMEWORK
© Copyright 2016 EMC Corporation. All rights reserved. 31
DCOS: THE DATACENTER OPERATING SYSTEM
learn more at mesosphere.com
© Copyright 2016 EMC Corporation. All rights reserved.
Data Persistence in the New Container WorldWednesday 3PM
Joshua BernsteinVP of Technology for ETD
Tobi Knaup CEO & Co-Founder of Mesosphere
Guru Session
© Copyright 2016 EMC Corporation. All rights reserved.
@EMCcode@cantbewong
mesosphere.comemccode.comcommunity.emccode.com
Come visit us at Booth #1044 or in the vLab
Questions?
© Copyright 2016 EMC Corporation. All rights reserved.© Copyright 2016 EMC Corporation. All rights reserved. 34
Recommended