73

Dev309 from asgard to zuul - netflix oss-final

Embed Size (px)

DESCRIPTION

Overview of Netflix Source core Infrastructure, Tooling and Availability components, fast and easy deployment via Docker containers

Citation preview

Page 1: Dev309  from asgard to zuul - netflix oss-final
Page 2: Dev309  from asgard to zuul - netflix oss-final
Page 3: Dev309  from asgard to zuul - netflix oss-final

Growth

Page 4: Dev309  from asgard to zuul - netflix oss-final

Data Center

MySQL

Middle Tier

Web UI / Front End API

Page 5: Dev309  from asgard to zuul - netflix oss-final

Regional Load Balancers Regional Load Balancers

Page 6: Dev309  from asgard to zuul - netflix oss-final

API or Web Calls

memcached

Cassandra

Web service

S3 bucket

Page 7: Dev309  from asgard to zuul - netflix oss-final

We’re here to help you get to global scale… Apache Licensed Cloud Native OSS

http://netflix.github.com

Page 8: Dev309  from asgard to zuul - netflix oss-final
Page 9: Dev309  from asgard to zuul - netflix oss-final
Page 10: Dev309  from asgard to zuul - netflix oss-final
Page 11: Dev309  from asgard to zuul - netflix oss-final
Page 12: Dev309  from asgard to zuul - netflix oss-final

Micro service Implementation

Web App Front End

(REST

services)

Hys

trix

Call “Auth Service”

Ribbon REST client with Eureka

Fallback Implementation Karyon

App Service (auth-service)

Eureka Server(s)

Page 13: Dev309  from asgard to zuul - netflix oss-final

Karyon Eureka

Ribbon

Hystrix

Page 14: Dev309  from asgard to zuul - netflix oss-final
Page 15: Dev309  from asgard to zuul - netflix oss-final
Page 16: Dev309  from asgard to zuul - netflix oss-final
Page 17: Dev309  from asgard to zuul - netflix oss-final
Page 18: Dev309  from asgard to zuul - netflix oss-final

Speed

Hardware Will Fail Slowly Changing Large Scale

Everything Is Broken Rapid Change Large Scale

Everything Works

Slowly Changing Small Scale

Hardware Will Fail Rapid Change Small Scale

Sca

le

Page 19: Dev309  from asgard to zuul - netflix oss-final
Page 20: Dev309  from asgard to zuul - netflix oss-final
Page 21: Dev309  from asgard to zuul - netflix oss-final
Page 22: Dev309  from asgard to zuul - netflix oss-final

Regional Load Balancers Regional Load Balancers

Page 23: Dev309  from asgard to zuul - netflix oss-final
Page 24: Dev309  from asgard to zuul - netflix oss-final
Page 25: Dev309  from asgard to zuul - netflix oss-final
Page 26: Dev309  from asgard to zuul - netflix oss-final

Micro service Implementation

Web App Front End

(REST

services)

Hys

trix

Call “Auth Service”

Ribbon REST client with Eureka

Fallback Implementation Karyon

App Service (auth-service)

Eureka Server(s)

Page 27: Dev309  from asgard to zuul - netflix oss-final
Page 28: Dev309  from asgard to zuul - netflix oss-final
Page 29: Dev309  from asgard to zuul - netflix oss-final
Page 30: Dev309  from asgard to zuul - netflix oss-final
Page 31: Dev309  from asgard to zuul - netflix oss-final
Page 32: Dev309  from asgard to zuul - netflix oss-final

Client

A p a c h e

Eureka (Service Registry)

Server (Karyon)

Apache

Tomcat

Ribbon Load

Balancing

Eureka Integration

Metrics (Servo)

Bootstrapping (Governator)

Metrics (Servo)

Admin Console HTTP

Eureka Integration

Registration

Fetch Registry

H T T P

C L I E N T

Hy s t r i x

EVCache

Page 33: Dev309  from asgard to zuul - netflix oss-final

Eureka (Service Registry)

Client (Ribbon 2.0) Server (Karyon)

Ribbon Transport

Load Balancing

Eureka Integration

Metrics (Servo)

Bootstrapping (Governator)

Metrics (Servo)

Admin Console

HTTP

Eureka Integration

Registration

Fetch Registry

Ribbon

Hystrix

EVCache

RxNet ty

RxNetty

UDP

TCP WebSockets

SSE

Page 34: Dev309  from asgard to zuul - netflix oss-final

                           @ArchaiusBootstrap                              @KaryonBootstrap(name  =  "hello-­‐netflix-­‐oss")                              @Modules(include  =  {HelloNossApp.KaryonJerseyModuleImpl.class,  KaryonWebAdminModule.class,  KaryonEurekaModule.class})                              public  final  class  HelloNossApp    {                                        public  static  class  KaryonJerseyModuleImpl  extends  KaryonJerseyModule  {                                                      protected  void  configure()  {                                                                super.configure();                                                                bind(AuthenticationService.class).to(AuthenticationServiceImpl.class);                                                      }                                                      public  int  serverPort()  {  return  8888;  }                                                      public  int  shutdownPort()  {  return  8899;  }                                                      public  void  configureInterceptors(GovernatorHttpInterceptorSupport<ByteBuf,  ByteBuf>  interceptorSupport)  {                                                                interceptorSupport.forUri("/hello").interceptIn(AuthInterceptor.class);                                                      }                                      }                              }  

 

Page 35: Dev309  from asgard to zuul - netflix oss-final

interface  

recommendationsByUserId @Var "userId"

@Http(method  =  HttpMethod.GET,  uriTemplate="/users/{userId}/recommendations")  @Hystrix(fallbackHandler  =  RecommendationFallbackHandler.class)  

@EvCache(name  =  "movie-­‐rec",  appName  =  "movieService",  cacheKeyTemplate  =  "{userId}")  

MovieService  movieService  =  Ribbon.from(MovieService.class);  Observable<Recommendations>  result  =        movieService.recommendationsByUserId(“user1”)          .toObservable();  

Page 36: Dev309  from asgard to zuul - netflix oss-final

createHttpResourceGroup "movieService"

newRequestTemplate "recommendationsByUserId""GET"

"/users/{userId}/recommendations"

requestBuilder"userId" “user1”

Page 37: Dev309  from asgard to zuul - netflix oss-final
Page 38: Dev309  from asgard to zuul - netflix oss-final
Page 39: Dev309  from asgard to zuul - netflix oss-final
Page 40: Dev309  from asgard to zuul - netflix oss-final

AZ1 AZ2 AZ3

Regional Load Balancers

Zuul – Traffic Shaping/Routing

Data Data Data

Geo-located

Chaos Kong

Chaos Kong

AZ1 AZ2 AZ3

Regional Load Balancers

Zuul – Traffic Shaping/Routing

Data Data Data

Customer Device

Page 41: Dev309  from asgard to zuul - netflix oss-final
Page 42: Dev309  from asgard to zuul - netflix oss-final
Page 43: Dev309  from asgard to zuul - netflix oss-final

A/B Test Engine

Personalization Engine User Info Movie

Metadata Movie

Ratings Similar Movies

API

Reviews

Page 44: Dev309  from asgard to zuul - netflix oss-final

A/B Test Engine

Personalization Engine User Info Movie

Metadata Movie

Ratings Similar Movies Reviews

User Info

API

Page 45: Dev309  from asgard to zuul - netflix oss-final

A/B Test Engine

Personalization Engine

Movie Metadata

Movie Ratings

Similar Movies Reviews

User Info

API

Page 46: Dev309  from asgard to zuul - netflix oss-final

User Info

API

A/B Test Engine

Personalization Engine

Movie Metadata

Movie Ratings

Similar Movies Reviews

Page 47: Dev309  from asgard to zuul - netflix oss-final

A/B Test Engine

Personalization Engine

Movie Metadata

Movie Ratings

Similar Movies Reviews

User Info

API

Page 48: Dev309  from asgard to zuul - netflix oss-final
Page 49: Dev309  from asgard to zuul - netflix oss-final

User Info

API

A/B Test Engine

Personalization Engine

Movie Metadata

Movie Ratings

Similar Movies Reviews

Page 50: Dev309  from asgard to zuul - netflix oss-final

A/B Test Engine

Personalization Engine

Movie Metadata

Movie Ratings

Similar Movies Reviews

User Info

API

Page 51: Dev309  from asgard to zuul - netflix oss-final

Personalization Engine

User Info

Movie Metadata

Movie Ratings

Similar Movies

API

Reviews A/B Test Engine

Page 52: Dev309  from asgard to zuul - netflix oss-final

Personalization Engine

User Info

Movie Metadata

Movie Ratings

Similar Movies

API

Reviews A/B Test Engine

Fallback

Page 53: Dev309  from asgard to zuul - netflix oss-final

Personalization Engine

User Info

Movie Metadata

Movie Ratings

Similar Movies

API

Reviews A/B Test Engine

Fallback

Page 54: Dev309  from asgard to zuul - netflix oss-final
Page 55: Dev309  from asgard to zuul - netflix oss-final
Page 56: Dev309  from asgard to zuul - netflix oss-final
Page 57: Dev309  from asgard to zuul - netflix oss-final
Page 58: Dev309  from asgard to zuul - netflix oss-final

Sketchy Scumblr

+

Page 59: Dev309  from asgard to zuul - netflix oss-final
Page 60: Dev309  from asgard to zuul - netflix oss-final

‒ ‒ ‒ ‒ 

‒ ‒ ‒ 

Page 61: Dev309  from asgard to zuul - netflix oss-final
Page 62: Dev309  from asgard to zuul - netflix oss-final

http://netflix.github.com

Page 63: Dev309  from asgard to zuul - netflix oss-final

https://github.com/Answers4AWS/netflixoss-ansible/wiki/AMIs-for-NetflixOSS

http://answersforaws.com/resources/netflixoss/cloudformation/

Page 64: Dev309  from asgard to zuul - netflix oss-final

https://github.com/Netflix-Skunkworks/zerotodocker

https://hub.docker.com/u/netflixoss/

Page 65: Dev309  from asgard to zuul - netflix oss-final
Page 66: Dev309  from asgard to zuul - netflix oss-final

Mac OS X

Virtual Box

Ubuntu 14.04

single kernel

Con

tain

er #

1 Fi

lesy

stem

+ p

roce

ss

Eur

eka

Con

tain

er

Zuul

C

onta

iner

Ano

ther

C

onta

iner

...

Page 67: Dev309  from asgard to zuul - netflix oss-final
Page 68: Dev309  from asgard to zuul - netflix oss-final
Page 69: Dev309  from asgard to zuul - netflix oss-final
Page 70: Dev309  from asgard to zuul - netflix oss-final

Talk Time Title

Page 71: Dev309  from asgard to zuul - netflix oss-final

http://netflix.github.com

Page 72: Dev309  from asgard to zuul - netflix oss-final

@NetflixOSS

http://netflix.github.com

Page 73: Dev309  from asgard to zuul - netflix oss-final

Please give us your feedback on this presentation