Microservice Resiliency - qconsp.com › sp2017 › system › files › presentation-slides ›...

MicroserviceResiliency

FromFronttoBackEnd

QConSãoPaulo,2017

LanceBall,SeniorSoftwareEngineer,RedHat

WhoamI?

SeniorSoftwareEngineer,RedHat

WhoamI?

SeniorSoftwareEngineer,RedHat

WhoamI?

SeniorSoftwareEngineer,RedHat

WhoamI?

SeniorSoftwareEngineer,RedHat

µService

“ softwareapplicationsassuitesofindependentlydeployableservices

https://martinfowler.com/articles/microservices.html

µService

“ softwareapplicationsassuitesofindependentlydeployableservices

https://martinfowler.com/articles/microservices.html

Butwhatdoesthismean?!

What'sinanapplication?

Stuff

Monolithicapplication

Scalingamonolith

Microserviceapplication

Scaledmicroservices

Wait...isn'tthistheUXtrack?

ServiceLifecycle

ServiceLifecycle

Clientmakesarequest

ServiceLifecycle

ClientmakesarequestServerprovidesaresponse

ServiceLifecycle

ClientmakesarequestServerprovidesaresponseOftenusingHTTPtransport

ServiceLifecycle

ClientmakesarequestServerprovidesaresponseOftenusingHTTPtransportOftenwithJSONdataformat

IntheBrowser

IntheBrowser

XMLHttpRequest

IntheBrowser

XMLHttpRequestJQuery

IntheBrowser

XMLHttpRequestJQueryAJAX

MicroserviceRequests

(simplified)

OperationalComplexity

MicroservicesVisualized

https://twitter.com/ThePracticalDev/status/845285541528719360

Problems

Problems

Timeouts

Problems

Timeouts

Networksaturation

Problems

Timeouts

Networksaturation

Programmererror

Problems

Timeouts

Networksaturation

Programmererror

Diskfailure

Problems

Timeouts

Networksaturation

Programmererror

Diskfailure

Transitivedependencies

Cascadingfailures

Howtodealwithallthis

Howtodealwithallthis

Limitsinglepointsoffailure

Howtodealwithallthis

Limitsinglepointsoffailure

Shedloadwhenpossible

Howtodealwithallthis

Limitsinglepointsoffailure

Shedloadwhenpossible

Providefallbackbehavior

Howtodealwithallthis

Limitsinglepointsoffailure

Shedloadwhenpossible

Providefallbackbehavior

Optimizefailurediscovery

CircuitBreaker

CircuitBreaker

Callsthatcouldfailarewrapped

CircuitBreaker

Callsthatcouldfailarewrapped

Circuitopensatafailurethreshold

CircuitBreaker

Callsthatcouldfailarewrapped

Circuitopensatafailurethreshold

Furthercallsshortcircuitforawhile

CircuitBreaker

Callsthatcouldfailarewrapped

Circuitopensatafailurethreshold

Furthercallsshortcircuitforawhile

Later,circuittriesagainandtripsimmediatelyifthereisfailure

CircuitState

Asyncoperationthatcouldfail

//UseJQuerytogetcartinfo$.get('http://mystore.com/cart').then((json)=>{//updatetheUIwithJSONdata}).catch((e)=>{//oopssomethingwentwrongconsole.error(e);})

Asyncoperationthatcouldfail

//UseJQuerytogetcartinfo$.get('http://mystore.com/cart').then((json)=>{//updatetheUIwithJSONdata}).catch((e)=>{//oopssomethingwentwrongconsole.error(e);})

Shedloadwhenpossible

Aside-Promsies

//UseJQuerytogetcartinfo$.get('http://mystore.com/cart').then((json)=>{//updatetheUIwithJSONdata}).catch((e)=>{//oopssomethingwentwrongconsole.error(e);})

CircuitBreakerExample

//UseJQuery'sajaxwrapperandcircuitbreaker//defaultsforfailurethreshold,timing,etc.constcircuit=circuitBreaker($.get);

circuit.fire('http://nodejs.org/dist/index.json').then((json)=>{//updatetheUIwithJSONdata})//onfailure,justlogtoconsole.catch(console.error);

CircuitBreakerExample

//UseJQuery'sajaxwrapperandcircuitbreaker//defaultsforfailurethreshold,timing,etc.constcircuit=circuitBreaker($.get);

circuit.fire('http://nodejs.org/dist/index.json').then((json)=>{//updatetheUIwithJSONdata})//onfailure,justlogtoconsole.catch(console.error);

CircuitBreakerExample

//UseJQuery'sajaxwrapperandcircuitbreaker//defaultsforfailurethreshold,timing,etc.constcircuit=circuitBreaker($.get);

circuit.fire('http://nodejs.org/dist/index.json').then((json)=>{//updatetheUIwithJSONdata})//onfailure,justlogtoconsole.catch(console.error);

Promisesvs.Callbacks

//WrapNode.js'fs.readFileasapromise-returningfunctionconstreadFile=circuitBreaker.promisify(fs.readFile);

constcircuit=circuitBreaker(readFile,options);

circuit.fire('./package.json','utf-8').then(console.log).catch(console.error);

CircuitBreakerFallback

Providesdefaultbehaviorincaseoferror

circuit.fallback((file)=>`Sorry,Ican'tread${file}`);

//Fallbackfunctionisstillasuccesscasecircuit.fire('./package.jsob').then((data)=>console.log(`package.json:\n${data}`)).catch((err)=>console.error(`ERR:${err}`));

CircuitBreakerFallback

Providesdefaultbehaviorincaseoferror

circuit.fallback((file)=>`Sorry,Ican'tread${file}`);

//Fallbackfunctionisstillasuccesscasecircuit.fire('./package.jsob').then((data)=>console.log(`package.json:\n${data}`)).catch((err)=>console.error(`ERR:${err}`));

Caching

Alwaysreturnsthesamevalue

constnow=circuitBreaker(Date,{cache:true});

Caching

Alwaysreturnsthesamevalue

constnow=circuitBreaker(Date,{cache:true});

circuit.fire().then(console.log);//MonApr10201712:10:26GMT-0400(EDT)circuit.fire().then(console.log);//MonApr10201712:10:26GMT-0400(EDT)circuit.fire().then(console.log);//MonApr10201712:10:26GMT-0400(EDT)

Whenisthisuseful?

Frequenthits,infrequentchangeE.g.username

constusername=circuitBreaker(fetchUsername,{cache:true});

//periodicallyclearthecachesetInterval(_=>username.clearCache(),5000);

Events

Circuitbreakersareeventemitters

//UpdatetheUIspecificallyfortimeouterrorscircuit.on('timeout',()=>$(element).prepend(mkNode(`${route}istakingtoolongtorespond.`)));

`fire``reject``timeout``success``failure`

`open``close``halfOpen``fallback``snapshot`

Events

Circuitbreakersareeventemitters

//UpdatetheUIspecificallyfortimeouterrorscircuit.on('timeout',()=>$(element).prepend(mkNode(`${route}istakingtoolongtorespond.`)));

Status

//createa10secwindowwith10bucketsof1secconstcircuit=circuitBreaker(asyncFunc,{rollingCountTimeout:10000,rollingCountBuckets:10});

//statusiscalculatedeverytimestatusisaccessedconststatus=circuit.status

//printtheentirestatisticalwindowconsole.log(status.window);

//printtherollingstatsconsole.log(status.stats);

Status

//createa10secwindowwith10bucketsof1secconstcircuit=circuitBreaker(asyncFunc,{rollingCountTimeout:10000,rollingCountBuckets:10});

//statusiscalculatedeverytimestatusisaccessedconststatus=circuit.status

//printtheentirestatisticalwindowconsole.log(status.window);

//printtherollingstatsconsole.log(status.stats);

Status

//printtherollingstatsconsole.log(status.stats);

//{failures:3,//fallbacks:4,//successes:44,//rejects:4,//fires:48,//timeouts:1,//cacheHits:0,//cacheMisses:0}

Dashboard

http://techblog.netflix.com/2012/12/hystrix-dashboard-and-turbine.html

Demo

Obrigado&Questions

http://lanceball.com/qcon-saopaulo-2017/https://github.com/lance/qcon-saopaulo-2017Twitter-@lanceballGitHub-@lance