JavaOne 2008 - TS-6048 - Complex Event Processing at Orbitz

COMPLEX EVENT PROCESSING AT ORBITZ WORLDWIDE

Matt O’Keefe, Senior ArchitectDoug Barth, Technical Lead

TS-6048

2008 JavaOneSM Conference | java.sun.com/javaone | 2

How to make pager duty suck less


$10.8 Billion in Gross Bookings in 2007


Dozens of Apps, Hundreds of VMs and Thousands of Services


The Need for Abstraction

Webapp

Travel Business Services

Switching Services

ab

stra

ctio

n

Transaction Services

Suppliers


Complex Event Processing is…

…about sensing and responding to threats and opportunities in real time.


The Big Picture

Event ProcessorsMonitored Apps

OperationsCenter

Graphite

JMX, ssh

ERMA

SNMP

Portal


ERMA

Extremely Reusable Monitoring API


The Monitor Interface


Using EventMonitors

protected void doValidate(RequestContext context, Object formObject, Errors errors) throws Exception {

super.doValidate(context, formObject, errors);

if (errors.hasErrors()) { EventMonitor validationMonitor = new EventMonitor("ValidationErrors"); validationMonitor.set("errors", errors.getAllErrors()); validationMonitor.fire(); }}


The CompositeMonitor Interface


Using TransactionMonitors

public Hotel findHotelById(Long id) { TransactionMonitor monitor = new TransactionMonitor(getClass(), "findHotelById"); monitor.set(“id”, id); try { Hotel hotel = em.find(Hotel.class, id); monitor.succeeded(); return hotel; } catch (RuntimeException e) { monitor.failedDueTo(e); throw e; } finally { monitor.done(); }}


TransactionMonitorTemplate

public void cancelBooking(final Long id) { TransactionMonitorTemplate.INSTANCE.doInMonitor( getClass(), "cancelBooking", new TransactionMonitorCallback() { public Object doInMonitor(TransactionMonitor monitor) { monitor.set("id", id); Booking booking = em.find(Booking.class, id); if (booking != null) { em.remove(booking); } return null; } });}


Interceptors

public interface HandlerInterceptor {

boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception;

void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception;

void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception;}


Listenerspublic interface FlowExecutionListener { public void requestSubmitted(RequestContext context); public void requestProcessed(RequestContext context); public void sessionStarting(RequestContext context, FlowDefinition definition, MutableAttributeMap input); public void sessionCreated(RequestContext context, FlowSession session); public void sessionStarted(RequestContext context, FlowSession session); public void eventSignaled(RequestContext context, Event event); public void stateEntering(RequestContext context, StateDefinition state) throws EnterStateVetoException; public void stateEntered(RequestContext context, StateDefinition previousState, StateDefinition state); public void paused(RequestContext context, ViewSelection selectedView); public void resumed(RequestContext context); public void sessionEnding(RequestContext context, FlowSession session, MutableAttributeMap output); public void sessionEnded(RequestContext context, FlowSession session, AttributeMap output); public void exceptionThrown(RequestContext context, FlowExecutionException exception);}


Annotations

@Monitoredpublic interface UnderpantsGnomes { void collectUnderpants(); void ?(); void profit(); }


Spring/AspectJ

<aop:config> <aop:aspect id="transactionMonitorActionAspect" ref="transactionMonitorActionAdvice"> <aop:pointcut id="transactionMonitorActionPointcut“ expression="target(org.springframework.webflow.execution.Action) and args(context)"/> <aop:around pointcut-ref="transactionMonitorActionPointcut“ method="invoke"/> </aop:aspect></aop:config>

<bean id="transactionMonitorActionAdvice" class= "c.o.webframework.aop.aspectj.TransactionMonitorActionAdvice"/>


Joining Monitors - Across Threads

TransactionMonitor monitor = new TransactionMonitor("Profit"); ErmaCallable callable = new ErmaCallable(new CollectUnderpantsCallable()); try { Future future = executorService.submit(callable); future.get(FUNDING_DURATION, TimeUnit.MILLISECONDS); monitor.addChildMonitor(callable.getMonitor()); monitor.succeeded(); } catch (TimeoutException e) { monitor.failedDueTo(e); throw e; } finally { monitor.done(); }


Joining Monitors - Distributed Servicespublic DispatcherResponse doFilter(DispatcherRequest request, Object resource, FilterChain chain) throws Throwable { TransactionMonitor monitor = new TransactionMonitor(getName(request)); MonitoringEngine mEngine = MonitoringEngine.getInstance(); Map inheritableAttributes = mEngine.getInheritableAttributes(); Map serializableAttributes = mEngine.makeSerializable(inheritableAttributes); HashMap ermaAttributes = new HashMap(serializableAttributes); request.addParameter(ERMA_FILTER_PARAM_KEY, OLCSerializer.serialize(ermaAttributes));

DispatcherResponse response = chain.doFilter(request, resource);

… return response;}


Distributed Services (cont.)

public DispatcherResponse doFilter(DispatcherRequest request, Object resource, FilterChain chain) throws Throwable { …

DispatcherResponse response = chain.doFilter(request, resource);

Object responseBlob = response.getParameter(ERMA_FILTER_PARAM_KEY); Monitor responseMonitor = (Monitor) OLCSerializer.deserialize((byte[]) responseBlob); monitor.addChildMonitor(responseMonitor); Throwable throwable = response.getThrowable(); if (throwable == null) { monitor.succeeded(); } else { monitor.failedDueTo(throwable); } return response;}


The MonitorProcessor Interface

public interface MonitorProcessor { public void startup();

public void shutdown();

public void monitorCreated(Monitor monitor);

public void monitorStarted(Monitor monitor);

public void process(Monitor monitor);}


A MonitorProcessorAdaptor Examplepublic class ResultCodeAnnotatingMonitorProcessor extends MonitorProcessorAdapter {

public void process(Monitor monitor) { if (monitor.hasAttribute("failureThrowable")) { Throwable t = (Throwable) monitor.get("failureThrowable"); while (t.getCause() != null) { t = t.getCause(); } monitor.set("resultCode", t.getClass().getName()); } else { monitor.set("resultCode", "success"); } }}


LoggingMonitorProcessor Output

process: c.o.monitoring.api.monitor.TransactionMonitor -> blockedCount = 9 -> blockedTime = 17 -> cpuTimeMillis = 1470.0 -> createdAt = Sun Mar 09 16:13:17 CDT 2008 -> endTime = Sun Mar 09 16:13:30 CDT 2008 -> failed = false -> hostname = oberon -> latency = 13180 -> name = httpIn_/shop/airsearch/search/air/pageView_airResults -> remoteIpAddress = 10.222.186.147 -> sessionId = D417C9CC585F82E1 -> threadId = 2a20ec -> validationFailure = false -> vmid = wl -> waitedCount = 10 -> waitedTime = 10414


EventPatternLoggingMonitorProcessor Outputwl|httpIn_/shop/airsearch/search/air/pageView_airResults|13180 wl|RoundTripAirSearchAction.resolveRoundTripAirLocations|136 wl|jiniOut_LocationFinderService_findAirports|84 tbs-shop-13.31|jiniIn_LocationFinderService_findAirports|31 tbs-shop-13.31|jiniOut_AirportLookupService_findLocationByIATACode|9 market-8.4|jiniIn_AirportLookupService_findLocationByIATACode|3 tbs-shop-13.31|jiniOut_AirportLookupService_findLocationByIATACode|15 market-8.4|jiniIn_AirportLookupService_findLocationByIATACode|10 wl|jiniOut_LocationFinderService_findAirports|48 tbs-shop-13.31|jiniIn_LocationFinderService_findAirports|16 tbs-shop-13.31|jiniOut_AirportLookupService_findLocationByIATACode|14 market-8.4|jiniIn_AirportLookupService_findLocationByIATACode|10 wl|AirSearchExecuteAction.search|10422 wl|jiniOut_ShopService_createResultSet|9798 tbs-shop-13.31|jiniIn_ShopService_createResultSet|9601 tbs-shop-13.31|com.orbitz.tbs.host.shop.ShopServiceImpl.createResultSet.AIR|9361 tbs-shop-13.31|com.orbitz.tbs.spi.SpiShopService.createResultSet.AIR|9333 tbs-shop-13.31|jiniOut_LowFareSearchService_execute|9175 air-search-7.2.1|jiniIn_LowFareSearchService_execute|9094 air-search-7.2.1|LowFareSearchRequest|9048 air-search-7.2.1|com.orbitz.afo.lib.search.service.LowFareSearchServiceImpl|9038 air-search-7.2.1|com.orbitz.afo.lib.search.service.LowFareSearchServiceImpl.execute|9037 wl|jiniOut_ShopService_viewResultSet|607 tbs-shop-13.31|jiniIn_ShopService_viewResultSet|486 wl|pageView_airResults wl|jsp.render.air200.page|2475


EventPatternLoggingMonitorProcessor Output

wl|AirSearchExecuteAction.search|NoSearchResultsAvailableException wl|jiniOut_ShopService_createResultSet|NoSearchResultsAvailableException tbs-shop|jiniIn_ShopService_createResultSet|NoSearchResultsAvailableException tbs-shop|c.o.t.h.s.ShopServiceImpl.createResultSet.AIR|NoSearchResultsAvailableException tbs-shop|c.o.t.s.SpiShopService.createResultSet.AIR|NoSearchResultsAvailableException tbs-shop|jiniOut_LowFareSearchService_execute|SearchSolutionNotFoundException air-search|jiniIn_LowFareSearchService_execute|SearchSolutionNotFoundException air-search|LowFareSearchRequest|SearchSolutionNotFoundException

Follow the trail of Exceptions… don’t bother the on-call engineers for the higher layers… save time by narrowing your log search query!


Event Processing


EventFlow


Expression Language

!failed

latency > 1000 && failed

totalFailed/total

sum(int(failed))

stddev(latency)

indexof(name, ‘jini’) == 0

strftime(“%Y%m%d, windowTimestamp)


Event Processing DemoStreamBase Studio


Custom Functions

public class HelloWorld { public static String sayHello(String name) { return "Hello, “ + name; }}


Custom Operators

public void processTuple(int inputPortId, Tuple tuple) throws StreamBaseException {

String name = tuple.getString("name"); Tuple output = outputSchema.createTuple(); output.setField("message", "Hello, "+name); sendOutput(OUTPUT_PORT, output);}


Visualization


SNMP


Graphite


Graphite RESTful URLstarget=tbs-shop.all.jiniIn.ShopService.createResultSet#all.count


Graphite RESTful URLs - Wildcardstarget=tbs-shop.all.jiniIn.ShopService.*.count


Graphite RESTful URLs - Functionstarget=derivative(tbs-shop.all.jiniIn.ShopService.*.count)


Graphite RESTful URLs – Pie Chartstarget=cpuTime:790&target=waitedTime:4043&target=blockedTime:0&target=other:31


Graphite RESTful URLs – Raw CSV Datatarget=tbs-shop.all.jiniIn.ShopService.*.count&rawData=true

85 86 82 84 87 83 79 82 79 82 94 91 81

0 0 0 0 0 0 0 0 0 0 0 0 0

None None None None None None None None None None None None None

0 0 0 0 0 0 0 1 0 0 0 0 0


85 86 82 84 87 83 79 81 79 82 94 91 81






89 81 85 83 89 84 78 81 76 84 90 92 84




0 0 0 0 0 0 0 0 0 0 0 0 0


Graphite CLI - Dashboards


Monitoring Portal


RSS


Command Line Stream Discovery

[mokeefe@egcep03 ~]$ sbc list input-streamsinput-stream DeleteFromThresholdsinput-stream GarbageCollectorStatsinput-stream ListActiveAlarmsInputinput-stream MemoryPoolStatsinput-stream MemoryStatsinput-stream MonitorInputinput-stream MonitoringEngineManager_lifecycleinput-stream ReloadLog4jinput-stream ThreadStatsinput-stream ThresholdInput…


Command Line Schema Discovery[mokeefe@egcep03 ~]$ sbc describe MonitorInput<stream name=“…" schema=“…" uuid="..."> <schema name="schema:MonitorInput" uuid=“…"> <field name="name" size="256" type="string"/> <field name="vmid" size="128" type="string"/> <field name="hostname" size="128" type="string"/> <field name="threadId" size="32" type="string"/> <field name="createdAt" type="timestamp"/> <field name="endTime" type="timestamp"/> <field name="latency" type="int"/> <field name="cpuTimeMillis" type="int"/> <field name="failureThrowable" size="16384" type="string"/> <field name="failed" type="bool"/> <field name="resultCode" size="128" type="string"/> <field name="sessionId" size="32" type="string"/> <field name="locale" size="24" type="string"/> <field name="remoteIpAddress" size="15" type="string"/> <field name="posCode" size="4" type="string"/> </schema></stream>


Command Line Queries

[mokeefe@egcep03 ~]$ sbc dequeue MonitorInput --where "sessionId == '570EA1FABEE015B9' and vmid='tbs-shop-13.31'"jiniOut_AirportLookupService_findLocationByIATACode,tbs-shop-13.31,egbsoneg01.prod.o.com,67484c,2008-03-10 15:40:42.489-0500,2008-03-10 15:40:42.489-0500,2008-03-10 15:40:42.502-0500,13,null,null,null,false,null,null,GBP,570EA1FABEE015B9,English (United Kingdom),10.235.1.119,null,EBUKjiniIn_LocationFinderService_findAirports,tbs-shop-13.31,egbsoneg01.prod.o.com,67484c,2008-03-10 15:40:42.480-0500,2008-03-10 15:40:42.480-0500,2008-03-10 15:40:42.503-0500,23,null,null,null,false,null,success,GBP,570EA1FABEE015B9,English (United Kingdom),10.235.1.119,null,EBUK…


Event Pattern Monitoring


Event Pattern Monitoring (cont)


Event Pattern Monitoring Work In Progress

OrbitzWorldwide

Real-time Clickstream Analysisw/Correlation


Recap

Event ProcessorsMonitored Apps

OperationsCenter

Graphite

JMX, ssh

ERMA

SNMP

Portal


Future Directions

Orbitz Worldwide will open source ERMAOrbitz Worldwide will open source Graphite Orbitz Worldwide is considering open sourcing of

orbitz-lib-streambaseEsper integration


For More Information

The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems, David Luckham, Addison Wesley Professional, May 2002, ISBN: 0201727897 http://www.complexevents.comERMA and Graphite: Watch http://forum.complexevents.com/ for an announcementhttp://corp.orbitz.com/careers/ StreamBase: http://www.streambase.com/Esper: http://esper.codehaus.org/


Matt O’Keefe, Senior ArchitectDoug Barth, Technical Lead

TS-6048


Appendix: Filters public void doFilter(ServletRequest servletRequest, ServletResponse

servletResponse, FilterChain filterChain) throws … { TransactionMonitor monitor = new TransactionMonitor(getClass(), "doFilter"); try { filterChain.doFilter(servletRequest, servletResponse); monitor.succeeded(); } catch (IOException e) { monitor.failedDueTo(e); throw e; } catch (ServletException e) { monitor.failedDueTo(e); throw e; } catch (RuntimeException e) { monitor.failedDueTo(e); throw e; } finally { monitor.done(); } }


Appendix: Interceptors

public class ERMAInterceptor extends HandlerInterceptorAdapter {

public boolean preHandle(…) throws Exception {

TransactionMonitor monitor = new TransactionMonitor("httpIn"); monitor.setInheritable(POSCODE, getPosCode()); monitor.setInheritable(LOCALE, getLocale()); monitor.setInheritable(CURRENCY, getCurrency()); monitor.setInheritable(CHANNEL, getChannel()); monitor.setInheritable(SESSION_ID, getSessionId()); monitor.setInheritable(IP_ADDR, getClientAddress()); }


Appendix: Interceptors (cont.)

public void postHandle(…) throws Exception { View view = modelAndView.getView(); if (view == null) { String viewName = modelAndView.getViewName(); EventMonitor monitor = new EventMonitor("pageView_" + viewName); monitor.fire(); } else if (RedirectView.class.isAssignableFrom( view.getClass())) { String redirectUrl = extractDispatcherPath(request); EventMonitor monitor = new EventMonitor("redirect_" + redirectUrl); monitor.fire(); }}


Appendix: Interceptors (cont.)

public void afterCompletion(…, Exception exception) throws Exception {

TransactionMonitor httpInMonitor = (TransactionMonitor) MonitoringEngine.getInstance(). getCompositeMonitorNamed("httpIn"); if (exception == null) { httpInMonitor.succeeded(); } else { httpInMonitor.failedDueTo(exception); } httpInMonitor.set(Monitor.NAME, constructNewMonitorName(httpInMonitor)); httpInMonitor.done();}


Appendix: Listeners

public void stateEntering(RequestContext context, StateDefinition nextState) throws EnterStateVetoException {

EventMonitor monitor = new EventMonitor("flowExecution.stateEntering"); StateDefinition currentState = context.getCurrentState(); monitor.set("currentStateId", currentState.getId()); monitor.set("nextStateId", nextState.getId()); monitor.fire();}

Technology

JavaOne 2008 - TS-6048 - Complex Event Processing at Orbitz