Object, measure thyself

Object, Measure Thyself

Greg Opaczewski – Orbitz Worldwide

Michael Ducy – BMC Software

Open Source

• ERMA Project :

http://launchpad.net/erma

• Graphite Project : http://launchpad.net/graphite

Complex Environment

$10.8 Billion in Gross Bookings in 2007

Myths of Instrumentation

• No Time For Instrumentation

• No Value ($) in Instrumentation

• Instrumentation Causes Bugs

Myth: No Time For Instrumentation

ERMAExtremely Reusable Monitoring API

TransactionMonitor monitor = new TransactionMonitor(“HotelService.purchase”);

try {response = hotelSupplier.reserve(hotel);monitor.succeeded();

} catch (ServiceException e) {monitor.failedDueTo(e);throw e;

} finally {monitor.done();

}

ERMA

Self-Instrumentation by:

• Hooks – Interceptors and Listeners

• Abstraction – Abstract the details away from developers

• AOP – Aspect Oriented Programming

Frameworks - Hooks

• Spring Framework

Frameworks - Abstraction

Self-Instrumentation by:

• Aspect Oriented Programming (AOP)

<aop:config>

<aop:aspect id="transactionMonitorActionAspect"

ref="transactionMonitorActionAdvice">

<aop:pointcut id="transactionMonitorActionPointcut“

expression="target(org.springframework.webflow.execution.Action)

and args(context)"/>

<aop:around pointcut-ref="transactionMonitorActionPointcut“

method="invoke"/>

</aop:aspect>

</aop:config>

Myth: No Time For Instrumentation

Myth: No Value ($) in Instrumentation

Event Aggregation

Event Aggregation

Storage and Visualization: Graphite

Graphite

Graphite

Graphite Demo

Value to the Business

• Fixing Production Problems Fast

• Capacity Planning

• Business Product teams rely on ERMA data

Myth: No Value ($) in Instrumentation

Myth: Instrumentation Causes Bugs

Avoid Boilerplate

@Monitored

public interface HotelService {

void purchase(Itinerary itinerary);

void cancel(Itinerary itinerary);

}

Avoid Boilerplate

public interface HotelService {

@Monitored(includeArguments = true)

void purchase(Itinerary itinerary);

void cancel(Itinerary itinerary);

}

Uncovers Bugs

• Allows you to base line across builds

• MASF and SPC

• Event Pattern Monitoring

Base Lining

• Compare present performance vs. historical performance

• Validate testing via theoretical models

MASF and SPC

Need for Abstractionabstraction

Webapp

Travel Business Services

Switching Services

Transaction Services

Suppliers

Event Pattern Monitoring

wl|httpIn.shop.search.air.redirect_searchFailure

wl|AirSearchExecuteAction.search

wl|com.orbitz.ojf.OJFClient.getInternal

wl|jiniOut_ShopService_createResultSet

tbs-shop|jiniIn_ShopService_createResultSet

tbs-shop|jiniOut_LowFareSearchService_execute

air-search|jiniIn_LowFareSearchService_execute

air-search|com.orbitz.afo.lib.SearchFilter

air-search|com.orbitz.afo.lib.LowFareSearchServiceImpl.execute

air-search|jiniOut_AirportLookupService_findLocationByIATACode

market|jiniIn_LocationService|DbPoolExhaustedException

Myth: Instrumentation Causes Bugs

Final Thought

Performance monitoring is easy when the objects practically measure themselves.

Thank You

• Special thanks to:– Fellow Co-Authors – Matthew O’Keefe and

Stephen Mullins– Neil Gunther – Mentoring and Candid Editorial

Review– Lead Graphite Developer – Chris Davis

Websites

• ERMA Project :

http://launchpad.net/erma

• Graphite Project : http://launchpad.net/graphite

?michael@ducy.org

gopaczewski@orbitz.com