Design and Implementation of a ServiceMonitoring Console within a Service Oriented

Architecture Framework

Arvind Krishnaa .J31508104017

Guided By

Dr. Chitra BabuHOD/CSE

SSN College of Engineering

First Review - 23rd February, 2012

OLAP2Db Interface

Figure: OLAP2Db Interface withcube delay

I Data from each data centerCAL log accumulated inPHX colo.

I Data in PHX colo stored incentralized data cube.

I Data from cube pushed ontorelational DB, by OLAP2Dbscripts.

I Data loss occurs here.

Minimizing the data loss

I Run CAL cube data availability scripts.

I CALDb records were found to be consistent withstatic CAL data.

I Data loss was isolated to partioned relationaldatabase.

I 2/3 cron jobs pushing data to the database wereidle.

I Once the cron jobs were restarted, the relationaldatabase synced the data accurately.

Minimizing the data loss

Figure: Before cron jobs wereactivated

Figure: After cron jobs wereactivated

SMC Vs TMC : Feature Comparison

Turmeric Monitoring Console

Figure: Java Class Hierarchy

Data Collection Possibilities

Figure: Using Cassandra Storage

I A real-time Cassandracluster where metrics arestored.

I Metrics reader class readsdata from each cluster.

I Offline aggregatoraccumulates the data.

I Metrics writer pushes backthe data to each node.

I Previously stored data ineach node is deleted

I No single point of failure

Data Collection Possibilities

Figure: Using Message Queue

I Implement a message queueon each server.

I This message queue willpush the metric data onto acentral database.

I This database acts as anaggregated storage provider.

I Single point of failure, atthe centralized database.

Minimizing the data loss

Using OpenTSDB

I Distributed, scalable high performance timeseries database, implemented over HBase.

I Allows simple storage and retrieval of metrics.

I Easy to graph the trends.

I Difficult to set up hardware components.

I Complicated installation procedure.

Data Collection Workflow

Figure: Activity diagram representing the data collection workflow

Implementation Schedule

References

[1]Jeffrey Dean, Sanjay Ghemawat, Google Inc., MapReduce:Simplified Data Processing on Large Clusters, In Sixth Symposiumon Operating System Design and Implementation(OSDI’04), SanFrancisco, CA, December, 2004

[2] Eben Hewitt, Cassandra: The Definitive Guide, OReillyPublications, November 2010.

[3] eBay Open Source Project, Turmeric SOA platform,http:

//www.ebayopensource.org/index.php/Turmeric/HomePage

References

[4] eBay Open Source Project, Documentation of Turmeric SOAplatform,https://www.ebayopensource.org/wiki/display/

TURMERICDOC110GA/Turmeric+Documentation+Overview

[5] eBay Open Source Project, Turmeric Source Code,http://www.github.com/ebayopensource

[6] Internal eBay documentation

[7] Google Web Toolkit,http://code.google.com/webtoolkit

[8] Apache Cassandra,http://cassandra.apache.org/