Case Study

Log Monitoring Using Hadoop

www.hexaware.com | marketing@hexaware.com

The customer is a leading profitability management and optimiza-tion software solutions company in the emerging pricing and revenue optimization sector. Founded in 1985, they offer big data solutions to optimize sales, pricing, and quoting, rebates and revenue management across more than 30 industries. With 27 years of data science experience, the client can unlock buying patterns and preferences within transaction data to reveal various opportunities.

Client Overview

Business Challenges• The client’s flagship pricing solution — Pricing & Revenue Optimization Solution Suite (PSS) — is used by major players in the airline and hospitality industry. As a consequence, downtime for the application means potential loss of business for their clients• The client’s system was experiencing frequent system errors, warnings, and performance degradation, resulting in application outages that went undetected until they were reported• Consequently, support technicians could only respond to issues in a reactive manner, and were unable to prevent issues from occurring or proactively communicate issues to their end customers• Further, the turnaround time to identify and resolve issues was too long, as the technicians had to crawl through verbose system logs to identify and trace specific warnings and errors• The log data growth was rapid and required scalable storage. A solution platform compatible with such an environment was urgently needed.

Solution OverviewThe main objective was to create a scalable approach for monitoring important events from log files. We devised a Hadoop-based solution that would enable the analysis of Task Manager and Tomcat log files. The architecture framework below highlights the data flow and key components of the solution using Cloudera Hadoop distribution and Splunk.

Workflow for the analysis:• PPSS generates Task Manager logs• Flume picks up these logs and transfers them to a Hadoop cluster• Splunk picks these new files up from the Hadoop cluster for analysis• Splunk detects the new Task Manager log files added to the cluster for analysis• Analyst can now search for specific events• Analyst can create reports based on events in the logs such as Duration of tasks, frequency of task execution and failures• Analyst can use these insights to predict application outages.

www.hexaware.com | marketing@hexaware.com

About HexawareHexaware is one of the leading, global providers of IT, Application, Infrastructure, BPO and Digital services. Our business philosophy of Shrink IT, Grow Digital

allows customers to significantly shrink commodity IT spend while partnering with them to embrace digitalization. The Company focuses on key domains such

as Banking, Financial Services, Capital Market, Healthcare, Insurance, Manufacturing, Retail, Education, Telecom, Travel, Transportation and Logistics.

Hexaware focuses on delivering business results and leveraging technology solutions by specializing in services like; Application support, development and

maintenance, Enterprise Solutions, Human Capital Management, Business Intelligence & Analytics, Digital Assurance (Testing), Infrastructure Management

Services, Digital and Business Process Services. Founded in 1990, Hexaware has a well-established global delivery model armed with proprietary tools and

methodologies, skilled human capital and SEI CMMI-Level 5 certification. For additional information logon to: www.hexaware.com

India Headquarters152, Sector – 3Millennium Business Park‘A’ Block, TTC Industrial AreaMahape, Navi Mumbai – 400 710Tel : +91-22-67919595Fax : +91-22-67919500

EU HeadquartersLevel 19, 40 Bank Street,Canary Wharf,London - E14 5NRTel: +44-020-77154100Fax: +44-020-77154101

APAC Headquarters180 Cecil Street,#11-02, Bangkok Bank Building,Singapore 069546Tel : +65-63253020Fax : +65-6222728

NA HeadquartersMetro 101, Suite 600,101 WoodAvenue South, Iselin,New Jersey - 08830Tel: +001-609-409-6950Fax: +001-609-409-6910

Technology EnvironmentHardware• PCs• VPN Access to Client Servers

Software• Splunk 5.0• Splunk Hadoop Connect App• Cloudera Distribution (CDH4.0)• Putty• Winscp• Firefox or IE• Adobe Flash Player11 and above.

Benefits:• The end result was a solution scalable to any amount of data since it was tested with Hadoop (Cloudera Distribution of Apache Hadoop)• The analysis was useful in finding errors and warnings within system logs in an automated fashion and predicting application outages• The user friendly charts allowed Support Technicians to identify and analyze most common/critical errors• Error tracking generated by Pricing Solution Suite on customer deployed solutions prompted alert to customers on new or otherwise hidden issues and also fixed these up for them.

Client Logs

SplunkReporting

HDFS

Framework:

PIG

AnalyticalReporting

Splunk Hadoop App

Real-timereporting

Explore