OSDConf 2014 “In love with Open Source : Past,

Present and Future…“

~/Piyush

http://piyush.me

In real open source, you have the right to control your own destiny. ~Linus Torvalds

OSDConf 2014 : http://piyush.me 2

Mostly Everyone Uses/Loves Open Source Directly or Indirectly

• Linux

• Android

• Mozilla Firefox

• LibreOffice/OpenOffice

• Apache HTTP server

• Asterisk (PBX)

• VLC media player

• Wordpress

OSDConf 2014 : http://piyush.me 3

4 C’s of Open Source

• Community • Collaboration • Collective Ownership of intellectual property

rights – Distributed Credit

• Continuous, public process

– Software releases part of the process – Open peer review – Public Bug Tracking – Testing

OSDConf 2014 : http://piyush.me 4

My entry to open source…

• I started with using Linux

• then started using a classic LAMP stack as Dev

• And then in sea of Infrastructure (WebOps) it was open source all around

• Now playing with BigData Stack again … its OSS … making our life easier !

OSDConf 2014 : http://piyush.me 5

Journey….

• 2003-2004 -> Linux User back there in my college days

• 2005 -> became first Red Hat Certified Engineer of college

• 2006-2008 -> NexTag Journey as QE

• 2008-2013 -> MakeMyTrip, Website Operations Team

• 2013-till now -> BI / Lead Data Platform Team

OSDConf 2014 : http://piyush.me 6

2006 -> 2008

• Started as QE – Quality Engineer at NexTag.com (27 months) part of QA-Infrastructure team having worked on managing QA Servers, Testing of NexTag Application (Search Indexes, Cache System, Monitoring Applications) and Automation

• Performance Testing of OS, Apache HTTP server, Tomcat + Every New Release

• Managing F5 BigIP LTM in QA & Google Mini (Search Engine)

OSDConf 2014 : http://piyush.me 7

Key Learning's • COMMUNITY : IRC #centos channel freenode

– Inspired by awesome z00dax : Karanbir Singh

• HABIT : IRC #noc channel for production updates @ NexTag Infra • Learned the art of

– ./configure && make && make install

• Love for BASH #!/bin/bash • SOA – Service Oriented Architecture • Performance Testing - Jmeter/Replay ( Load / Stress ) • Automated Testing – Sahi / Selenium • Code Coverage - Cobertura • Importance of managing Infrastructure like your web applications

– Continuous updates / improvements – Automation – Monitoring

OSDConf 2014 : http://piyush.me 8

2008 -> till date

• I played various roles in Web Operations (which demanded multidisciplinary skills )

– System Engineer

– Security Operations

– Lead Infrastructure

– ISM – Information Security Manager

– Lead DataPlatformTeam

OSDConf 2014 : http://piyush.me 9

hacker@makemytrip

DDoS Protection

Firewall - ASA

IPS – Intrusion Detection / Prevention System

WAF – Web Application Firewall

Load Balancer

Web Server

Engine / SOA / APIs

RDBMS - Transactional Database

NoSQL

Caching Technologies

Virtualization / Cloud Stack

Email / DNS / AUTH / Config Management / CI/Deployment

CDN – content delivery network

Log Management - SIEM

Typical Web Infra Stack

Obviously including Hardware, Network & Power OSDConf 2014 : http://piyush.me 10

MMT OSS powered Tech Stack • Programming Languages

– JAVA / PHP / Python

• Operating System – CentOS 5/6 • Caching proxy

– ATS – Apache Traffic Server

• Webserver – Apache HTTPD – Nginx

• Application Server – Apache Tomcat / Jetty – JBoss

• RDBMS – MySQL

OSDConf 2014 : http://piyush.me 11

• NoSQL – SOLR / ElasticSearch (Lucene) – Search – Cassandra – Couchbase

• Monitoring – Graphite / Diamond – Nagios – NOCMATE-Rx

• Lmetric – Logging Infrastructure – Syslog – Logstash – ElasticSearch – Kibana – Web Dashboard

OSDConf 2014 : http://piyush.me 12

• Cheetah – Selenium , TestNG , Pyvnc2swf – Jenkins

• Neerikshan – Observium (network monitoring)

• Version Control – GIT, SVN , CVS – GitLab – Git Management

• Continuous Integration – Jenkins

• Performance Testing – Jmeter / ab / soapui

• IDE – Eclipse / vim

OSDConf 2014 : http://piyush.me 13

• Virtualization – XenServer

• Configuration Management - Puppet • Application Frameworks

– Spring (JAVA) – Laravel (PHP) – Django / Flask (Python)

• Knowledge Management – Twiki / Wordpress

• CMS – Drupal

• Security – OSSIM / AlienVault – OWASP ZAP

• …..and many more ….

OSDConf 2014 : http://piyush.me 14

Cheetah

• cheetah : Test automation and business service monitoring tool

– uses Selenium and Jenkins(Hudson) integration for Real Time monitoring of website with nice Reporting and Alerting

– record sessions using Pyvnc2swf

OSDConf 2014 : http://piyush.me 15

Few other hacks

• BizEye SynC Tool

• MyClient

• Image Thumbnail Processing

– ImageMagick

• MyIdeas

– Drupal

OSDConf 2014 : http://piyush.me 16

LBManager • Managing LTM and GTM using Django /

PyControl

• Single interface for managing the LoadBalancers hosted in multi-colocations / IDCs

OSDConf 2014 : http://piyush.me 17

Graphite/Diamond & NOCMATE-Rx • Monitoring for Operational Excellence

– Easy interface to collect data / Metrics – Trends, Dashboards, Alerting , Reporting – User specific/LOB specific dashboards

• Desi principle: If it moves, we track it :) • “You cannot manage what you do not measure and

to measure you need to monitor” • 5 Categories of Metrics

– System | Network | Security | Business | Application

• Poller’s – DBPoller, SNMP, Script etc.

OSDConf 2014 : http://piyush.me 18

Graphite : Trends / Analysis / Dashboards

OSDConf 2014 : http://piyush.me 19

Lmetric (Logging Metric) • Centralized Log and event storage and search !

• Powered by :

– Logstash

– ElasticSearch

– Kibana – Web Dashboard

OSDConf 2014 : http://piyush.me 20

Personare : BigData Stack • To collect valuable user interaction information for

subsequent use in personalization, recommendations, analysis and reporting.

• Layered Architecture (Lambda):

– Batch Layer : Hadoop M/R

– Speed Layer : Storm NRT Computation

– Serving Layer

OSDConf 2014 : http://piyush.me 21

Personare : Architecture

OSDConf 2014 : http://piyush.me 22

OSS powered BigData Stack • Log Collection: Logstash, Syslog-NG

• Distributed messaging system : Kafka , RabbitMQ

• ETL : Pentaho/ Talend

• Hadoop Ecosystem

• NRT Computation: Twitter Storm

• Document-Oriented NoSQL DB : Couchbase

• Distributed Search: ElasticSearch

• Distributed messaging system : Kafka , RabbitMQ

• NoSQL : Cassandra, Redis, Neo4J (Graph)

• API Management : WSO2 API Manager

• Programming Languages : Java , Python, R

OSDConf 2014 : http://piyush.me 23

My next hope ! …goal ?

• Would I be able to apply what all I have learned in technology through OSS to solve some issues faced by masses around the world ..?

– Using technology to make world a better place for everyone !

and I am sure OSS will have major role to play in it !!

OSDConf 2014 : http://piyush.me 24

Thanks!

Questions (If any ) ?

Contact:

@piykumar

http://piyush.me

OSDConf 2014 : http://piyush.me 25