Managing and Monitoring Application Performance

Sebastian Marek, Internal Systems Technical Architect

MANAGING AND MONITORING APPLICATION

PERFORMANCEhttp://www.flickr.com/photos/wraith34/9377593973/

a Pole living in Sheffield

over 12 years in software development

big fan of process automation

TDD and CI

occasionally contributes to open-source projects

wants to be a knight @proofek

h"ps://joind.in/9301

HTTPS://WWW.CODECLUB.ORG.UK/

A nationwide network of volunteer-led after school coding clubs for children aged 9-11

Volunteer Code Club Projects Venue Children

PERFORMANCE MONITORINGOverview

network level

system level

application level

End-user level

http://www.flickr.com/photos/global-jet/483825828/

PERFORMANCE MONITORINGNetwork level

failed TCP connections and packet losses

bandwidth usage

firewalls and loadbalancers

connectivity to external serviceshttp://www.flickr.com/photos/sbrown6079/2038502257/

PERFORMANCE MONITORINGSystem level

CPU load

memory consumption

disk space

http://www.flickr.com/photos/viagallery/3178699697/

PERFORMANCE MONITORINGApplication level

new deployments

response time

key transactions

memory usage

http://www.flickr.com/photos/qualityfrog/3546657245/

PERFORMANCE MONITORINGEnd-user level

response time

user experience

http://www.flickr.com/photos/alesk/356136498/

remote debugging

code coverage

tracing

profiling

Xdebug

Installation

XDEBUG

# pecl install xdebug

zend_extension=”/usr/lib/php/extensions/xdebug.so”

configure in xdebug.ini

web server restart (optional)

Configuration

XDEBUG

;Profiling settings (always on)xdebug.profiler_enable = 1xdebug.profiler_output_dir = /var/log/xdebug/

;Profiling settings (triggered by GET/POST/COOKIE)xdebug.profiler_enable = 0xdebug.profiler_output_dir = /var/log/xdebug/xdebug.profiler_enable_trigger = 1

Browser integration

XDEBUG

Profiling data

XDEBUG

fl=php:internalfn=php::spl_autoload_register881 3

fl=/Users/Sebastian/git/app/src/include/HTMLPurifier/HTMLPurifier.standalone.phpfn=HTMLPurifier_Bootstrap::registerAutoload878 21cfl=php:internalcfn=php::spl_autoload_functionscalls=1 0 0880 1cfl=php:internalcfn=php::spl_autoload_registercalls=1 0 0881 3

Profiling tools

XDEBUG

KCacheGrind for *nix systems

qcachegrind for MacOsX (and apparently Windows)

MacCallGrind for MacOsX

WebGrind via browser

qCacheGrind

XDEBUG

WebGrind

XDEBUG

profiling

web interface

xhProf

Installation

XHPROF

# pecl install xhprof

extension=”/usr/lib/php/extensions/xhprof.so”

configure in xhprof.ini

web server restart (optional)

Configuration

XHPROF

; XHProfextension = xhprof.soxhprof.output_dir = /var/log/xhprof/

Profiling tools

XHPROF

default web interface

xhGui (old, also known as XH UI)

xhGui (new, with MongoDB backend)

Default web interface - Installation

XHPROF

git clone git@github.com:facebook/xhprof.git

XHPROF

;Webserver (VirtualHost)<VirtualHost *:80> DocumentRoot “/Users/Sebastian/git/xhprof/xhprof_html” ServerName xhprof.localdomain

<Directory /Users/Sebastian/git/xhprof/xhprof_html> Order allow,deny Allow from all </Directory>

ErrorLog “/private/var/log/apache2/xhprof-error_log” CustomLog “/private/var/log/apache2/xhprof-access_log” common</VirtualHost>

XHPROF

;auto_prepend file - enable xhprof$xhProfHome = __DIR__ . ‘/..’;$profiler_namespace = ‘myapp’; // namespace for your application

if (extension_loaded(‘xhprof’)) { include_once “$xhProfHome/xhprof_lib/utils/xhprof_lib.php”; include_once “$xhProfHome/xhprof_lib/utils/xhprof_runs.php”; xhprof_enable(XHPROF_FLAGS_CPU | XHPROF_FLAGS_MEMORY);}

XHPROF

;auto_prepend file - collect datafunction xhprofShutdownFunction(){ global $profiler_namespace; $xhProfUrl = “http://xhprof.localdomain/”;

if (extension_loaded(‘xhprof’)) { $xhprof_data = xhprof_disable(); $xhprof_runs = new XHProfRuns_Default(); $run_id = $xhprof_runs->save_run($xhprof_data, $profiler_namespace); $profiler_url = sprintf(“$xhProfUrl/index.php?run=%s&source=%s”, $run_id, $profiler_namespace); echo “<a href=’$profiler_url’ target=‘_blank’>Profiler output</a>”; }}register_shutdown_function(‘xhprofShutdownFunction’);

XHPROF

;enable profiling in your application (VirtualHost)<VirtualHost *:80> DocumentRoot “/Users/Sebastian/git/my_app/src” ServerName my_app.localdomain

<Directory /Users/Sebastian/git/my_app/src > Order allow,deny Allow from all </Directory>

php_admin_value auto_prepend_file /Users/Sebastian/git/xhprof/xhprof_html/header.php</VirtualHost>

Default web interface - Home page

XHPROF

Default web interface - Drilldown

XHPROF

Default web interface - Call graph

XHPROF

Default web interface - Full Call graph

XHPROF

XH UI - Installation

XHPROF

git clone git@github.com:preinheimer/xhprof.git

XHPROF

;Webserver (VirtualHost)<VirtualHost *:80> DocumentRoot “/Users/Sebastian/git/xhui/xhprof_html” ServerName xhui.localdomain

<Directory /Users/Sebastian/git/xhui/xhprof_html> Order allow,deny Allow from all </Directory>

ErrorLog “/private/var/log/apache2/xhui-error_log” CustomLog “/private/var/log/apache2/xhui-access_log” common</VirtualHost>

XH UI - Configuration

XHPROF

# vim xhui/xhprof_lib/config.php

and reuse already provided xhui/external/header.php

XHPROF

php_admin_value auto_prepend_file /Users/Sebastian/git/xhui/external/header.php</VirtualHost>

XH UI - Home page

XHPROF

XH UI - Transaction detail

XHPROF

XH UI - Transaction detail

XHPROF

XHG - Installation

XHPROF

git clone git@github.com:preinheimer/xhgui.git

XHG - Installation

XHPROF

;Webserver (VirtualHost)<VirtualHost *:80> DocumentRoot “/Users/Sebastian/git/xhgui/webroot” ServerName xhgui.localdomain

<Directory /Users/Sebastian/git/xhgui/webroot> Order allow,deny Allow from all AllowOverride All </Directory>

ErrorLog “/private/var/log/apache2/xhgui-error_log” CustomLog “/private/var/log/apache2/xhgui-access_log” common</VirtualHost>

XHG - Requirements

XHPROF

MongoDB server

PHP MongoDB extension

PHP mcrypt extension

XHG - Installation

XHPROF

# cd xhgui

# php install.php

XHG - Installation

XHPROF

php_admin_value auto_prepend_file /Users/Sebastian/git/xhgui/external/header.php</VirtualHost>

XHG - Home page

XHPROF

XHG - Transaction trace

XHPROF

XHG - Transaction history

XHPROF

XHG - Transaction call graph

XHPROF

Glossary

PROFILING

call count - number of times a function/request was called

wt (Wall Time/Wall Clock Time) - time it took to execute a function/request

cpu - CPU time spent executing a function/request

mu - Amount of memory used during function/request execution

pmu - The peak amount of memory used during function/request execution

exclusive - total time spent excluding time to execute other functions

inclusive - total time spent including time to execute other functions

Process

PROFILING

Profile and record the results

Identify functions/requests using the most CPU time/memory (exclusive)

Find out what’s causing the issue

Refactor and optimize the code - look for external resources - database, web services, filesystem

Profile and compare the results

start all over again

server monitoring

application monitoring

real user monitoring

NewRelic

Supported web applications

NEW RELIC

Supported PHP framework

NEW RELIC

Installation

NEW RELIC

# RHEL

$ rpm -Uvh http://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-repo-5-3.noarch.rpm

$ yum install newrelic-php5

$ /usr/bin/newrelic-install

# Ubuntu / Debian

$ wget -O - https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -

deb http://apt.newrelic.com/debian/ newrelic non-free

$ apt-get install newrelic-php5

$ /usr/bin/newrelic-install

Server Monitoring

NEW RELIC

Server Monitoring - Main Dashboard

NEW RELIC

Portal01

Portal02

SugarCRM

Support Centre

Server Monitoring - Overview

NEW RELIC

SugarCRM (App02)

Application Monitoring

NEW RELIC

Server Trace - Summary

NEW RELIC

App01(5.3.19+a2h(2.2.3):SugarCRM)

Server Trace - Details

NEW RELIC

Server Trace - SQL queries

NEW RELIC

Application Monitoring - Error Tracking

NEW RELIC

Real User Monitoring - App Dashboard

NEW RELIC

Understanding RUM

NEW RELIC

Web application - The time spent in the application code

Network - The time it takes for a request to make a round-trip over the internet.

DOM processing - Time spent in the browser parsing and interpreting the HTML.

Page rendering - Time spent in the browser displaying the HTML, running in-line JavaScript and loading images.

Understanding RUM

NEW RELIC

Source: http://blog.newrelic.com

Real User Monitoring - Browser Page Load

NEW RELIC

Real User Monitoring - Browser Page Load

NEW RELIC

Satisfied zone: < T

Tolerating zone: T - F (4T)

Frustrated zone: > F

Apdex score

NEW RELIC

Satisfied customers + (Tolerating customers/2)

Total number of customers

Apdex by regions

NEW RELIC

Apdex - baseline

NEW RELIC

check the average response time

set your Apdex to average response time

find areas that require performance fixes

when performance improves lower your Apdex

New Relic - Map

NEW RELIC

memcached.local

crm.local

Keeping on top of the changes

NEW RELIC

Server Monitoring - plugins

NEW RELIC

Server Monitoring - APC

NEW RELIC

Server Monitoring - MySQL

NEW RELIC

Server Monitoring - Apache

NEW RELIC

PERFORMANCE MONITORINGSummary

Xdebug - application profiling during development stage

xhProf - application profiling during development stage and on live production platform

New Relic - end-to-end application monitoring on live production platfrom

PERFORMANCE MONITORINGAlternatives

DataDog - http://www.datadoghq.com/product/

GraphDat - http://www.graphdat.com

Scout - https://scoutapp.com/

Nagios - http://www.nagios.org/

Graphite - http://graphite.wikidot.com

ResourcesXdebug - http://www.xdebug.org/docs/profilerkCacheGrind - http://kcachegrind.sourceforge.netWebGrind - https://github.com/jokkedk/webgrindMacCallGrind for MacOsX - http://www.maccallgrind.com

xhProf - https://github.com/facebook/xhprofxhUI - https://github.com/preinheimer/xhprofxhGui - https://github.com/preinheimer/xhgui

New Relic - http://newrelic.comNew Relic Documentation - https://newrelic.com/docsApdex - http://apdex.org

http://techportal.inviqa.com/2009/12/01/profiling-with-xhprof/http://techportal.inviqa.com/2013/10/01/profiling-php-applications-with-xhgui/

h"ps://joind.in/9301

Managing and Monitoring Application Performance

Technology

Managing Union Parishad Monitoring System ... - 103.109.52.4

Managing and Monitoring Effluent Treatment Plants

Managing Application Systems

Stratos: A Rich Web-based UI for Managing and Monitoring ... · 3 Neil MacDougall Senior Technical Engineering Manager Cloud Application Platform, SUSE Stratos Managing and Monitoring

Monitoring for managing Mediterranean Marine Protected Areas - 2014 MedPAN network regional experience-sharing workshop Monitoring for managing Mediterranean

systems for monitoring and managing of photovoltaic · PDF file system for monitoring and managing of photovoltaic plants 1 14|02|14 systems for monitoring and managing of photovoltaic

Monitoring and Managing Memory

A New Grid Portal for Managing and Monitoring Application ... · A New Grid Portal for Managing and Monitoring Application Execution on Global Grids through ... accessing the Grid

Thanking, monitoring and managing donors

IBM PowerVM Virtualization Managing and Monitoring

Monitoring and Managing Java Applications

11 MANAGING AND MONITORING DHCP Chapter 2. Chapter 2: MANAGING AND MONITORING DHCP2 MANAGING DHCP: COMMON DHCP ADMINISTRATIVE TASKS Configure or modify

Monitoring and Managing Storage

Developing, Managing and Monitoring Service Centers

Managing Oracle WebLogic Server with Oracle Enterprise Manager · J2EE Application Monitoring Oracle Enterprise Manager provides a wide range of application and server metrics for

Managing Application Development

Managing and Monitoring TeamPage

Managing Pavements and Monitoring Performance

Wily | CA - Nortalfiles.webmedia.ee/micro/techday/pdf/WILY, Managing... · · 2012-10-12Wily | CA Application Performance Monitoring ... 24 x 7 Monitoring ... SAP Enterprise Portal)

Managing, Monitoring, and Troubleshooting the Exchange ...media.techtarget.com/searchExchange/downloads/ManagingExchangeOrg.pdf.....Managing, Monitoring, and Troubleshooting the Exchange