©2017Percona1

MarcosAlbe

ASupportEngineerWalkthroughonpt-stalk

PrincipalSupportEngineer- Percona

©2017Percona2

WhoisSpeaking?• Marcos Albé

- Principal Support Engineer @ Percona

- 7 years here- Focus on performance and observability

©2017Percona3

Agenda

©2017Percona4

Agenda

1. WhatdoweUSE

2. Who’sstalkin’?

3. Howpt-stalkcanbeUSEd

4. Walkthrough

©2017Percona5

WhatdoweUSE

©2017Percona6

Weuseoneofthese...

WhatdoweUSE

©2017Percona7

Utilization

WhatdoweUSE

©2017Percona8

UtilizationSaturation

WhatdoweUSE

©2017Percona9

Utilization

SaturationErrors

WhatdoweUSE

©2017Percona10

WhatdoweUSE

©2017Percona11

Who’sstalkin’?

©2017Percona12

Who’sstalkin’?

check out those resources!

©2017Percona13

Who’sstalkin’?•Aspersa stalk+Aspersa collectinone!(thanksDaniel&Baron)•#Thename'stalk'isbecause'watch'isalreadytaken,and'stalk'isfun.•Nowpartofpt-toolkit•Monitorsvariableorascript’ssingle-valuedoutput•Triggerscapturewhensetthresholdisreached- filesfrom/proc:diskstats,interrupts,meminfo,slabinfo,procstat,procvmstat.- vmstat,iostat,top,netstat,pmap,mpstat,sysctl,df,hostname,mysqladmin ext,mysqladmin debug,lsof).- SHOWENGINEINNODBSTATUS,SHOWENGINEINNODBMUTEX,SHOWGLOBALVARIABLES,SHOWFULLPROCESSLISTandSHOWOPENTABLES.

•Productionsafe!•Customtriggersandcustomcaptures•Companiontools(pt-sift,pt-mext,pt-diskstats,pt-summary);Bashdissection

©2017Percona14

Who’sstalkin’:commonparameters•Stalkingpart- --daemonize |--no-stalk- --function=status- --interval=1- --variable=Threads_running- --threshold=25- --cycles=5•Collectingpart- --run-time=30- --sleep=300- --iterations=- --plugin=- --dest=/var/lib/pt-stalk

•Valuesshownarethedefaultsforeach(wheregiven)

©2017Percona15

Who’sstalkin’:workflow

©2017Percona16

Who’sstalkin’:workingexamplesICollectdataduring60secondsfor3consecutivetimes,startingimmediately.

pt-stalk--no-stalk\--dest=$HOME/percona \--user=root\--ask-pass\--run-time=60--sleep=60--iterations=3

©2017Percona17

Who’sstalkin’:workingexamplesIIRunasadaemonandtriggercollectionifInnoDB hasmorethan5pendingfsyncs.

pt-stalk--daemonize \--function=status\--variable=Innodb_data_pending_fsyncs \--threshold=5\--cycles=2\--interval=1\--iterations=5\--plugin=/usr/local/bin/raid_card_status.sh

©2017Percona18

Who’sstalkin’:customtriggers#!/bin/bash# Save this bash snippet as /root/delayed_slave_trg.sh;trg_plugin() {

mysql $EXT_ARGV -Ee "SHOW SLAVE STATUS" | \grep "Seconds_Behind_Master" | \awk "{ print $2 }";

}#uselikept-stalk --function=/root/delayed_slave_trg.sh --etc --etc

©2017Percona19

Who’sstalkin’:customcapture#!/bin/bash# save as /root/collect_perf.shbefore_collect() {

# run perf stat; match sleep length to --run-time

/usr/bin/perf stat --all-cpus \--output=${d}/${p}-stat_a \-- sleep ${OPT_RUN_TIME};

}#uselikept-stalk --plugin=/root/collect_perf.sh --etc --etc

©2017Percona20

Howpt-stalkcanbeUSE’d

©2017Percona21

Howpt-stalkcanbeUSE’d

Linu

x K

erne

l CPU

CPUInterconnect

DRAM

I/O Bridge

I/O Controller Network Controller

I/O Bus

ExpanderInterconnect

Disk Disk Swap Port Port

Applications

System libraries

System Call Interface

VFS SocketsScheduler

File Systems TCP/UDP

Volume Manager IP Virtual Memory

Block Device Interface Ethernet

Device Drivers

Memory Bus

Interface Transports

©2017Percona22

Linu

x K

erne

l CPU

CPUInterconnect

DRAM

I/O Bridge

I/O Controller Network Controller

I/O Bus

ExpanderInterconnect

Disk Disk Swap Port Port

Applications (MySQL!)

System libraries

System Call Interface

VFS SocketsScheduler

File Systems TCP/UDP

Volume Manager IP Virtual Memory

Block Device Interface Ethernet

Device Drivers

Memory Bus

Interface Transports

ps

df

/proc/diskstats

SHOWENGINEINNODBSTATUS

/proc/interrupts

iostat

lsof /proc/meminfo

mpstat

SHOWENGINEINNODBMUTEX

mysqladminext

SHOWOPENTABLES

pmap

/proc/procstatSHOWFULLPROCESSLIST

/proc/vmstat/proc/slabinfo

sysctl

SHOWENGINETOKUDBSTATUS

top

SHOWGLOBALVARIABLES

vmstat

SHOWSLAVESTATUS

netstat

strace

tcpdump

gdb(backtraces)

oprofile

©2017Percona23

Howpt-stalkcanbeUSE’d:HereBeDragons!

strace tcpdump

gdb(backtraces)oprofile

©2017Percona24

Awalkthrough

©2017Percona25

AwalkthroughUsingpt-stalktosolveproblemswith...●...HighCPU●...HighIO●...Network

©2017Percona26

Demo

©2017Percona27

GetYourTicketsforPerconaLiveEurope!

ChampioningOpenSourceDatabases▪ MySQL,MongoDB,OpenSourceDatabases▪ TimeSeriesDatabases,PostgreSQL,RocksDB▪ Developers,Business/CaseStudies,Operations▪ September25-27th,2017▪ RadissonBlu RoyalHotel,Dublin,Ireland

LastYear’sConferenceSoldOut!ReserveyourspotASAP.

©2017Percona28

TalktoPerconaExpertsatAWSRe:Invent!DatabasePerformanceforCloudDeployments▪PerconaSupportandManagedServices• AmazonRDS,Aurora,RollYourOwn•MySQL/MariaDB/MongoDB• Reducecostsandoptimizeperformance

▪PerconaMonitoringandManagementDemos• Point-in-timevisibilityandhistoricaltrendingofdatabaseperformance• Detailedqueryanalytics

▪Booth#1138