How to Diagnose Problems Quickly on Linux Servers

Richard Cunningham (@rythie)

Not working

Running Slow

Two Types of Problems

Basics

ps – List of processes (use ps -ef)

top – Similar to ps but shows memory usage and is interactive

dmesg – Messages directly from the kernel

lsof – list open file and network ports

/var/log/ - Where most/all of the logs live

du – you may be out of disk space!

Understanding 'top'

top - 17:59:20 up 272 days, 22:17, 1 user, load average: 0.27, 0.33, 0.29Tasks: 142 total, 1 running, 141 sleeping, 0 stopped, 0 zombieCpu(s): 1.4%us, 0.5%sy, 0.3%ni, 95.3%id, 1.9%wa, 0.0%hi, 0.1%si, 0.5%stMem: 2051088k total, 1986640k used, 64448k free, 48156k buffersSwap: 4194300k total, 129028k used, 4065272k free, 810744k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1156 mysql 20 0 793m 467m 5060 S 2.0 23.4 16960:15 /usr/local/mysql-5.1 4130 richie 20 0 10916 940 668 R 2.0 0.0 0:00.01 top -cbn1 17311 backend 20 0 115m 40m 6240 S 2.0 2.0 12:54.17 /usr/local/bin/php 1 root 20 0 10408 508 480 S 0.0 0.0 6:32.75 init [3] 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kthreadd] 3 root 20 0 0 0 0 S 0.0 0.0 5:56.83 [ksoftirqd/0]

/var/log

ls -ltr /var/log – see most recently modified

tail -f /var/log/message – watch for new lines at the end of a log file

grep – most of those logs have too much irrelevant stuff in them, grep helps

Slowness

iostat – I/O device statistics

free – how much free memory you have

sar – historical logs of performance

Understanding 'free'

$ free -m total used free shared buffers cachedMem: 3019 2908 111 0 225 1187-/+ buffers/cache: 1495 1524Swap: 0 0 0

What are System Calls?

Compiled Program (including libraries)

Kernel

open(), read(), write()...

Hardware

011100101101010...

strace – typical output

open("/etc/configfile.conf", O_RDONLY) = 3

read(3, "#this is the config file for...”, 4096) = 32

write(1, "hello", 5) = 5

strace - invocation

strace -p <process id>

strace <program name>

Wireshark

Wireshark

Wireshark

Wireshark

Usetcpdump -s0 -w filename

To capture network traffic to a file for later analysis in wireshark (i.e. copy it to you local

machine)

Wireshark available for OSX, Windows and Linux and is Open Source

The End

Richard Cunningham(@rythie)