Basic System Administration

Your daily commands as root

Becoming root

• Avoid login as root over network (denied by default)

• Use "/bin/su -" command from your regular account

- “-” runs user LOGIN scripts

- root can su to any userid without password.

- Note utility for NOLOGIN ID’s.

• $HOME is sometimes not “/root” but instead “/”; so watch what you delete!

• Remove the current working directory (".") from your PATH

• Never execute any regular user's program as root (possible Trojan Horse)

• Use SSH, not TELNET over network to avoid sniffers

root access - sudo

• visudo (as root) creates /etc/sudoers file in the following format:

usernames/group servername = (usernames command can be run as) command

• To implement: sudo <command>

• Groups are the same as user groups and are differentiated from regular users by a % at the beginning. The Linux user group "users" would be represented by %users.

• You can have multiple usernames per line separated by commas.

• Multiple commands also can be separated by commas. Spaces are considered part of the command.

• The keyword ALL can mean all usernames, groups, commands and servers.

• If you run out of space on a line, you can end it with a back slash (\) and continue on the next line.

• sudo assumes that the sudoers file will be used network wide, and therefore offers the option to specify the names of servers which will be using it in the servername position in Table 9-1. In most cases, the file is used by only one server and the keyword ALL suffices for the server name.

• The NOPASSWD keyword provides access without prompting for your password.

• Same concept as the SUID bit in permissions (4000).

sudo examples

• sudo command … to run the command as USER

Examples:

- user1 localhost=/sbin/halt … user1 can halt local system

[user1@student1]$ sudo /sbin/halt

password:

[user1@student1]$ System going down now!

- user2 ALL= NOPASSWD: /sbin/halt … user2 can halt any system w/o password

- user3 instructor = /usr/sbin/* … user 3 can run any command in /usr/sbin

System Administration tools

• man: Sections - 1 commands, 2 system calls, 3 C library routines, 4 devices

and networks, 5 file formats, 6 games and demos, 7 miscellaneous, 8 system

administration

• info – textinfo man page

• vi editor (front-end to a lot of utilities)

• su, sudo

• df/du, mount

• dump/restore, dd, cpio, tar, rmt, find, rsync

• ps, at, batch, crontab, anacron, watch, kill, nice, nohup, killall,

• useradd, usermod, userdel

• groupadd, groupmod, groupdel

• who, whoami

• syslog

• system configuration files – /etc

System information • hostname

• uname –a

• dmesg

• who, whoami

• last (reboot)

• which, where cmd

• hwclock

• date

• ulimit (user limits)

• sysctl (system limits/settings)

• cgroups

• /etc/sysconfig

• /etc/security

• /proc

• ps, pstree,

System monitoring

• sar

• pmap

• vmstat,

• mpstat

• iostat

• nstat (network),

• pidstat

• free

• lsof

• top, ntop, iftop, latencytop

• ulimit –a (view), ulimit –n (set) ulimits –Hn, ulimits -Sn … per user limits,

/etc/security/limits.*

• See /etc/security/limits.conf: sysctl –q (view), sysctl –w (set) … system

limits, /etc/sysctl.conf

• strace (debugging)

/etc/sysconfig

• The /etc/sysconfig directory is where many of the files that control the system

configuration are stored for daemon processes or system services like networking.

Contents vary depending on products installed.

/etc/sysconfig is usually “sourced” (. /etc/sysconfig) in SYSV startup scripts.

• Files in the /etc/sysconfig/ Directory

amd , apmd ‘ authconfig , clock , desktop , devlabel, dhcpd, firstboot, gpm, harddisks,

hwconf, i18n, init, ip6tables-config , iptables-config, irda, keyboard , kudzu, mouse,

named, netdump, network, network-scripts, ifup-xxxx, ntpd, pcmcia, radvd, rawdevices,

selinux, logrotate, samba , sendmail, spamassassin, squid , tux , vncservers, xinetd

/proc

• /proc is a virtual filesystem. It's sometimes referred to as a process information pseudo-

file system. It doesn't contain 'real' files but runtime system information (e.g. system

memory, devices mounted, hardware configuration, etc) for all processes started by init

including PID and startup commands. /proc was developed as a LINUX extension to

keep track of all the complex processes started in the system

• For this reason it can be regarded as a control and information centre for the kernel. In

fact, quite a lot of system utilities are simply calls to files in this directory. For example,

'lsmod' is the same as 'cat /proc/modules' while 'lspci' is a synonym for 'cat /proc/pci'. By

altering files located in this directory you can even read/change kernel parameters

(sysctl) while the system is running.

• The most distinctive thing about files in this directory is the fact that all of them have a

file size of 0, with the exception of kcore, mtrr and self.

/etc/security • Central directory for system defaults

• The limits.conf file defines process resource limits for users. (see ulimit)

• opasswd - Store old passwords.

• access.conf used to allow or restrict access to the system.

• chroot.conf used to restrict users to there home directories

• console.apps contains files which are same as service names.

• console.perms and console.perms.d directory determine the permissions that

will be given to

• The rest are PAM (Program Authentication Module) related.

sysctl – system limits • sysctl –q, sysctl –w, sysctl –p file, sysctl -A

• /etc/sysctl.conf

For network:

# Enable IP spoofing protection

• net.ipv4.conf.all.rp_filter=1

# Disable IP source routing

• net.ipv4.conf.all.accept_source_route=0

# Ignoring broadcasts request

• net.ipv4.icmp_echo_ignore_broadcasts=1

net.ipv4.icmp_ignore_bogus_error_messages=1

• # Make sure spoofed packets get logged

net.ipv4.conf.all.log_martians = 1

• # disable IPv6

net.ipv6.conf.all.disable_ipv6=1

Kernel isolation (test carefully on test system):

• Turn on execshield

kernel.exec-shield=1

kernel.randomize_va_space=1

ulimit – user limits

• ulimit - set user limits

• -c maximum core file size (in 512-byte blocks)

• -d maximum size of data segment or heap (in kbytes)

• -f maximum file size (in 512-byte blocks)

• -n maximum file descriptor plus 1

• -s maximum size of stack segment (in kbytes)

• -t maximum CPU time (in seconds)

• -v maximum size of virtual memory (in kbytes)

• -S soft limit

• -H hard limit

• /etc/security/limits.conf

c(ontrol)groups

• Cgroups allow you to allocate resources—such as CPU time, system memory, network bandwidth, or

combinations of these resources—among user-defined groups of tasks (processes) running on a

system.

• A *cgroup* associates a set of tasks with a set of parameters for one or more subsystems. A

*subsystem* is a module that makes use of the task grouping facilities provided by cgroups to treat

groups of tasks in particular ways. A subsystem is typically a "resource controller“ in a hierarchy of

processes.

• A cgroup is mounted as a virtual filesystem and can be modified to re-alllocate kernel resources.

Each cgroup is represented by a directory in the cgroup file system containing the following files

describing that cgroup:

- tasks: list of tasks (by pid) attached to that cgroup

- releasable flag: cgroup currently removeable?

- notify_on_release flag: run the release agent on exit?

- release_agent: the path to use for release notifications (this file exists in the top cgroup only) Other

subsystems such as cpusets may add additional files in each cgroup dir.

PAM

• Pluggable Authentication Module

• Centralized authentication mechanism

• “Plug in” different authentication methods

• Different services can have different

authentication policies

• Highly secure systems can require multiple

passwords to authenticate

PAM Framework

• ftp login ssh

pam_unix

conf libpam

pam_ldap pam_securetty

Applications

PAM Library

Modules

PAM Stack

• Modules are stacked (order is important)

• Sample PAM configuration in /etc/pam.d:

interface control flag module name

auth required pam_nologin.so

auth required pam_securetty.so

auth sufficient pam_unix.so

auth required pam_ldap.so

Security Enhanced LINUX

• Kernel level security included since 2.60 kernel

• Not an application interface. But sends access return codes to applications.

• Can be combined with ACLs

• /etc/selinux directory

• Old gui system-config-selinux, new gui: policycureutils-gui

• BE CAREFUL with changes. Especially deleting files.

• See “enforcing=0” or “selinux=0” on Grub edit menu

• Protects, files, processes, applications

• Based on security “context”

An SE Linux security context is comprised of three parts: an "identity", a "role", and a "type" for users and files or "domain" for processes. Default context for root, the role is sysadm_r, and the domain is sysadm_t.

• Configuration directory: /etc/selinux directory

SELINUX Modes/Types

• Enforcing: enable and enforce the SELinux security policy on the system, denying access and logging actions in /var/log/audit/audit.log

• Permissive: enabled but will not enforce the security policy, only warn and log actions. Used for troubleshooting SELinux issues

• Disabled: SELinux is turned off

• Targeted: Specific processes

• MLS/STRICT: VERY secure - systemwide

SELINUX Policy

• Policy: a set of rules the SELinux security engine that

defines types for file objects and domains for processes,

and user defined (identities) roles to limit the domains that

can be entered.

• Strict - minimum access

• Targeted – specific processes

• Unconfined – not under SELINUX control, outside

SELINUX context

SELINUX Access Control

• Type Enforcement (TE): Type Enforcement is the

primary mechanism of access control used in

the targeted policy

• Role-Based Access Control (RBAC): Based around

SELinux users (not necessarily the same as the Linux

user), but not used in the default targeted policy

• Multi-Level Security (MLS): Not commonly used and

often hidden in the default targeted policy.

• Shown with –Z option (ls –Z, ps –Z etc)

SELINUX Commands

• sestatus: show SELINUX status

• getenforce: show SELINIX status

• setenforce: set SELINUX status

• semanage: command line policy management

• chcon: change SELINUX context

• restorecon: restore default SELINIX context

• audit2allow: Generate SELINUX policy from /var/log/audit/audit.log

• sealert: troubleshooting tool

• ls –Z, ps –Z: show SELINUX context for files, processes

• id: show the current user id context.

Process info: ps -aux(BSD)

• Common options:

-a print all processes involving terminals

-e print environment and arguments

-l long listing

-u print user information

-xi nclude processes with no terminals

• Meaning of user information columns:

%CPU percentage use of CPU

SZ total size (in 1024 byte pages) of the process

RSS total resident size (in pages) of the process

STAT state of the process

TIME time, including both user and system time

Process info: ps –ef (System V)

• Common options:

-e print all processes

-f print full listing

-l long listing (more info than -f)

• pstree

• Meaning of full listing columns:

S state

PRI priority

SZ total size (in 4096 byte pages) of the process

RSS total resident size (in pages) of the process TIME starting time

TIME cumulative execution time

Process Management

• at – schedule onetime batch job (scripts or commands).

Example: at now +1 minutes –f somecommandfile.txt

“batch” interactive at command on some systems.

atq, atrm etc

• anacron – (Linux) workstation scheduler. See /etc/anacrontab

• watch - execute a program periodically, display results fullscreen

• cron – scheduler routine must be a started process. See /etc/crontab/, /etc/cron.d, /etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly

- crontab [-e] [-l] [-r] [–u user] [filename]. Creates cron table “-e”uses vi syntax.

- Each line contains:

mm(0-59) hh(0-23) dd(1-31) mm(1-12) day (0-6, 0=Sunday**) command

* - Is treated as a wild card. Meaning any possible value.

*/5 - Is treated as ever 5 minutes, hours, days, or months.

2,4,6 - Treated as an OR, if placed in the hours, this could mean at 2, 4, or 6 o-clock.

9-17 - Treats for any value between 9 and 17. If placed in day of month, days 9 -17.

** Note 1-7 on some UNIX systems

Example */5 * * * * echo “hi there” > /dev/tty2 2>&1

Process Management

• kill pid– stop a process. -9 kills absolutely. killall

• nice pid or command – set process priority

• nohup command – run a process after logging off. Nohup.out contains job

output.

• “sighup” option on some commands for forcing process to reinitialize.

• “zombie” processes – “kill”ed or abended processes with no parent.

Usually requires a reboot to reclaim resources. Can cause system instability.

See also: Ctrl Z, Ctrl C, fg, bg, &

Backup/Restore • dump/restore - backs up file systems, has interactive mode, can do

incremental backups, maintains "sparse files", is most commonly used utility

• cpio - can back up individual files/directories, handles special files, packs data tighter than tar, skips bad spots on media on restore, use with find (some versions of find have -cpio option for this purpose)

• tar - backs up directory trees, does not back up special files, poor error handling with media errors, does not pack blocks (GNU tar solves some of these problems). Some LINUX/UNIX systems have built-in compress with –z flag.

• dd - copies/converts files, can go from one medium to another, processes whole entity or select blocks, can swap bytes and do ASCII/EBCDIC conversions. Performs physical backup of raw devices.

• rmt - used for remote tape operations. Varies by OS.

• rsync – used for directory synchronization, e.g. “hot” folders

Backup Strategy

• Physical (dd, cpio) – usually devices (as root)

dd if=devicefile if=outputfile bs=blocksize count=#blocks

• Logical (rmt, tar, dump/restore). Backup marker.

create: tar –cvf tarfilename.tar [directory list] | compress

list: tar –tvf tarfilename.tar

extract: tar –xvf tarfilename.tar

• Can use logical backups in conjunction with find command –exec option (next panel) for differential or incremental backups

• Backup types: Full (everything)

Incremental (Difference since last backup)

Differential (Difference since last full backup)

Full + Incremental or Differential = Backup set

find

• Syntax: find starting-dir(s) matching-criteria-and-actions Matching criteria

-atime n file was accessed n days ago

-mtime n file was modified n days ago

-size n file is exactly n 512-byte blocks

-type c file type (e.g., f=plain, d=dir)

-name nam file name (e.g., `*.c')

-user usr file's owner is usr

-perm p file's access mode is p

-print display pathname

-exec cmd execute command ({} expands to file)

• find examples

find . -name \*.c -print

find / -size +1000 -mtime +30 \ -exec ls -l {} \;

find / \( -name a.out -o -name core \ -o -name' \) -type f -atime +14 \ -exec rm -f {} \;

find / \( -perm 2000 -o -perm 4000 \) \ -print | diff - files.secure

Disk management

• df

• mount / umount

• du | sort –rn | more

• find / -name core -exec rm -f {} \;

• Filesystems: /home, /var, /tmp (noexec), / (never full!)

• mkdev, mkfs, fdisk

User management • Set system account parameters (e.g., password aging, account expiration,

quotas, login scripts - /etc/profile, /etc/bashrc etc)

• Determine login name, user ID (UID), group ID (GID)

• Assign password (passwd)

• /etc/passwd - logname:passwd:uid:gid:user info:home:shell

• Passwords stored in /etc/shadow (pwconv)

• Commands: useradd, usermod, userdel, chage, passwd

• /etc/group: group:passwd:gid:members

• Commands: groupadd, groupmod, groupdel

User Security • ALWAYS use /etc/shadow (pwconv command)

• Password aging:

get: chage -l userid

set: chage -M 60 -m 7 -W 7 userid

• Lock/unlock and account

passwd –l userid

passwd –u userid

• Limit password reuse:

vi /etc/pam.d/system-auth (RHEL/Fedora)

vi /etc/pam.d/common-password (Ubuntu)

Add: password sufficient pam_unix.so use_authtok md5 shadow remember=10

• Verify root ID’s: awk -F: '($3 == "0") {print}' /etc/passwd

• Verify no password ID’s: awk -F: '($2 == "") {print $1 }' /etc/shadow | grep /etc/passwd

Make sure they are /bin/nologin

• No ROOT LOGIN, use su or sudo.

• Disable ROOT login under FTP., SSH, GUI (procedures vary)

• Configure password policy (LINUX: see pam_cracklib.so, other vary).

• See defaults in /etc/login.defs for /etc/shadow values.

SYSLOG

• (r)syslog is a utility for tracking and logging all manner of system messages from the merely informational to the extremely critical. Sysogs stored in

• In LINUX, system logs are stored in /var/log. System messages are recorded in /var/log/messages. Other OS’es may use different files in different directories (e.g. /var/adm).

• Each system message sent to the syslog server has two descriptive labels associated with it that makes the message easier to handle.

- The first describes the function (facility) of the application that generated it. For example, applications such as mail and cron generate messages with easily identifiable facilities named mail and cron.

- The second describes the degree of severity of the message.

SYSLOG

• Severity: Level: Keyword: Description

• 0 emergencies System unusable

• 1 alerts Immediate action required

• 2 critical Critical condition

• 3 errors Error conditions

• 4 warnings Warning conditions

• 5 notifications Normal but significant conditions

• 6 informational Informational messages

• 7 debugging Debugging messages

SYSLOG • configuration file /etc/rsyslog.conf or/etc/syslog.conf.

• File consists of two columns.

- First lists the facilities and severities of messages to expect

- Second lists the files to which they should be logged.

- LINUX default directory is /var/log

• Example:

*.info;mail.none;authpriv.none;cron.none /var/log/messages

• Note other services may record messages in other files (e.g. sendmail)

• Syslog is also a network service. A common implementation is to forward system info to a common syslog server. TCP or UDP can be used.

• Logs compressed, ,stored and optionally e-mailed by the logrotate function. Definitions stored in /etc/logrotate.conf and /etc/logrotate.d

System shutdown

• Shutdown will run SysV K* scripts.

• shutdown {–h|-r} {time in minutes|now}

• See also wall command

• Other commands: halt, reboot, Ctrl-Alt-Del may bypass some processing. Not

recommended for production systems.

• Reboots recorded in /var/log/wtmp or utmp

• last (reboot) displays info