7
System Administration Toolkit: Monitoring disk space and usage Martin Brown ([email protected]), Freelance Writer, Freelance Developer Summary: Look at methods for determining disk usage across multiple UNIX® systems and how to create a simple warning system to alert you of potential problems. Keeping an eye on your file systems and ensuring they don't fill up is a trivial, but vital process in the day-to-day management of your UNIX systems. In this article, you'll look at methods for keeping an eye on disk space, discovering which files, users, or applications are using up the most space, and how to make use of quotas and other solutions to find the information you need. Date: 13 Jun 2006 Level: Intermediate Activity: 4665 views Comments: 0 (Add comments) Average rating (based on 71 votes) About this series The typical UNIX® administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Getting started: using df Monitoring disk space is a vital part of your job as a UNIX administrator. This article gives you the tools you need to be successful, including the use of df, du, find, and even the use of quotas. Let's get started by taking a look at how useful df can be. Use of the df tool can be a bit like a nervous twitch for many administrators, as it gives you a snapshot view of the storage space used and available across all of your file systems through a single command. Depending on your UNIX environment, the default output from df might contain a range of different information. Most modern df variants show disk space, usage and availability, and usually the mountpoint (and sometimes the device). For example, Mac OS X, based on BSD, shows the information in Listing 1. Listing 1. Default disk space information for Mac OS X Filesystem 512-blocks Used Avail Capacity Mounted on /dev/disk0s2 268435456 119741344 148182112 45% / devfs 195 195 0 100% /dev fdesc 2 2 0 100% /dev <volfs> 1024 1024 0 100% /.vol /dev/disk0s3 43474520 7251448 36223072 17% /Volumes/Untitled Note that the size is displayed in 512 byte blocks, not kilobytes. Nearly all traditional UNIX operating systems report the size in 512 byte blocks rather than kilobytes by default. To get the information in 1K blocks, use the -k command-line option, as shown in Listing 2. Listing 2. Using the -k option with the df command $ df -k Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/disk0s2 134217728 59870704 74091024 45% / devfs 97 97 0 100% /dev fdesc 1 1 0 100% /dev <volfs> 512 512 0 100% /.vol /dev/disk0s3 21737260 3625724 18111536 17% /Volumes/Untitled Some variants might also support alternative block sizes, such as megabytes (use -m) and gigabytes (use -g), as demonstrated in Listing 3. Listing 3. Using the -g option with the df command $ df -g Filesystem 1G-blocks Used Avail Capacity Mounted on /dev/disk0s2 128 57 70 45% / devfs 0 0 0 100% /dev fdesc 0 0 0 100% /dev <volfs> 0 0 0 100% /.vol /dev/disk0s3 20 3 17 17% /Volumes/Untitled Obviously, as you increase the block size shown, the level of detail in the information starts to reduce, but using alternative sizes can be a useful way to quickly monitor disks, particularly very large ones. Other versions of the df tool (notably Solaris) report the bare block and file availability information by default. Most file systems have a limited number of files that can be stored (the number is so large that you are unlikely to reach the limit), so it is possible to fill up a system with files, but still have disk capacity and be unable to store any more files. System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html 1 de 7 28/08/09 18:59

Disk Quotas on Unix-Mac OS X

Embed Size (px)

DESCRIPTION

Look at methods for determining disk usage across multiple UNIX® systems and how to create a simple warning system to alert you of potentialproblems. Keeping an eye on your file systems and ensuring they don't fill up is a trivial, but vital process in the day-to-day management of your UNIX systems.In this article, you'll look at methods for keeping an eye on disk space, discovering which files, users, or applications are using up the most space, and how tomake use of quotas and other solutions to find the information you need.

Citation preview

Page 1: Disk Quotas on Unix-Mac OS X

System Administration Toolkit: Monitoring disk space and usageMartin Brown ([email protected]), Freelance Writer, Freelance Developer

Summary: Look at methods for determining disk usage across multiple UNIX® systems and how to create a simple warning system to alert you of potentialproblems. Keeping an eye on your file systems and ensuring they don't fill up is a trivial, but vital process in the day-to-day management of your UNIX systems.In this article, you'll look at methods for keeping an eye on disk space, discovering which files, users, or applications are using up the most space, and how tomake use of quotas and other solutions to find the information you need.

Date: 13 Jun 2006Level: IntermediateActivity: 4665 viewsComments: 0 (Add comments)

Average rating (based on 71 votes)

About this series

The typical UNIX® administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are keyutilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of thetricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available toolsacross a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment.

Getting started: using df

Monitoring disk space is a vital part of your job as a UNIX administrator. This article gives you the tools you need to be successful, including the use of df, du,find, and even the use of quotas. Let's get started by taking a look at how useful df can be.

Use of the df tool can be a bit like a nervous twitch for many administrators, as it gives you a snapshot view of the storage space used and available across all ofyour file systems through a single command. Depending on your UNIX environment, the default output from df might contain a range of different information.

Most modern df variants show disk space, usage and availability, and usually the mountpoint (and sometimes the device). For example, Mac OS X, based onBSD, shows the information in Listing 1.

Listing 1. Default disk space information for Mac OS X

Filesystem 512-blocks Used Avail Capacity Mounted on/dev/disk0s2 268435456 119741344 148182112 45% /devfs 195 195 0 100% /devfdesc 2 2 0 100% /dev<volfs> 1024 1024 0 100% /.vol/dev/disk0s3 43474520 7251448 36223072 17% /Volumes/Untitled

Note that the size is displayed in 512 byte blocks, not kilobytes. Nearly all traditional UNIX operating systems report the size in 512 byte blocks rather thankilobytes by default. To get the information in 1K blocks, use the -k command-line option, as shown in Listing 2.

Listing 2. Using the -k option with the df command

$ df -kFilesystem 1K-blocks Used Avail Capacity Mounted on/dev/disk0s2 134217728 59870704 74091024 45% /devfs 97 97 0 100% /devfdesc 1 1 0 100% /dev<volfs> 512 512 0 100% /.vol/dev/disk0s3 21737260 3625724 18111536 17% /Volumes/Untitled

Some variants might also support alternative block sizes, such as megabytes (use -m) and gigabytes (use -g), as demonstrated in Listing 3.

Listing 3. Using the -g option with the df command

$ df -gFilesystem 1G-blocks Used Avail Capacity Mounted on/dev/disk0s2 128 57 70 45% /devfs 0 0 0 100% /devfdesc 0 0 0 100% /dev<volfs> 0 0 0 100% /.vol/dev/disk0s3 20 3 17 17% /Volumes/Untitled

Obviously, as you increase the block size shown, the level of detail in the information starts to reduce, but using alternative sizes can be a useful way to quicklymonitor disks, particularly very large ones. Other versions of the df tool (notably Solaris) report the bare block and file availability information by default. Mostfile systems have a limited number of files that can be stored (the number is so large that you are unlikely to reach the limit), so it is possible to fill up a systemwith files, but still have disk capacity and be unable to store any more files.

System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

1 de 7 28/08/09 18:59

Page 2: Disk Quotas on Unix-Mac OS X

You can see a sample of the default output of Solaris in Listing 4.

Listing 4. Default output of Solaris

$ df/ (/dev/dsk/c0t0d0s0 ):14877208 blocks 914042 files/devices (/devices ): 0 blocks 0 files/system/contract (ctfs ): 0 blocks 2147483618 files/proc (proc ): 0 blocks 16109 files/etc/mnttab (mnttab ): 0 blocks 0 files/etc/svc/volatile (swap ): 5737216 blocks 147177 files/system/object (objfs ): 0 blocks 2147483532 files/usr (/dev/dsk/c0t0d0s3 ): 9076010 blocks 863695 files/dev/fd (fd ): 0 blocks 0 files/var (/dev/dsk/c0t0d0s4 ): 8110796 blocks 483714 files/tmp (swap ): 5737216 blocks 147177 files/var/run (swap ): 5737216 blocks 147177 files/export/home (/dev/dsk/c0t0d0s7 ):69362510 blocks 4272812 files

Using the -k command-line option switches the display to one similar to that shown with earlier examples, where the data is summarized into a more easilyreadable format (see Listing 5).

Listing 5. Using the -k option

$ df -kFilesystem kbytes used avail capacity Mounted on/dev/dsk/c0t0d0s0 7644629 206026 7362157 3% //devices 0 0 0 0% /devicesctfs 0 0 0 0% /system/contractproc 0 0 0 0% /procmnttab 0 0 0 0% /etc/mnttabswap 2868600 1016 2867584 1% /etc/svc/volatileobjfs 0 0 0 0% /system/object/dev/dsk/c0t0d0s3 8261237 3723232 4455393 46% /usrfd 0 0 0 0% /dev/fd/dev/dsk/c0t0d0s4 4130238 74849 4014087 2% /varswap 2867584 0 2867584 0% /tmpswap 2867624 40 2867584 1% /var/run/dev/dsk/c0t0d0s7 35611388 930133 34325142 3% /export/home

All variants of df accept a directory or path, and then the information about that disk space for the file system that contains that path is displayed. For example, thefollowing shows you how to get the space on the root file system:

$ df -k /Filesystem kbytes used avail capacity Mounted on/dev/dsk/c0t0d0s0 7644629 206026 7362157 3% /

Or, the following uses df to show the space on a user's home directory:

$ df -k ~mcFilesystem kbytes used avail capacity Mounted on/dev/dsk/c0t0d0s7 35611388 930133 34325142 3% /export/home

Using the df output

The df tool is actually short for disk free, and that is precisely what the tool shows. If you look at some sample output again, you can extract the importantindividual elements:

Filesystem kbytes used avail capacity Mounted on/dev/dsk/c0t0d0s7 35611388 930133 34325142 3% /export/home

The first column, usually with a header according to the block size data, shows the total size of the disk. The used column shows the number of blocks that havebeen used on the file system or device. The avail column shows the number of unused (available) blocks on the file system.

The capacity column gives a percentage calculation of how much disk space out of the total size has actually been used. On its own, this information can give aquick indication of the space available.

By default, UNIX creates file systems that can be filled up to 90 percent capacity with data by users. The remaining 10 percent is reserved by the system for useby root to perform any emergency maintenance. If a user tries to add or create a file that goes beyond that level, an error is returned (out of file system space).

That emergency maintenance space is there so it can be used to recover from an overfull disk. For example, as an administrator, you could use the space to createa quick backup, or to compress existing data without having to use a secondary file system for the purpose. You can adjust the amount of free space that isreserved (using the tunefs tool), or when creating the file system. This is vital for large disks, where a 10 percent buffer could equal many gigabytes of potentialspace. It is generally a good idea to reserve at least one percent to give you a minimal buffer before you completely run out of space. Before you get that far,especially if a lot of your disk space is being used, you should determine who is using all of that space.

System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

2 de 7 28/08/09 18:59

Page 3: Disk Quotas on Unix-Mac OS X

Using du

The du command, instead of showing disk free space, shows disk usage information. The du tool is used to determine the disk usage for files and directories. Touse it, change to a directory and run the tool (see Listing 6).

Listing 6. du command

$ cd /var$ du16 ./lost+found4 ./sadm/install/admin22 ./sadm/install/logs28448 ./sadm/install4 ./sadm/pkg/SUNWocfd/install4 ./sadm/pkg/SUNWocfd/save/pspool/SUNWocfd/install16 ./sadm/pkg/SUNWocfd/save/pspool/SUNWocfd18 ./sadm/pkg/SUNWocfd/save/pspool20 ./sadm/pkg/SUNWocfd/save28 ./sadm/pkg/SUNWocfd4 ./sadm/pkg/SUNWcsu/inst...

The output, shown in Listing 6 above, has been trimmed. By default, du shows the file usage for every file and directory beneath the current, or specified,directory. The value given is the size of the file in the default block size for that system, just as that used with df. This might, or might not, be 1K; you can forcethe display to 1K blocks using the -k command-line option.

You will probably want to summarize the information by the top-level file, or directory, that you are examining. Use the -s option to switch on the summary view.Here's a summarized version, shown in the /var directory, of a Solaris installation:

$ du -sk 70818 .

Note that it shows the summary information for the current directory (.). To get summary information for all of the files and directories, use the * wildcard (seeListing 7).

Listing 7. Using the * wildcard to get summary information

$ du -sk *382 adm950 apache683 apache26837 appserver1 audit162 cache3 cc-ccr2 crash4 cron31 dmi22 dt6 fm2 imq1 inet3 krb54 ld1 ldap937 lib6 log8 lost+found2 lp2 mail1 mysql1 news3 nfs38 nis2 ntp10034 opt1 preserve96 run49687 sadm15 saf3 samba2 sma_snmp131 snmp39 spool4 statmon663 svc14 tmp10 uucp24 yp

Be careful when using that tool, particularly in user directories, as the information shown will not include hiddenfiles -- in other words, those files and directorieswith a single period prefix. You might want to use the following line in a user directory to get all of the summary information:

$ du -sk * .[a-zA-Z0-9]*

System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

3 de 7 28/08/09 18:59

Page 4: Disk Quotas on Unix-Mac OS X

One final very useful option for du is -d, which prevents du from following file system boundaries. For example, to determine the disk usage on the root filesystem but not any other file systems, you would use the -d option:

$ du -dsk /

Some systems do not have this option but do have the -x command-line option, which only includes the file usage for files on the same device or file system as thepath you specify.

Finding disk usage for a specific user

To find the disk space being used by a single user, you need to combine du with the find command to only report disk usage for a specific user:

$ find . -user mc -type f -exec du -k {} \;

The -user option allows you to specify that find will only report files that are owned by the specified user. The -type option forces find to only return the path ofitems of a specific type (in this case, files); this prevents du from including directories, which might be owned by one user, but contain files for many users. Then,for each found path, the du command is executed to report the disk usage.

To get summary information, in other words, the total space used by a specific user, you can use awk to total the information and print out the final value:

$ find . -user mc -type f -exec du -k {} \;|awk '{ s = s+$1 } END { print "Total used: ",s }'Total used: 123721

You use the same principle with groups using the -group option to find:

$ find . -group mcslp -type f -exec du -k {} \;|awk '{ s = s+$1 } END { print "Total used: ",s }'Total used: 542485

There is, however, an easier way, if you enable disk quotas.

Using quotas

The quota system automatically monitors the disk usage for users on a file system by a file system basis. The quota environment not only enables you to monitordisk usage, but you can also set limits for usage that provide a warning and, more explicitly, prevent users from using disk space over their assigned quota value.The lower limit (warning) is called the soft limit and the higher limit (which prevents creation of files over that level) is called the hard limit. Some systems mightalso allow you to control the number of files owned by each user.

The precise method required to enable quotas on your machine is dependent on the operating system you are using. Most UNIX systems include quota support bydefault. Linux® systems might need to build a new kernel that includes quota support. Most, however, use a single file on each file system, usually called quotas,into which you place the limits for each user.

To enable quotas, you should first create this file and ensure that the quotas file is only editable by root:

$ touch /export/home/quotas$ chmod 600 /export/home/quotas

Then switch on quotas using the quotaon command:

$ quotaon /export/home

Finally, you must edit the appropriate quota for each user. You can do this by using the edquota command, specifying the user:

$ edquota mc

Your default editor (or vi, if you haven't set an alternative) will be opened with a simple form for setting quota values. You can see below that quotas have beenenabled on the users home directory file system with a soft limit of 200,000KB and a hard limit of 400,000KB. The settings of zero for file limits indicate thatthere is no set limit and the user can set and create as many files as they like.

fs /export/home blocks (soft = 200000, hard = 400000) inodes (soft = 0, hard = 0)

System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

4 de 7 28/08/09 18:59

Page 5: Disk Quotas on Unix-Mac OS X

If you have more file systems with quotas, then you will need to configure more rows for the file systems.

If you want to configure the quota for multiple users, first set the quota for a single user, then use the -p command-line option to edquota. This uses the specifieduser as the basis for the new users. For example, to set the quota using the settings for mc, for usernames slp, tw, and sh, do the following:

$ edquota -p mc slp tw sh

File limit warnings with quotas

When the user creates a file that goes over their soft limit, they will get the following warning:

quota_ufs: Warning: over disk limit (pid 1738, uid 101, inum 94, fs /export/home)

Note that the user is given seven days to correct the problem; you can alter the duration by using edquota -t.

If the user tries to create a file that pushes them over the hard limit, the system terminates the write process and truncates the file to the limit:

$ mkfile 210000k overlimitquota_ufs: over hard disk limit (pid 1843, uid 101, inum 130, fs /export/home)overlimit: initialized 191873024 of 215040000 bytes: Disc quota exceeded

Any user can check their own quota limits and disk usage by running the quota command:

$ quotaOver disk quota on /export/home, remove 199993K within 7.0 days

Quota administration

The systems administrator can check the quota of any user using the quota command; you should use the -v command-line option to provide a full report on thefile system, usage, and limit information (see Listing 8).

Listing 8. Using the -v option

$ quota -v mcDisk quotas for mc (uid 101):Filesystem usage quota limit timeleft files quota limit timeleft/export/home 399993 200000 400000 6.9 days 151 0 0

To get a report about disk and quota usage use on a file system basis detailing all users, use the repquota command, specifying the file system to report (seeListing 9).

Listing 9. Using the repquote command

$ repquota -v /export/home /dev/dsk/c0t0d0s7 (/export/home): Block limits File limitsUser used soft hard timeleft used soft hard timeleftmc +- 399993 200000 400000 6.9 days 151 0 0

To ensure that the quota information is kept up to date, you should use the quotacheck command. This validates the file storage data with the quota informationused to report quota information. You should run this automatically using cron -- once a day should be fine (the process is comparatively time consuming).

Quota tricks and traps

The quota system provides the best combination of monitoring and automatic administration, but you should be careful about the file systems on which youenable quotas and the limits that you set, as they might actually end up hindering the work of your users rather than controlling their disk usage.

For example, programmers might need significantly more space to actually build an application than you might have configured for their use. You can get aroundthis issue by providing an unrestricted area for compilation (by setting an alternative temporary directory) while retaining quotas on their home directories, or bysetting appropriate soft limits while setting the hard limits very high -- maybe even the maximum size of the file system.

The result should be a warning when the soft limit is reached, but no enforcement of the hard limit. Users should still be able to build and create temporary filesbut, as they are naturally deleted during the build process, they should never be prevented from creating the files they need.

Quotas can also be used to help monitor and alert you to usage of special user accounts. I have, in the past, used quotas to monitor Web user accounts, so callednobody accounts and others, to ensure that they do not have the ability to write files to file systems they should not otherwise have access to. To do this, set thehard limit to 1KB for the file system and user you want to protect.

System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

5 de 7 28/08/09 18:59

Page 6: Disk Quotas on Unix-Mac OS X

Automatic monitoring

Monitoring disk space usage by hand is fine, but you don't want to be constantly running df (or even du) to determine the disk space being used, or that isavailable. You can automate the process and automatically send the administrator (or an administration group) an e-mail when the available space gets below acertain level. The script in Listing 10 monitors disk space, and you can set the warning (warninglimit) and emergency (lowlimit) limits, as well as the list of filesystems that are checked.

Listing 10. Monitoring disk space

#!/bin/sh

warninglimit=500000lowlimit=250000

filesystems="/export/data /export/home /"

for fs in $filesystemsdo size=`df -k $fs|grep $fs|awk '{ print $4; }'` if [ $size -le $lowlimit ] then mailx -s "URGENT: Low disk space for $fs ($size)" break fi if [ $size -le $warninglimit ] then mailx -s "WARNING: Low disk space for $fs ($size)"fidone

The key is the line that extracts the amount of free disk space for each file system:

size=`df -k $fs|grep $fs|awk '{ print $4; }'`

The script uses df, extracts only the line you want with grep, and then uses awk to extract the fourth column of data, which is the amount of free space.

You can then check the free space against the warning/low limits and generate a suitable error. To prevent the system from raising both a low and a warning error,the script checks for a low warning first and uses break to skip to the next file system in the loop before the warning test is tried.

Summary

Monitoring disk space is a vital part of your job as an administrator; run out of disk space and you could seriously impair the work of your users and, in severecases, lose data, or crash systems as they run out of the vital disk space required to operate.

You can determine the free space and usage for entire file systems using df, but this only gives you part of the picture. To determine where that disk space is beingused, you need to use the du tool to study individual directories. Through the use of find, you can even use du to find out how much space specific users are using.For more automated user-level disk usage and control, quotas are a much better option.

Resources

Learn

System Administration Toolkit: Check out other parts in this series.

developerWorks Podcasts: Listen to interesting interviews and discussions for software developers.

IBM Redbooks: IBM has a Redbook on AIX for Sun Solaris Administrators that contains useful information on the differences between AIX from theperspective of Solaris system administrators.

HP-UX 11i System Administration Handbook and Toolkit: Marty Poniatowski's (Prentice Hall PTR, ISBN: 0-13-060081-3) book contains information notonly about HP-UX, but also compares the HP-UX commands and generated output to other UNIX operating systems.

AIX® and UNIX articles: Check out other articles written by Martin Brown.

Search the AIX and UNIX library by topic:System administrationApplication developmentPerformancePortingSecurityTipsTools and utilitiesJava technologyLinuxOpen source

AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and

System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

6 de 7 28/08/09 18:59

Page 7: Disk Quotas on Unix-Mac OS X

expanding your UNIX skills.

New to AIX and UNIX: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.

AIX 5L™ Wiki: A collaborative environment for technical information related to AIX.

Safari bookstore: Visit this e-reference library to find specific technical resources.

developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.

Podcasts: Tune in and catch up with IBM technical experts.

Get products and technologies

IBM trial software: Build your next development project with software for download directly from developerWorks.

Discuss

Participate in the discussion forum.

Participate in the developerWorks blogs and get involved in the developerWorks community.

Participate in the AIX and UNIX forums:AIX 5L -- technical forumAIX for Developers ForumCluster Systems ManagementIBM Support AssistantPerformance Tools -- technicalVirtualization -- technicalMore AIX and UNIX forums

About the author

Martin Brown has been a professional writer for more than seven years. He is the author of numerous books and articles across a range of topics. His expertisespans myriad development languages and platforms -- Perl, Python, Java™, JavaScript, Basic, Pascal, Modula-2, C, C++, Rebol, Gawk, Shellscript, Windows®,Solaris, Linux, BeOS, Mac OS X and more -- as well as Web programming, systems management, and integration. He is a Subject Matter Expert (SME) forMicrosoft and regular contributor to ServerWatch.com, LinuxToday.com, and IBM developerWorks. He is also a regular blogger at Computerworld, The AppleBlog, and other sites. You can contact him through his Web site.

Trademarks | My developerWorks terms and conditions

System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

7 de 7 28/08/09 18:59