Detecting Dio Performance Problems

Methods for Detecting Demoted Direct I/O Performance Problems Barry J. Saad, ATS pSeries Advanced Technical Support Harold R. Lee, ATS pSeries Advanced Technical Support

Direct I/O (DIO) is an alternative method for accessing data stored in files on a file

system that is functionally equivalent to raw I/O for devices. The primary benefit of DIO

is reduced CPU utilization; however, if read and write requests are not properly constructed

the applications DIO request will be “demoted” resulting in significant negative impact to

application and system performance.

This paper provides a brief introduction to DIO, and introduces the techniques and tools

required to detect problems associated with demoted DIO read and write requests. To

accomplish this goal, the following topics will be discussed:

• DIO Primer - Understand the basics concepts of DIO

• Performance Impact of DIO – Understand the performance impact associated with

DIO, including the impact of DIO demotions.

• Using DIO – How to access files using DIO

• Test Scenarios – Sample program to exercise DIO

• Tools – AIX tools and techniques to monitor and troubleshoot DIO problems

DIO Primer

The default option when mounting a file system and accessing files is to allow the virtual

memory manager (VMM) to cache file-pages in main memory. In this situation, when the

application makes a read request, the VMM will determine if the file-page is in memory, if

so you have a cache hit and the data is copied from the main memory to the application’s

buffer. If the file-page isn’t in memory an I/O request will be issued to the physical disk

drive – the data will then be loaded into cache and copied to the application’s buffer. When

an application writes data, it is written from the application’s buffer to filesystem cache,

and then control returns to the application (i.e., the application doesn’t have to wait for the

physical write to occur, which improves performance). The actual writes to disk are done

later under the control of the system (i.e. the random/sequential write behind mechanism or

the syncd process).

Version 1.1 – 4/6/2005 1 of 27 © 2005 IBM Corporation

Methods for Detecting Demoted Direct I/O Performance Problems

DIO is an alternative method that allows an application to bypass the normal AIX file

caching mechanism. As shown in Figure 1, an application makes a read request (using the

read system call) to the kernel. The kernel then initiates a read request to the physical

disk drive, and finally the data is transferred from the disk drive directly to the

application’s buffer.

Kernel

Physical DiskDrive

Application

Application Buffer

Application makes a read request

Kernel issues a readrequest to the physical

disk drive

Data is moved fromphysical drive to the

application buffer

1

2

3

Figure 1: DIO Read Request

When an application issues a DIO write request, as shown Figure 2, the data is copied

directly from the application’s buffer to the physical disk drive. The drawback in this case

is that the application must wait for the I/O to complete before continuing execution.

Kernel

Physical DiskDrive

Application

Application Buffer

Application makes a write request

Kernel issues a writerequest to the physical

disk drive

Data is moved fromapplicatoin buffer to the

physical drive

Application waitsuntil I/O is

complete beforecontinuing

1

34

2

Figure 2: DIO Write Request

Version 1 – 3/17/2005 2 of 27 © 2005 IBM Corporation


Eliminating the copying of data from the application buffer to the system cache is the

primary benefit of using DIO. This benefit results in decreased VMM activity which

translates into reduced CPU utilization. A second benefit of DIO is that it can increase the

effectiveness of the system cache for other applications not using DIO. Anytime a file is

read or written, that file competes for space in the cache, and other file’s data may be

pushed out of cache. If the newly cached data has very poor reuse characteristics, the

effectiveness of the cache can be reduced. DIO allows applications to more effectively

control caching, thus releasing more cache space for files where caching policies are more

beneficial.

What type of application can benefit using DIO? Applications that read data once,

maintain their own cache, or need to use synchronous writes would benefit the most. For

these types of application, the system’s cache can result in the following undesirable

overhead: (1) the data has to be copied from system cache into the application’s buffer, (2)

the data may be stored in both the application’s cache and the system’s cache, and (3) an

application with poor reuse characteristics may have to force other pages out-of-memory.

Performance Impact of Direct I/O

When a DIO request is properly constructed the primary performance impact from DIO is

associated with the difference between cached and non-cached file pages.

• For every DIO read request a synchronous read from disk must occur. With file

caching it is possible that the read request could be satisfied from cache. The

difference between a read that is satisfied from file cache vs. a read from the

physical disk is a significant impact to an application’s response time performance.

Cache hits improve application response time.

• DIO bypasses the normal file system (JFS/JFS2) read-ahead algorithms, which

pre-fetches data into cache when the virtual memory manager detects sequential

access to a file. Applications can compensate for loss of read-ahead by issuing

larger read request. At a minimum, DIO reads can issue requests for 128K, which

will match the default read-ahead characteristics of the JFS/JFS2 file systems.



In addition, applications can also improve performance either by using multiple

threads or the asynchronous IO subsystem ( i.e. aio_read() system call).

• For every DIO write request a synchronous write to disk must occur. With file

caching the write is typically completed once the data is copied to the system’s

cache. The different between a write to cache vs. write to the physical disk is a

significant impact to an application’s response time performance. Applications can

improve response time performance by using either multiple threads or the

asynchronous IO subsystem ( i.e. aio_write() system call).

The majority of the benefits from DIO require the application to be aware of and capitalize

on using DIO. Unless you know the specific read and write characteristics of your

application or have the ability to tune them, your best choice for file access is using

conventional I/O, which is cached by the filesystem.

Finally and perhaps the most significant problem when using DIO is an application that does

not adhere to alignment and length restrictions for I/O requests. An application program

can query the offset, length and address alignment requirements using the finfo()

subroutine. When used with the FI_DIOCAP command, the finfo() subroutine returns

information in the diocapbuf structure as described in sys/finfo.h. The structure contains

the following fields:

• dio_offset – DIO seek alignment required

• dio_max – Maximum DIO transfer size

• dio_min – Minimum DIO transfer size

• dio_align – DIO memory alignment required

Failure to meet these requirements will cause the file reads and writes to use the normal

filesystem cache mechanism in a very inefficient manner. A demoted DIO read causes the

VMM to take over the read request, which results in the creation of filesystem cache pages

to receive the data, the data being moved into the filesystem cache from the disk, then

copied from the filesystem cache into the application’s buffer, then discarded from file

cache, and finally the filesystem cache pages are deleted and returned to the free list. A

demoted DIO write causes the VMM to create filesystem cache pages, read a page from



physical disk, copy the data from the application’s cache into the system cache, write the

data to physical disk synchronously, and then discarding the file cache. The additional

overhead associated with demoted DIO read and write requests require additional CPU

usage and increase application response time.

Using DIO There are three different methods by which DIO can be used. The specific method that is

best for your application depends on the scope of the files requiring DIO access and your

ability to modify the application. The three options, described below, are:

• Method 1: Mount the file system with the –o DIO option

• Method 2: Mount a portion of the file system with the –v namefs option

• Method 3: Modify the application to use DIO

Method 1: Mount the file system with the –o DIO option Use this method when all of the files in the file system need DIO access and the application

program can not be modified.

For example, mount –o dio /dev/lv05 /app1 In this case all the files in the /app1 directory will be accessed using DIO. Method 2: Mount a portion of the file system with the –v namefs option Use this method when only a portion of the files in a file system need DIO access and the

application program can not be modified.

For example, mount –v namefs –o dio /app1/read_once_files /rof In this case, you would move files that require DIO access into a common directory (e.g.,

/app1/read_once_files), and then mount that sub-directory using –v namefs option of the

mount command. Now, applications reading or writing files from /rof will use DIO.



Method 3: Modifying the Application to use DIO This is the preferred method for access files; however, the application does have be

modified to open a file with the O_DIRECT flag, and adhere to the correct alignment

requirements as specified in the data returned by the finfo() system call.

For example, open( “/home/bjsaad/file1”, O_RDONLY|O_DIRECT)

In this case, the file is open read only with DIO access. Since this file is part of a normally

mounted file system it’s possible for other applications to use this file. To avoid

consistency issues, if there are multiple opens of a file and some of them are not DIO the

file will stay in its normal cached access mode. Only when the file is open exclusively by a

DIO program will the file be placed in direct I/O mode. This also applies if the file is

mapped into virtual memory using either shmat() or mmap() system call. If the conflicting

or non-DIO access is eliminated the system will attempt to move the file into DIO mode.

Changing the file from normal mode to Direct I/O mode can be rather expensive because it

requires writing all modified pages to disk and removing all the file’s pages from memory.

Test Programs The test programs used to exercise DIO are described below, and the source code for each

program is provided in Appendix A and Appendix B.

• Appendix A – the directio program - reads a specified file using either the

specified or default read block size and initial seek position parameters. The

optional parameters are provided to illustrate the impact when either the block size

or initial read position is set incorrectly. The command syntax is

directio [-b block size] [-s seek position] –f filename

• Appendix B – the directio_w program - writes a specified file using either the

specified or default file size, write block size and initial seek position parameters.

The optional parameters are provided to illustrate the impact when either the block

size or initial seek position is set incorrectly. The command syntax is

directio_w [-w file size] [-b block size] [-s seek position] –f

filename



For example,

• Using the directio program to read a file, the command below will open the file

“/home/bjsaad/proj/test.file”, and read the entire file with a block size of 4096

bytes. In this case, 4096 will result in DIO reading the file on correctly aligned

boundaries.

# ./directio –b 4096 –f /home/bjsaad/proj/test.file

• Using the directio_w program to write a file, the command below will open the file

“/home/bjsaad/file1”, and write 1MB (1048576 bytes) with a block size of 4096

bytes. In this case, 4096 will result in DIO writing the file on correctly aligned

boundaries.

# ./directio –w 1048576 –b 4096 –f /home/bjsaad/file1



Method to Detect Performance Problems The AIX trace command is the primary method for detected demoted DIO requests. This

section will provide examples of how to use the trace command to detect DIO demotion

problems, and in conjunction with the filemon and tprof commands, illustrate the impact

associated with demoted DIO requests.

Test Case 1: Reading a JFS file with an alignment size of 4096 # trace –aj3B1,3B2,3B3 # ./directio –b 4096 –f /home/bjsaad/proj/test.file # trcstop # trcrpt –o trcrpt.tc1 Explanation of the commands:

1. The trace –aj3B1,3B2,3B3 command starts trace in the background, and limits

the to trace hooks 3B1, 3B2 and 3B3. In this case we’re limiting the trace hooks

for two reasons (1) we’re only interesting is seeing the DIO related hooks, and (2)

we’re limiting the size of the trace file (located in /var/adm/ras/trcfile). To know

what trace hooks to use you need to review the trace hook id header file

(/usr/include/sys/trchkid.h), which is shown below in Figure 3.

2. The ./directio –b 4096 –f /home/bjsaad/proj/test.file command reads

the specified file using a 4096 block size.

3. The trcstop command stops the background trace.

4. The trcrpt –o trcrpt.tc1 command formats the raw trace output in a readable

format.

Now, determining whether your DIO requests have been demoted is a straight forward

exercise. Review the trace file, shown in Figure 4, and every time you see 3B3 the DIO

request has been demoted to an inefficient cached I/O request is being executed. As the

trace file snippet shows, you have no 3B3 events, which means that DIO is functioning as

designed.



Trace Hooks JFS DIO

Figure 3: Trace Hook Id Header File

No trace hook 3B3; therefore, DIO requests are functioning correctly. 3B1 and 3B2 hooks are normal.

Figure 4: trcrpt.tc1 Output

Finally, you will notice that the all the trace hooks are “UNDEFINED” this is because the

trace hooks haven’t defined in the trace format file (/etc/trcfmt).



Test Case 2: Reading a JFS file with an alignment size of 2048 # trace –a # ./directio –b 2048 –f /home/bjsaad/proj/test.file # trcstop # trcrpt –d3B1,3B2,3B3 –Oexec=on,pid=on –o trcrpt.tc2 Explanation of the commands:

1. The trace –a command starts trace in the background. Unlike the first test case,

we’re collecting all the trace hooks.

2. The ./directio –b 2048 –f /home/bjsaad/proj/test.file command reads

the specified file using a 2048 block size.

3. The trcstop command stops the background trace,

4. The trcrpt –Opid=on –o trcrpt.tc2 command formats the raw trace output

into a readable format, and limits the output to hooks 3B1, 3B2, and 3B3. Limiting

hooks on output make the report easier to analyze, and allows the option ‘-Opid=on’

to provide the correct data, which places a new column in the trace output file that

contains the process ID number, which is beneficial in identifying the specific

process that has demoted DIO requests.

In this case the DIO read is being made using a block size of 2048. As the trace file shows,

Figure 5, 3B3 events have occurred, which indicate that the DIO requests have been

demoted. When a request is demoted the following events occur:

• The first read request for 2048 bytes cause the DIO to be demoted

• <Extra Work> A page is allocated in system cache

• Read request initiated for 4096 bytes

• <Extra Work> Data is copied into the system cache

• <Extra Work> Data is copied from the system cache to the application buffer

• <Extra Work> The system cache data is discarded

• The next read request for 2048 bytes causes the DIO to be demoted

• <Extra Work> A page is allocated in the system cache



• <Extra Work> Read request initiated for 4096 bytes. NOTE: You’re re-reading the

same page of data! For every 2048 byte read request that is demoted you cause 2

physical I/O to disk. So, in the first two reads the application is requesting 4096

bytes of data; however, the system is forced to read 8192 bytes of data. Figure 6,

shows a snippet form filemon that illustrates this point.

• <Extra Work> The system cache is discarded

• The cycle will continue unless you the application reads on the correct alignment

boundaries

3B3 Events indicate demoted DIO requests

PID is 21948




# of 512 blocks read from the physical device = equals the size of the file.

# of 512 blocks read from the physical device = 2 x the size of the file. Note not exactly 2x because the file isn’t a even multiple of 4k. Increase # of physical reads is due the DIO demotion!

Figure 6: Filemon Output



Test Case 3: Reading a JFS file with an alignment size of 4096 and an offset of 1024 # trace –aj3B1,3B2,3B3 # ./directio –s 1024 –b 4096 –f /home/bjsaad/proj/test.file # trcstop # trcrpt –o trcrpt.tc3

In this example, the application reads on 4096 block size; however, the offset alignment is

incorrect. Therefore, all the I/O requests will be demoted. The snippet, shown in Figure 7,

shows 3B3 events, which indicate demoted DIO requests.




Test Case 4: Reading a JFS2 file with an alignment size of 4096 # trace –aj59B,59C # ./directio –b 4096 –f /home/hrlee/proj/test.file # trcstop # trcrpt –o trcrpt.tc4 The same technique is used for JFS2; however, the trace hooks ids have changed. To

determine the correct hook id we check the file /usr/include/sys/trchkid.h, as shown in

Figure 9. The trace hook header file indicates that we need to look for hook 59B sub-hook

3 events to determine whether a DIO request has been demoted. Search for this sub-hook

has been made easier because the appropriate entries have been made to the trace format

file (/etc/trcfmt), shown in Figure 10. So, to determine whether a DIO request has been

demoted review the trace file output looking of the phase “DIO DEMOTED”.

The trace report, shown in Figure 8, shows no DIO DEMOTED events.

No DIO DEMOTED events




For JFS2 trace using hooks 59B and 59C. Sub-Hook 3 indicates that DIO output has been demoted. The JFS2 hook have been added to the /etc/trcfmt file; therefore, you search for DIO DEMOTED in the trace file.

Figure 9: Track Hooks Ids

For Hook 59B sub-hook 3 the trace format file specified what to print. Search the file for DIO DEMOTED to determine whether DIO request has been demoted.

Figure 10: Trace Format File



Test Case 5: Reading a JFS2 file with an alignment size of 2048 # trace –aj59B,59C # ./directio –b 2048 –f /home/hrlee/proj/test.file # trcstop # trcrpt –o trcrpt.tc4

This example, reads the file using a block size of 2048. As shown in Figure 11, DIO

DEMOTED events have occurred, which indicate that the DIO requests has been demoted,

and inefficient cached I/O has being preformed.

DIO DEMOTED




Test Case 6: Writing a 1 MB JFS2 file with an alignment size of 2048 # trace –aj59B,59C # directio_w –w 1048576 –b 2048 –f /home/hrlee/test_file # trcstop # trcrpt –o trcrpt.tc7 In this example, we’re writing 1MB to a file using 2048 block size. As shown in Figure 12,

the DIO DEMOTED event has occurred, which means that the DIO write request is using

inefficient cache I/O. In addition, as shown in the filemon output , Figure 13, the system is

not only force to write more data than required, but it also has to read data before each

write. All the operations are synchronous, which result in a significant impact to the

application’s performance.

Demoted Write Request – DIO DEMOTED




Incorrect alignment results in system writing twice as much data as requested by the application. 4096 blocks x 512 bytes / block = 2MB

Incorrect alignment results in the VMM read data prior to writing

Figure 13: Filemon Output for write

DIO Demotion Response Time and CPU Utilization Penalty The following figures illustrate that demoted DIO requests impact both response time and

CPU utilization. The increased response time is due to the fact that the system is forced

to read or write more data than specified by the application. The increased CPU utilization

is related to the additional VMM overhead.

• Figure 14, shows the response time using the timex. The output clearly indicates

that response time increases dramatically when an application has demoted DIO

requests.

• Figure 15, shows a CPU utilization of 14 ticks for correctly aligned DIO requests.

• Figure 16, shows a CPU utilization of 15 ticks for demoted DIO requests.



Impact of demoted DIO requests Increased Response Time 14.15 secs w/demoted DIO requests .68 secs w/normal DIO requests

Figure 14: Output from timex

Consumed CPU = 14 ticks

Figure 15: tprof output for 4K read



Consumed CPU = 51 ticks Increased utilization is due to added VMM overhead required to handle demoted DIO requests.

Figure 16: tprof output for 2K reads

In summary, we have explained how DIO works and when used with correctly aligned

requests reduce CPU and response time requirements for an application. Used incorrectly,

with the badly aligned requests, demoted DIO requests have a significant negative impact

on application performance.

Before using DIO in a production environment it is important to understand the read and

write characteristics and tuning option available for an application. Again, application that

have poor cache reuse, common in technical workloads, or application that have their own

cache are best positioned to use DIO.

Finally, we presented the techniques and tools used to detect performance problems with

demoted DIO requests. As shown, the trace command is a simple and effective tool that

can quickly highlight applications that have demoted DIO requests.

If you have any questions please contact either Barry J. Saad at ([email protected]) or

Harold R. Lee ([email protected]).


mailto:[email protected]

mailto:[email protected]


References

• General Programming Concepts: Writing and Debugging Programs (SC23-4128-08) • Improve Database Performance on File System Containers in IBM DB2 Universal

Database Stinger using Concurrent I/O on AIX – dated August 2004



Appendix A: Example program directio.c

#include <stdio.h> #include <fcntl.h> #include <unistd.h> #include <sys/finfo.h> int main( int argc, char *argv[]) { int fd ; int blk_size = 4096 ; extern int optind ; extern char *optarg ; char file_in[255]; char *buffer ; int i; int offset ; struct diocapbuf diobuf; int c ; int r; offset = 0 ; printf("My Process Id is %u\n", getpid() ) ; while( ( c=getopt( argc, argv, "s:b:f:")) != EOF ) { switch(c) { case 'b': blk_size = atoi(optarg); break; case 'f': strcpy( file_in, optarg) ; break ; case 's': offset = atoi(optarg); break ; case '?':



printf("Usage: directio [-s seek location -b size_to_read] -f file_name\n"); exit(-1); break ; } } if( (buffer=(char *) malloc( blk_size ) ) == NULL ) { printf("Error: Unable to malloc space\n"); exit(-1); } printf("Opening file %s for direct I/O reading\n", file_in); printf("Reading with block size of %d starting at %d\n", blk_size, offset); if( ( fd = open( file_in ,O_RDONLY|O_DIRECT ) ) < 0 ) { printf("Error: Unable to open file %s\n", file_in); exit(-1); } if( finfo( file_in, FI_DIOCAP, &diobuf, sizeof( diobuf )) < 0 ) { printf("Error: Unable to get finfo information\n"); } else { printf("dio_offset: %llu\n", diobuf.dio_offset ) ; printf("dio_min: %llu\n", diobuf.dio_min ) ; printf("dio_max: %llu\n", diobuf.dio_max ) ; printf("dio_align: %u\n", diobuf.dio_align ) ; } printf("File is open and ready for reading ... hit enter to continue\n"); getchar(); if( lseek( fd, offset, SEEK_SET) < 0 ) { printf("Error: Unable to to seek to location %d\n", offset); } /*



* Read the file */ do { r = read( fd, buffer, blk_size ) ; } while( r ); printf("Ready to exit ... hit any key to continue\n"); getchar(); exit(0); }



Appendix B: Example Program directio_w.c #include <stdio.h> #include <fcntl.h> #include <unistd.h> #include <sys/finfo.h> int main( int argc, char *argv[]) { int fd ; int blk_size = 4096 ; extern int optind ; extern char *optarg ; char file_in[255]; char *buffer ; int i; int write_size; int offset ; struct diocapbuf diobuf; int c ; int r; offset = 0 ; printf("My Process Id is %u\n", getpid() ) ; while( ( c=getopt( argc, argv, "s:w:b:f:")) != EOF ) { switch(c) { case 'b': blk_size = atoi(optarg); break; case 'f': strcpy( file_in, optarg) ; break ; case 's': offset = atoi(optarg); break ; case 'w':



write_size = atoi(optarg); break ; case '?': printf("Usage: directio_w [-w write_size -s seek to location -b size_to_read] -f file_name\n"); exit(-1); break ; } } if( (buffer=(char *) malloc( blk_size ) ) == NULL ) { printf("Error: Unable to malloc space\n"); exit(-1); } memset( buffer, 'a', blk_size ) ; printf("Opening file %s for direct I/O writing \n", file_in); printf("Writing %d bytes with block size of %d starting at %d\n", write_size,blk_size, offset); if( ( fd = open( file_in ,O_CREAT|O_RDWR|O_DIRECT ) ) < 0 ) { printf("Error: Unable to open file %s\n", file_in); exit(-1); } if( finfo( file_in, FI_DIOCAP, &diobuf, sizeof( diobuf )) < 0 ) { printf("Error: Unable to get finfo information\n"); } else { printf("dio_offset: %llu\n", diobuf.dio_offset ) ; printf("dio_min: %llu\n", diobuf.dio_min ) ; printf("dio_max: %llu\n", diobuf.dio_max ) ; printf("dio_align: %u\n", diobuf.dio_align ) ; } /* printf("File is open and ready for writing ... hit enter to continue\n"); getchar(); */ if( lseek( fd, offset, SEEK_SET) < 0 ) {



printf("Error: Unable to to seek to location %d\n", offset); } /* * write the file */ r = 0 ; do { r = r + write( fd, buffer, blk_size ) ; } while( r < write_size ); exit(0); }


Documents

Detecting Dio Performance Problems