1 Disk I/O Performance focusing on Caching Saturday, October 03, 2015 Guy Peleg President Maklee Engineering [email protected]

1

Disk I/O Performancefocusing on

Caching

Friday, April 21, 2023

Guy PelegPresident

Maklee Engineering

[email protected]

2

Why should I care?

Computers became very fast over the last decade

I/O performance remained (almost) the same

Disk I/O is an extremely! slow operation

3

Why should I care?

1.6Ghz rx7640 single cell

CPU cycle is 0.5 ns

Memory latency is ~ 250 ns

EVA5000 read latency 5 – 10 ms

Now let’s change the scale and assume 0.5 ns = 1 sec

4

Why should I care?

1.6Ghz rx7640 single cell

CPU cycle is 1 second

Memory latency is 8.3 minutes

EVA5000 read latency 23 days !!!!!

5

Why should I care?

Test program written in CCreates a 1GB container fileOpens the newly created filePositions at EOFWrites a small record

Closes file

1.6Ghz rx2600

OpenVMS V8.3-1H1

6

0

50

100

150

200

250

300

350

400

450

500Out of thebox

I/O tuning

Seconds to complete the test program

less is better

Why should I care?

7

I/O Best Practices

“The fastest I/Os are those that are avoided”OpenVMS info desk Oct. 2004

Cache as much as you canCaching was invented to attempt and reduce the need for physical I/Os.

This session is all about providing techniques for avoiding/reducing I/O

8

High Water Marking

The file highwater mark attribute guarantees that a user can not read data that was not written by the user.

The init command turns this feature on by default

Tradeoff between security and performacne

INIT/NOHIGH

SET VOLUME/NOHIGH

9

0

50

100

150

200

250

300

350

400

450

500Out of the box

I/O tuning

Highwatermarking turnedoff

Seconds to create a container file

less is better

The impact of highwater mark

10

Data caching

Data could be cached by:The ApplicationRMS (or Database cache)OpenVMS file caching (XFC)Hardware (disk controller)

No one knows your application better than you !!

Try to satisfies I/O requests as close to the application as possible

11

XFC

eXtended File Cache

Introduced with OpenVMS V7.3

Replaces VIOC

64 bit

No closed file limit

Will cache I/Os larger than 35 blocks

12

XFC

Supports readahead

(currently) does not support writeback

May grow up to 50% of physical memory

Grows and shrinks according to free physical memory.

May be forced to return memory instantly

No caching for files opened for write cluster wide

13

XFC Reserved Memory

Using reserved memory for XFC is slightly faster

Reserved Memory for Permanent memory SYSMAN> RESERVED_MEMORY ADD VCC$MIN_CACHE_SIZE/SIZE=1500 SYSMAN> RESERVED_MEMORY ADD VCC$MIN_CACHE_SIZE/SIZE=1500

/ALLOCATE/NOPAGE_TABLES/NOZERO/NOGLOBAL/ALLOCATE/NOPAGE_TABLES/NOZERO/NOGLOBAL

VCC_MAX_CACHE must be greater than the reserved memory size.

14

XFC SYSGEN paraeters

$ mc sysgen sh vccParameter Name Current Default Min. Max. Unit Dynamic-------------- ------- ------- ------- ------- ---- -------VCC_FLAGS 2 2 0 -1 BitmaskVCC_MAXSIZE 6400 6400 0 3700000 BlocksVCC_MAX_CACHE -1 -1 0 -1 Mbytes DVCC_MAX_IO_SIZE 256 127 0 -1 Blocks DVCC_MAX_LOCKS -1 -1 50 -1 Locks DVCC_READAHEAD 1 1 0 1 Boolean DVCC_WRITEBEHIND 1 1 0 1 Boolean DVCC_WRITE_DELAY 30 30 0 -1 Seconds DVCC_PAGESIZE 0 0 0 -1 DVCC_RSVD 0 0 0 -1 D$

15

$show memory /cache/full

$ sh mem/cache/full System Memory Resources on 13-MAY-2008 12:44:19.80

Extended File Cache (Time of last reset: 8-MAY-2008 22:57:33.05) Allocated (GBytes) 1.35 Maximum size (GBytes) 4.00 Free (GBytes) 0.21 Minimum size (GBytes) 0.00 In use (GBytes) 1.13 Percentage Read I/Os 26% Read hit rate 95% Write hit rate 0% Read I/O count 2901784 Write I/O count 8118032 Read hit count 2781426 Write hit count 0 Reads bypassing cache 180 Writes bypassing cache 6860600 Files cached open 848 Files cached closed 993 Vols in Full XFC mode 0 Vols in VIOC Compatible mode 3 Vols in No Caching mode 0 Vols in Perm. No Caching mode 0

I/O Statistics - Distributions (MAX_IO_SIZE: 127)

-------------------------------------------------Transfer Size: Reads Read Hits Writes 1 Block IO: 193447 191602 435529 2 Block IO: 77685 76719 2148301 3 Block IO: 37242 37063 322191 4 Block IO: 108729 108557 238616 5 Block IO: 11719 11592 139933 6 Block IO: 77626 77453 92434 7 Block IO: 45780 45618 141327 8 Block IO: 191750 191351 136810 9 Block IO: 5359 5280 207786 10 Block IO: 17835 17787 140148 11 Block IO: 5343 5285 60112

16

Show memory /cache/full

… 27 Block IO: 4670 4655 14815 28 Block IO: 5657 5625 15013 29 Block IO: 5218 5192 14984 30 Block IO: 4717 4696 14841 31 Block IO: 4708 4679 14344 32 Block IO: 527966 467655 996501 33 Block IO: 48 33 2269… 63 Block IO: 466 432 6366 64 Block IO: 1075132 1065615 22811 65 Block IO: 2 0 1144 124 Block IO: 200562 190862 193467 125 Block IO: 1 0 493 126 Block IO: 1 0 504 127 Block IO: 0 0 512

>127<256 Block IO: 21 263932>255 Block IO: 49 176683-------------------------------------------------Totals: 2901784 2781426 8118032

17

Forcing XFC to shrink

$ show mem/cache System Memory Resources on 13-MAY-2008 12:42:54.59

Extended File Cache (Time of last reset: 13-MAY-2008 10:13:01.11)

Allocated (MBytes) 222.32 Maximum size (MBytes) 4096.00 Free (MBytes) 0.19 Minimum size (MBytes) 3.12 In use (MBytes) 222.12 Percentage Read I/Os 87% Read hit rate 29% Write hit rate 0% Read I/O count 90334 Write I/O count 12864 Read hit count 26537 Write hit count 0 Reads bypassing cache 589 Writes bypassing cache 9969 Files cached open 383 Files cached closed 631 Vols in Full XFC mode 0 Vols in VIOC Compatible mode 2 Vols in No Caching mode 0 Vols in Perm. No Caching mode 0

SYSBCK> mc sysgenSYSGEN> SET VCC_MAX_CACHE 100SYSGEN> W ASYSGEN> EXIT

$ set cache/reset

18

Forcing XFC to shrink

$ sh mem/cache System Memory Resources on 13-MAY-2008 12:43:22.10

Extended File Cache (Time of last reset: 13-MAY-2008 10:13:01.11)

Allocated (MBytes) 95.63 Maximum size (MBytes) 100.00 Free (MBytes) 0.02 Minimum size (MBytes) 3.12 In use (MBytes) 95.60 Percentage Read I/Os 87% Read hit rate 29% Write hit rate 0% Read I/O count 90391 Write I/O count 12894 Read hit count 26584 Write hit count 0 Reads bypassing cache 589 Writes bypassing cache 9996 Files cached open 383 Files cached closed 181 Vols in Full XFC mode 0 Vols in VIOC Compatible mode 2 Vols in No Caching mode 0 Vols in Perm. No Caching mode 0

19

Cache pollution

The Cache is a limited resource

Avoid polluting the cache

OpenVMS attempts to reduce pulltionVMS BACKUP

Caching disabled for all backup IO

Some utilities are not “cache safe”COPYCONVERT

Consider disabling caching temporarilyvcc_max_io_size = 0 mount/nocacheset file/cache=no

OpenVMS provides DCL commands to view the contents of the cache

20

Free Hot File Tracking Utility

$ sh mem/cache=(volume=*,topqio)

System Memory Resources on 26-APR-2008 01:39:15.03

Extended File Cache Top QIO File Statistics:

_$1$DGA242: (DISK$SEC6), Caching mode is VIOC Compatible

_$1$DGA242:[VMS$COMMON.SYSEXE]RIGHTSLIST.DAT;1 (open)

Caching is enabled, active caching mode is Write Through

Allocated pages 9 Total QIOs 107

Read hits 92 Virtual reads 107

Virtual writes 0 Hit rate 85 %

Read aheads 0 Read throughs 107

Write throughs 0 Read arounds 0

Write arounds 0

_$1$DGA242:[VMS$COMMON.SYSEXE]VMS$OBJECTS.DAT;2 (open)







Write arounds 0

_$1$DGA242:[VMS$COMMON.SYSEXE]VMS$AUDIT_SERVER.DAT;1 (open)







Write arounds 0

Total of 3 files for this volume

21

Free Hot File Tracking Utility

_$1$DGA142: (DISK$MVMS), Caching mode is VIOC Compatible

_$1$DGA142:[VMS$COMMON.SYSLIB]DECC$SHR.EXE;1 (open)







Write arounds 0

_$1$DGA142:[VMS$COMMON.SYSLIB]LIBRTL.EXE;1 (open)







Write arounds 0

_$1$DGA142:[VMS$COMMON.SYSLIB]CMA$TIS_SHR.EXE;1 (open)







Write arounds 0

Avoid caching files that pollute

the cache

22

I/O response time

SDA> xfc show volume/brief

Summary of XFC Cached Volumes (CVBs)------------------------------------Volume Name CVB Open Closed Total Read Read Write ... Response (Milliseconds)... Files Files I/Os Hits Count Count Hits disk AverageDISK$FOOBAR FFFFFFFEE01895E0 0 0 0 0 0 0 (N/A) (N/A) (N/A)DISK$GP FFFFFFFEE0189380 0 0 0 0 0 0 (N/A) (N/A) (N/A)DISK$ORACLE FFFFFFFEE0189120 26 3 1872255 0 0 1872255 (N/A) 0.0000 0.0000DISK$ORA2 FFFFFFFEE0188EC0 73 177 22015701 14108183 21116834 898891 0.0232 0.5811 0.2236DISK$SYS FFFFFFFEE0188C60 0 0 0 0 0 0 (N/A) (N/A) (N/A)DISK$ORAIDX FFFFFFFEE0188A00 0 0 1 0 1 0 (N/A) (N/A) (N/A)DISK$SYSBCK FFFFFFFEE01887A0 2 0 0 0 0 0 (N/A) (N/A) (N/A)DISK$LEEHE FFFFFFFEE0188540 4 3 27676052 27667501 27674665 1387 0.0118 0.4007 0.0120DISK$MIKA FFFFFFFEE01882E0 0 0 0 0 0 0 (N/A) (N/A) (N/A)DISK$SYS831H1 FFFFFFFEE0188080 313 183 2736618 2668894 2713025 23594 0.0179 0.5425 0.0308

SDA>XFC SHOW VOLUME/BRIEF

23

The XFC “overhead”

0

2

4

6

8

10

12

Cachingdisabled

Cachingenabled - firstattempt

Cachingenabled -second attempt

Elapsed time to copy 150MB file, rx2600, HSG80, OpenVMS V8.3

24

0

50

100

150

200

250

300

350

400

450

500Out of the box

I/O tuning

High WaterMarking disabled

Disable caching


less is better

The XFC “overhead”

25

Shadowed RAM disk

Applications that mainly read data would benefit from using shadowed RAM disk.

The Shadow server will read from memory and will write to both devices.

Forces data to remain resident in memory.

Significantly boosts performance when files are opened cluster wide by multiple users.

XFC will not help

26

Physical Disk Vs. RAM disk

C application that processes records read from sequential fileEach I/O 124 BlocksRX2600, OpenVMS V8.3, HSG80

0

5

10

15

20

25

$1$DGA425

Single memberDSA666

DSA666 withRAM disk

File in Cache

Elapsed time to read 250MB file (less is better)

27

RMS

RMS holds great potential for reducing I/Os & improving performance

The C RTL uses RMS

Most C applications would benefit from RMS tuning

28

RMS

RMS parameters related to performance:

FAB/RAB parameters (should you have access to the code):ASY, RAH, WBH, DFWALQ & DEQMBC & MBFNOSHR, NQL, NLK

SET RMS …/SYSTEM | /PROCESS/BUFFER_COUNT=n/BLOCK_COUNT=n

SYSGEN> SET RMS_SEQFILE_WBH 1

Global Buffers

29

gZIP & RMS

gZIP is written in C – I/Os eventually reach RMS

1.6 Ghz rx2600, MSA30, OpenVMS V8.3

Test 1Compress 5.67 GB savesetDecompress 2.74 gZIP archiveDefault O/S & RMS settings

Test 2Compress 5.67 GB savesetDecompress 2.74 gZIP archiveSET RMS/BLOCK=127/EXTEN=60000/BUFFER=8, RMS_SEQFILE_WBH=1

30

gZIP & RMS

0

5

10

15

20

25

30

35

40Test 1 - Compress

Test 1 -Decompress

Test 2 - Compress

Test 2 -Decompress

Elapsed Time in Minutes (less is better)

31

FTP Performance

0

5

10

15

20

25

30

35

40

45

50Throughput

Elapsed Time

32

0

50

100

150

200

250

300

350

400

450

500Out of the box

I/O tuning

High WaterMarking disabledDisable caching

RMS Tuning


less is better

Impact of RMS tuning

33

I/O Consolidation - PC Sampling – “Before”

PC Sampling Information (CPU_CYCLES):-------------------------------------Start VA End VA Image Count Percent----------------- ----------------- ---------------------------------------- ----------- --------FFFFF802.89400000 FFFFF802.89415FFF Kernel Promote VA 2 0.78%FFFFFFFF.80000100 FFFFFFFF.800111FF SYS$BASE_IMAGE 10 3.89%FFFFFFFF.80011200 FFFFFFFF.80068FFF SYS$PLATFORM_SUPPORT 1 0.39%FFFFFFFF.800A0000 FFFFFFFF.801D3CFF SYSTEM_PRIMITIVES_MIN 86 33.46%FFFFFFFF.801D3D00 FFFFFFFF.802537FF SYSTEM_SYNCHRONIZATION_MIN 21 8.17%FFFFFFFF.80365600 FFFFFFFF.80451FFF EXCEPTION 1 0.39%FFFFFFFF.80452000 FFFFFFFF.80543EFF IO_ROUTINES 6 2.33%FFFFFFFF.80553200 FFFFFFFF.80699DFF PROCESS_MANAGEMENT 43 16.73%FFFFFFFF.80699E00 FFFFFFFF.807A25FF SYS$VM 2 0.78%FFFFFFFF.807A2600 FFFFFFFF.807F05FF LOCKING 14 5.45%FFFFFFFF.80847600 FFFFFFFF.80927CFF F11BXQP 22 8.56%FFFFFFFF.80927D00 FFFFFFFF.80B40CFF RMS 10 3.89%FFFFFFFF.80B9CC00 FFFFFFFF.80C091FF SECURITY 4 1.56%FFFFFFFF.80CB0A00 FFFFFFFF.80D2C9FF SYS$XFCACHE 11 4.28%FFFFFFFF.810C1B00 FFFFFFFF.811080FF SYS$DKDRIVER 1 0.39%FFFFFFFF.8406E000 FFFFFFFF.8425FFFF LIBRTL 3 1.17%FFFFFFFF.84260000 FFFFFFFF.84285FFF LIBOTS 2 0.78%FFFFFFFF.844FA000 FFFFFFFF.845ABFFF PTHREAD$RTL 18 7.00% Not Found 4

34

I/O Consolidation - PC Sampling – “After”

PC Sampling Information (CPU_CYCLES):-------------------------------------Start VA End VA Image Count Percent----------------- ----------------- ---------------------------------------- ----------- --------00000000.00000000 00000000.7ADCBFFF Process Space 28936 34.50%FFFFF802.89400000 FFFFF802.89415FFF Kernel Promote VA 236 0.28%FFFFFFFF.80000000 FFFFFFFF.800000FF SYS$PUBLIC_VECTORS 13 0.02%FFFFFFFF.80000100 FFFFFFFF.800111FF SYS$BASE_IMAGE 3272 3.90%FFFFFFFF.80011200 FFFFFFFF.80068FFF SYS$PLATFORM_SUPPORT 56 0.07%FFFFFFFF.800A0000 FFFFFFFF.801D3CFF SYSTEM_PRIMITIVES_MIN 7848 9.36%FFFFFFFF.801D3D00 FFFFFFFF.802537FF SYSTEM_SYNCHRONIZATION_MIN 2349 2.80%FFFFFFFF.8027C700 FFFFFFFF.8029CAFF SYS$EI1000.EXE 2 0.00%FFFFFFFF.80365600 FFFFFFFF.80451FFF EXCEPTION 153 0.18%FFFFFFFF.80452000 FFFFFFFF.80543EFF IO_ROUTINES 3664 4.37%FFFFFFFF.80543F00 FFFFFFFF.805531FF SYSDEVICE 838 1.00%FFFFFFFF.80553200 FFFFFFFF.80699DFF PROCESS_MANAGEMENT 5334 6.36%FFFFFFFF.807A2600 FFFFFFFF.807F05FF LOCKING 1 0.00%FFFFFFFF.80927D00 FFFFFFFF.80B40CFF RMS 2 0.00%FFFFFFFF.80B9CC00 FFFFFFFF.80C091FF SECURITY 479 0.57%FFFFFFFF.80CB0A00 FFFFFFFF.80D2C9FF SYS$XFCACHE 3 0.00%FFFFFFFF.80D58E00 FFFFFFFF.80E1C6FF SYS$CLUSTER 1 0.00%FFFFFFFF.81049500 FFFFFFFF.810C1AFF SYS$PKMDRIVER 4 0.00%FFFFFFFF.810C1B00 FFFFFFFF.811080FF SYS$DKDRIVER 1 0.00%FFFFFFFF.8110BF00 FFFFFFFF.8113D3FF SYS$LAN_CSMACD.EXE 1 0.00%FFFFFFFF.8113D400 FFFFFFFF.811B03FF SYS$PEDRIVER.EXE 6 0.01%FFFFFFFF.81356900 FFFFFFFF.813BCAFF SYS$LOADABLE_IMAGES:SYS$LTDRIVER.EXE 2 0.00%FFFFFFFF.813E3E00 FFFFFFFF.816028FF TCPIP$INTERNET_SERVICES 6 0.01%FFFFFFFF.81640300 FFFFFFFF.817141FF TCPIP$NFS_SERVICES 2 0.00%FFFFFFFF.8406E000 FFFFFFFF.8425FFFF LIBRTL 18395 21.93%FFFFFFFF.84260000 FFFFFFFF.84285FFF LIBOTS 2251 2.68%FFFFFFFF.844FA000 FFFFFFFF.845ABFFF PTHREAD$RTL 9452 11.27%FFFFFFFF.845AC000 FFFFFFFF.850BDFFF DECC$SHR 555 0.66% Not Found 633

35

RMS Global Buffers

RMS Global Buffers provide another layer of caching between RMS & XFC

Does it make sense to use both?

YES !

36

RMS Global Buffers and/or XFC?

RMS Global buffers WILL cache files open for writeshare in a cluster.

RMS GB require knowledge of application & I/O to apply them effectively.

$ SET FILE/GLOBAL_BUFFER=n

XFC is S2 Space based and has no effective limit to its size other than physical memory available

Prior to V8.3 RMS GB are limited to a max value of 32767 buffers per file.

37

RMS Global Buffers AND XFC make a good pair

Remember slide 13?RMS is closer to the application.

Use global buffers for files open for shared write in a cluster – RIGHTSLIST.DAT is a good example with a common system disk.

XFC will cache as much as it can once RMS passes I/O to it

38

The Impact of RMS Global Buffers

39

The Impact of RMS Global Buffers

40

File Expansions

The volume extension size specifies the number of blocks to be added when RMS automatically extends a file.

Larger values result in fewer file extensions.

SET VOLUME/EXTENSION=n

FAB$W_DEQ

Maximum extension size – 65,535 blocks

41

File Expansions

$ sh dev dka100/full

Disk $1$DKA100: (STRIKE), device type DEC RZZ28B, is online, allocated, deallocate on dismount, mounted, file-oriented device, shareable, available to cluster, error logging is enabled.

Error count 0 Operations completed 57 Owner process "SYSTEM" Owner UIC [SYSTEM] Owner process ID 00000222 Dev Prot S:RWPL,O:RWPL,G:R,W Reference count 2 Default buffer size 512 Total size 14.05GB Sectors per track 32 Total cylinders 28786 Tracks per cylinder 32 Logical Volume Size 14.05GB Expansion Size Limit 16.96GB Allocation class 1

Volume label "XBC3_RESD$" Relative volume number 0 Cluster size 4 Transaction count 1 Free space 106.93MB Maximum files allowed 2947659 Extend quantity 5 Mount count 1 Mount status Process Cache name "_$1$DKA0:XQPCACHE" Extent cache size 64 Maximum blocks in extent cache 21900 File ID cache size 64 Blocks in extent cache 0 Quota cache size 0 Maximum buffers in FCP cache 1724 Volume owner UIC [SYSTEM] Vol Prot S:RWCD,O:RWCD,G:RWCD,W:RWCD Volume access control list: (IDENTIFIER=%X80010033,ACCESS=READ+WRITE+CREATE+DELETE)

42

Oracle 10g Database Export

Minutes to complete database export

less is better

0

5

10

15

20

25

30

35

Extend = 5

Extend = 2048

20% Improvement

43

Larger Block Size with RMS

Going beyond 127 blocks I/O requires using RAB64

Implemented in VMS COPY/BLOCK qualifier

The Port drivers currently support a maximum of 256 blocks per I/O

Larger I/Os are segmented

44

0

5

10

15

20

25

30

35OpenVMS Backup

COPY/BLOCK=256

COPY/BLOCK=2560

Seconds to copy 5GB file from disk to disk

less is better

Impact of larger block size

45

0

50

100

150

200

250

300

350

400

450

500Out of the box

Final result

High WaterMarking disabledDisable caching

RMS Tuning


less is better

Evolution of the test program

46

FastPath

Spread I/Os across all CPUs

$ show fastFast Path preferred CPUs on MAKLEE 13-MAY-2008 21:20:41.82HP rx3600 (1.59GHz/9.0MB) with 4 active CPUs

Device: Fastpath CPU:EIA0 1EIB0 0FGC0 1FGA0 0FGB0 3FGD0 3PKA0 2

OpenVMS TCP/IP is currently running on CPU 1

47

Summary

Disk I/O is an extremely slow operation

Don’t stall the applicationUse Asynchronous I/Os whenever possible

Consolidate your I/OsPerform larger and fewer I/Os

Cache information as close to the application as possible

See us at www.maklee.com for:

• Performance improvements

• Oracle Tuning

• Platform Migration

• Custom Engineering solutions

• Custom Training

Questions?

http://www.maklee.com/

Documents

1 Disk I/O Performance focusing on Caching Saturday, October 03, 2015 Guy Peleg President Maklee Engineering [email protected]