By Hook or By Crook: Overcoming Resource Limitations

Curtis A. Smith, Defense Contract Audit Agency, La Mirada, CA.

ABSTRACT Have you ever had trouble getting a SAS job to eomplete, although yourSAS code is bug free? Often, especially when processing large amounts of data, your operating environment just does not have enough resources to complete your SAS job the way you have written it. Likely, a solution to the resource problems you have ormightencounterexists. Herein,the author will share some of his favorite tricks to squeeze more out of the operating environment and for working SAS code around resource limitations. These tricks will include SAS code, SAS

and operating system parameters. This cover subjects such as reducing the

of SAS data sets needed, reducing the SAS data sets, sorting efficiencies,

optilml2:1ng data library parameters, and cleaning the work environment.

INTRODUCTION In my endeavors to get my large SAS jobs to complete, I encounter four categories of problems: temporary work space shortages, permanent storage space shortages, memory shortages, and (uh, I can't remember the other one' oh, yeah) miscellaneous stuff. So, I will approach the problems I encounter and the solutions 1 use in these four groupings. Some of the problems and solutions presented are unique to IBM MVS, · some are unique to MS-Windows, but most are common to all operating systems.

TEMPORARY WORK SPACE SHORTAGES SAS, of course, uses ns WORK library as the place to create and store temporary work data sets. A SAS job can require many temporary SAS data sets, even some that you may not reference in a DATA step. For example, if you create a SAS data set to replace an existing permanent SAS data set, SAS will first create a

temporary SAS data set. Then SAS will move the temporary SAS data set over the existing SAS

set, but only after the DATA step has cornpiE!ted successfully.

~~~:~~~s'!::~ temporary storage space is ::.~ Many approaches to solve the

nmn"''"' of out of work space are available. These fall into at least four categories: increase the size and optimize the temporary work space; reduce the size of the data sets being stored in temporary work space; decrease the number of data sets processed in the temporary work space; and clean up the temporary work space during the SAS job.

Increase the Size of and Optimize the WORK Library When you launch SAS it will create its WORK library using the parameters provided to it. You can increase the size of the WORK library and alter the parameters of the WORK library to make the WORK library larger and more efficient.

IBM MVS WORK Librarv Space Under IBM MVS you can increase the size of the space (SPACE) parameter of the physical file created for the WORK library. If you are running SAS as a batch job, you can change the SPACE parameters, in the DO statement for the WORK data set. Simply increase the primary and/or secondary allocations and specify the desired allocation units. For example,

lllfORK I)D 1l II

UNI'f-SYSD!\,S~AC»>(cij.; l¥00, SOH' ........ . non- 1Rii:Ci1f""l'!ii p;:;o~-;P,Si . • • • · .. ·.

r,ll,t>·C· .. L*2 ... • .. ··. 1. 6 .. ···4··· 8 ... ·.;·····))···· L .. ·. K .. s. I.· ZE-Z ... 1 ~· ... s. J • . .-.• ... "' ..... ··...... .. . . ...... . .. .... ... . ....

439

Here, we are allocating cylinders, with 100 primary and 50 secondary. If you use a procedure in your JCL to run SAS you may need to change your JCL to run the SAS program so that you can overnde the DO statements used in the procedure. An example of JCL running the SAS program is shown below. Note the first line, where the SAS program is run.

1/SASJO& II 1/STEPLIB //CONFJ.G II //C'I'IlJINS //S.IISI\UTOS //SASH!ILP //SASMsG• 1/WORI< l/ II .. ·.

EXEC PIJH=SASROST 1 TIHE•2 99, .··

I? ARM= ( 'OPJ.I.$1' ~ULI;S'I'IMER 'l DO DSN=$U$1? ;sAs .LISilJIRY,.DISPFsl!li. DD'DSN-SUSl'.si\S.CNTL(BA'I'C!f1>A),!>IS1>"SHI( DO DSN*NIILLJ'ILS, Disi'#SIIR ·.· .· .. . . .· .. •· UD DS!I*SUSP. SAS :.SASC ;TRANSLl!i, DISP•SNR. DO .DS.li*SUSP.SAS .AU'I'OLIS, DlSP=SIIR

· [)D I)S!I*,SI1511.SI\S. Silsll.ELP, DIS11..,SM • DO. DS}l'•!IQ'J!f!.>sAs. SIISI,ISG.• DISf!•SIIR DD tJI'!ITooS'(SDA,SPACE-IC'll;, UllQ,.S.<Hl~.•i·••'

OOB" (~FM•I'S;l!SO\Ui'!'l'S; !,RECL .. 27 648; ' '.B· t.J(SIZE"'2·'1· 6.4 .. fl.L .... . . . . .· ·. .. . .. . .·

An alternative is to pass the SPACE parameter to SAS on the EXEC card and eliminate the WORK DO statement. Do this as follows:

In this case, the parameters passed are blocks. To control all ofthe DCB parameters for the WORK library, I recommend using the DO statement for the WORK library rather than just passing the SPACE parameter.

When running SAS interactively, specify the size of the WORK library using the desired SPACE parameter for the desired amount of blocks. Accomplish this with an option on the SAS invocation command. For example,

IBM MVS WORK librarv Block Size A great way to save space is to tune the block size (tiL.II.:SILt:W' parameter associated with the library allocation. The block size is set when you create a physical file. This relates to the number of observations transferred together as a group. Each block is separated by a marker, known as the interblock gap, that occupies space but does not hold any data. The smaller the block size, the more interblock gaps that will exist in the same amount of physical space. Therefore, smaller block sizes result in more interblock gaps in your SAS data libraries, and thus, more unusable space. Also, larger block sizes transfer more observations as a group and can increase 1/0 throughput (decreasing EXCPs). To save space In the WORK library and improve performance, larger SAS data sets work better with a large block size, like 27648 as In the example above. Set the LRECL parameter to match the block size or ignore the LRECL parameter and let SAS set it for you. You can read a thorough and technical discussion of block sizes In Michael A. Raithel's book 'Tuning SAS Applications in the MVS Environment," which is available through the SAS Institute.

Another factor that influences throughput is the number of buffers SAS uses. The number of buffers is set for the life of the SAS data set using the SAS bufno= option. This is also thoroughly covered in Michael Raithel's book.

Check out the benchmarks below. I used a SAS data set containing 791 ,656 observations and created a new SAS data set from it in the WORK library using a simple DATA step. Notice the impact the block size and number of buffers has on EXCPs, tracks, and memory. If memory is not an issue, larger number of buffers along with the larger block size makes a big difference. But also notice what happens if you use a block size greater than Y. of a track. In my benchmarks I used a 3390 disk drive that has a Y. track size of 27,648.

iikiiliff • e~io~? ~cii?/ trackS' .. AA~£1~1~1

6,144 3 2,648 1,455 3,362

27,648 3 1,593 1,299 3,694

29,184 3 1,603 2,484 3,642

6,144 15 686 1,455 3,830

27,648 15 476 1,299 4,366

29,184 15 526 2,484 4,330

6,144 30 440 1,455 4,414

27,648 30 335 1,299 5,150

29,184 30 390 2,484 5,138

When using a 3390 disk drive, the block size of 27,648 is optimal. Notice the amount of storage space reduced using this block size. While this amount may not seem too great in this benchmark, the SAS data set I used was not all that big. I see greater space reduction with larger files. And even though it may not seem like much space reduction, it may be enough to get a job to run that would otherwise fail for lack of WORK space. But even more impressive is the reduction in EXCPs. Reducing EXCPs will mean your job will run faster and you will be charged less for 110. Increasing the number of buffers helps even further. Before leaning of these tuning techniques, I was using the default block size (6144) and the default number of buffers (3). This job would have required 1,455 tracks and 2,648 EX CPs. Now, I use a block size of 27,648 and 30 buffers. This job now requires 1 ,299tracks and 335 EX CPs.

IBM MVS Sort Space Sequential SAS data libraries do not store observation information in the header of the SAS data library. This means that SAS does not know how many observations are in a SAS data set stored on tape. This can create some problems. When you sort a SAS data set (SORT, MEANS, SUMMARY procedures, etc.) SAS will allocate the sort work data sets it needs, determine their size, pass the SIZE parameter to the host sort utility, and choose the sort program (unless these options are overridden with an OPTIONS statement). SAS chooses the values for these options based upon the number of observations in the SAS data set. However, when the SAS data set is on tape, SAS does not know the number of observations. Thus, SAS cannot properly set the needed sort parameters. It is very easy for a sort routine to fail for lack of sort space when your SAS data sets are stored on tape. SAS will return the following message:

IJ6.#t .• ~iJ51 n<;>:l:;.¢map~~~e•~uccessfl,l~ly. see ~s~aq~~-''\1~.-i;t:lle JoJ? •:Coi)so~e t<ll;J or t/SYSOUT ~,ta1:4!t.set;,. i: · .. ·· · · · ...

To overcome this, specify the following sort options with an OPTIONS statement.

SORTPGM= This option tells SAS to use the host sort utility or the SAS sort program. Typically, the host sort utility, such as DFSORT, SYNCSORT, or DJSORT, will

440

do better on large SAS data sets. Select either 'SAS' for the SAS sort utility or 'HOST' for the host sort utility.

SORTWKNO= This parameter tells SAS how many sort work data sets to use. This is a numeric value from 1-6.

SORT= This parameter specifies the minimum size of all sort work data sets. This is a numeric value for the number of cylinders. SAS will divide the si:~e

you select by the number of sort work data sets to figure out the size of each sort work data set.

SORTSIZE= This passes a value to the host sort utility for its SIZE parameter. This is a numeric value.

Here is an example:

.0!1'l'IbNS SO!l'l'PGM=HOS'l'·:: SORTSlZiil"'251iK •: :. s~ttO=.Ii soilw .. 2to\li · · · ·

__ ;.> .· . c' •

This will cause SAS to use the host sort utility, with 256K for its SIZE parameter, and aRocate six sort work data sets of 400 cylinders each. The amount of space SAS needs for sorting is a bit larger than the amount of space the SAS data set would occupy if it were stored on disk. Trial and error may be necessary to find the right values to use.

Really Big Rles on MVS Sometimes your files may be so large that the limit of six sort workers that SAS can allocate just isn't enough because you may not be able to allocate enough cylinders in the SORT= parameter. It might be very difficult to get a large enough allocation on with only six sort workers. For example, you may not be able to allocate 6000 cylinders for your 6 sort workers because the operating system may not be able ti find that much available

·contiguous space. Fortunately, there is a solution. By allocating the sort workers externally by the host and using the host sort routine you can allocate as many as ninety-nine sort workers. Here's how to do it in your JCL with a DO statement.

The DO name prefix can be determined by checking With your site administrator or by looking at the output JES messages from a

previous SAS job. The SAS default is 'SASS'. This prefix can also be set using the SORlWKDD= option. In this

• example, I have specified eight sort workers with 800 primary cylinders each. You will probably find that

!) . it is easier to get the total allocation space you · ;j' need if you allocate more sort workers with smaller

cylinder amounts rather than fewer sort workers :!I with larger cylinder amounts.

Next, within your SAS code specify the SAS option "DYNALLOC. • This option tells SAS to let the host sort utility dynamically allocate the sort workers, rather than having SAS allocate its maximum six sort workers. For example:

loP~~~-~·· ~l\to~; Sl)~~f~AS~I:fjl

To identify and overcome other problems 11ssociated with SAS data sets stored on tape, see the author's paper entitled Data Ubraries on Tape in the proceedings from the WUSS '98 conference and the

SUGI 24 conference.

MS-Windows WORK library Space Under MS-Windows or other microcomputer operating systems there are ways to increase the size of the WORK library and optimize ~even though it resides on your microcomputer's hard disk. Your WORK library will be limited to the available hard drive space. Wrthin your SAS configuration file (e.g., config.sas under version 6, SASv7.cfg under version 7, SASv8.cfg under version 8) is an option that selects the hard drive and folder for the SAS WORK library. The option will look something like this: You can chan e the drive and folder of the SAS

WORK library. One way to increase the size of the WORK fibrary is to direct SAS to place the WORK library on a hard drive with sufficient space. If your primary hard drive does not have as much tree space as your SAS jobs require for WORK space, place the WORK library on another local hard drive or on a network drive. Also, if you have more than one hard drive with enough space for your WORK library choose the fastest hard drive to improve perfonnance (you would want to do the same for your MS-Windows virtual memory).

Another way to optimize your WORK library is to keep your hard drive cleaned and optimized. For example, delete unnecessary files from the hard drive. If your version of MS-Windows operating system support's FA T32 and yet your hard drives are configured as FAT16, changed them to FAT32. Doing so can recover as much as 40 percent of the hard drive. Windows 98 comes with a utility to convert a FAT16 hard drive to FAT32. Partition Magic will safely convert a FAT16 hard drive to FAT32 as will some freeware products. Also, use utilities such as ScanDisk, Dfrag, Norton Utilities, and SpinRite to keep the hard drive perfonning as efficiently and error free as possible.

Reduce the Size of the Data Sets in the WORK Ubrary

There are ma.ny ways to reduce the size of SAS data seAts . . .. . that you place in your temporary work space. Below I share my favorite strategies.

Compressing Files Under some operating systems, ~ including IBM MVS and MS-Windows, SAS can use compression algorithms to compress SAS data sets. Using the COMPRESS= system or data set option, any SAS data set created on disk will be compressed. SAS data set compression can greatly reduce the size of SAS data sets. lfthose SAS data sets are placed in the WORK library, you can use the COMPRESS= option to extend the WORK library space. To use the COMPRESS= system or data set option, set the option to either "YES" or "BINARY." The COMPRESS= YES value uses an algorithm that works better with SAS data sets that primarily have character variables. On the other hand, COMPRESS=BINARY uses a different algorithm that works better with SAS data sets that have many variables including many numeric variables. My experience has been that COMPRESS=YES results in about 50 percent less storage space.

An option to use with COMPRESS= REUSE=. Specifying this option allows reuse space within the compressed SAS ua•a ,__

set that has been freed by deleting observation. Otherwise, SAS cannot reclaim the deleted space.

Consider the following examples.

441

After running the DATA step with the COMPRESS= set to "YES" or "BINARY" you will note a message in your SAS log that looks something like this:

I'JOTE;. !j.'he da~ll. ~~t WORK. FIL£3 Mi •ri'!~/i!l ob:ser.,ations ·anj;t.:ZO Vjlria))le$; ..• ,,,. •.. •·····' NO"rE; l(OIIIpr!!sl1l,lll;l•data a,et WORK. t:J:,L£;l····· ··. i. qe()!!ea!Jed ~fz..,.\t)!.; 3~.70: pereet~t·•·iCpmprot~s-!~'44 ,3~ • :pagea1 ~cOII!p~i!!l<i "!!U'I.'d • .r 30 page!!.

I thought you might want to see some benchmark results. I took four SAS data sets with different number of observations and variables. I started with uncompressed SAS data sets, then compressed with nonnal compression, then compressed with binary compression. Filet and File3 have 20 variables, 5 of which are numeric; and File2 and File4 have 29 variables, 10 of which are numeric.

:·-:-·-·-·-·:-:·-:-·-:-:-·-:-:-:-::-:-:.-:

i~i$ ·········· ii~Jii¥ File1 120,641 15 5 31.91% 23.72%

File2 183,476 19 10 52.10% 49.22%

File3 1,542,257 15 5 31.38% 21.02%

File4 6,976,838 19 10 48.81% 46.61%

Notice that File2 and File4 with more variables and more numeric variables resutt in greater savings using either compression method. Now take a look at the space savings (in this example, from MS­Windows).

::::::: ~~1:; :SAS.S ........ Dat•S" &l1!11002oiiPM s.o.$-o•a·sor .. 6111fl!Ji:i'Ci(!M

27~ ~~~--~ .. ·1VJS/11)2.33-~ 113.321k8 SASS-o ... s.. 811Srooi!l81;'M

91ii.<461KS SAS SPttanbata-Set. .6119100 2*"PM ''".869Kll SASS ...... e ... s•· •6119J0023:1.PM 52!1.1531:& ·SP.S-s., ... ~~«oseoc .s!19J00252Piol 1~ SAS'S-l)ataht ~~a!IPM 55t.a:n<B .SAS$ ...... D .... Sol· 8.<1Mil~07PM

Delete Unneeded Variables All soon as possible in your DATA steps and procedures delete any SAS data $&I variables that you do not need. Use the data set DROP= option to identify which variables to delete or use the KEEP= option to identify which variables to retain. Both will accomplish the same thing, one will be easier to use than the other depending the number of the existing variables you want to eliminate.

Frequently, when creating a subset or summary SAS data set or just processing a SAS data set with a

procedure, you do not need all of the variables. So delete those you do not need from the source SAS data set. However, be sure not to delete any variables that you need to process within the DATA step or procedure. For example, If you need to perform a WHERE selection on a variable but do not want that variable in the output SAS data set you will need to keep it on the input SAS data set but can drop it from the output SAS data set. Deleting unneeded variables can have a dramatic imp'lct on the size of the SAS data set. For example, a variable of only five bytes in a SAS data set of one million observations will require five million bytes, or approximately 5MB. Consider the following example.

~i.i'$QII~ llil'!'ci\""!WSS ,Willl D'rL · · :: : . .·•. , , .. ··, . • . :.: ······:·.l~fm01'Aifl'"•~·~~··~J···

. ili.!T..w\JSS.SO!\TEI)i~"~J..; . lit ~ w~r 1'~llo!t coA'rll!'

~ , ..;:1Ym:'im:_ 7~---~_-!;

• hl.lu.~wus~: SOR;flll fllsstNG ~til\¥: et•l'I!C</tit J\~r ·~RIMe: .cMTE: .... ·. · .. ·

, ·'1~.¥:'1Jl(l'l'll~s ··.~11mdll!ll?:•~rwrl\Mr·.~rlYrtJl\i>IJ' m;;sMlfl": ;:>¢(11'!!lJT'. OUT"!IIUSS. WIP:$1Jl.! .. · : ; /: .: .• J.J:!IDEX" tl'BCOi:>&J !lR¢r>e .'i:YPJ> ! ,. • i :. f ;S~Y~I)(II'ijl\$:Y~DTOH!\lf-:YTOOTI\MT

1\U!Ii: •.•••..

Notice In the SORT procedure that we do not drop the REC _TYPE variable from the input (DATA=) SAS data set because we need it for the WHERE statement. If we had included tt in the DROP= option, we would have received an error that the REC_ TYPE variable was not on the input SAS data set In this example, the SUMMARY procedure will produce a SAS data set w~h only the character variables identified in the BY statements and the four numeric variables identified on the VAR statement (plus, of course, the _FREO_ and _TYPE_ variables created by the SUMMARY procedure, unless we drop them). So, why bother dropping the unwanted variables during the SORT procedure? Because by dropping those variables, our intermediate WORK SAS data set is greatly reduced in size.

Let's take a look at the WIP DTL and the SORTED file from Windows Explorer's point of view. Dropping five variables made a significant difference.

612GI007;17 AM 6120101ll:45AM 6120/00 7: .. 5AM

Now look at the file properties from SAS' point of view. First notice the WIP _DTL SAS data set. It contains 22 variables.

WIPcDTL ... DATA WIPACOOUNTS lli«ANO lEO¢< Vll 25JAN200H!II;50o1 G 2WUN200(l07:17c21 6066128. 22 C1fAA 123 0 y., No

Now, notice the SORTED SAS data sets. II contains 16 variables.

442

Delete Unneeded Observations Any observations that are not needed in your temporary SAS data sets just take up space. So, delete them as soon as possible. Do this wtth a WHERE statement when processing SAS data sets in a DATA step or procedure or an IF statement when processing external files in a DATA step. Consider your data needs for the SAS data sets you will create in your temporary work space. If you will not need all of the observations in the SAS data se~ get rid of those you do not need. Consider the sample SORT procedure in the above section.

Decrease the Number of Data Sets Processed In WORK There are several ways you can reduce the number of data sets SAS will need to put into the WORK library. Below are my favorite strategies for reducing the number of SAS data sets in the WORK library.

Use an Atternate Temporarv Library When you do not specify a library for a SAS data set, SAS will place the data set in the WORK library. Of course, you can also specify the WORK library in the two level SAS data set name to place the SAS data set in the WORK library. When you are short of space in WORK and need to have more than one SAS data set in temporary storage simultaneously an approach is to create addnional temporary work libraries. Below are ways to do this under IBMIMVS and under M5-Windows.

IBMIMVS Creating an alternate temporary work library is a very simple process. Just allocate a temporary SAS data library and use it for some SAS data sets that you need to place in temporary work space. For example,

l,.IBNAI:f~ 'r~P • ~~NG .~Si~.rt"EN~E. . DISP=(~;()~LE~;[)El,.ETE! S.Pl\!!II'".J:<:\'l,., ~~lll),~MJ; ,S'L

< " "" " ,•' ' o,~ ,' '- "-" ' '"' "' ""' '' "•" f ' >

Then in your SAS program, you might do something like the following:

If you need more than one additional temporary work library, simply allocate as many as you need. If you need to put some really big files in a temporary work library and cannot get enough disk space, allocate the temporary work libraries to tape. Generally, you will want to do this in the JCL as follows.

MS-Windows Creating an alternate temporary work library is a very simple process. Just allocate a temporary SAS data library on a different hard drive than your SAS WORK library. If you use the same hard disk drives, then the temporary library you allocate and the WORK library will be competing for the same limited space. Then use it for some SAS data sets that you need to place in temporary work space. For example (assuming your SAS WORK library is on the C: drive and you have a D: drive available):

jL:~~E TEMP·"P:\LONG\WIIIOOWS\FOLDERNAMf;"il

Then in your SAS program; you might do something like the following:

~I1!~.1\l4E CD ~D:\LONG\WINIJOlis\TDt.DERNAME\TABLES# i DATA IIORKiFIL&l 'JfE!4P. FlLE~i . . sST (:p,JIASli'ER .

SSfiECll'· ('lll;VIsiON)I . . • •.. . .. ··· . • IIHEN J"At~t.ouTPuT woJ<K,FILEl; WJIEN· ("00".1• OUTPOT TE14P.Fl~~~ O'l'I!Eitli1SI!H>SLETEI . . .

EltDi ... 1\ll.lf;.

Output Sorted Sas Data Sets When you sort a SAS data set using the SORT procedure if you do not specify an output SAS data set using the OUT= option SAS will overwrite the input SAS data set with the sorted SAS data set. To do this, SAS must create a temporary sorted SAS data set until the SORT procedure successfully completes. Then SAS will overwrite the input SAS data set with the sorted SAS data set. This method works, of course, but creates a temporary sorted file in the WORK library. If you specify an output file for the sorted SAS data set, you will still have two SAS data sets created: the input SAS data set and the sorted output SAS data set. However, in this situation you can place the sorted SAS data set somewhere other than the WORK library, such as another temporary SAS data library, a tape SAS data library, or a permanent disk SAS data library. Consider the following example.

l!R()C S()RT '.I>Al!'A~WORK.wlP_:.rm,. !)U'l',.l!'(\1'~. SORTED;· .. ··•.· B!i: •P~Oilt: 1\Ct;!()Ull'f i'l\l:ME CI>ATEI . •·•. ~.~ ··•: . ~L. : ... . --- . .. .. .. ·•

Delete Permanent SAS Data Sets Before Replacing Them Whenever you update or otherwise replace a permanent SAS data set, SAS will create a temporary SAS data set to hold the new data until the DATA step or procedure successfully completes. Then SAS will overwrite the old SAS data set with the one just created. This causes SAS to create a temporary SAS data set in the WORK library (I noticed that MS-Windows places this temporary SAS data set in the same library as the permanent SAS data set you are creating). Lefs look at a real-life example under MS-Windows.

01\Tl\ • WUSS. Wff I>T,l.; • • SET TAPE. Wii? I>TJ;;

WIIERE REC TYi?E·NE' RUN; -

If the WIP _DTL SAS data set already existed in the WUSS library, the following would happen in the WUSS library.

Notice the temporary file created because the target output SAS data sets already existed. This, of course, is wasting space.

443

Instead, delete the existing SAS data set using the DATASET$ or the DELETE procedure before running the DATA step or procedure that will update or replace the SAS data set. This method cannot be used, of course, if the existing SAS data sells needed as the basis for updating or replacing the SAS data set. Consider the following example.

Plloc iOATJ\sE'l's L18AARY.,WUSS NOliiS.T; : • · !l~n~Ts'wr~:ll'liu; · • · · .. · · · · .. ~ ··· · · · ··· ·

qtn'ti -RUN; OAT}\ WUSS.!flf[)TL;

sET 'fAPE .lUP. <DT~i WHj;:JI.E 1\E:c.'i'xi?l!ius· •

RUNi -

Clean Up the WORK Library This tip is so obvious that overlooking it is easy. Simply delete SAS data sets in the WORK library or other temporary work space when you no longer need them. Do this, of course, using the DATASET$ or the DELETE procedure. Consider the following typical job stream. Here you are sorting a permanent SAS data set to a WORK SAS data set that you will summarize. Then, you will sort another permanent SAS data set to a WORK SAS data set that you will summarize. The first WORK SAS data set is not needed again after the first SUMMARY procedure has completed.

PROC SORT DATA=w!ISS.WIP. O'l'L oll'f•JIICRI(,p~l; BY PBCOOE ACCQJJNT PJI.~ifE COATE; ...

PROC. s~ttY DA'l'A"ftORlt' ~l Mxss~; eY PBCODE . ACCOUNT PJI.!HE CW.TEi . · • • •..•. · .. :• · ..•• · iri\Jt . YTI>T()AMT; . · . ." • . . . . • . • . . .. ·• • :0 : • •

OUTPUT.OUT"WUSS.WIP_;_SJJM·SJJM•Y'l'~;

RUN;. . . .. . . . . .. •·•···· •• . •• . PRpc SOR'J' I>ATA"WUSS>lt!D 'I>TL OlJ'I'.O.Wo~:iSOkTED~f •:

. I>Y PflCOI>E A(:;:GOUNT• PI\~HE CPA'l'Et •.. ··.· .. ••••• .... •·•· •.. ~ P~··S~RY DJ\TA"IIORK.•S.ORTE~ Ml$S~NI.>f .

BY PBCOI>t: J\CCOtlNT PRIME c!)/>.:fp;;. > . " • • • VIIR ~TDTQAMT i : . · . > . . : · . , OUTPUT ·· OUT=WUst). Ill I> • SOM · SUK'-Y:!'D'l'OJ\MT ;•

RUN I ·. ~L .. ·.··•·· · L- ."·· . . / ...

We can see the result in the WORK library, in this example, from MS-Windows.

u-~ Utili lSI ~t,---SL ....... 1:WM, u fOffi'- flaMiH:~ 2Q fQfflio ~ftltAM

~QW~o.t.hr~ll4l~ lauzJQ -SAS.~W ll'liJ,WI:RN4

In this example, you have a very large file, SORTED1, that is just taking up space while you sort and summarize your second SAS data set. Instead, you can insert a DATASETS or DELETE procedure after summarizing the first SAS data sat.

PROC SORT. DATA,.WlJSS;tf!P D'l't OOT"'JIICRlC..C~J. IIY PllCOI'Jll 1\q;91JNT. PRlift: QDATEt ..•••... ·.····. ·• ...

PROC •·. SUMMARY :··oAT!'>~/i$'(1R'J'BD.:l•M!SSING; BY PBCOPEACCOUNTPRIMECDATEJ. Vl\R Y'l'I>Tolllfr; . ·. . . .. . . . . . . , ouTPlJll'·:.ouT .. wuss;wJ:r. SJJM s~v'!'I>Tol!M'l'•

RUN; ·• ·. .•... .• . .. •·· -. .··· ·.·. . . PllOC !;IE~gXE DATA•II'Q&{,~~~ ; •· . . ···.· RUN;. ..·.······: . .·. · · ... ·.; · • . . · .: o .. :·;' :·: PROC SoRT PAl!'A•WUSS• lNI>,,_WL 01)1'=WoRI:'tSOR'rSP2t

BY PSCQDt: • ACCoUNT l?RIM& .COATE;· PROC SVMMMY DATA-w<lRK.SoltTllD2 MISSUI~)' · ~y ~CoJ;>r;: .ACCOUNT PR~MI! COATE/ VAR YTDTOAM'.l! i OUTI'!IT otlT•liUSS. INO SOM SIIM=YT[)TO~;

RUN) -

PERMANENT STORAGE SPACE SHORTAGES Most of the tips mentioned earlier for reducing space in temporary work space will also work for reducing space in permanent storage. I do have a couple additional strategies for improving permanent storage.

Use Removable Storage Devices Yes, not all permanent SAS data sets need to be stored on permanent disk drives. Under IBM MVS and other operating systems you can use tape devices. LimHations imposed by tape devices exist because they are sequential storage devices, but they work very well for storing very large SAS data sets. To identity and

overcome problems associated wHh SAS data sets stored on tape, see the autho~s paper entitled Data Libraries on Tape in the proceedings from the WUSS '98 conference and the SUGI 24 conference.

Under MS-Windows and other microcomputer operating systems you can store large SAS data sets on removable storage devices

such as ZIP drives, Jazz drives, and writeable COs. While you might need to create the permanent SAS data sets on permanent disk and then move them to removable media using imaging software, this strategy is workable.

Create Summary Files If you need all of the data in a large SAS data set but do not need all of the detail you can create summary SAS data sets from the detail file and keep only the summary SAS data sets on your permanent disk. For example, suppose you have a SAS data set with forty character variables and six numeric variables and you typically only need ten ofthe character variables with all the numeric values. Simply use the SUMMARY procedure to summarize the detail SAS data set by the ten character variables and sum the numeric variables. Place the summarized SAS data set in a permanent SAS data library on disk and keep the detail SAS data set in a permanent SAS data set on tape or other removable storage device.

Using the SAS program code example in the section "Cleaning Up

the WORK Library", produces tremendous storage savings. Look at the difference in disk space storage from MS-Wmdow's point of

view.

Summarizing the detail (DTL) SAS data sets by a few variables greatly reduces the size of the summary (SUM} files.

MEMORY SHORTAGES There was something I wanted to say about memory shortages, but I can't remember ... oh yeah. Frequently, you may either find your job faRing for lack of memory or find your job running slowly because the operating system is forced to use virtual memory rather than physical memory. While adding more physical memory always helps, this may not be an option. There are other strategies to use to help manage memory.

MVS Region Size When running SAS in batch mode under IBM MVS you can increase the size of the REGION parameter to allow more memory for your job. You can set this in your JCL on the job card. Below is an example.

444

Check with your site administrator to find vaHd values for the REGION parameter.

MS-Windows Memory MS-Windows and MS-Windows applications are not very good about using memory and cleaning up after themselves. Many programs load large libraries into memory although you may not need all the functions stored in the libraries. Some programs do not unload everything from memory when you exit the program. With all this junk in physical memory, other programs and data, like SAS

and your SAS data sets, may find themselves loaded Into virtual memory (which, of course, is much slower then physical memory).

SAS users have a cheap solution to this problem. Shareware and freeware programs are available that routinely flush unneeded stuff from physical memory and move them to virtual memory. Doing so allows your SAS program to load itself and your SAS data sets into physical memory, thus improving performance. My two favorite programs are RamBooster and FreeMem. I have the free 'light' version of FreeMem, called FreeMem Standard. This version does not have all of the features of the shareware version, called FreeMem Professional. RamBooster is free and I find it better than FreeMem Standard. However, I do not know how it compares to FreeMem Professional. You can find RAMBooster at http://www.sci.fi/-born/rambooster/index.htm and FreeMem at http: /lwww. 3bsoftware. com.

Take a look at the screen from RAMBooster. This is what my PC looked like after working on this paper for a while. Notice how little RAM I had left of my 32MB PC. (I know, 1 desperately need an

Now, with one click of a button, I got a good bit of my memory back.

Sort Before Using the SUMMARY Procedure When using the SUMMARY procedure you can specify the summary variables with the BY or the CLASS statements. Wrthin the SUMMARY procedure the BY statement requires that the input SAS data set must already be sorted or indexed in the same sequence as you specify in the BY statement. In contrast, within the SUMMARY procedure the CLASS statement will sort the observations in the input SAS data set according to the variable~ you specify in the CLASS statement. Is there a difference? Well, using the CLASS statement saves you from running a SORT procedure or indexing the input SAS data set. However, SAS must still sort the input SAS data set. It is just going to do H within the SUMMARY procedure. And it will do this in the WORK library. I have found that using the CLASS statemerrt running under IBM MVS with large SAS data sets takes longer than using the BY statement, including the time necessary to first sort the input SAS data set. I have also found that using the CLASS statement running under IBM MVS with large SAS data sets takes much more memory than using the BY statement, Including the memory necessary to first sort the input SAS data set. In contrast, I have found that using the CLASS statement running under MS-Windows takes less time and memory than using the BY statement combined with the time and memory needed to first sort the input SAS data set. However, if the input SAS data set is already sorted, then using the SUMMARY procedure with the CLASS statement takes much more time and memory than does when using the BY statement.

Here's another point about WORK space. If you use a SORT procedure and then the SUMMARY procedure with the BY statement (rather than the CLASS), you can direct the sorted SAS data set to a temporary library other than the WORK library (see the discussion above). Consider the example below.

PROO"IloaT.···M'J.'A~ORK ;wfl' DTt..···p!J.'l"';~Ef>!P,·SORTE\"); .c.J:)~.cl',II~PE .• ACCPtitf'l.'··;pRJiiJ;: CJ;)j\TE("•••'·.·.·. \.·

l';fi;OC • SOMNAllY. [)A')'A,.TtMP, $1)R1l'EO •IUSSI,IIG; i .l!Y I?BCOil!j?:i\ll;izootr.l' Plfl;I<IE COATE; •..... ·•

.• •·. VJ\R . YTP{!TIIIUl. Y'.rD'l'OllR~ . YT[)()TAMT YTDTOAMTi

.. OU!,rPUTO!J~IiCC•HI&FY.;;WIP $UM • • JIND~•u•ecooE! ... r.Ji<oP- l'Yi?.&.! . S~~YToorHRS yT!)'l'QilRS ""'l'l'r:\OTAMT YTOT01\MT;

RUN I

445

For those of you who like benchmarking these things, I did some benchmarking. Under IBM MVS I ran a SORT procedure then a SUMMARY procedure using the BY statement and compared to running a SUMMARY procedure with the CLASS statement. I did three comparisons: one with a SAS data set containing 81,765 observations, another with a SAS data set containing 318,799 observations, and another with a SAS data set containing 1 ,000,955 observations. As noted above, the most dramatic difference seems to be that larger SAS data sets require the SUMMARY procedure to use far more memory when using the CLASS statement than when using the BY statement.

File1 81,765 SORT 1.63 869

SUMMARY/BY 0.95 1,076

Total I \ ?:~~> . ~&~ " !?<

SUMMARY/CLASS I ~~!!~ L 1~?~ * File2 318,799 SORT 6.96 843

SUMMARY/BY 3.56 1,076

Total ; SUMMARY/CLASS >··1

File3 1,000,955 SORT 16.57 869

SUMMARY/BY 10.92 1,076

Total

••••••••••••••

i 901(

"''; 40::0:: )> ii .,. I

MISCELLANEOUS STUFF There are always a few miscellaneous things that do not fit irrto any other groups: so here they are.

MVS TIME Parameter It is really a bummer when your MVS SAS job stops suddenly only because you did not allocate enough CPU time to the job. Wrthin lhe JCL for your batch job you may need to adjust the time for the SAS program and/or you may need to adjust the time for the entire batch job. You will find the TIME parameter for the entire job on the JCL job card. And you will find the TIME parameter for the SAS task, or step on the EXEC card. Look at the example below.

//MYJOB JO$'(Wo97~8S,, S1QHti316ji13~q, .•.•.•..•.•. · II '\SM .IS C®L. ·~ . . . 4Z77''iREGlON•?T6$0K;TIMBii06oO'i /lSA$5'/.'SP · • El(EC PG!f.>SASR!>Si; 1'IIIB#2~,9J • >. ) • . . • • • •

t'~;{!' Pl'LI,S;t' .~~il;<$'t.l:.t('1 ••. · .. ·.•. ; : 1/.S'l'EPLIB.. J)O ps!I'-SUst>:.si!..S>Ll$1UU\:t:,ii[)~$P"$~i ••·•

MVS Execution Your sHe administrator probably has a handy-dandy SAS procedure

for you to use to launch SAS in your JCL. However, if r• you run the procedure you may be stuck with some

f •. ·.··.·.·default values you do not want to use, silch as the .. DCB parameters for the WORK library. Instead,

• execute the SAS program and specify the · necessary DD statements, thereby taking control of

the SAS invocation. Consider the following SAS •• ·:: JCL.

lii&M . . . ·lt)l~.··.•l,'Gif

~i~~£~i.za ~~;~ 1f>=Of!'.tG. uo•·•·!)Eiri•s!Js~ •• s~s,~r;.<llA'f u: · ; ....• 00 D!lli,.llo~ti'J;r.li:,;PX~I'=Sijll • •. ••;. ·.; •••. #~AANlJ ·l,ll)•l)SN"'l;!J;l~. ~~S: SAS!Z:Dr\IIS~:J;S, DISp;.s#l! l>!l!l\81)~ ll!)HDSN .. SUSl'iSl\S,AUTOL:J;BiPIS.,'*SIUI.·· ~ 1/SA~I!Elll' . l,ll) .DSN .. Sl)S)';!!l\S! ,PISI',;.s!iR ltSASNllG ; .cpp· DSN"ll!JSP, .SAS.. DlSI',SIIII• i .• l~ · · · •·oo ·tm!T"SYSDJ\,S . L; no\>; lH-i>·L n · • ncsfi tJ!j(;F:!f"f:.•;;;oso!l<i;=Ei!l,LR£CL•27.64s:•

. SLI(SIZE=27641l) ·

I especially like to increase the default size of the WORK library and increase the block size to reduce the size of the interblock gaps. You can also specify a different location for the CON FIG file so you can change the defauH options set up at your stte.

Eliminate the Need for Multiple Tape Devices Your site has a finite number of tape devices. When you need multiple tape libraries in the same SAS job and if you are not careful and plan, your job will require the operating system to allocate as many tape drives as you specify tape libraries. You may be waiting awhile for the tape drives to become available. Consider the example of reading a huge SAS data set into a SAS DATA step where you want to split the file into fwe new SAS data sets. Each of the SAS data sets will be stored in a separate tape library. You specify each libref and SAS data set name on the DATA step like this:

DATA • Tl\P£1 'UX.E 'l'APE2. FILE 'l'AP£3. FlLE .. TAl'E4 .FILE TAPE5. FILE}

sET TM•stn~ tNl'UT.; MORE $AS STATEMENTS

RUN;

Here, you will need six tape devices at once. Your job will wait, maybe for hours, for six drives to become available. Then, when your job grabs six tape drives. everyone at your site will wait for your job to release the drives. Not a very good plan. Instead, you can create the subsets one at a time. This will require only two tapes at once. You can write your code with a SAS macro like this:

%M,r.C~0St!BSST( DAT,'Iyt.LIS •• E"IL$i

SST TII,1'15IN: IN.P!JT i ~~,tORE •SAS STATEMENTS·. RUN.r• %MEND: %LEt I!II!m<TAPE~; %131Jl}SSt:, UET I.IB-'!'l\PE;!; %SUBSE'l:i ETC.

To keep the operating system from still allocating all your tapes at once and on different tape drives, use the UNIT=AFF (AFFINITY) parameter In your JCL or SAS LIB NAME statement on the second through the last tape library to which you are writing. The JCL would look like this:

/}~1\PEI!l /ZTAPEl /l'fl!,l'E2

tlitAPEl

Do DSN•LONG •. M\1$, yii,Ii;~i\loJE; TAI'EIN, OISP..QLD PO DSN=LONG.MVS,FltSl!~E •• 'l'AI'El,OlSP=O!;O •oo .. OSN=LONG .HVS, FILEIII\ME. '!'APE2, !)lSP=OLDi

!,IN~T,.AFF"T/I.PEl . ~0 DS!i,.LO!IG.MV!l lF!LEIIA!fE; 'l'A!'E:ioOISP~OI,D, •i· UNI'l'•AFF!'TAPE2 . • •.. . . • . ·.. > ·.: 90• !>SN":LONG. M'll!, F:LLENI\Mli!iTAPE~,DISPoiOLD,

.· i t!ill'l'•AFi-;TAP!i!3 ••• • ; ·• .• .. ·· , •. · ~> , , ·· !>!> tis'N#L()NG•~>Mi•.FILEliAME:TAI't5,ors

IJ!IlT=AFF!'TAPE 4 . • . , .

446

The UNIT=AFF option will tell the operating system to load the tape on the same unit as it loaded the. referenced tape. In our example, we will need two tape devices, oneforTAPEIN and oneforTAPE1 through TAPES, reading them one tape at a time.

This example may be a good use of tape devices, but it is a poor use of processing because we must read the huge source SAS data set ftve times. Instead, we could use our original DATA step idea, but use TAPE1 for the first of ourfwe output SAS data sets and put the others in temporary disk libraries. Then, we could use the COPY procedure to copy the four temporary SAS data sets to our four other tape libraries, one at a time. Or, we could use five temporary libraries, eliminating the need for all but one tape device. If we do that we will need to alter our UNIT=AFF a bit. Whether we use four or all five temporary libraries, we still use the UNIT=AFF in our JCL to keep the tapes mounted on the same tape device.

CONCLUSION SAS is such a powerful tool and functions so well with the operating systems on which it runs (most everything except a Commodore 64 and a TRSH 80) that there are ways around every problem. problems associated with limited resources and solutions to those problems presented above are those that I have encountered frequently over my nine years of SAS programming.

REFERENCES PROC DATASETS:

SAS Procedures Guide, Version 6, Third Edition, Chapter 17 SAS On-Line Documentation, Version 7. 1 PROC SUMMARY: SAS Procedures Guide, Version 6, Third Edition, Chapters 21 and 36 SAS On-Line Documentation, Version 7.1

BLKSIZE SAS Companion for the MVS Environment, First Edition, Chapter 17 Tuning SAS Applications in the MVS Environment, Michael A. Raithel, Chapter 4

SORT Options: SAS Companion for the MVS Environment, First Edition, Chapter 17

LIBNAME Statement: SAS Language Reference, Version 6, First Edition, Chapter 9 SAS Companion for the MVS Environment, First Edition, Chapter 17 SAS On-Line Documentation, Version 7.1

SAS MACRO Variables: SAS Guide to Macro Processing, Version 6, Second Edition, Chapter 2 SAS Language Reference, Version 6, First Edition, Chapter 20

ACKNOWLEDGMENTS SAS and SASJGraph are registered trademark or trademark of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. IBM and MVS are registered trademarks or trademarks of International Business Machines Corporation.

Other brand and product names are registered trademarks or trademarks of their respective companies.