WAVV 2007, Green Bay, WI VSE/VSAM – Inside & Out John Mycroft, Software Developer CSI International @e-vse.com

WAVV 2007, Green Bay, WI

VSE/VSAM – Inside & OutVSE/VSAM – Inside & Out

John Mycroft, Software DeveloperJohn Mycroft, Software Developer

CSI InternationalCSI International

www.e-vse.comwww.e-vse.com

[email protected]@e-vse.com


AcknowledgementAcknowledgement

With grateful thanks to Dan With grateful thanks to Dan Janda, The Swami of VSAM, from Janda, The Swami of VSAM, from whom most of this presentation whom most of this presentation was stolenwas stolen


AbstractAbstract This presentation gives an overview of VSAM & its components. This presentation gives an overview of VSAM & its components.

We take a look at what a VSAM file really looks like and how to We take a look at what a VSAM file really looks like and how to soup up its performance.soup up its performance.

We also look at some common mistakes and how to avoid them.We also look at some common mistakes and how to avoid them. This presentation and its materials are copyrighted and This presentation and its materials are copyrighted and

developed by John Mycroft from a presentation originally developed by John Mycroft from a presentation originally copyrighted by Dan Janda. Permission is granted for WAVV to copyrighted by Dan Janda. Permission is granted for WAVV to reproduce this presentation for distribution to its members at no reproduce this presentation for distribution to its members at no charge.charge.

Trademarks:Trademarks: IBM, VSE, VSE/ESA, zVSE, CICS & DL/I are trademarks or registered IBM, VSE, VSE/ESA, zVSE, CICS & DL/I are trademarks or registered

trademarks of the IBM Corporationtrademarks of the IBM Corporation The Swami of VSAM is a trademark of Dan Janda.The Swami of VSAM is a trademark of Dan Janda.


VSE/VSAM OverviewVSE/VSAM Overview

Virtual Storage Access MethodVirtual Storage Access Method For disk filesFor disk files

Sequential – “Entry Sequence Dataset” or ESDSSequential – “Entry Sequence Dataset” or ESDS Begin at the beginning, go on til you get to the end and Begin at the beginning, go on til you get to the end and

then stopthen stop Indexed – “Keyed Sequence Dataset” or KSDSIndexed – “Keyed Sequence Dataset” or KSDS

Process by key or sequentially or a mixtureProcess by key or sequentially or a mixture Direct – “Relative Record Dataset” or RRDS (fixed) or Direct – “Relative Record Dataset” or RRDS (fixed) or

VRDS (variable)VRDS (variable) Calculate a record’s location in the file to access itCalculate a record’s location in the file to access it

Alternate index (AIX) – gives an alternative route to a Alternate index (AIX) – gives an alternative route to a KSDSKSDS Allows unique & non-unique keysAllows unique & non-unique keys


VSE/VSAM Functional areasVSE/VSAM Functional areas CatalogCatalog

Volume & file informationVolume & file information Usage statisticsUsage statistics

Disk space managementDisk space management Space allocation including secondary Space allocation including secondary

allocationsallocations VSAM and VSAM/SAM filesVSAM and VSAM/SAM files System filesSystem files LibrariesLibraries


VSE/VSAM Functional areasVSE/VSAM Functional areas

IntegrityIntegrity PerformancePerformance

Data transfer sizeData transfer size BufferingBuffering Backup / restoreBackup / restore File sharing between jobs and File sharing between jobs and

systemssystems


Processing a VSAM fileProcessing a VSAM file Sequentially (ESDS)Sequentially (ESDS)

Forward or backwardForward or backward Keyed access (KSDS)Keyed access (KSDS)

Direct by full or partial (generic) keyDirect by full or partial (generic) key Sequentially, forward or backwardSequentially, forward or backward Skip sequential, forward or backwardSkip sequential, forward or backward

Addressed access (RRDS, VRDS)Addressed access (RRDS, VRDS) Direct, by record addressDirect, by record address Sequential & skip sequentialSequential & skip sequential

Alternate Index AccessAlternate Index Access Same as keyed accessSame as keyed access Also direct access by non-unique keyAlso direct access by non-unique key


How VSAM stores dataHow VSAM stores data We’re going to look at We’re going to look at

How VSAM stores records logically on diskHow VSAM stores records logically on disk Performance considerationsPerformance considerations

How VSAM physically stores data on diskHow VSAM physically stores data on disk Disk space usage calculationsDisk space usage calculations Optimizing disk capacityOptimizing disk capacity Performance considerationsPerformance considerations

VSAM jargonVSAM jargon Control IntervalControl Interval Control AreaControl Area CI & CA splitsCI & CA splits FreespaceFreespace RDF, CIDFRDF, CIDF


VSAM JargonVSAM Jargon Control Interval (CI)Control Interval (CI)

““Smallest unit of data transfer between main & Smallest unit of data transfer between main & disk storage”disk storage”

In other words, when you read a record, VSAM reads In other words, when you read a record, VSAM reads the whole CI that contains that recordthe whole CI that contains that record

Think of it as the same as a block of records in a Think of it as the same as a block of records in a sequential file if you like (though it’s laid out sequential file if you like (though it’s laid out differently)differently)

A CI can initially contain 1 or more recordsA CI can initially contain 1 or more records More can be insertedMore can be inserted Some or all can be deletedSome or all can be deleted When you try to add a new record to a CI with no When you try to add a new record to a CI with no

room, a “CI split” takes place – more about that laterroom, a “CI split” takes place – more about that later


Layout of a control intervalLayout of a control interval

ALL VSAM FILES ARE VARIABLE LENGTHALL VSAM FILES ARE VARIABLE LENGTH Even if all the records are the same sizeEven if all the records are the same size Rec 1 – Rec n Rec 1 – Rec n 1 to n logical records of any length1 to n logical records of any length FreespaceFreespace Unused space in CI for inserting records Unused space in CI for inserting records

or making existing records longeror making existing records longer RDFsRDFs 3 byte record descriptor field3 byte record descriptor field

ESDS/KSDSESDS/KSDS one per record length, one for all one per record length, one for all consecutive records of same lengthconsecutive records of same length

RRDSRRDS one per numbered record slotone per numbered record slot CIDFCIDF 4 byte Control Interval Descriptor Field4 byte Control Interval Descriptor Field

Rec 1Rec 1 Rec 2Rec 2 Rec 3Rec 3 Rec …Rec … FreespaceFreespace RDFsRDFs CIDFCIDF


Control Area (CA)Control Area (CA)

CA size is the smallest of :CA size is the smallest of :One cylinder orOne cylinder orThe size of the primary allocationThe size of the primary allocationThe size of the secondary allocationThe size of the secondary allocation

The number of CIs per CA depends on the device and the CI and The number of CIs per CA depends on the device and the CI and CA sizesCA sizesIt is generally a good idea to go for the biggest CA possibleIt is generally a good idea to go for the biggest CA possible

A CA is a group of CIs. In a KSDS, all the data CIs in a CA are indexed by one index CI

CI 0CI 0 CI 1CI 1 CI 2CI 2 CI 3CI 3 CI 4CI 4 CI 5CI 5 CI 6CI 6 CI 7CI 7 CI 8CI 8 CI 9CI 9

CI10CI10 CI11CI11 CI12CI12 CI13CI13 CI14CI14 CI15CI15 CI16CI16 CI17CI17 CI18CI18 CI19CI19



Index Control Interval (Index CI)Index Control Interval (Index CI)

CI 0CI 0 CI 1CI 1 CI 2CI 2 CI 3CI 3 CI 4CI 4 CI 5CI 5 CI 6CI 6 CI 7CI 7 CI 8CI 8 CI 9CI 9



A CI in an index containing pointers to The next level in the index orThe Data CI in the CA – this is referred

to as a Sequence Set CI

Index CIIndex CI


Index and data structureIndex and data structure

Balanced treeBalanced tree Sparse indexSparse index

Always just 1 high-level index CIAlways just 1 high-level index CI There can be 0 to many intermediate There can be 0 to many intermediate

level index CIslevel index CIs There can be one or more low-level There can be one or more low-level

(sequence set) index CIs.(sequence set) index CIs. If there is only 1 sequence set CI, it is If there is only 1 sequence set CI, it is

also the high-level index CIalso the high-level index CI


And now the bit you’ve all And now the bit you’ve all been waiting for……been waiting for……


Performance rules of thumbPerformance rules of thumb

Use largest data CI possible, Use largest data CI possible, especially for sequential workespecially for sequential work

Use as small an index CI as you Use as small an index CI as you can (but not too small!)can (but not too small!)

Use large data CA – allocate Use large data CA – allocate primary and secondary as at primary and secondary as at least 1 cylinderleast 1 cylinder

Avoid too many extents / Avoid too many extents / allocationsallocations


Allocation calculationsAllocation calculations

CI freespace = CI freespace = CI Size * Freespace % CI Size * Freespace %

Number of records per CINumber of records per CI ““Fixed” length:Fixed” length:

(CI Size -10 –Freespace) / LRECL(CI Size -10 –Freespace) / LRECL Variable length:Variable length:

(CI Size -7 –Freespace) / (Average (CI Size -7 –Freespace) / (Average LRECL +3)LRECL +3)


Allocation calculationsAllocation calculations

Calculate Freespace in each CACalculate Freespace in each CA Get number of CIs per CA from Get number of CIs per CA from

LISTCAT or device characteristics LISTCAT or device characteristics (3390, 12 x 4K CIs/track, 180/cyl)(3390, 12 x 4K CIs/track, 180/cyl)

CA freespace = No of CIs per CA * CA CA freespace = No of CIs per CA * CA Freespace %, rounded upFreespace %, rounded up

Number of CIs loaded per CA =Number of CIs loaded per CA =CIs per CA – CA freespaceCIs per CA – CA freespace

Number of records loaded per CA = Number of records loaded per CA = Loaded CIs in CA * No of recs in CILoaded CIs in CA * No of recs in CI


VSAM CatalogsVSAM Catalogs

Exactly one master catalogExactly one master catalog Assigned at IPL with DEF CAT orAssigned at IPL with DEF CAT or DEFINE MCAT IDCAMS commandDEFINE MCAT IDCAMS command

User catalogs – 0 to manyUser catalogs – 0 to many No more than 1 per volumeNo more than 1 per volume Catalog can own multiple spaces Catalog can own multiple spaces

on a volumeon a volume Many catalogs can own space on a Many catalogs can own space on a

volumevolume


VSAM CatalogsVSAM Catalogs

Catalog contains :-Catalog contains :- Self-describing recordsSelf-describing records User catalog pointersUser catalog pointers Volume definitionsVolume definitions Space definitionsSpace definitions Cluster (file) definitionsCluster (file) definitions Component (data, index) definitionsComponent (data, index) definitions AIX & Path definitionsAIX & Path definitions


Catalog recommendationsCatalog recommendations

Use naming conventionsUse naming conventions Name Cluster, Data and Index components Name Cluster, Data and Index components

explicitlyexplicitly Use partition / system independent names Use partition / system independent names

where applicablewhere applicable SeparateSeparate

Files seldom defined or deletedFiles seldom defined or deleted Files often defined or deletedFiles often defined or deleted Online critical filesOnline critical files Batch filesBatch files

Multiple baskets – all the eggs won’t get Multiple baskets – all the eggs won’t get brokenbroken


More recommendationsMore recommendations

Don’t use recoverable catalogsDon’t use recoverable catalogs Hangover from 2314 / 3330Hangover from 2314 / 3330

Backup is vastly betterBackup is vastly better IDCAMS, Faver, Maxback, Dr D, IDCAMS, Faver, Maxback, Dr D,

user-written …user-written …


CI & CA splits and freespaceCI & CA splits and freespace

You try to insert a record in a CI You try to insert a record in a CI or extend a record already thereor extend a record already there

If there is enough free space in If there is enough free space in the CI, everyone moves up, the CI, everyone moves up, record is inserted and CI record is inserted and CI rewrittenrewritten

BUT what if there isn’t enough BUT what if there isn’t enough free space????free space????


CI & CA splitsCI & CA splits

CI split – 4 physical IOsCI split – 4 physical IOs Set “Split in progress”, write CISet “Split in progress”, write CI Move half of records to new CI & Move half of records to new CI &

write itwrite it Update sequence set, write index Update sequence set, write index

CICI Erase moved records from old CI, Erase moved records from old CI,

turn off “Split in progress”, wite old turn off “Split in progress”, wite old CICI

BUT…..BUT…..


Failure in CI splitFailure in CI split

System failureSystem failure Corrected next time CI is updatedCorrected next time CI is updated

No free CI in the CANo free CI in the CA CA split is neededCA split is needed

Remember – 1 physical IO = Remember – 1 physical IO = 30,000 – 40,000 CPU 30,000 – 40,000 CPU instructions…instructions…


CA SplitCA Split MANY physical reads and writesMANY physical reads and writes

Set “Split in progress”, write sequence Set “Split in progress”, write sequence set CIset CI

Maybe get new extentMaybe get new extent Format new CA at HURBA positionFormat new CA at HURBA position Read / write half of CIs to new CARead / write half of CIs to new CA Write new sequence set CI for new CAWrite new sequence set CI for new CA Update higher level index CIsUpdate higher level index CIs Erase moved CIs from old CA, write Erase moved CIs from old CA, write

empty CIsempty CIs Write updated original sequence set CIWrite updated original sequence set CI


RecommendationsRecommendations

Don’t worry about CI splitsDon’t worry about CI splits Avoid excessive CA splits by Avoid excessive CA splits by

defining CA freespacedefining CA freespace Don’t do a reorg just because Don’t do a reorg just because

you have done n CI / CA splitsyou have done n CI / CA splits


To reorg or not to reorg?To reorg or not to reorg?

““We’ve done 1000 CA splits – We’ve done 1000 CA splits – better reorg!”better reorg!”

Inserts tend to be clusteredInserts tend to be clustered CI / CA split creates freespace CI / CA split creates freespace

where it is needed, allows faster where it is needed, allows faster insertsinserts

Reorg gets rid of freespace, Reorg gets rid of freespace, causing more CI / CA splitscausing more CI / CA splits


RecommendationsRecommendations

Avoid frequent reorgsAvoid frequent reorgs Once a split has occurred, the Once a split has occurred, the

processing cost has been paidprocessing cost has been paid Understand your applicationUnderstand your application

1 “hot spot”1 “hot spot” Little distributed freespace – let it splitLittle distributed freespace – let it split

Many hot spotsMany hot spots Little distributed freespace – let it splitLittle distributed freespace – let it split

Even distribution – no hot spotsEven distribution – no hot spots Use distributed freespaceUse distributed freespace


FreespaceFreespace

•3% of each CI is empty

•5% of CIs in each CA are empty

•3% of 2048 = 61 bytes = 0 records (or, at most, 1)

•5% of 315 CIs per CA = 16 CIs


FreespaceFreespace

3% CI freespace where CISZ=2048 and average LRECL=120

No room in this CI for an average length record


Altering freespaceAltering freespace

Initial freespace set via DEFINE Initial freespace set via DEFINE eg 10% of CI and 5% of CAeg 10% of CI and 5% of CA

If inserts are clustered, considerIf inserts are clustered, consider DEFINE with 0% freespace, thenDEFINE with 0% freespace, then Load the file then Load the file then ALTER freespace to non-zeroALTER freespace to non-zero


StringsStrings

VSAM allows multiple concurrent VSAM allows multiple concurrent processing e.g.processing e.g. CICS transactionsCICS transactions BrowsingBrowsing UpdatingUpdating Placeholders (“strings”) hold file Placeholders (“strings”) hold file

location infolocation info


Shared / non-shared resourcesShared / non-shared resources

Non-shared resources (NSR)Non-shared resources (NSR) Each string has its own buffersEach string has its own buffers Multiple copies of a CI may be in Multiple copies of a CI may be in

memorymemory Works well for batchWorks well for batch

Local Shared Resources (LSR)Local Shared Resources (LSR) Many strings share a pool of buffersMany strings share a pool of buffers Only 1 copy of a CI in the poolOnly 1 copy of a CI in the pool Ideal for onlineIdeal for online


Recommendations - NSRRecommendations - NSR

Non-shared resourcesNon-shared resources Each string must have enough index Each string must have enough index

buffersbuffers Bad – 1 buffer (old default)Bad – 1 buffer (old default) OK – 1 buffer per index level (new default)OK – 1 buffer per index level (new default) Good – enough buffers for all high level Good – enough buffers for all high level

indexes + 1 moreindexes + 1 more Best – enough buffers to hold entire indexBest – enough buffers to hold entire index


Recommendations - LSRRecommendations - LSR

Local Shared Resource buffersLocal Shared Resource buffers Same index buffer needs as NSR Same index buffer needs as NSR

(buffers are per pool, not per (buffers are per pool, not per string)string)

Monitor VSAM LSR stats to make Monitor VSAM LSR stats to make sure BUFNI keeps up with index sure BUFNI keeps up with index growthgrowth

Monitor data buffers for high hit Monitor data buffers for high hit ratesrates


IO with NSRIO with NSR

VSAM uses chained IO to read VSAM uses chained IO to read ahead and write behindahead and write behind Better to read many CIs in one IO Better to read many CIs in one IO Block bigBlock big

Large CI sizesLarge CI sizes Be aware that VSAM will split CIs into Be aware that VSAM will split CIs into

smaller blocks to save spacesmaller blocks to save space Eg 3390 with 32K CI gets written as 2 x Eg 3390 with 32K CI gets written as 2 x

16K blocks giving 1.5 CIs = 48K/track16K blocks giving 1.5 CIs = 48K/track

Buffer bigBuffer big ½ to 1 cyl of BUFND to minimize IO½ to 1 cyl of BUFND to minimize IO


IO with LSRIO with LSR

VSAM reads 1 CI at a time, even VSAM reads 1 CI at a time, even for sequential processingfor sequential processing


Monitor your statsMonitor your stats

LISTCAT before and after critical jobLISTCAT before and after critical job Data & Index EXCPs – the fewer the better. Data & Index EXCPs – the fewer the better.

Index EXCPs should be close to number of Index EXCPs should be close to number of index CIs.index CIs.

Job Accounting dataJob Accounting data IO count by deviceIO count by device Overal CPU & IO activityOveral CPU & IO activity

CICS statsCICS stats Shows logical / physical IO counts by fileShows logical / physical IO counts by file LSR pool hits and missesLSR pool hits and misses

VSAM buffer stats – in VSE/ESA examples docVSAM buffer stats – in VSE/ESA examples doc LSR is in 31 bit – use LOTS but don’t pageLSR is in 31 bit – use LOTS but don’t page


Sharing VSAM datasetsSharing VSAM datasets

VSAM can share files among partitionsVSAM can share files among partitions And among VSE systemsAnd among VSE systems

BUTBUTTANSTAAFL (Robert Heinlein)TANSTAAFL (Robert Heinlein)Sharing is not a performance Sharing is not a performance

option (Dan Janda)option (Dan Janda) It’s your gun and your foot It’s your gun and your foot

(Steve Huggins)(Steve Huggins)



Sharing is based onSharing is based on The type of sharing you ask for The type of sharing you ask for

(SHAREOPTIONS)(SHAREOPTIONS) VSE Lock Table within a single VSE systemVSE Lock Table within a single VSE system VSE Lock File when sharing across VSE VSE Lock File when sharing across VSE

systemssystems VSE sharing mechanism is not VSE sharing mechanism is not

compatible with zOS or zVMcompatible with zOS or zVM



Sharing at OPEN / CLOSE timeSharing at OPEN / CLOSE time Entries checked and placed in / removed Entries checked and placed in / removed

from lock tablefrom lock table If DASD volume is added as shared (ADD If DASD volume is added as shared (ADD

cuu,SHR), it is added to lock filecuu,SHR), it is added to lock file VSE & VSAM allow concurrent VSE & VSAM allow concurrent

processing to protect against processing to protect against concurrent updates messing up the fileconcurrent updates messing up the file



Integrity classes – your choiceIntegrity classes – your choice NO INTEGRITY – VSE & VSAM provide no data NO INTEGRITY – VSE & VSAM provide no data

protection: it’s all up to you. Your data can be protection: it’s all up to you. Your data can be messed up.messed up.

WRITE INTEGRITY – VSE & VSAM protect against WRITE INTEGRITY – VSE & VSAM protect against concurrent updatesconcurrent updates

READ INTEGRITY – VSE & VSAM make sure your READ INTEGRITY – VSE & VSAM make sure your programs always see the latest version of a programs always see the latest version of a recordrecord

The priceThe price Higher levels & broader scopes of integrity lead Higher levels & broader scopes of integrity lead

to more CPU and IO activityto more CPU and IO activity


SHAREOPTIONSSHAREOPTIONS

Ready – Fire – AimReady – Fire – Aim Set in DEFINE CLUSTERSet in DEFINE CLUSTER Get it wrong & be prepared to sufferGet it wrong & be prepared to suffer If a disk drive isn’t shared between If a disk drive isn’t shared between

VSEs, don’t ADD it with SHR as this VSEs, don’t ADD it with SHR as this causes lock file IOcauses lock file IO


SHAREOPTIONS & LockingSHAREOPTIONS & Locking

SHR(1) 1 output OR many input SHR(1) 1 output OR many input External lock at OPEN, unlock at CLOSEExternal lock at OPEN, unlock at CLOSE

SHR(2) 1 output AND many inputSHR(2) 1 output AND many input External lock at OPEN, unlock at CLOSEExternal lock at OPEN, unlock at CLOSE

SHR(3) No checking or locking SHR(3) No checking or locking Prepare for garbage dataPrepare for garbage data

SHR(4) Many output in one VSE & many SHR(4) Many output in one VSE & many input OPENs across all VSEsinput OPENs across all VSEs External lock at OPEN, unlock at CLOSEExternal lock at OPEN, unlock at CLOSE External lock at access, unlock at releaseExternal lock at access, unlock at release

SHR(4 4) Many output OPENs across all SHR(4 4) Many output OPENs across all VSEs + many input OPENsVSEs + many input OPENs Locks same as SHR(4)Locks same as SHR(4)


Alternate indexes (AIX)Alternate indexes (AIX)

An AIX is a VSAM KSDS, acting as a An AIX is a VSAM KSDS, acting as a “pointer file” for another file“pointer file” for another file Target file (“Base Cluster”) can beTarget file (“Base Cluster”) can be

KSDS – pointers are KSDS key valuesKSDS – pointers are KSDS key values ESDS – pointers are Relative Byte AddrsESDS – pointers are Relative Byte Addrs

Great for multiple or non-unique keysGreat for multiple or non-unique keys BUTBUT

Processing via an AIX needs IO to both the Processing via an AIX needs IO to both the AIX and to the base clusterAIX and to the base cluster


Setting up an AIXSetting up an AIX

DEFINE CLUSTER for base clusterDEFINE CLUSTER for base cluster DEFINE AIX for the alternate indexDEFINE AIX for the alternate index

Give base cluster’s name & alternate keyGive base cluster’s name & alternate key Data & Index CI sizesData & Index CI sizes

DEFINE PATHDEFINE PATH Allows specifying of NOUPGRADE pathsAllows specifying of NOUPGRADE paths

BLDINDEXBLDINDEX Reads primary & alternate key info from base Reads primary & alternate key info from base

clustercluster Sorts into alternate key sequenceSorts into alternate key sequence Loads alternate indexLoads alternate index


AIX recommendationsAIX recommendations

To process the base cluster in AIX To process the base cluster in AIX order, it is better to sort it and use order, it is better to sort it and use the SORTOUT filethe SORTOUT file

Remember VSAM processes base Remember VSAM processes base clusters directly based on AIX valuesclusters directly based on AIX values

Base cluster will need lots of index Base cluster will need lots of index buffers for batch processing. Give buffers for batch processing. Give Base cluster large BUFFERSPACE on Base cluster large BUFFERSPACE on DEFINE or ALTERDEFINE or ALTER


AIX and CICSAIX and CICS

““SPHERE” – a base cluster and all its SPHERE” – a base cluster and all its AIXs related to itAIXs related to it

RequirementsRequirements Each sphere must be wholly within one Each sphere must be wholly within one

LSR poolLSR pool Use Dataset Name SharingUse Dataset Name Sharing

In CICS 2.3, add BASE= to FCT entry forIn CICS 2.3, add BASE= to FCT entry for Base cluster file entryBase cluster file entry Each related path file entryEach related path file entry

This is automatic in CICS TSThis is automatic in CICS TS SHR(2) is usually bestSHR(2) is usually best Make sure your CICS and VSAM service Make sure your CICS and VSAM service

is current!is current!


Contacting the presenterContacting the presenter

You can contact me by email at You can contact me by email at [email protected]@e-vse.com

Dan Janda’s website has much of Dan Janda’s website has much of this info – this info – http://business.epix.net/~theswamihttp://business.epix.net/~theswami

And, if you want to find me this And, if you want to find me this evening…evening…

mailto:[email protected]

http://business.epix.net/~theswami


You’ll find me hereYou’ll find me here

Documents

WAVV 2007, Green Bay, WI VSE/VSAM – Inside & Out John Mycroft, Software Developer CSI International @e-vse.com