SIR Database Management A Refresher Course for SIR DataBase Administrators © Tom Shriver, DataVisor 2001, 2002

SIR Database Management

A Refresher Coursefor SIR DataBase Administrators

©Tom Shriver, DataVisor 2001, 2002

SIR Database Management

A Refresher Coursefor SIR DataBase Administrators

©Tom Shriver, DataVisor 2001, 2002

SIR Database ManagementA Refresher Course

for SIR DataBase Administrators

I. SIR Files

II. Master III. SIR Menu System IV. Batch Processing & Scheduled Events V. Managing Schema Modifications VI. Backups VII. DB Integrity, Corruptions & Recovery VIII. SIR Utilities for Database Management IX. SIR Software Upgrades & Local

Testing

©Tom Shriver, DataVisor 2001

SIR Database Administration ©2001 Tom Shriver, DataVisor 4

Typical SIR Software Files & Locations

Directoriesc:\sir2002 SIR executables and support filesc:\sir2002\help SIR on-line help (html)c:\sir2002\tbbb Tool Bar Button Binary (*.bmp picture) filesc:\sir2002\images Picture files (*.bmp)c:\sir2002\test Example database COMPANYc:\sir2002\apiHost Language application interface support

Important Files:Executables: sir.exe; sirbatch.exe; sirmaster.exe;

sirsql.exe; sirgraph.exe; sirforms.exe;*.dll files

sirproc.srp System Procfile (SYSPROC) & the SIR Environment PQLsir.lcn Contains Licensing Information


SIR Database Files

Typically kept in a database specific folder A database has 4, 5 or 6 files Database Name plus numbered extension

c:\dv\company.sr1 Schema, rectype & variable specsc:\dv\company.sr2 Schema, textual specs (e.g. labels)c:\dv\company.sr3 Data primary index & datac:\dv\company.sr4 Procfile procedure definitions, PQLc:\dv\company.sr5 Journal data & schema updates)c:\dv\company.sr6 Alternate Indexes (SIR2002)

Database name here is COMPANY SIR database Prefix is c:\dv\


Data File (SIR3) Internal Usage

Low Level Storage Unit is a SIR Data Block Will be multiples of Op System data block size Will be large enough to hold 4 of largest database records

Contains B-Tree Index Two Levels, Max 6 Levels Data Record Search is b-tree search to data block and sequential

within data block

Contains Data Records Records never span Data Blocks Data Blocks are split leaving empty space:

the Loading Factor

Data Blocks Cached during use; User can control cache


Journal File (SIR5) Internal Usage

Journal is a Sequential Binary File Contains copies of Every DB Transaction

Data & Schema; not Procfile Each Entry Stamped with Update Level New Records: Record copied to Journal Modified Rec: Old Rec & New Rec Copied (from-to) Deleted Rec: Copy of Deleted Rec Journalled

Journal Can Grow Much Faster than Database View with ITEMIZE FILE Utility


Backup and Archive Files

Database & Application files backed up Separate Device (tape, disk, computer) Separate Facility (different building)

Additional Copies Locally Depending on resources (available disk space) Depending on criticality (time to recover from tape)

Master: How it WorksPart 1

SIRDatabase

User BUser A User C

A SIR DataBase Allows

Multiple Readers and

a Single Writer

User AReads & Writes

Users B & CRead Only

Master: How it Works

Part 2

SIRDatabase

User BUser A User C

A SIR Database Under Master Appears to Allow

Multiple Readers and Writers

Users & Master Exchange

Messages & Data

Master is the Only Writer

(& Also Reads)

MASTER

Another SIR Database

Master may Control Several

Databases


Protecting & Controlling DataBasewith Master

Don’t allow Users “into” DataBase without Master

Once in DataBase Don’t Allow Access to the Data Entry System

Reports Don’t Need Master Disconnect Master Before Running Query Re-Connect After the Query Can be Automated


Master Administration: StartUp

Start from an OS command file (BAT, COM) on the Server

Assign a Master Administration Password Set Difference File Copy (DFC) Interval

On Windows NT set up as a System Service


Master Administration: Monitoring Master

From Master Window on Server Pause Master ( ctl-C ) List Users ( LIST ) Kill User ( KILL ) Copy Difference File ( COPY ) Resume Master ( Enter )

From DataBase “Settings”, “Master Settings” Connect / Disconnect Monitor Users, DFC Control Users, DFC


Master Administration: Orderly Shut-Down

Interactively Check Users, Make sure Everyone is Off Force a Difference File Copy Use “NoLogons” Exit if there are users

In Batch Use PQL Commands & Functions

o Monitor Userso Remove Users

DBMS Command EXIT MASTER NOLOGONS Explicit EXIT MASTER


SIR Menu and Toolbar SystemThe Big Picture

Covered Separately at UK SIR Conference 2002


Batch Processing

BAT Files Executed from Server SIRBATCH Execution Statement Specificationss

o Database name (db = dbname )o Prefix ( p = ‘folder_spec’) db

directoryo Master (mst = ‘master_name’)o Proc to Run (in = ‘file_name’)o Output File (out = ‘file_name’)


BAT File Examples

rem Run Daily CathLab Volume Reportcd \\ind1nt99\hbaseadm\cvl\\ind1nt99\sir2000\sirbatch.exe in=cvl.pql out=cvl.out

- - - - - - - - - - - - - - - - - - - - - - - - - - -

rem Run DB Integrity Checkcd s:\hbaseadm\utils:\sir2000\sirbatch.exe in=stopmst.pql out=stop.outs:\sir2000\sirbatch.exe in=verify.pql out=verify.outs:\sir2000\master.bat


Scheduled Events

Use a Job Scheduler Hourly, Daily, Weekly, Monthly, Selected Days You set Time, Frequency, BAT File name to run

Possibilities Nightly Backup Routines Interfaces Daily Reports Integrity Checks Data Audits


Managing the Database SchemaSchema Modifications

Three Types of Schema Modifications Changes to Primary Key Structural Change to Records

(add/delete variables, change data types) Cosmetic Changes

(value labels, variable labels, BDI directives)

Index Mods Lock Rectype & Require Restrucure Record Structure Changes handled Dynamically Cosmetic Changes are Immediate Master Does not Handle all Changes Equally Well


CODEBOOK LEVEL & UPDATE LEVEL

Code Book (Schema) Level Incremented with each Schema Run

See current level with system global CBLEVEL

Update Level Incremented with each: Data Update Run

Data records "stamped" with update level of last save - see with RECLEVEL function in PQL

Schema Run - see current update level with List Stats Utility

Max Update Levels (pre SIR2002): 32,767 Reset with Export/Import or Reload Option


Update Level Usage by System

Sequences of Schema Mods Maintained Identified by Update Level SIR eventually matches data records to schema level

when it restructures records

Executable Procedures & Subroutines Keep "Header" information

-- Record Types referenced in routine-- Schema Level of those records

Automatically deleted if Schema mods affect routine Header Info matched to schema every time used


Handling Schema Modifications

Keep Schema Mod Frequency Down Save up modifications & do several at once Post a schedule for users

Always & Immediately Restructure with UNLOAD / RELOAD

Not absolutely required, but safe Minor problems exacerbated until restructure happens


Database Indexes

Primary Index kept in the SR3 Data File Composed of:

CASE ID (if case structured database)RECTYPE NumberKEYFIELD Values

Index size is fixed, usually as big as largest key Data Records stored in this sorted order

Secondary Indexes ( SIR2002 ) kept in SR6 File Keyed by any variables Options for Unique / Non-Unique Indexes Can be dropped and added


Backups

Regular System “Tape Backups” Make sure not ‘incrementals’ on db files

Copies to Other Directories on Server For Quick Access in an Emergency To Assure Successful System Backups Minimize Database Downtime

Journal Files Journal File is the SR5 File Keep Nightly Copies, About a Weeks’ Worth Delete Current Journal, Will Restart Automatically


Backups: UNLOAD FILE Utility

The SIR Recommended Backup Utility Creates Sequential Database in a Single File Additional Functionality:

Restructures DB Data after Schema Modifications Releases Empty Data Blocks Option to Reset Update Level Can backup multiple databases to single file

RELOAD Procedure restores database in Restructured form


Backups: Database Integrity Checks

Verify File Daily (Every Night) Turn Master Off Run VERIFY FILE Turn Master back On

Check Verify File Results Every Day Could be shown to specific users automatically


Backups: Verify File Utility & Options

VERIFY FILECIRKEY | list case id valuesRCF | list CIR record countsCIRDATA | list CIR variable valuesRECKEY | list record key field valuesRECDATA | list record variable valuesCHECK | check all variable values vs. schemaCOUNT | process a subset of casesCCF | clear corruption flagPATCH | fix repairable corruptions

Rerun Verify File after PATCH Listing Options Produce Huge Output Use CCF with caution. It hides problems.


Backups: Verify File Output

Data records on database 19614Cases on database 53Index blocks read 10Data blocks read 618

Average index block loading 0.74Average data block loading 0.81Warning messages 0Correctable errors 0Non-correctable errors 0

Verification complete with no errors


Corrupt Database Recovery

Methods in Order of Preference VERIFY FILE with PATCH JOURNAL RESTORE with DB Backup JOURNAL UPLOAD / DOWNLOAD SIR FILE DUMP / Batch Data Input Re-Enter Data

Disaster Planning: Create a Document Disaster Practicum: Run a "Fire Drill"

Call Information Services for Backup Files Recover with Journal Restore (can you recover 3 days)


EXPORT Utility

Creates a textual set of SIR commands that when run, re-builds the database

Schema Commands (like WRITE SCHEMA) Text Members of the Procfile (like PWRITE) Export data is in a special text format

Intended to Move a Database between SIR Platforms (different operating systems)

Frequently used as an alternate backup method


IMPORT Procedure

Several Options Run the Export File Use Import Option on Database Menu Batch Run

c:\sir2002\sirbatch.exe in=mydb.exp out=mydb.out


SIR FILE DUMP &Batch Data Input

SIR FILE DUMP Creates formatted files per Schema specifications

o Must maintain Schema column specso Variables not assigned columns are not dumpedo Overlapping fields (including keys) may be overwritten

Specify Rectypes to Dump Boolean, List, Count & Sample for Record Filtering

Batch Data Input Inputs data from files formatted as per Schema Control with Add / Replace Options


SUBSET and MERGE

SIR SUBSET Creates Sequential "Unload" type of file Options to filter records to create subset Options to rename records and variables

SIR MERGE Merges SUBSET data into a different database Not available through menu system

Potential Use Recover specific data records from backups


Database Content & Status Reporting Utilities

SIR Utilities from Commands & Menu SystemLIST STATS Total Records, DB Settings, Rectype Summary StatsSIR SCHEMA LIST Schema Description; Options for Level of DetailWRITE SCHEMA Schema Command FileITEMIZE FILE Sequential File Description (Unloads, Journals,

Subsets)SIR FILE LIST Data Listed & LabeledVERIFY FILE DB Integrity with Space Usage Statistics

Build Your Own Utility with PQL PQL Functions Schema Functions (records, fields, keys, labels)

Tabfile FunctionsSystem FunctionsEnvironment FunctionsClient Server (Master, ODBC, Application)

System Globals CBLEVEL, CPTIME, TERROR, SIRVER, MASTER


SIR Software Upgrades & Local Testing

Major New SIR Releases Every 18-24 Months Frequent Bug Fix Upgrades Available on Web-Site

SIR Welcomes Beta Testers When Putting up New Version of SIR

Install SIR in a new directory Create a test version of your database

Export on Old SIR & Import with New Version Recompile all Executables and Subroutines Test Entire Application before Going Live

Documents

SIR Database Management A Refresher Course for SIR DataBase Administrators © Tom Shriver, DataVisor 2001, 2002