Statistics Function Implementation

Rolando V. Raqueño Saturday, April 22, 2023

Statistics Function Implementation

Traditional Programming Approach


minimum.pro

function minimum, xn = n_elements(x)answer = x(0)for i=1L, n-1 do beginif( answer gt x(i) ) then $answer = x(i)

endforreturn, answer

end


sum.pro

function sum, xn = n_elements( x )answer = 0for i = 0, n-1 do beginanswer = answer + x(i)endforreturn, answer

end


Overflow Problems

• What if the array is an integer array?• Sums will have a maximum limit based on

the number of bits specified by the integer type.


Corrected sum.pro

function sum, xn = n_elements( x )answer = 0.0Dfor i = 0L, n-1 do beginanswer = answer + x(i)

endforreturn, answer

end


mean.pro

function mean, xn = n_elements(x)answer = sum(x) / nreturn, answer

end


Integer Problems

IDL> b=9/5IDL> print,b 1


Integer Problems

IDL> b=9/double(5)IDL> print,b 1.8000000


Corrected mean.pro

function mean, xn = n_elements(x)answer = sum(x) / double(n)return, answer

end


sum_squares.pro

function sum_squares, xsquares = x^2.0answer= sum( squares )return, answer

end


variance.pro

function variance, xn = double(n_elements(x))answer=(sum_squares(x)-(sum(x)^2.0/n))/

(n-1)return, answer

end


sd.pro

function sd, xanswer = sqrt(variance(x))return, answer

end


Shell Programming


UNIX Tip of the Day• Directory maneuvering commands pushd, popd, and, dirs

% cd /usr/tmp% pwd/usr/tmp% pushd ~rvrpci% pwd/cis/staff/rvrpci


UNIX Tip of the Day

% dirs/cis/staff/rvrpci /usr/tmp% pushd% pwd/usr/tmp% dirs/usr/tmp /cis/staff/rvrpci% pushd /usr/local/bin


UNIX Tip of the Day

% dirs/usr/local/bin /usr/tmp /cis/staff/rvrpci% pwd/usr/local/bin% pushd % dirs/usr/tmp /usr/local/bin/cis/staff/rvrpci


UNIX Tip of the Day% dirs/usr/tmp /usr/local/bin/cis/staff/rvrpci% pwd/usr/tmp % pushd +2% pwd/cis/staff/rvrpci


UNIX Tip of the Day% dirs/cis/staff/rvrpci /usr/tmp /usr/local/bin% popd% dirs/usr/tmp /usr/local/bin


IMPORTANT UNIX Concepts• Environment and Shell Variables

– These allow you to customize your UNIX environment

– They are different in terms of their SCOPE• SCOPE determines the visibility of a variable


Other IMPORTANT UNIX Concepts• Environment Variable

– Examples are TERM and DISPLAY– Set a particular variable to a value by using the setenv command

– You can print the value of a particular variable or all the environment variable using the printenv command


% Environment Variables %• Examples

– To set environment variables% setenv TERM vt100% setenv DOG Goofy– print out the terminal type% printenv TERM vt100– print out all environment variables% printenv


Shell Variables

• Shell variables are similar to Environment variables except they have a limited scope, i.e., they exist only in the shell which they are defined.

• Environment variables on the other hand, exist in all its children shells

• To illustrate this concept, let us look at the following example


Environment vs. Shell Variables% set prompt = "Parent Shell > "Parent Shell > setenv DOG GoofyParent Shell > set mouse=MickeyParent Shell > printenv DOGGoofyParent Shell > echo $mouseMickeyParent Shell > xterm &

(YOU SHOULD NOW HAVE A NEW xterm WINDOW)THIS IS KNOWN AS

“SPAWNING A NEW (OR CHILD) PROCESS”


Environment vs. Shell Variables(IN THE NEW xterm WINDOW, DO THE FOLLOWING)

% set prompt = "Child Shell > "Child Shell > printenv DOGGoofyChild Shell > echo $mousemouse: Undefined variable.


Environment vs. Shell VariablesChild Shell > setenv DOG PlutoChild Shell > set mouse=MinnieChild Shell > printenv DOGPlutoChild Shell > echo $mouseMinnieChild Shell > exit

(THE xterm WINDOW SHOULD NOW GO AWAY - THIS PROCESS HAS NOW BEEN KILLED)


Environment vs. Shell VariablesParent Shell >Parent Shell > printenv DOGGoofyParent Shell > echo $mouseMickeyParent Shell >


Environment & Shell Variables

• Why is this important?– UNIX uses Environment and Shell Variables

control a number of processes– Customizes your working environment– Variables used for UNIX Scripts

• They are typically defined and initialized in your .login and .cshrc files


Useful Shell Variables

filec#Allows file completionpath #List of command

directoriescdpath #List of candidate

directories to cd intohistory #Number of commands to

remember


What is shell programming?

• Shell programming – automate a set of UNIX commands.– Just like any programming language– “wrappers”

• black box a customized collection of UNIX commands.

– Example of shell programs.login.cshrc


.login fileset path=($HOME/bin /usr/local/bin \/usr/ucb /usr/sbin /bin /usr/bin \/usr/bin/X11 .)stty dec newtset -I -Qset mail=/usr/spool/mail/$USERset editmode = emacsumask 077biff ndate


.cshrc file

if ($?prompt) then set notify set history = 100 set savehist = 100 alias pd pushd alias pop popd alias vt100 "set term = vt100"endif


When these files are executed?

.cshrc – is automatically executed when you start a new

shell

.login – only gets executed once when you first login

Can be re-executed by giving the source command% source .cshrc


Other useful .login and .cshrc entries

set filecset cdpath=(~ ~rvrpci/pub ~/mythesis)

Other common entries

set path=( $path /usr/local/bin)set path=(/usr/local/bin $path)


User defined shell program

• Determine name of command• Determine input, output, and option

arguments• Determine UNIX commands to execute• Establish error trapping• Make shell program executable


A simple shell program

• dd command to swap bytes

% dd if=input.dat of=output.dat bs=2 conv=swab

• Very difficult to remember• Very little utility to non-UNIX geeks

(normal people)


We would rather see...

% swap_bytes input.dat output.dat


Special Shell Variables Set


$0 $1 $2$argv[1] $argv[2]command


Another Special Shell Variables


$#argvIndicates how many arguments are present

In this case, 2


shell program swap_bytes

#!/bin/csh -fdd if=$1 of=$2 bs=2 conv=swab


Making swap_bytes shell script executable

% ls -l swap_bytes-rw------- ... swap_bytes% chmod u+x swap_bytes% ls -l swap_bytes-rwx------ ... swap_bytes


To run swap_bytes

• swap_bytes becomes just another unix command!



Limitation of swap_bytes

• No error trapping• Should give usage when typing command

% swap_bytesusage: swap_bytes input_file output_file


Improvement to swap_bytes

#!/bin/csh -f if ( $#argv != 2 ) then echo "usage: $0 input_file output_file" exit 1 endif dd if=$1 of=$2 bs=2 conv=swab


Commad exit status

• By convention

exit 0Indicates successful command completion

exit 1 (or non-zero)Indicates some error condition


Informational message from swap_bytes

• UNIX style informational message

% swap_bytesusage: swap_bytes input_file output_file


Interactive swap_bytes

• If you want a “friendlier” shell program– Have it query the user for the inputs

• Another special shell variable can be used

$<


Interactive swap_bytes#!/bin/csh -fif ( $#argv != 2 ) then echo -n "Please enter the input file> " set input=$< echo -n "Please enter the output file> " set output=$<endifdd if=$input of=$output bs=2 conv=swab


Interactive swap_bytes example

• User simply types the command

% swap_bytesPlease enter the input file> input.datPlease enter the output file> output.dat


UNIX Quotes


A note about quotes in UNIX

% set a=ls% echo a% echo $a% set b=“$a”% echo $b% set b=‘$a’% echo $b% set b=`$a`% echo $b


A note about shell variables

• Shell variables can also double up as arrays

• Using the previous example,% echo $b% echo $b[1]% echo $#b% echo $b[$#b]


A more complex shell program• In pbmplus utilities,

rawtopgm conversion existspgmtoraw conversion does not

• A version of pgmtoraw in a programming language like C– Time consuming– Program will likely be used infrequently

• Solution: shell program


pgmtoraw shell script design

• Define input and output files• Figure out dimensions of input image • Determine number of bytes for input image• Determine number of bytes for header• Need to strip out the header bytes• Write out headerless image data


Define input and output files pgmtoraw

#!/bin/csh -fset command_name=$0set number_args=$#argvif( $number_args != 1 ) then echo “usage:$command_name input_file_name” exit 1endif...


Dimensions of input image ( pnmfile)

% more test_data.pgmP23 32551 2 34 5 67 8 9

% pnmfile test_data.pgmtest_data.pgm: PGM plain, 3 by 3 maxval 255


pgmtoraw (continued)

set input_file=$1set pnm_info = `pnmfile $input_file`set image_type = $pnm_info[2]set data_type = $pnm_info[3]set width = $pnm_info[4]set height = $pnm_info[6]set maxval = $pnm_info[8]set pixel_bytes = 1@ image_bytes = $width * $height


pgmtoraw (continued)set file_info=`wc -c $input_file`set bytes_in_file = $file_info[1]@ header = $bytes_in_file - $image_bytesdd if=$input_file bs=$pixel_bytes skip=$header


Resulting pgmtoraw utility

• Uses existing routines to obtain informationpnmfilewcdd

• Functional tool written in 20 command lines


Current Limitations of Resulting pgmtoraw utility

• No check between “ASCII” vs. “RAW” pgmif( data_type == ‘plain,’) ...

• No provisions for multibyte per pixel case– Use pnmfile results to check for above cases– endian case needs to be addressed for multibyte

case• Above conditions can be addressed by suite

of UNIX commands


Shell Scripts Wrappers and IDL

• Another utility formerly missing in pbmplusjpegtopnm or pnmtojpeg

• IDL has jpeg read/write capability– Create a “wrapper” that makes an idl

program pbmplus-like


pnmtojpeg.pro

pro pnmtojpeg, input_file, output_file

read_ppm, input_file, image

write_jpeg, output_file, image

end


Usage of pnmtojpeg.pro in IDL

IDL> pnmtojpeg,‘image.pnm’,’image.jpg’


Usage of pnmtojpeg.pro in IDL

IDL> pnmtojpeg,‘image.pnm’,’image.jpg’

• For IDL to automatically find pnmtojpeg.pro

– It must be in the current working directory

– Directory containing pnmtojpeg.pro must be defined in the ENVIRONMENT VARIABLE•IDL_PATH


IDL_PATH environment variable

setenv IDL_DIR /cis/common/rsi/idl_5setenv IDL_PATH \+$IDL_DIR/lib:\+$IDL_DIR/examples:

\+/dirs/common/rsi/idl_5:\+/dirs/common/lib/idl:\+~/lib/idl


pnmtojpeg.csh#!/bin/csh -fecho pnmtojpeg “,” “’”$1”’” “,” “’”$2”’” | idl


Usage of pnmtojpeg.csh

% pnmtojpeg.csh image.pnm image.jpg


Limitation of pnmtojpeg.csh

• Does not conform to pbmplus piping, i.e.,% tifftopnm file.tif | pnmscale 2.0 > new_file.pnm

• No error trapping


Usage cases of pnmtojpeg(no command line arguments)

% tifftopnm file.tif | pnmscale 2.0 | pnmtojpeg > new_file.jpg


Usage cases of pnmtojpeg(1 command line argument)

% pnmtojpeg image.pnm > image.jpg


Usage cases of pnmtojpeg(2 command line arguments)

% pnmtojpeg image.pnm image.jpg


Yet another wrapper pnmtojpeg

#!/bin/csh -f

# If user interrupts process, jump to stop

onintr stop# $0 is the command name# $#argv is the number of arguments# $$ is the process id


Code for no argument case

if($#argv == 0) then set input_file = /usr/tmp/$0_input_$$ set output_file = /usr/tmp/$0_output_$$ cat > $input_file pnmtojpeg.csh $input_file $output_file cat $output_file...


Code for 1 argument case

else if($#argv ==1) then set input_file = $1 set output_file = /usr/tmp/$0_output_$$ pnmtojpeg.csh $input_file $output_file cat $output_file...


Code for 2 argument caseelse set input_file = $1 set output_file = $2 pnmtojpeg.csh $input_file $output_fileendif

#clean up when finishedstop: rm -f /usr/tmp/$0_input_$$ rm -f /usr/tmp/$0_output_$$


pnmtojpeg summary

• Produced a “new” pbmplus utility• Used UNIX shell scripting

– Argument handling– Scratch space /usr/tmp– Process id handling– Clean up

• Integrated IDL program and commands• 21 lines of UNIX commands


Summary

• The “dot” files• Basics of Shell Scripting• Special Shell Variables• Seamless integration of UNIX to other

utilities (IDL)


Other Shell Constructs to keep in mind

• foreach• while• case

Documents

Statistics Function Implementation