10/15/05 1 Making a build of WDSS-II from the source Jeff Brogden National Severe Storms Laboratory & University of Oklahoma

10/15/05 http://www.wdssii.org/ 1

Making a build of WDSS-II from the source

Jeff BrogdenNational Severe Storms Laboratory & University of Oklahoma


Concurrent Versions System

CVS – Concurrent Versions System CVS is a version control system which allows us to record the history of

source file changes. Allows multiple developers to make changes to the code at the same time. Can access it across a network.

WDSS-II's repository is on tensor.nssl.noaa.gov

Not accessible from outside the lab Source code for infrastructure, algorithms is not freely distributed

Make sure that you are allowed to put source code on a machine before you make a build on that machine

No source code on machines belonging to private companies, for example.

Make build on NSSL machines, then move the build over.


CVS Access

To view the repository and logs, you can use the web. http://tensor.nssl.noaa.gov/

Two ways to access CVS Using pserver

setenv CVSROOT :pserver:jbrogden@tensor:/var/cvs export CVSROOT=:pserver:jbrogden@tensor:/var/cvs

Using ssh (preferred) setenv CVS_RSH ssh setenv CVSROOT jbrogden@tensor:/var/cvs Your LDAP username and password will work Use ssh techniques to give yourself password-less access

Log into CVS cvs login


Build Scripts

Builds are best created with the perl script build_distribution.pl

Interface to this script with build_distribution.shShell script to help make builds easierAllows you to set the appropriate optionsAlways use the most current

build_distribution.pl to create builds.


Build script options

Options are set in build_distribution.sh:perl build_distribution.tmp\

user=":pserver:jbrogden"\

loc="/mnt/home/jbrogden/CLEAN/WDSS2"\

packages="w2 w2algs w2tools"\

static=no

'perl build_distribution.pl -help' will give a list of options you can set.


Options

user (jbrogden) – user name on tensor (LDAP user name).

loc (/home/jbrogden/WDSS2) – location of build in your directory.

branch () – empty indicates latest.

asof () – empty indicates latest, (any format that cvs -D can accept,

including 20051017 or yesterday)

static (yes) – or shared libraries.

extras (--disable-orpg --disable-hires --disable-wish --enable-

nexrad --enable-netssap)


Options - cont...

packages (gdal g2clib netcdf config orpginfr udunits w2 w2ext

w2algs w2tools)

optimized (no) – uses O2 for gcc

debug (yes) – debug flag (-g) to compiler

doc (no) – build the documentation

tar (no) – make distribution tar file.

cxxflags () – additional compile flags

checkout (yes) – check out or just build


Options cont....

researchonly (no) – if turned on, this build is created without

realtime support, for research users only. The free version of WDSS-II is created with researchonly

gdal (yes) – build applications that rely on gdal.

numcpus (1) – number of CPUs your machine has.

distclean (no) – do a 'make distclean' in the directory to remove

detritus from earlier builds.


Using CVS directly

Documentation available on tensor's web page. http://tensor.nssl.noaa.gov/cvs.html

Basic Commands checkout

'cvs checkout w2' Creates a directory called 'w2' and populates it with files.

Committing changes in existing file cvs commit filename starts an editor so you can enter a log message. use '-m "message" ' to avoid starting editor


Using CVS - cont...

viewing differences cvs diff filename

runs the diff command to compare your copy to latest checked in copy of file.

tkdiff filename will launch graphical diff If you don’t have tkdiff, Google it and install it (just a script)

adding files to existing directories – two step process cvs add filename

use '-kb' if it's a binary file cvs commit filename

actually check file in and make it available remove files – three step process

cvs remove filename rm filename cvs commit filename


Using CVS - cont...

checking the status of files cvs status [filename] cvs log [filename]

updating and merging cvs update filename – updates a specified file cvs update – recursively update all files from current directory cvs update directory – update a specific directory

Works pretty well, but there can be conflicts. saves original file in '.#filename.version (ie .#driver.c.1.4) new file contains conflict markers to show where conflicts exist.

'<<<<<<<', '=======', '>>>>>>>'.


WDSS-II Packages

gdal – for converting to GeoTIFF (optional)

g2clib – for converting to GRIB2 (optional)

netcdf – for reading/writing NetCDF files

config – used to get autoconfig stuff working

orpginfr – used for LB and event-handling infrastructure

udunits – unit conversion package

w2 – all WDSS-II infrastructure

w2ext – extra data formats (Level-III, ORPG...) (optional)

w2algs – algorithm development

w2tools – display and its supporting infrastructure & a few other

data source management tools


Adding an algorithm

Writing algorithms covered in earlier session.

We recommend: make most of code a library have apps as a subdirectory use an existing algorithms Makefile.am file as a template for

your own modify w2algs/configure.in to add your Makefile add all your code to the CVS repository run ./autogen.sh and make to be sure things work correctly.

Look in w2/w2config/algcreatorfiles/README for more information.


Doxygen

Doxygen allows us to use code comments to auto-generate

documentation.

Current documentation is available at

http://tensor.nssl.noaa.gov/cgi-bin/doxlist.pl

Example of documentation:

/** SampleClass * Implements a series of funcitons that operates on Samples * @see BigClass * @author LastName */class SampleClass : public BigClass{ public: /** Construct a SampleClass with the number of parameters. * @param num How many samples there are. * @param val The default value. */ SampleClass(int num, double val);}


Environment Variables

If, for some reason, you have things installed in non-standard

places, you can use environment variables to inform the autoconfig

process of the build where they exist.

Some examples: GLDIR – location of GL libraries FREETYPEDIR – location of FreeType Font library NCDIR – location of NetCDF libraries

For the most part, you shouldn't ever need to do this.

If you are porting to a different platform where this is where these

files will always be, add the directory to config/w2.m4.


Authorization files

WDSS-II has an authorization mechanism that allows you to

control access to different features.

You can distribute authorization files to users that "turn-off"

capabilities or "turn-on" things without having to recompile.

For algorithms that wish to restrict users, there is a config file. each line states the restriction. (ie "realtime false")

All algorithm's authorization keys are collected into single key file. The authorization key value (or the file) should NOT be

distributed Keep the authorization key file safe.


Use W2Authorization class just after you get all the startup options#ifndef algorithm_AUTH_KEYW2Authorization auth("alg_name", algorithm_AUTH_KEY);// Make sure authorized for more than 3 tilesif (auth.getString(“numTiles”) != “ANY” && num_tiles> atoi(auth.getString(“numTiles”).c_str()) ) throw std::string(“Exceeded tiles limit!”);#endif

Make a new file called w2/w2config/auth/default/alg_name

Run w2auth to create a default auth for this filew2auth -i w2config/auth/default -o w2config/auth \

-k w2config/auth/default/key

Check the added/modified files in the auth and default directories

into CVS

How to restrict code


Format of Authoriztion file

Field-name values-seperated-by-commas

no extra spaces in the values field

true/false are special literals

ANY is a special value

domain nssl.noaa.gov,ou.eduuser ANYuntil ANYrealtime truenumTiles 5


Making a build for external users

Generate authorization files Use w2auth to generate 128-bit keys that are hard to break Keep keyfile in a safe place – no way to recreate it if you lose it. Use different keyfiles for different licensees Write plain-text authorization files (format on previous slide) Run w2auth to create encrypted .auth files

Make a build Specify tar=yes Specify keyfile=location_to_key

Replace auth directory in tar file Replace the w2config/w2auth directory in temp_build Recreate tar file – this is what you need to distribute

No source code to be distributed outside NSSL

Binary distribution outside NSSL must be approved


Extracting an algorithm?

The algorithm scientific part is separate from the WDSS-II infrastructure is dependent on it for input, output, data formats, notification,

factories, data objects, etc.

This common infrastructure Makes algorithm development easy and productive Makes “extracting” the scientific part almost impossible.

You will have to code up the WDSS-II services again Not recommended Unless you can code up the services based on similar facilities that

already exist in another system (e.g: AWIPS) If this is the case, we can help you create the appropriate adapters.


Developer services

Changes checked into cvs since the previous day are e-mailed

out to all users of cvs Tell ITS that you need to be added to the “cvs” group Name of file, and message listing reason for change Make sure to explain changes Use this log to keep up to date on changes to entire system

Documentation automatically built on tensor The documentation on tensor is up-to-date daily Document all classes that others will use! The documentation on wdssii.org reflects the latest research

build.

Documents

10/15/05 1 Making a build of WDSS-II from the source Jeff Brogden National Severe Storms Laboratory & University of Oklahoma