• CentOS/RedHat 6 was released on 2010-11-09, as of

2012-12-20 they have released 1048 updates, 1.35

updates per day, 16% of the entire OS.

• System management tools and configuration

management tools (Rocks, Warewulf, Puppet,

CFEngine, etc.) do not directly address the problem of

“collateral” damage caused by updates.

• Virtualized computing means a proliferation of system

images/configurations. None have detailed

descriptions of how they where defined. This

compounds the problem.

• How does one determine an application’s specific

dependencies (static/dynamic) on system software?

• Systems are continuously updated by administrators

to add new capabilities, fix bugs, and close security

holes. Did a change “break” the application?

• Can one use an application’s dependencies

(fingerprint) to build a compatible system?

• System requirement: Python version >= 2.4

• Create, display content, of swirl file

• Verify availability of requirement, verify integrity of

requirement of a swirl file

• Support dependency detection for ELF format

(32bit/64bit)

• Dynamic dependencies detection using /proc/*/maps

• Public Git Repository

• Detect dynamic dependency

Extend dynamic dependency detection tracing running

application using ptrace system call

• Build a system around an application

Automatically create a virtual appliance which can run

a given application

• Support scientific verifiability and reproducibility

Create an archive of an application with all its

dependencies which can be executed on another

(newer) system

• An application can be statically or dynamically

analyzed to create a fingerprint of its dependencies

and requirements. All this information is saved in a

Swirl file.

• A Swirl file contains: a list of all the dependencies,

hashes of files providing dependencies and package

names of each dependencies

• The swirl can then be reused to:

1. Detect system changes which are part of the

application dependencies

2. Verify if all dependencies are available on another

system

3. Create a customized appliance which has all the

dependencies of a given application.

Fingerprinting Scientific Codes to Verify and Create

Compatible System Software Environments Philip Papadopoulos, San Diego Supercomputer Center and Calit2; University of California San Diego (ppapadopoulos@ucsd.edu)

Luca Clementi, San Diego Supercomputer Center; University of California San Diego (lclementi@ucsd.edu)

1. Papadopoulos P. M., Katz M. J., Bruno G., “NPACI Rocks: Tools and

Techniques for Easily Deploying Manageable Linux Clusters.” IEEE

Cluster 2001, Newport Beach, CA, Oct 2001.

2. Grethe J. S., Baru C., Gupta A., James M., Ludaescher B., Martone

M. E., Papadopoulos P. M., Peltier S. T., Rajasekar A., Santini S.,

Zaslavsky I. N., Ellisman M. H., “Biomedical Informatics Research

Network: Building a National Collaboratory to Hasten the Derivation

of New Understanding and Treatment of Disease.” From Grid to

Healthgrid: Proceeding of Healthgrid 2005.

3. Papadopoulos P. M., “Extending Cluster to Amazon EC2 Using the

Rocks Toolkit”, International Journal of High Performance Computing

Applications, 2011

4. Allen J. M. “OS and Application Fingerprinting Techniques” SANS

Institute Info Sec Reading Room

Fingerprint Approach

References

Overall Usage Current Features Problem Description

Background

• Extensive set of unit tests

• Continuous integration each code modification is tested

at UW-Madison NMI Build & Test Lab (a distributed,

multi-platform facility designed to provide automated

software build and test services), running unit tests on:

• RHEL 5.3 32bit/64bit

• RHEL 6.3 32bit/64bit

• Debian 5.0 64bit

• XSEDE Preliminary Testing

Custom applications with Fingerprint verified nightly

NAMD (Molecular Dynamics)

APBS (Poisson Boltzman Solver)

• XSEDE system in use:

Gordon (San Diego Supercomputer Center)

Open Sience Grid (Indiana University)

Steele (Purdue)

Quality Assurance and Validation

[tg-steele ~]$ fingerprint -y -v -f namd_compute.swirl

The file namd_compute.swirl failed.

Missing Dependecies:

libfftw3f.so.3()(64bit)

liblimic2.so.0()(64bit)

libmpich.so.3()(64bit)

libmpichcxx.so.3()(64bit)

libmpl.so.1()(64bit)

libopa.so.1()(64bit)

[tg-steele ~]$ module load mpich2/1.4.1p1_intel-12.1

[tg-steele ~]$ module load fftw/3.3.1_intel-12.1_openmpi-1.4.4

[tg-steele ~]$ module load intel/12.1

[tg-steele ~]$ fingerprint -y -v -f namd_compute.swirl

The file namd_compute.swirl failed.

Missing Dependecies:

liblimic2.so.0()(64bit)

[tg-steele ~]$ ldd namd2 | wc

43

Example Usage – Compatibility Testing

Fingerprint created on Gordon (CentOS 5.6)

Load Modules on Steele (RHEL 5.8)

RDMA Module only on Gordon

Future Directions

NAMD has 43 System Library Dependencies

* Fingerprint to system definition will be developed in year 2 of project

*