Xports: Network-Based Macro-Molecular Structure Determination

R. Bramley, D. F. McMullen, J. Huffman

Indiana University - BloomingtonI. Foster, G. von Laszewski, M. Westbrook

Argonne National LaboratoryE. Westbrook,

Lawrence Berkeley Laboratory

Overview

• Application and goals– X-ray crystallography; remote instrument control and collaboration

• Software architecture– Explore use of DoE Common Component Architecture framework

– Extensions to non-compute components, network awareness

– Explore use of CoG Kits (commodity Grid components)

• Network requirements and research– End-to-end reservation

– QoS requirements

– Capability for dynamic network access, control

Application and Goals

• X-Ray crystallography to determine molecular structure

• Better resolution from high-energy, high-flux beam lines at LBL and ANL

• Current procedure:– Ship crystal sample to LBL

– Lab tech mounts sample, runs predetermined scan procedure

– Data collected, written to DAT, shipped FedEx

– If sample is flawed/mounted incorrectly, multi-hour run wasted

Application and Goals

• Desired procedure– Scientist sends sample to LBL

– With scientist’s advice via teleconferencing, technician mounts sample

– Scientist sends initial set of scan commands

– Preliminary frames are filtered via local/remote components, get sent to scientist who manipulates images to check

• mounting is proper

• sample is still good

• whether sample is subject to twinning

– Scientist sends new scan parameters - or terminates beam-line session

– At any time, scientist can allow remote colleagues to view partial results and confer

Application

Beam Source

Sample Crystal

Goniostat

CCD Array

LocalStore

CCD DataCollector

User CtlScan Set

StructureAnalysis

Local DSP,Buffering

CollaborativeVisualization

Application

Beam Source

Sample Crystal

Goniostat

CCD Array

LocalStore

CCD DataCollector

User CtlScan Set

StructureAnalysis

Local DSP,Buffering

CollaborativeVisualization

Image Framesto Struct. Analysis

Image Framesfrom Local St

Image Frames

Ctl Signals

Status

User Ctl

Munged Frames

Xports Software Architecture

• Common component architecture (CCA) mechanisms• Utilize CoG Kits: Commodity Grid Kits • CCA defines minimal feature set for HPC software

“components” - but is not a component framework• Provides for interoperability between components

developed by different sites using different frameworks. • CORBA, DCOM, Java Beans are related systems but

don’t handle HPC and parallel components. Fail on “minimum surface area” test.

• Basic ideas: CCAServices objects, and CCAPorts

Component System Advantages

• Dynamic composition: run-time construction• Run-time customization via configuration parameters• Components may have state, multiple interfaces. • Can move location without recompiling/relinking• “Multicast” capability• Supports composition of components in different

languages and software systems

CCA Ports

• Output ports provide methods to other components; example might be SendData( )

• Input ports are remote references to output port methods• Only required method: PortInfo( ) giving port type and

name as strings. Type matching via the strings.• Fast “direct connect” for components in same address

space; has uses port = connected provides port

OutPort InPortSource Target

Comm Channel

CCA Services

• Each component is provided with “CCA.services” object– Used by component to tell framework about the ports it has

– Access to services provided by framework for components

• Services methods include (SIDL interface descriptions)– array<PortInfo> getProvidesPorts( );

– array<PortInfo> getUsesPorts( );

– Port getPort(in string name);

– registerUsesPort(in PortInfo name_and_type);

– addProvidesPort(in ProvidesPort inPort, in PortInfo name);

– releasePort(in string name);

– ComponentID getComponentID( );

Indiana CCA Toolkit

• An implementation of CCA with– Directory Service: for browsing remote directories containing data

about components and their specifications.

– Registry Service: for browsing remote directories with data about running instances of components.

– Creation Service: instantiates another component.

– Connection Service: connects port of one component to another.

– Event Channel: publication and filtering of events. Push-based publisher channel/subscriber model with type filtering.

• Services above are themselves “pseudocomponents” - CCA components with some internal method access

CCAT Extensions for XPorts

• CCA model based on computational components– Components consisting of parallel programs

– Mixture of shared memory, distributed memory components

• Xports must extend to – Remote instrument access

– Remote control

– Large-scale, multi-source data management

– Network capabilities access

Xports Components

• Components match block model shown earlier– Handle difficulty of proprietary HW/SW of beam-line equipment

– Control port takes user-specified scan parameters

– Frame output port multiplexed to avoid loss of data from network problems

• May require caching/sequencing component• Data analysis components dynamically instantiated

where most useful - on site, or at user location.• User interface components: dynamically allow scripting,

GUI, or auto-default as needed.

Xports Components

• Network capabilities and CCA components– Network aware computational/data store components

• Dynamic negotiation of protocols and capabilities, on a per-message basis if necessary

• Ability to restore lost links, re-establish state

• Interfaces to DiffServ, other systems (?)

• Ability to vary QoS as analysis proceeds

– Network status/statistics interface components

• Give user information about overall system

– Co-allocation of multiple QoS streams with differing service levels (more later)

Network requirements and research

• Bulk data transfer

• QoS requirements

• Network related component enhancements

Bulk data transfer

• Rates: 12MB/sec for several hours– Up to 16 MB/frame uncompressed (for 4k x 4k detector)

– 9-900 frames

– One frame per 1.5 sec

• Other apps (e.g., CMT) have one frame per 0.1 sec, but total bandwidth reservation time reduced by same factor

• Frames “demultiplexed” for reliability to – local buffer/cache

– local or remote computational component

– mass storage component

– (beam-line time rarer, more expensive than compute/network time).

Xport QoS Requirements

• Multiple streams: – CCD frame transfer: initial frames of critical importance

– 2 to 3 video cameras: operator, crystal, crystal alignment, user(s)

– Audio

– Real-time visualization and shared virtual spaces (collaborative CAVE data presentation and analysis)

• Co-allocation of multiple service levels– Video and audio

– Bulk data transfer to/from HPSS and local cache

– Data transfer between components during execution of distributed computations

Xports QoS Between Functional Blocks

CCD Detector

crystal monitoring camera

conference camera

Goniostat control and status,

temperature

monochromator

Beamline Network app. services End-user services

View CCD images

Scan setup and control

Mass storage for CCD output

Structure solution and refinement

Offload data

Structure visualization

Structure refinement control

Work with beamline tech.

On-line data reduction

QoS required

QoS not required

Image analysis

Network related component enhancements

• Determine and verify routing and link characteristics– Components may connect temporarily, drop and reconnect

– During its lifetime a component should be aware of routing or link capacity changes

– Determine link characteristics (throughput, latency(?), optimum protocol and socket settings)

• DiffServ QoS support at component level– Determine classes of service and permission to request them intra- and

inter- domain (?!)

– Set a class of service for a flow

– Determine ad hoc performance of a “QoS” flow

Xport Summary

• Application has full range of network requirements• Good candidate for CCA component architecture

extended to non-compute services• Network requirements range from premium to default• Require simultaneous management of multiple QoS

streams with differing priorities.• Component-to-component CCA port connection will have

QoS requirements that vary dynamically