OOI CyberInfrastructure: Technology Overview - Hyrax January 2009 Claudiu Farcas OOI CI Architecture & Design Team UCSD/Calit2

OOI CyberInfrastructure:Technology Overview - Hyrax

January 2009 Claudiu Farcas

OOI CI Architecture & Design TeamUCSD/Calit2

2

OOI-CYBERINFRASTRUCTURE

Hyrax Overview

• OPeNDAP Hyrax is a server for scientific data provisioning (using DAP2 standard)

• Uses a modular architecture to support different application-level protocols– Data access using DAP– Catalogs using THREDDS– Browsing using HTML and ASCII

• Modules for data access– Different file types– Potential for database and scripting

• Modules for commands– Commands provide varying operations for different

protocols

3


Hyrax High-level Architecture

• Two cooperating components:– OpenDAP Lightweight Front-end Server (OLFS)

provides DAP, plain ASCII & HTML interfaces– Back-end server (BES) reads data from repositories

• Both parts can be customized– Front-end: different network protocols– Back-end: different data formats/systems

OLFS BES Data

4


Hyrax Architecture

• Front-end (OLFS):– Runs inside Tomcat servlet container– Authentication & Authorization performed via

Tomcat– Provides DAP2 compliant interface– SOAP interface for programmatic access– Extensible via pluggable dispatch handlers

• Back-end (BES):– Data resource access.– Extensible (see Patrick’s slides)

• Single/Multiple Machine Installations Possible• Communication between OLFS & BES via

TCP/IP or UNIX Sockets


OLFS

Java Servlet Engine

BES

Unix Daemon

BES Commands

XML- encapsulated object

DataStore

File system with data files,SQL Database, …

DAP2

THREDDS

HTML

Optional THREDDScatalogs

Hyrax Architecture

6


OLFS DAP Request Processing

• DAP requests are made using URLs– URL: <pathname>.<request>[?<constraint>]

• Pathname: names a unique data source• Request: what the client wants• Constraint: optional modification to the request

• OLFS – parses the URL into the pathname, request and

constraint– uses knowledge of the BES commands &

capabilities to instruct the BES how to build the response

– packages the resulting response from the BES (e.g., builds a HTTP ‘text/plain’ response document and inserts the correct content).

7


Extending the OLFS

• Extension ‘modules’ written in Java• Added to a directory within Tomcat• The new modules have complete access to the request

information• Both HTTP GET and POST requests• There is some significant processing done before the

handler is called:– Conditional GET Requests– Authorization & Authentication

8


BES Architecture

Network Protocol andProcess start/stopactivities

Data Store Interfaces

BES Framework

PPT*Initialization/Termination

DAP2Access

NetCDF3 HDF4 FreeForm…

DataCatalogs

Commands**BES Commands/ XML Documents

*PPT: point to point transport**Some commands are built in, most load at run-time

9


BES Components

• Modular architecture in C++• Modules interact with the BES and can interact with

other modules• Modules can add/modify functionality of BES• New modules are loaded at runtime when needed• The command handler can be replaced to replace the

BES’s default command syntax• Handlers are not limited to DAP

10


BES Handler Interaction• Response handlers:

– Respond to commands from the ‘command handler’– Know how to create the response object – Do not necessarily fill in the response object, but know how the

response should be filled in (by asking appropriate request handlers)

– Build & transmit new responses• Request/Format handlers

– Register interests and capabilities (i.e., know-how to fill a response)

– Read data and provide object content• Handlers can:

– Build responses directly from data– Use output from other handlers– Query one or more handlers for fill-in information

• Reporters: Record usage information (e.g., logging, statistics)• Aggregators: Build aggregate responses

11


BES Extensibility

• New request handlers (data handlers like netcdf, hdf4)– Request/data handlers know how to fill in a response object.

(e.g., how to open a netcdf file, access attributes within that file, and add those attributes to the DAS response object)

• New response handlers (flat, Tab, Info)– Response handlers know how to create the response object

and what needs to happen to get that response object filled in. (e.g., for a DAP request, the DAP response handler knows to create a DAS object and contact the request handler for each container to fill in the DAS object)

• New commands (e.g., hello world example)• Container and Definition storage• Aggregation engines (multiple can be installed)• Methods of returning data (return as nc)• Initialization/Termination callbacks• Exception Handlers (registered with the Exception

Manager)• Reporters (data access statistics, metrics, logging, etc)

12


BES Example

Definitions def as c1, c2

get das for def

Containers c1,file1,csv c2,file2,csv ce,file3,nc c4,file4,ffdef

def

command

Response Handlers das, BESDASResponseHandler help, BESHelpResponseHandler

Request Handlers csv, CSVRequestHandler das, fill_das_func help, fill_help_func

das

das

csv

csv

das

c1

das

c1

c1

c2

c2c2

das

show help

help

help

help

csv

csv

13


BES Built-in Commands

• String commands sent from client to server– show help; (version, process, status, keys)– set container …– show containers;– delete container <container_name>;– delete containers;– define …– show definitions;– delete definition <def_name>;– delete definitions;– set context <name> to <value>;– show context;

14


BES DAP Commands

• DAPCommandModule adds:– get das for <def_name> [return as type];– get dds …– get ddx …– get dods …– show catalog [for “node”]; (info)

• dap-server modules (www,usage,ascii) add:– get info_page …– get ascii …– get html_form …

• Data handlers (nc, ff, csv) don’t add any new commands.

15


Thank you