ORION CI Conceptual Architecture Team Progress Report

28 Mar 2006

ORION CI

ORION CI Conceptual Architecture Team

Progress Report

AgendaContext

Presentation (Options)Discussion

28 Mar 2006

ORION CI

Context

28 Mar 2006

ORION CI

ORION Context

28 Mar 2006

ORION CI

28 Mar 2006

ORION CI

Expanding Role of Environmental Observing

• From individual expedition to collaborative observation. – Long-term persistent presence, continuous 24/7 time series– Increasing numbers and diversity of sensors– Multiple, complex data streams– Many more types of users and applications– Ability to control instruments, observing systems, and large-scale

experiments.

• Ubiquitous data requiring greater differentiated expertise.– From kilobits to terabytes in a matter of hours.– Collaborative discovery by groups of researchers with different

vocabularies and expertise looking at complex data streams.

• A need for cooperating networks of data/processing/analysis systems that are different than in the past.

28 Mar 2006

ORION CI

Team Members

Matthew Arrott (UCSD, Chair)Alan Chave (WHOI)John Graybeal (MBARI)Eric Guillemot (NEPTUNE Canada)Ingolf Krüger (UCSD)Benoît Pirenne (NEPTUNE Canada)

28 Mar 2006

ORION CI

Team Mandate

• Develop a set of information system capabilities.• Develop conceptual level design options to

support the Ocean Observatories Initiative (OOI) systems operations and RFP development.

• Develop a CI Work Breakdown Structure (WBS).• Create resource estimates for each line in the

WBS to prepare a Rough Order of Magnitude (ROM) cost and timeline for the project.

28 Mar 2006

ORION CI

What do you want us to talk about?

28 Mar 2006

ORION CI

Presentation Options

• How will using ORION be different from the way science is done today?

• What will CI do for us as individuals and as a community?

• How has CI been implemented in other environments?

• What is the current architectural concept?• How can we be sure this is a viable,

implementable infrastructure?• How did you go about putting this together?

28 Mar 2006

ORION CI

How will using ORION be different from the way science is done today?

28 Mar 2006

ORION CI

Getting A Book: The Way It Used To Be

QuickTime™ and aTIFF (LZW) decompressorare needed to see this picture. QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

I need to special order a book.What book do you want?What book do you want?

It’s by Rachel Carson, Silent Spring.Yes, we can order that. It’s 23.95.Yes, we can order that. It’s 23.95.

When will you have it?Maybe Tuesday, but if it’s not in stock, not for a month.Maybe Tuesday, but if it’s not in stock, not for a month.

Can you send it to me?No, you’ll need to pick it up.No, you’ll need to pick it up.

OK, can I pay by phone?Yes, we’ll need your credit card and address.Yes, we’ll need your credit card and address.

OK, here’s my card info and address…OK, we’ll call you when it’s in.OK, we’ll call you when it’s in.

28 Mar 2006

ORION CI

Getting A Book: The Way It Is Now

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

28 Mar 2006

ORION CI

Doing Science: The Way It Has Been

Hey Francisco, you still running that mooring in Monterey?Yeah, John, what do you need?Yeah, John, what do you need?

Is it collecting current data?Yeah, you looking for profiles?Yeah, you looking for profiles?

Yup, are you doing that with an ADCP?Yes, we’ll probably start profiling soon, too.Yes, we’ll probably start profiling soon, too.

No, don’t need that. How do I get the data?You want the raw data or QCd? Real-time?.You want the raw data or QCd? Real-time?.

Oh, the QC’d, delayed mode data, definitely, last 3 years worth.That’d be off our dods site, here’s the address..That’d be off our dods site, here’s the address..

Documentation come with that?We got some, I’ll have to get one of my techs to send it to you…We got some, I’ll have to get one of my techs to send it to you…

QuickTime™ and aTIFF (Uncompressed) decompressor




28 Mar 2006

ORION CI

Doing Science: The Way It Is and Will Be



QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

28 Mar 2006

ORION CI

Doing Science: Another Option



28 Mar 2006

ORION CI

Or…



28 Mar 2006

ORION CI

Interfacing with ORION

Working with the system will be at least as easy as it is now.

• Building an instrument?– Instrument interfaces are defined– Libraries and adapters are provided to help you.– Core instruments will be taken care of.

• Discovering data?– Lots of options, most of which actually work.– Just “get the data” (as you want it) once you find it.– Not just ORION data, data from lots of places.

• Running an observatory?– Access to system status.– Ability to coordinate the entire system as needed.

28 Mar 2006

ORION CI

Return to menu

Forward to science scenarios

28 Mar 2006

ORION CI

A Science Scenario

• What we have here– An example for illustration– Only one of many applicable science scenarios

• The example– “Developing a complex instrument for ORION”– Actually a platform, but we’ll call it an instrument here– Covers most of the lifecycle of the instrument

• And furthermore…– Accessing and using data from the instrument

28 Mar 2006

ORION CI

Developing a Complex Instrument for ORION

• Platform carrying multiple sensors• Commandable• Not always connected• Multiple non-trivial data sets• Data transport varies

– may be stored and copied out of the instrument later– may be streamed in (near) real time

28 Mar 2006

ORION CI

Steps Along The Way



28 Mar 2006

ORION CI

Develop: Create Instrument



What does my instrument have

to do?

In any case, I have to get money (may involve research).

Finally, I’m building the instrument.

ORION just sent me instructions on testing

my instrument.

What does ORION say my instrument

must do?

Can I really justify building this? (May call for research.)

ORION IO may give me funds (e.g., for

core systems)!

I guess I’d better test it.

Darn, need to fix that./

Yay, it’s ready!

28 Mar 2006

ORION CI

Commission: Test and Validate Instrument



Does system know about my

instrument?

Does my instrument talk to system correctly?

Put instrument in the water and

hook it up.

Make sure the instrument is “behaving.”

Make sure my instrument is

working correctly.Tell system about

my instrument.Record the transaction.

28 Mar 2006

ORION CI

Publish: Announce Availability of Instrument



Declare what is the final version of my instrument

Group this with the other instruments

of similar products.

Tell everyone this instrument

is available.

Describe my instrument (using ORION forms).

Make sure this description is valid

and complete.

Enter my instrument into the

approved list.

28 Mar 2006

ORION CI

Discover: Find the Instrument



What kind of an instrument am I

looking for?

Son of a gun, I didn’t know they have a forbitz!

Let’s look over here, this list

looks relevant.

Describe the details I’m looking for.

What’s the best way to find it?

(How & where?)

OK, I’ll browse/ search using this

interface.

Start the search already.

Darn, need to try again./

Yay, found it!

28 Mar 2006

ORION CI

Acquire: Get OK to Command Instrument



Please let me control the frobnitz

(private) instrument.

“Authorizing access,

please wait.”

“Instrument documentation is

in the mail.”

“Can user control this

thing?”

Never mind, I didn’t want to use that

instrument anyway.

Your instrument access list includes:

frobnitz ID#2365

Hi, Pat? Jan

here.

Of course.

Do you agree to call the provider now and

give credit later?

“Sure, if user calls me and credits me.”

28 Mar 2006

ORION CI

Use: Command Instrument



This is an instrument I can

command.

I’d better cite that scientist

like I agreed to.

Thank you for giving me credit on your paper.

“set sample rate on ‘instrument 23493’ =10

Hz for 3600 secs”

Changing sample rate to

10 Hz for 1 hour.

I’m getting the data back, I can start my

analysis now.

Do I need to do more?

OK, don’t need to command it

any more.

28 Mar 2006

ORION CI

Govern: Restrict Instrument Control



Is it still OK to use the instrument

(so much)?

Define access rights and policies

for instrument

“Is it, or is it not, OK to use instrument

right now?

Is it OK for this user to use the

instrument?

28 Mar 2006

ORION CI

What About the Data?

Return to menu

Forward to data access scenarios

28 Mar 2006

ORION CI

Publish: Announce Data Available



Decide on the final format for data stream(s).

Group this data with the other

similar products.

Tell everyone this data product

is available.

Describe data stream (using

ORION forms).

Make sure this description is valid

and complete.

Enter my stream into the list of

available data.

28 Mar 2006

ORION CI

Discover: Find the Data



What kind of an data am I looking

for?

Son of a gun, I didn’t know they

have carbon data!

Let’s look over here, this data set relevant.

Describe the data sets I want

in detail.

What’s the best way to find it?

(How & where?)

OK, I’ll browse/ search using this

interface.

OK, it knows me and is letting me see the data set.

Start the search already.

Darn, need to try again./

Yay, found it!

28 Mar 2006

ORION CI

Acquire: Obtain the Data



Please give me access to the

frobnitz data stream.Yes.“Authorizing

access, please wait.”

Oh look, the download button’s

enabled.

Can user have

data?

Do you accept this data is not quality

controlled?

Oh, I didn’t want the data if it isn’t

Quality Controlled.

Click on link, watch data stream in.

(done)Sure, if user

agrees to disclaimer

28 Mar 2006

ORION CI



Use: Modify Data, Present ResultsThis is just data.

On we go.

I’d better cite that scientist

like I agreed to.

Thank you for giving me credit on your paper.

I’m getting data, my software’s running, I detected upwelling! I write my paper.

Can I turn this stream

off?

Yup, don’t need to access it any

more.

28 Mar 2006

ORION CI

Return to menu

28 Mar 2006

ORION CI

What will CI do for us as individuals and as a community?

28 Mar 2006

ORION CI

For the individual

• Provide workflow and resource management tools to automate experimental design and execution

• Provide a secure work environment that automatically mediates conflicts

• Generalize and automate query and publish/subscribe processes for retrospective and real-time data

28 Mar 2006

ORION CI

28 Mar 2006

ORION CI

• Dynamic Data Driven Assimilation System (DDDAS) use case

• Workflow binds diverse fixed and mobile instruments to assimilation model and data repository on shore

• Resource conflict mediation is automated

28 Mar 2006

ORION CI

28 Mar 2006

ORION CI

• Distributed remote multi-mission laboratories distributed on an RCO

• Resource intensive, shared use operations require automated brokering

• Quality of service policy constraints are important

28 Mar 2006

ORION CI

For the community

• Facilitates collaborative experimentation and communication

• Automated integration of ORION policies and external obligations

• Ability to archive data with reliable discovery services for future use

28 Mar 2006

ORION CI

28 Mar 2006

ORION CI

• Automated tracking and coordination of the state of observatory resources

• External resources can be integrated with observatory assets and operations

• Science use case becomes the observatory use case

28 Mar 2006

ORION CI

Return to menu

28 Mar 2006

ORION CI

How has CI been implemented in other environments?

28 Mar 2006

ORION CI

CI: Evolved from Grid Computing

• Grid Computing is a term for loosely coupled distributed computing across a diverse community of resources, owners

• Many Grid Computing efforts and products– Computational Grid (TeraGrid, Open Science Grid)

• Decomposition of large problems into many small atomic tasks– Data Grid (Storage Resource Broker, OGSA-DAI)

• Federation & cataloguing of distributed data repositories– Service Grid (On-Demand and Location-based service models—IBM,

Microsoft)• Generalization of computational and data grids as “Service” patterns• Emergence of Service Oriented Architectures

– Autonomic Grid (DMTF, WSDM from HP, IBM, CA, Oracle, Opsware)• Autonomous resource management, load balancing and fault

detection isolation and recovery applied to networks of coupled resources

• Grid computing contributes to overall cyberinfrastructure

28 Mar 2006

ORION CI

ORION CI Building on Other CI Projects

• GriPhyN, Atlas, Ligo, CMS – Data distribution and shared computational grid

• NVO (National Virtual Observatory) – Community data model, and shared data repositories and

applications

• BIRN (Biomedical Information Resource Network)– Federated data repositories of disparate data models into a

common meta-catalog– Resolution of disparate data models through mediation

• GEON (GEOsciences Network)– Extends data mediation model with ontologies

• TeleScience and NEES– Developed real-time control of remote instrumentation and the

coupling of remote physical and simulated systems

28 Mar 2006

ORION CI

Environmental Cyberinfrastructures like ORION

• What other projects are most like this?– LEAD (Linked Environments for Atmospheric Discovery)– NEON (National Ecological Observatory Network)

• What makes them similar?– Significant increase in the number and diversity of

instruments and data products – Real-time stream processing and inquiry– Interaction with the sensing environment– Semantic heterogeneity (mixed vocabularies)

28 Mar 2006

ORION CI

Return to menu

28 Mar 2006

ORION CI

What is the current architectural concept?

28 Mar 2006

ORION CI

External Architecture

28 Mar 2006

ORION CI

Lifecycle of ORION Activities (v.B)

28 Mar 2006

ORION CI

Internal Architecture

28 Mar 2006

ORION CI

System Components

28 Mar 2006

ORION CI

Deployment Architecture

28 Mar 2006

ORION CI

Enterprise Service Bus

28 Mar 2006

ORION CI

Return to menu

Go to Scenarios

28 Mar 2006

ORION CI

Viable Architecture

How can we be sure this is a viable, implementable architecture?

28 Mar 2006

ORION CI

Reasons for Confidence

• Open and transparent design process• Ongoing review by external and internal experts and

stakeholders • Iterative functional deployments reflecting use priorities

and system risks • A rapidly maturing field

– Major subsystems have multiple implementations in existence – Science efforts on which to build (BIRN, GEON)– Many sophisticated commercial collaboration architectures,

particularly in web domain (Flikr, eBay, MySpace)– Pioneering efforts in ocean cyberinfrastructure (LOOKING on

research side, NEPTUNE Canada in operation)

• Innovation will be focused on increased usability, effectiveness and reliability

28 Mar 2006

ORION CI

Concerns

• Some technical areas deserve extra attention– Incorporation of streaming data, a new data type, into

scientific process– Specification and adoption of instrument services– Effective operation of systems of systems that will

scale with the anticipated growth of ORION– Interfacing with external organizations and resources

• Implementing Organizations take this on– Evaluation and response to the most challenging tasks– Collaborative development encouraged to maximize

success

28 Mar 2006

ORION CI

Return to menu

28 Mar 2006

ORION CI

How did you go about putting this together?

28 Mar 2006

ORION CI

Methodology & Timeline

• DoDAF model

• Development Process– Requirements– Operational activities– Organizational and operational entities

• Dictionary– Domain model– System components– WBS– Use domain experts for final costing

• Timeline– May 15, 2006

28 Mar 2006

ORION CI

Characterize the System

• Identify the main areas of Research Activity, Internal/External lines of Operations, and/or Support Functions.

• Describe the operations that are normally conducted in the course of achieving an experiment or research goal. – Description includes: Capabilities, operational activities,

input/output flows between activities, input/output flows from/to activities outside the scope of the architecture

• Characterize the command, governance, and business relationships among the human, organizations, and organizational type roles that are the key participants/stakeholders in the architecture.

• Depict the operational nodes, including organizations, with need lines between those nodes that indicate a need to exchange information.

28 Mar 2006

ORION CI

Entity Activities

28 Mar 2006

ORION CI

Lifecycle of ORION Activities

28 Mar 2006

ORION CI

Example of Activity

28 Mar 2006

ORION CI

Internal Interaction Model

28 Mar 2006

ORION CI

Reference: NOSA Logical Data Model

28 Mar 2006

ORION CI

Return to menu

28 Mar 2006

ORION CI

Discussion: Questions, Comments, Suggestions

Documents

ORION CI Conceptual Architecture Team Progress Report