Upload
datacenters
View
506
Download
3
Embed Size (px)
Citation preview
1 2005 NOAA Data and Information Users' Workshop
1
Comprehensive Large Array-data Stewardship
System (CLASS) http://www.class.noaa.gov
“What is CLASS and How Will It Serve NOAA Clientele?”
12 May 2005Richard G. Reynolds / CLASS Project Manager
2 2005 NOAA Data and Information Users' Workshop
2
Agenda
• Data Centers and Data Stewardship• CLASS System Introduction• Project Management• Accomplishments and Near-term Goals• Budgets and “Campaigns”• Functionality & User Services
3 2005 NOAA Data and Information Users' Workshop
3
NOAA’s National Data Centers
• NOAA’s National Data Centers are major archive, access, and assessment sites maintaining, processing, and distributing environmental and geospatial data. – National Climatic Data Center – WWW.NCDC.NOAA.GOV
• Asheville, NC
– National Coastal Data Development Center – WWW.NCDDC.NOAA.GOV
• Stennis, MS
– National Geophysical Data Center – WWW.NGDC.NOAA.GOV• Boulder, CO
– National Oceanographic Data Center – WWW.NODC.NOAA.GOV• Silver Spring, MD
4 2005 NOAA Data and Information Users' Workshop
4
NOAA’s National Data Centers(Continued)
• These Centers provide long-term stewardship for most of NOAA’s environmental and geospatial data, and a broad range of user services.
• Centers of data are facilities where extensive collections of given environmental parameter(s) are maintained because of individual or institutional research or operational requirements
• They also serve as Agency Record Centers
5 2005 NOAA Data and Information Users' Workshop
5NOAA’s National Data Centersare Environmental Data Stewards
Scientific Data Stewardship is ownership, knowledge, utilization, and
application of the data
CLASS is the Information Technology infrastructure
(hardware and software environment, and tools)
underpinning SDS
Data Rescue preserves and makes available
historical data sets from obsolete media
6 2005 NOAA Data and Information Users' Workshop
6
Scientific Data Stewardship (SDS)
• Observing system performance monitoring– Bias monitoring– Data character and metadata
• Climate Data Records (CDRs) – transition from research to products and services– Sentinel science teams– Blended products
• Provide useful information for national and regional management decisions– Applied climatology– Climate monitoring– Climate forcings and feedbacks
7 2005 NOAA Data and Information Users' Workshop
7
Principles of Environmental Data and Information Stewardship
1. Archive and access to fundamental measurements, products and metadata - CLASS
2. Data archaeology and improved use – CLASS / SDS3. Careful monitoring of observing system performance
for long-term applications - SDS4. Generation of authoritative long-term records through
validation of the calibration process, reprocessing, product generation and the blending of in situ and satellite measurements - SDS
5. Provide state of the environment information for decision makers, and place the current state in its historical context - SDS
8 2005 NOAA Data and Information Users' Workshop
8
CLASS Background, Mission, and
Overview
9 2005 NOAA Data and Information Users' Workshop
9
CLASS BackgroundThe CLASS project derives in part from an effort
by NOAA to centralize its numerous systems for
(satellite) data access.
The goal of this effort is to eliminate the various "stove-pipe” systems and
produce a unified "enterprise” access system for the NOAA environmental data holdings.
10 2005 NOAA Data and Information Users' Workshop
10
CLASS Mission Statement
NOAA's National Data Centers and their world-wide clientele of customers look to CLASS as the sole NOAA IT infrastructure project in which all NOAA’s current and future environmental data sets will reside. CLASS provides permanent, secure storage, and safe, efficient data discovery and access between the Data Centers and the customers.
11 2005 NOAA Data and Information Users' Workshop
11
CLASS Goals
• Give any potential customer access to all NOAA and non-NOAA data
through a single portal.
• Eliminate the need to keep creating “stovepipe” systems for each new type of data,
but, in as much as possible, use already polished portions/modules
of existing legacy systems.
• Describe a cost-effective architecture that can primarily handle large-array data sets,
with the capability of handling smaller ones as well.
12 2005 NOAA Data and Information Users' Workshop
12
CLASS Overview• CLASS is a web-based data archive and distribution
system for NOAA/NESDIS environmental data– Archive … ingest, storage, metadata management, and data
quality assurance– Distribution … access, visualization, and data delivery
• CLASS is an extension of an 1995 operational system … SAA (Satellite Active Archive)– Transition to the CLASS architecture began in 2001– CLASS subsumed SAA as the Operational Archive and Access
System for NOAA in 2004
• CLASS currently supports POES, DMSP, and GOES data sets
• CLASS will support additional campaigns, broader user base, new functionality as it evolves– CLASS concurrently supports ongoing operations and new
requirements implementation
13 2005 NOAA Data and Information Users' Workshop
13
CLASS Overview (Continued)
• Provide one stop shopping and access capability for NOAA and NESDIS environmental data and products
• Provide a common look and feel for accessing NOAA and NESDIS environmental data and products
• Provide an efficient architecture for archiving and distribution of NOAA and NESDIS environmental data and products
• Reduce implementation costs by using reengineering, and evolutionary effort
• Allow NOAA to fulfill its requirements regarding archive, access and distribution of large array data sets
14 2005 NOAA Data and Information Users' Workshop
14
CLASS Overview (Continued)
• Accommodate expanding number of data sources – MetOp, NPP, NPOESS, EOS, In-situ, NexRAD,
GOES-R, etc.
• Data volume is growing exponentially– Anticipating up to 100 Petabytes by 2015
• User volume is also growing exponentially
15 2005 NOAA Data and Information Users' Workshop
15
“CLASS” Synonyms
• Comprehensive Large Array-data Stewardship System (CLASS)
• Archive and Distribution Segment (ADS)
• NPP/NPOESS
• Archive and Access System (AAS) • METOP/IJPS
• Long-term Archive (LTA)• Jason
16 2005 NOAA Data and Information Users' Workshop
16CLASS … as part of the NOAA Observing System Architecture
NOSA … The “End-to-End System”-- Notional Architecture --
Larger System
Observing System
Data Handling System
Human
Environmental Phenomenon
Environmental Parameter
Sensing Element
Sensor
Platform / Station
part of
measures
is a
contains
characterizes
provides data to
Observation Control System
is controlled by
Location
located at
MobileFixed
is
SpaceAirGroundOcean
SpaceAirGroundOcean
Basic Service Requirement
< drives
provides data directly to
Userprovides info to is type of
Stake-holder
has
Operatoroperated by
situated on
Support
supported by
Owner
owned by
Stakeholder Requirement
< drives< drives
Processing Element
RemoteIn Situ
is type
CLASS
17 2005 NOAA Data and Information Users' Workshop
17CLASS Technical Description Functional Flow Diagram
Ingest and Store Data
VisualizationData
Data SetInventory
DataCaches
Orders
Maintain,Monitor,Control
ProcessOrders
AccessData
VisualizeData
Interfacewith Users
Data Productsand Metadata
DataProviders
USERS
CLASSOperators
Archive
CLASS Internet/Intranet
DataProvidersData
Providers
DataProvidersData
Providers
USERSUSERS
USERS
USERS
18 2005 NOAA Data and Information Users' Workshop
18CLASS Technical Description Functional Block Diagram
19 2005 NOAA Data and Information Users' Workshop
19
CLASS Project Management
20 2005 NOAA Data and Information Users' Workshop
20
CLASS Project• Consortium of NOAA Projects and Budget
Lines have been consolidated under the CLASS Project:
– CLASS
– Satellite Active Archive (SAA) – GOES Active Archive (GAA) – Earth Observing System (EOS) – Integrated Joint Polar Program (IJPS) /
MetOp AAS– GOES R-series (GOES-R) AAS– [NOAA Virtual Data System (NVDS) and NOAA E-commerce
System (NeS) moved to NCDC]
21 2005 NOAA Data and Information Users' Workshop
21
CLASS Project Plan
• 10 year “PAC” Plan (Procurement, Acquisition, & Construction)
– Road Map for CLASS Program acquisition
– Budgetary Funding Requirements for all CLASS elements
– Life Cycle Planning document
22 2005 NOAA Data and Information Users' Workshop
22
NOAA DataStewardship Committee
(Tom Karl/NCDC)
CLASS (Richard G. Reynolds/OSD)
NESDIS - Office of Systems Development,
Data Centers, and contractors
Archive Requirements
Working Group (ARWG)
(John Bates/NCDC)
Information Exchange
23 2005 NOAA Data and Information Users' Workshop
23
CLASS Project ManagementNOAA
Data Stewardship Committee
CLASS ProjectRichard G. Reynolds
Charles S. Bryant
CLASS Project Management Team (CPMT)
NGDC Development
Teams (Boulder, CO)
OSD/TMCDevelopment
Team(Fairmont, WV)
OSD/CSCDevelopment
Team (Suitland, MD)
System Integration & Test Team
(Suitland, MD)
OSDPD-CSC Operations
(Suitland, MD)
NCDC-TMC Operations
(Asheville, NC)
Archive RequirementsWorking Group (ARWG)
NESDIS ITATUsers
System Engineering Team (SET)
CLASS Operations Team (COT)
System Administration Team (SAT)
24 2005 NOAA Data and Information Users' Workshop
24
CLASS - Process DocumentationCLASS Master Project
Management Plan
CSC Contract Activity Plan
CLASS Software
Development Guide
CLASS Configuration
Management Plan
Other Activity Plans
Procedures
CLASS Quality
Management Plan
TMC Contract Activity Plan
25 2005 NOAA Data and Information Users' Workshop
25
Project Management (Continued)
• CLASS Project Oversight Groups– Data Stewardship Committee (DSC)– NESDIS Headquarters / AA and DAA– NESDIS Information Technology Architecture Team (ITAT)
- - - - - - - - - - • CLASS Project Management Team (CPMT)
– Management oversight and coordination– Risk management– Project tracking– Overall decision-making body for CLASS
• System Engineering Team (SET)– Technical oversight and coordination– Provides recommendations to CPMT on technical direction– Provides technical input to CCB
26 2005 NOAA Data and Information Users' Workshop
26
Project Management (Continued)
• Software Engineering Process Group (SEPG)– Process definition and improvement– Provides recommendations to CPMT on process definition
• Configuration Control Board (CCB)– Change review and control– Includes CPMT members and SET members
• Systems Administration Team (SAT)– Develop, Maintain, and Supervise Policies and Procedures
regarding configuration and operation of CLASS computing resources
• CLASS Operations Team (COT)– Develop and Implement operational policies and Standard
Operating Procedures for the operational CLASS environments – Includes operations support personnel, system administrators,
database administrators and archive managers
27 2005 NOAA Data and Information Users' Workshop
27
CLASSAccomplishments
28 2005 NOAA Data and Information Users' Workshop
28
CLASS Accomplishments• Completed overall design of CLASS top-level architecture • Prepared Key Project and System Documentation
– CLASS Business Case– System Requirements– Interface Control Documents (ICDs)– Concept of Operations (CONOPS)– Management Plans and Procedures
• Established a project wide risk management program• CSC Development Team Certified at SEI-CMM Level-3• Established the CLASS Operations Team (COT)• Completed … Summary “10-year” CLASS Project Budget
Requirement - $25M/year
29 2005 NOAA Data and Information Users' Workshop
29
CLASS Accomplishments (Continued)
• Delivered baseline systems to Suitland and Asheville– Established Operational Dual-site Configuration
• Established Operational, Integration and Test, and Development environments in Suitland– Completed migration from SAA to CLASS
• CLASS Operational with POES, DMSP, and GOES data sets
• Completed Preliminary & Critical Design Reviews for the IJPS/Metop – Archive and Access Segment
• Coordinated with NPP/NPOESS for defining the IDPS-to-CLASS Interface Control Document – Completed … NPP/NPOESS Campaign Implementation Plan
• Worked with NASA personnel to define initial requirements to archive EOS/MODIS Level-0 data.
30 2005 NOAA Data and Information Users' Workshop
30
CLASS Accomplishments (Continued)
• Completed … Software Release 2.0 / 2.1 / 2.2 – Operational Dual-site configuration -- 02 April 2004– CLASS Operational with POES, DMSP, and GOES data sets, plus
RadarSat (Synthetic Aperture Radar) and SeaWiFS (Ocean Color Product)
• Completed … Software Release 3.0 – 12 July 2004– Provides: Delivery Manifest and Web Enabled Subscription
Management • Completed … Software Release 3.1 – October 2005
– Provides: Ingest Enhancements to support IJPS NOAA data
• Completed … Integration of the Development Teams – CSC-TMC
• Began … SEI/CMMI Certification Process for the total Development Team
31 2005 NOAA Data and Information Users' Workshop
31
FY05 CLASS Goals & Plans• Prepare System security Certification & Accreditation (C&A)
– CLASS currently encompassed by the Satellite Active Archive C&A– Draft stand-alone CLASS C&A prepared (March 2005)– Final to be completed to support relocation to new NSOF Building in
September 2005
• Update the CLASS Long-term Architecture– Held 2-day architecture session confirming/upgrading the long-term
architectural plan for CLASS– FY2005 CLASS Long-term Architecture Plan to be released in September
2005
• Achieve Hardware/Software Commonality among all Nodes
• Project/Development Teams SEI-CMMI Certified– Initial review completed (January 2005)– Final assessment to be conducted in May 2005
• Relocate Suitland Node to Boulder (NGDC)
32 2005 NOAA Data and Information Users' Workshop
32
FY05 CLASS Goals & Plans (Continued)
• METOP-1 Pre-Launch Testing and Operational Readiness (Completed March 2005)– Capability contained in Release-3.2– Awaiting data flows from the Spacecraft for final functional verification
• NPP “Campaign” Development and Testing • EOS-MODIS “Campaign” Development and Testing• Establish an interface with NeS• Establish an interface with NMMR• Metadata “Campaign” development continues• Geospatial Capability development begins• Jason/OSTM ”Campaign” development begins• Establish a Development Environment at TMC/Fairmont
----------------• Operations Continue
33 2005 NOAA Data and Information Users' Workshop
33
FY05 Hardware/Software Plans• System SAN Capacity Upgrade (Completed January
2005) • Additional disk space at both CLASS operational sites• Data Direct Networks … 56 Tbytes (expandable to 302 Tbytes)
• CLASS Release 3.2 (Completed April 2005)• Support for Metop-1 data / readiness for IJPS End-to-End test • Subscription for GOES data w/ separate GVAR data ‘families;”
GOES-N• Upgrade all systems to AIX 5.1/5.2 (64-bit structures)
• CLASS Release 3.3 (Scheduled for July 2005)• McIdas-less ingest • Upgrades to the Help Pages/Static Pages • Map server upgrades; Point Searches• CLASS-NMMR Interface • Security enhancements, including capability to deliver data
encrypted• UTC Time utilization
34 2005 NOAA Data and Information Users' Workshop
34
FY06 CLASS Goals & Plans
• METOP-1 Operational Activation• NPP “Campaign” End-to-End Testing,
Compatibility Testing, and Operational Readiness • Metadata “Campaign” development continues• Geospatial Capability development continues• Jason/OSTM “Campaign” Testing• Data QA/QC “Campaign” begins• Reprocessing “Campaign” begins
35 2005 NOAA Data and Information Users' Workshop
35
FY06 Hardware/Software Plans• System Storage Capacity Upgrade
– Scheduled for September 2006– LTO-2 to LTO-3 Migration
• CLASS Release 4.0 (Scheduled for October 2005)• Basic NPP Support • Final IJPS/Metop Pre-launch Release • CLASS – NeS Interface • CLASS – NMMR Interface
• CLASS Release 4.1 (Scheduled for February 2006)• NPP Readiness for NCT-#3 • Initial Data Delivery Upgrades
• CLASS Release 4.2 (Scheduled for August 2006)• NPP Readiness for NCT-#4
• CLASS Release 4.3 (Scheduled for October 2006)• NPP Pre-Launch Release• Complete Data Delivery Upgrades
36 2005 NOAA Data and Information Users' Workshop
36
CLASS Statistics(Average Last 12 Months)
• Ingest – 71 GB/Day … 26 TB/Year
• Ingest – 860,000 Data Sets/Year
• Distribution (On-line & Subscriptions) - 44 TB/Year …. 3.63 TB/Month
• Distribution (On-line & Subscriptions) - 3,170,000 Data Sets/Year … 263,888 Data Sets/Month
37 2005 NOAA Data and Information Users' Workshop
37
CLASS BUDGETSand “Campaigns”
Budget numbers are shown for the purpose of establishing a reference for relative complexity of a
requirement and level of completeness, and do not represent NOAA, Department of Commence, or
The President’s position regarding specific Congressional Budget Requests.
38 2005 NOAA Data and Information Users' Workshop
38
Major Core-CLASS Project“Functional Campaigns”
• “Core CLASS” Baseline System Development, Expansion, & Evolution
–FY04-FY16 $94M•Metadata “Campaign”
–FY04-FY14 $12M•Reprocessing “Campaign”
–FY06-FY16 $35M•QA/QC
–FY06 …. $2M/year--------------------
•System O&M–FY04 ($2M)-FY14($10M) $11M/year thereafter
39 2005 NOAA Data and Information Users' Workshop
39
General Core-CLASS Activities
• Support for CLASS Development Activities– Architecture Design– System Design– Software Development and Integration– Quality Assurance– System Engineering Team Activities
• Support for CLASS Operations and Maintenance– Contractor and Government Project Support– Licenses– Hardware Maintenance– Hardware Refresh
40 2005 NOAA Data and Information Users' Workshop
40General Core-CLASS Activities (Continued)
• Metadata– Assure FGDC Compliance – Establish Rich Metadata Standards– Develop Integrated data and metadata access
• Geospatial– Capability development and implementation– Integration of all CLASS data
• Web– Integrate SABR into CLASS– Design and implement web mapping system – Enhancements to support Data Mining capability
• Data Mining– Architecture design and implementation– Application design and implementation
• Reprocessing– Architecture design and implementation– Establish operational processes and standards
41 2005 NOAA Data and Information Users' Workshop
41Major CLASS Project“Data Campaigns”
•Metop-1–FY01-FY07 $6.5M
•NPP –FY04-FY11 $15.8M
•EOS-MODIS –FY04-FY10 $16.7M
•NDE –FY06-FY12 $1.5M
•NEXRAD–FY06-FY09 $8.1M
• Insitu –FY06-FY14 $8.0M
•EOS Retrospective–FY07-FY13 $8.2M
•GOES-R–FY07-FY14 $41M
•Metop-2–FY08-FY14 $5.0M
•NPOESS-C1–FY08-FY12 $11.1M
42 2005 NOAA Data and Information Users' Workshop
42
General “Data Campaign” Requirements
• Requirements Definition and ICDs
• Data and Products Ingest
• Storage, Processing, and Communications Upgrades
• Metadata extensions
• Catalogue extensions
• Visualization extensions
• Reprocessing extensions
43 2005 NOAA Data and Information Users' Workshop
43
CLASS Budgets• FY01 $1.995M • FY02 $3.599M• FY03 $2.881M
-----------• FY04 $10.5M• FY05 $14.6M• FY06 $11.9M • FY07 $18.0M *• FY08 $20.8M *• FY09 $25.5M *• FY10+ $29.3M/yr
*– * FY07 PA&E Recommendations
– FY05 Funding Sources
– CLASS $6.6M– GAA $2.5M– EOS $3.0M– SAA $1.5M– OSD $1.0M **
** POES Ground Systems PAC – METOP/IJPS Archive & Access $200K
CLASS Hardware $800K
44 2005 NOAA Data and Information Users' Workshop
44
FY06-07 CLASS Budgets• FY07 $20.0M
• CLASS $15.1M
• SAA $1.5M
• EOS $1.0M
• GOES-R $2.0M
• OSD $0.4M
FY06 $11.9M
CLASS $6.6M
SAA $1.5M
EOS $3.0M
OSD $0.8M
45 2005 NOAA Data and Information Users' Workshop
45
CLASSFunctionality and
User Services
46 2005 NOAA Data and Information Users' Workshop
46
Core Functionality
• Data and metadata ingest, archive, and replication (NARA compliant)– Delivery Manifests– Data Quality– Companion Files– Metadata Management
• User Data Discovery– Access, Search, and Visualization
47 2005 NOAA Data and Information Users' Workshop
47
Core Functionality (Continued)
• User Access– Web– Single Sign-on/On-line– Subscription– Machine-to-Machine/Bulk Order
• User Search– Inventory– Temporal /Spatial – Data Type– Metadata Characteristics
• User Visualization– Imagery Chips– Geospatial Navigation– Overlays
48 2005 NOAA Data and Information Users' Workshop
48
Core Functionality (Continued)
• Data Delivery System– Improved use of LTO tape robotic system capabilities– Consolidate multiple order delivery systems (Online,
Subscription, Bulk Order)– Improved cache management– Delivery Options
• Physical media creation and shipment• Electronic delivery
– E-Commerce (NESDIS E-commerce System – NeS)• Pricing, Payment receipt, Payment processing
– User Profiles
• Data Denial
49 2005 NOAA Data and Information Users' Workshop
49
Core Functionality (Continued)
• Data Delivery to the User– Format options– Push– Pull– Physical Media – Delivery Notifications
• User Helps
• New Requirements … via the ARWG!
50 2005 NOAA Data and Information Users' Workshop
50
THANK YOU!