The Data Center of the 21st Century

John Bates NOAA

National Climatic Data Center

The Data Center of the 21st CenturyPrimary Functions – The 4As

• Acquire

• Archive

• Access

• Assess

A Reference Model for an Open Archival Information System (OAIS)

Within a Data Center

• Producer – provides the information to be preserved• Management – sets overall policy as one component within a

broader policy domain• Consumer – interacts with the archive system to find and

acquire information; a designated community is the primary set of consumers for a specific discipline

Data Center - External Interactions• Human to human

interactions are the essence of a data center

• Negotiations are required to achieve submission agreements with data producers and use case scenarios with consumers

• Technology then enables the implementation of the agreements and scenarios

Data Center – Management Interactions

• Provides the primary source of funding and provides guidelines for resource utilization

• Conducts regular performance reviews• Determines pricing and distribution

policies• Participates in conflict resolution involving

producers, consumers and internal administration

NOAA Observing System Council

Data Stewardship Committee

CLASS

Data Centers &Related Offices

Archive Requirements

Working Group

Information Exchange

Current NOAA Management Model

Data Center – Data Provider InteractionsCurrently No Data Submission Agreements

• Data Providers produce products and give them to the archive without complete metadata or a clear purpose (no designated communities).

• Archives end up with many products that they do not understand and can not provide quality service or science stewardship for.

• The problems increase with time as the number of products increases and they age.

Data Providers

Products

Archive Requirements

Science Requirements

No Effective Partnerships Between Data Providers and Archives

NOAA’s New Submission Agreement Process

• Data Providers produce products supported by validated science requirements and request that those products be archived and made available to a designated community by the archive.

• Submission agreements are negotiated for each product.

• Those submission agreements lead to allocated requirements for archive development. Hopefully, those requirements overlap, so that the development effort for each new product decreases with time.

Data Providers

Products

Archive Requirements

Submission Agreements

Science Requirements

Data Center Functional Entities

Lack of Metadata is the Achilles Heel of Long-Term Archives

• Representation information is required to transform digital bits into information

• Rich representation information can only be provided by the data producer and is required

• Currently producers are not supplying this, leading to a loss of how to interpret data objects

Required Metadata for Information Objects

Data Center – Consumer Interactions

• Consumer expectations are rapidly changing

• Consumers want to solve problems across a wide range of disciplines

• Consumers want, and expect, easy data discovery, 24x7 access, and metadata

• The IT revolution will continue to shape consumers expectations

Consumer Interest AreasReport for Nov 6, 2002 through Nov 6, 2003 Customer Type No. of Datasets Type of Data Est. Orders Agriculture 1 Image (1) Business 29 Images (20) Consultant (non-meteorologist) 11 Images (6) Consultant - meteorologist 332 Images (60) Contractor 23 Images (12) DoD 108 Images (15) Engineer 15 Images (10) Foreign 28 ISCCP (1) Foreign Research 147,000 DMSP - TDR/SDR (1) 7 Images (2) Foreign University 45 Images (8) Individual 96 Images (40) Insurance 1 Image (1) Legal 19 Images (10) Manufacturer 0 Congress 20 Images (6) Mining 0 NOAA 1000 McIDAS (30) Other Federal gov 5840 ISCCP (1) 22 Images (12) Research (US) 6 Images (1) State/local gov 17 McIDAS (4) 4 Images (3) Transportation 54 Images (7) University (US) 29 McIDAS (2) 10 Images (4) Media 100 Images (40) Utilities 0

Consumer Use Case Scenarios

Case Scenarios by Number of Users

0

10

20

30

40

50

60

70

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58

Time (BP)

Nu

mb

er o

f U

sers

Event

Steady

Episodic

Long-Term

Case scenarios by volume

0

20

40

60

80

100

120

140

0 10 20 30 40 50 60

Time (BP)

Daily

Ord

er V

olum

e

Event

Steady

Episodic

Long-Term

Conclusions• Data Centers now have a common Open

Archival Information System (OAIS) reference model

• Data Center personnel provide the information broker function between data producers and consumers

• Data Producers must provide rich representation information or digital data object information will be lost forever

• Data Centers must respond to increasing consumer demands for more information