Building a Corporate Metadata Repository

Building a Corporate Metadata Repository

Past

Present

Future

CMRCMR

Sam HighsmithSam HighsmithPrincipal ResearcherPrincipal ResearcherStatistical Research DivisionStatistical Research DivisionU.S. Census BureauU.S. Census [email protected]@census.gov

Agenda

• The Past

• The Present

• Requirements for 2000

• Questions

What is the Corporate Metadata Repository?

• Electronic card catalog• Developed in conjunction with work from

Sweden, Canada, Australia, and UN/ECE Metadata Workshop

• Implements Business Data Model for Surveys and Censuses

• Provides a process for registering data set, variables, and documents for access and reuse

Past Milestones

• Built a Metadata Repository model • Built a Web enabled Prototype• Formal Agreements with other Bureau users• DADS adopted the Metadata Repository model• Developed a Transition to Production plan• Contract with Oracle to build a CMR • Economic Directorate Pilot Application

Metadata Repository Model

• Business Data Model– CASRO, SRD, consultant, Reinvention Lab– Survey and Census business model

• Data Element Registry– ISO 11179

• Built BDM and DER, then merged to CMR

• Modeled with ERD tool

Milestones PROTOTYPE

• Interface to Multiple Statistical Information Systems - DADS, FERRET, PCDOCS

• Table of Contents organization

• Search capabilities

• Oracle generated database for repository

• Completely Web based

Milestones

• Formal Memorandum of Understanding with major Bureau directorates

Milestones

• DADS adopted the Metadata Repository model• Implemented by Oracle Consulting for AFF

– Dissemination tool for 2000 Census data

– Accessible from www.census.gov

• DADS American Fact Finder is completely metadata driven

• !!!!!!!!!!!!

Milestones

• Developed a Transition to Production plan• Metadata support staff

• metadata providers in directorates

• Still waiting…...

Milestones

• Built a Metadata Repository model • Built a Web enabled Prototype• Formal Agreements with other Bureau users• DADS adopted the Metadata Repository model• Developed a Transition to Production plan• Contracted with Oracle to Build CMR

Review of CMR Implementation StrategyReview of CMR Implementation Strategy

Current Business Process does not include an Integrated Current Business Process does not include an Integrated

Metadata Business ProcessMetadata Business Process

Review of CMR Implementation StrategyReview of CMR Implementation Strategy

Current Business Process does not include an Integrated Current Business Process does not include an Integrated

Metadata Business ProcessMetadata Business Process internallydevelopedsystems

customizedcommercial

systems

CASES

variety ofprogramming

languages

individualtool of choice

internallydevelopedsystems

customizedcommercial

systems

CASES

variety ofprogramming

languages


CATICAPIMailPAPIOCRICR

CADECSAQTDE

PFIRS

CATICAPIMailPAPIOCRICR

CADECSAQTDE

PFIRS


SAS

DEVSURV

COBOLFORTRANDECForms

StEPSECON DW



SAS

DEVSURV

COBOLFORTRANDECForms

StEPSECON DW


AFF

CENSAS

FERRET

Econ DW

CD-ROM

Internet

AFF

CENSAS

FERRET

Econ DW

CD-ROM

Internet

Census 2000 ACS

DemographicSurveys

Econ Census

Econ Surveys

Design Collect Process Share

Review of CMR Implementation StrategyReview of CMR Implementation StrategyBenefits of a CMRBenefits of a CMR

Review of CMR Implementation StrategyReview of CMR Implementation StrategyBenefits of a CMRBenefits of a CMR

A C T I V E

CorporateMetadata

Repository(CMR)

-------------------Statistical MD

Geographic MDTechnical MD-------------------

Business Rules-------------------

Product Layouts

A C T I V E

CorporateMetadata

Repository(CMR)

-------------------Statistical MD

Geographic MDTechnical MD-------------------

Business Rules-------------------

Product Layouts

Launch Dissemination Systems

Support One View to Internet Users

Support “Virtual Statistical Agency”and sharing data/MD across web sites

Support Generation of All Kindsof Products, including Reimbursables

Support sophisticated analysisof BOC data (e.g. time series and crossdataset comparisons)

“Faster to Market” through reuse

Proposed Technical/Software Architecture Four Ways an Application Can Use CMR MetadataProposed Technical/Software Architecture

Four Ways an Application Can Use CMR Metadata

Tightly Coupled with CMR

Loosely Coupled with CMR

1. Application written against CMR - uses it directly for metadata access and maintenance.

2. Application uses same extensible CMR physical model - can replicate metadata from CMR. (e.g. DADS/AFF,ECON,LEHD)

3. Application communicates with CMR through an API to exchange metadata. (e.g. FERRET)

4. Application communicates with CMR using a standard XML-based metadata interchange. (future - e.g. FEDSTATS,BLS...)

CMR Meta-ModelCMR Meta-Model

• Meets ISO/IEC 11179 Data Element Registry standards

• Meets FGDC Content Standard for Digital Geospatial Metadata (CSDGM)

• Can support emerging metadata standards

• Extensible

• Shareable

• COTS where possible

Economic Directorate Pilot

• Pilot versus Prototype

• Focus on two stages of the information collection life cycle of the ASM

• Added functionality to two existing METADATA applications– Dissemination and Refics

• Used JAD, RAD, code generation• The “workers” jumped on board!• Found multiple existing metadata applications

A METADATA EXAMPLE FROM

The Annual Survey Of Manufactures

MetadataReuse

Computerized Self Administered Questionnaire

MetadataReuse

Editing Annual Survey of Manufacturers

BALANCING SIMPLE:B1:SW=WW+OW YYN 00.00 00.00 N N B2:TE=PW+OE YYN 00.00 00.00 N N B3:PW=(PW1+PW2+PW3+PW4)/4 YNNNN 00.05 00.05 00.50 N N TTHVB4:LC=LE+VP NYY 00.05 00.00 N N B5:CM=CP+CR+CF+EE+CW YNNNNN 00.05 00.05 00.50 N N NTTHVB6:TIB=WIB+MIB+FIB YNNN 00.25 00.25 00.50 N N NTTHVB7:TIE=WIE+MIE+FIE YNNN 00.25 00.25 00.50 N N NTTHVBALANCING T-ADJUSTMENTS:B03A 0,4.1,4.1,4.1,4.1B03B 0,3.2,3.2,3.2,3.2B05A 0,1.0,0.8,0.5,0.5,0.9 0,4.4,4.6,4.6,4.5,4.6B05B 0,1.0,0.8,0.5,0.5,0.9 0,2.0,2.0,2.0,2.0,2.0B06A 0,4.3,4.3,4.3B06B 0,2.0,2.0,2.0B07A 0,4.3,4.3,4.3B07B 0,2.0,2.0,2.0BALANCING FENCE-CONSTANTS:0 0 11 0.9 1.12 0.75 1.253 0.5 1.54 0.2 55 0.05 106 0.0 20

MetadataReuse

ASM IRACS

MetadataReuse

MetadataReuse

A FewViews of the Pilot Application

Year 2000 Planning

What are we building this year?

Hint: A production system!Hint: A production system!

Metadata Portal SiteMetadata Portal Site• Web-enabled

• Access to all CMR tools

• Promote knowledge management

• Secure access (role-based & by individual)

• Support Collaboration and Personalization:

• BOC Public Access to Tools and Folders

• Collaboration areas for BOC Projects and Organizations

• Personalization

– Customize own CMR Web Site to short-cut access and add personal work

– Publish personal work, if desired

Metadata ManagementMetadata Management

DEPARTMENT

EMPLOYEES SALARY

DEPT

EMP SAL

Meta-ModelMeta-ModelManagementManagement

RepositoryRepositoryManagementManagement

Architectural Architectural SolutionsSolutionsManagementManagement

AdminTools

Bulk Load& UnloadServices

BrowsingTools

MetadataExchangeServices Publish on Web

Search structured & unstructured MDDrill-through to actual dataLaunch Applications

Security

NetworkSystemRepositoryMetadataFine Grain Ownership

Object-level VersioningMulti-path VersioningObject-level Check-in/Check-out

ConfigurationManagement

V1.0

V1.1 V2.0

Synchronize Repositories Master-Slave Relationships Same Meta-ModelMeta-Model BridgesXML-based Exchange

Formatted/Delimited ASCII FileXML-based FilePublish in many different standard formats

Create, Read, Update, DeleteArchiveBackup/Recovery

Object Layer and Open APIObject Layer and Open API

• Object layer over the CMR physical repository tables– The underlying meta-model could change with minimal

impact to the CMR tools and interfaces accessing the object layer.

• Provide fine-grain, role-based metadata security for access by CMR tools

• Publish an object oriented Open API for developers• Use wizards and generation tools as much as possible• Use meta-model generation tools (preferrable)

Metadata ManagementMetadata Management

• Web-enabled• Meta-model generated application• COTS repository administration• Supports CRUD manipulation of metadata• Supports copying and versioning metadata objects• Supports storage and retrieval of unstructured

metadata• Implements business rules, audit trails, and metadata

status for simple work flow

The CMR Metadata Management tool will enable the BOC business metadata owners and stewards to maintain their own metadata.

Oracle Designer Generated HTML/JS Applications

LogicalModels

MiddleTier

Deployment

ServerTier

Deployment

PhysicalModels

ClientTier

Deployment

FunctionalRequirements

UseCases

UMLObjectModel

ServerModel

WebModules

CMRRepository

TAPI(PL/SQL)

PL/SQLgeneratingHTML & JSApplication

Code

OASEnvironmen

tw/

PL/SQL Cartridge

&HTTP

Listeners

Web BrowserHTML

ApplicationHTTP

Net8

Net8

View Layer

Created/Generated usingOracle DesignerHand coded

Created/Generated usingRational Rose

Metadata Browsing and ReportingMetadata Browsing and Reporting

• Web-enabled• Meta-model generated application• Supports textual and structured metadata searches• Supports nested searches• Supports hierarchical, tree-based browsing (drill

up/down)• Supports search and browsing of Tier 1 documents

(accessible in the CMR repository or on the Census Intranet/Internet)

The CMR Metadata Browsing and Reporting tools will provide read-only metadata searching, hierarchical browsing, and advanced analysis and reporting of the CMR metadata.

Metadata Open InterchangeMetadata Open Interchange

• Use an open standard technology (such as XML) to interchange metadata with external systems

• Support bulk metadata object level refreshes• Support metadata validation and correction in staging

area prior to final load• Enforce role-based, object level security• Support web-enabled log of input/output activity• Web-enable UI as much as possible

The CMR Metadata Open Interchange provides an open, standards-based method for interchanging metadata between the CMR and external systems.

CMR <-> base SAS CMR <-> base SAS

• Web-enabled UI• Output metadata for a statistical dataset for use by

base SAS• Input metadata for a statistical dataset from base SAS• Support metadata validation and correction in staging

area prior to final load• Enforce role-based, object level security• Support web-enabled log of input/output activity

CMR and SAS metadata interchange tool will enable the metadata owner to perform a bi-directional metadata exchange, and will establish a common mechanism of loading metadata into SAS system and pushing it back to CMR repository.

Target: Production in 2000

?Questions

?

Documents

Building a Corporate Metadata Repository