1 CORE – COmmon Reference Environment How it works Jean-Pierre Kent 11 January 2012

1

CORE –COmmon Reference

EnvironmentHow it works

Jean-Pierre Kent11 January 2012

Contents

• Introduction– Design ≠ implementation

• Overview– A model of the user’s experience

• Presentation of the information model

2

Design ≠ Implementation

• Goal of design:

– Deliver a concept apt to contribute

to the industrialisation of official

statistics (ref: HLG-BAS Vision)

• Result:

– A model that exceeds the capacity

of a 1-year project3

Design ≠ Implementation

• Goal of implementation

– Deliver Proof Of Concept with:

• Platform independence

• Model-driven

• Model-driven

• Result:

– Implementation of a subset of the

model 4

design environment

execution environment

What is Platform Independence?• Once implemented, a service can run on any

platform (e.g. .Net, Java, ...)

• A process engine running on a platform (e.g. Java) can control services running on another platform (e.g. .Net)

• A process can be distributed:

– Manage microdata at Statistics Netherlands

– Do aggregation at ISTAT

– Produce the SDMX output at Eurostat

– Under control of a process engine running at INSEE

5

Model-Driven: cutting

costs• Traditionally:

1. Designer makes models

2. Developer creates the system

3. System runs.

• In a model-driven environment:

1. Designer makes models

2. System runs

6

Model-Driven:

effectivity

7

Fle

xibi

lity

Standardisation

Taylor-made

Standard package, ERP, CRM, DMS...

Model-Driven

Spaghetti

Model-driven: benefits

• Cost reduction: less manual work

• Reliability: manual work is error prone

• Time to market: less manual work

• Standardisation: system enforces standards

• Flexibility: incremental development, agile maintenance

• Reliability: build processes from well-designed and well-tested services

• Strict separation of design and execution

• Focus on process quality as a source of product quality

• … and some more

Questions (Part 1)

?9

Overview

• What users see and use

– This is not the information model

– Nor the technical model

– But a model of the user’s

experience

10

11

Data Set+PreparesProcess Administrator

CORE Run Time

Process Administrator

+Runs Process

Process Step Data Set

Data Set DescriptionOrdered list of column names and types

12

Data Set Kind

CORE Design time

+Designs

Process Designer

Process

Process Step

CORE Service

Service Designer

NSI Community

List of required column kinds

Column kind: generic definition of a column, consisting of a data type and a statistically defined role ( e.g. dimension, measure, classification level, calibration weight)

This is where GSIM

comes in

Ordered list of column names and types

Data Set Description +Designs

13

CORE: the whole

picture

Ordered list of column names and types

Data Set Description +Designs

List of required column kinds

Process Administrator

Process+Runs

Data SetProcess Step+Prepares

Process Designer

+Designs

NSI Community

Data Set Kind

CORE Service

Service Designer

Column kind: generic definition of a column, consisting of a data type and a statistically defined role ( e.g. dimension, measure, classification level, calibration weight)

How do services

interact?• You can use different tools for

different services

– e.g. SPSS, SAS, R ...

• Different tools expect different

data formats

• Conversions are inevitable14

Conversions are

expensive!• Between 2 formats

– A B: 2 conversions

• Between 3 formats

– A B: 6 conversions

C

• Between N formats: N2-N15

CORE reduces N2-N to

2*N• Standard CORE data format

• Conversion to and from CORE format

Tool XInput (X)Output (X)

Model (CORE)

Convertor

Input (CORE) Output (CORE)

Model (X)

Convertor

Convertor is tool-specific – not service-specific

Model (X)

Model (CORE)

Convertors are format-specific, not service-specific

Questions (Part 2)

?17

18

CORE: the whole

picture

Ordered list of column names and types

Data Set Description +Designs

List of required column kinds

Process Administrator

Process+Runs

Data SetProcess Step+Prepares

Process Designer

+Designs

NSI Community

Data Set Kind

CORE Service

Service Designer

Column kind: generic definition of a column, consisting of a data type and a statistically defined role ( e.g. dimension, measure, classification level, calibration weight)

The information model

• What the designer sees

– Data set description and

data set kind

– Service and data set kind

Data Set Description

Data Set Kind

CORE Service

Data set definition

package

e.g. microdataset, aggregated data set, classification structure

e.g. variable, measure, dimension, level, weight

The type of a column definition should be one of the allowed types of its column kind

Cell

value

Data set kind

namemin number of rowsmax number of rows

Column kind

namemax number of occurrencesmin number of occurrences

1..n1..n

column kinds

<<set>>

Column definition

namecolumn id

column kindcolumn definition

Row

row id

1..n1..ncols

<<ordered>>

Data set definition

namelink to documentationlocation

data set kind

1..n1..n

column definitions

<<ordered>>

Value type

name1..n1..n allowed types

<<set>>

type

Data set

0..n0..nrows

data set definition

Descriptor

name

0..n0..n

descriptors

descriptor type

Actual descriptor

value

descriptors

descriptor

Data Set Description

Data Set Kind

Communication

channels• Manage communication between a

service and the execution environment

• Give support to Plug-and-Play coupling

• Implement messaging

Data Set Kind

CORE Service

5 types of channels

Channel

Data set kind(f rom Data set def initions)

Rule kind(f rom Rules)

GSIM object description(f rom Messages)

Parameter(f rom Parameters)

Expected free-style argument(f rom Messages)

4 types of messages

• Service signature message

• Service configuration message

• Service execution message

• Service output message

Service signature

message• Service communicates to its

environment the channels

that it supports.

• Channels constrain the kinds

of information

– expected during execution or

during configuration

• e.g. data set kind

“microdata”

– to produce during execution.

• e.g. data set kind

“aggregate”

Service AEnvironment

I expect a fixed number of data sets of the following kind: […] I expect an unbounded number of rules of the following kind: […] I

expect values for the following other parameters: […]

Service signature message

What is your signature?

channels

Service configuration

message

• Environment communicates

details about data sets that

will be offered to the service

during execution.

– e.g. number of columns of a

data set

– value types for each of these

columns

• This information must fit in

the channels specified in the

service signature message

– e.g. a data set description

must match the expected data

set kind

Service AEnvironment

The data sets you’ll receive during execution are of the following form: […] Please receive the following rules and other parameters: […]

Service configuration message

channels

Service execution

message• Service is requested to

execute itself

• Service is offered a number of data sets and business objects as input.

• This information must comply with the service's signature message

• It must be consistent with the data set and business object details the service is configured with through the service configuration message

– e.g. a data set must match the expected data set description

Service AEnvironment

Please start execution and receive the following data sets as arguments: […]

Service execution message

channels

Service output

message • A service ends its

execution by sending a

service output message.

• The result of the service

execution is documented

by data sets which

match the service's

configuration message.

– e.g. the output data sets

are consistent with their

data set descriptions.

Service AEnvironment

Please receive the following data sets as my output: […]

Service output message

What is your output?

channels

Questions (Part 3)

?28