The ABC of Data Governance: driving Information Excellence

Alan Duncan, Director of Data Governance, UNSW

E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan

Uncontrolled when printed

The ABC of Data Governance and its role in driving Information Management Excellence

Alan D. Duncan February 2014

mailto:[email protected]

http://www.linkedin.com/in/alandduncan




A bit about me....

• Alan Duncan, Director of Data Governance, UNSW

• 21 years Information Management & Business

Consulting

– EDS, KPMG, CPW, Acuma, Pelion, SMS

– Scottish Power, United Distillers, O2, Astra Zeneca,

Carphone Warehouse, Vodafone, Riyad Bank

– Commonwealth Bank, NSW Roads & Maritime

Services, Centrelink, OATSIH, NSW Family &

Community Services, CASA, AMSA, FaHCSIA, DAFF,

Navy…

• Information-Management.com “Top 12 on Twitter”

• Best supporting Actor, 2005 Barnet Drama Festival






…and a bit about UNSW.






Agenda

• Why we need to rethink our Information

Management Strategies

• Techniques and capabilities we need to focus on

• What data should be stored, and how long should

we keep it?

• Information Management Excellence






Why we need to rethink our information

Management Strategies,

Sponsored by Thomas Edison

“The value of an idea

lies in the using of it.”






“A” is for “Action”

Identify measurable and targeted Business Outcomes

Why do we need information? For whom? What will we do

differently?

Establish DG Operating Model

Who is accountable? By what

processes?

Execute Activities & Tasks

How do we deliver? Who does the

work?

Confirm the Information Holdings & Gaps

What do we need to provide? (Content + Context)

Implement DG/IMCC Services

Catalogue:

What core capabilities do we need? “When it is obvious that the

goals cannot be reached,

don't adjust the goals,

adjust the action steps.”






“A” is for “Access”

• Stimulus to improve data quality

• Consistency of data definitions

• Openness and trust

• Transparency & accountability

• Opportunity value

• Proactive publication and Open

Data vs. “Need to know”

7

“Publish and be damned!”

http://www.ted.com/talks/tim_berners_lee_the_year_open_data_went_worldwide.html



http://www.ted.com/talks/tim_berners_lee_the_year_open_data_went_worldwide.html




“A” is for Analytics

8

“The alchemists in their

search for gold discovered

many other things of

greater value.”

• “Traditional” BI (reporting & ad-hoc analysis)

• Data Mining

• Statistical modelling

• Data visualisation

• Textual analytics

• What questions do we want to answer?

• What questions can we answer with the data

we’ve got?

• What other data would we need?

• What does the data tell us we should be

asking?






Summary: Information Use Cases • Based on our current understanding of business needs, the following classes of

Information Use Case are identified

• Detailed Requirements Analysis should be conducted on a project-by-project basis

to explore any detailed Use Cases within each class

• Not all detailed Use Cases need to be defined ahead of time

• Solutions should be flexible to accommodate new and changing Use Cases

Structured data

reporting

Strategic Intelligence

and Data Mining

Publish content to a community

Executive briefings

Education, Training, Learning

Search for content

previously created

Records Management, Compliance &

Audit

GIPA & Privacy

Responses

Ability to publish Filtering/screening/validation of what gets published

Feedback loop, measure of usefulness & continuous improvement

Shared understanding (IT & Business)






Techniques and capabilities,

sponsored by Mark Twain

“To succeed in life,

you need two things:

ignorance and

confidence.”






“A” is for “Asset”

Owners

AssetManagement

Tools

Governance

Admin

Experts

User Community

Information

Asset

Steward

OwnersOwners

The “Information Asset Community”

Directives

WMSAIRS

Example High Level Data Systems & Flows

Version 1.0

General

Mandatory

Core

Corporate Support

Automatic One-way Relationship

Automatic Two-way Relationship

Manual One-way Relationship

Manual Two-way Relationship

External

External System

AIRSInterchange

AUSSAR

AME

Cyber Exams

ATSL

CLIC

SDR

Publishing System

GMEL

TRIM AD

FRLI

Web Control Mgmt

System (WCMS)

FMIS

STI

ESIR

Comweb

ATO Business

Portal

Inventory Mgmt

System (IMS)

EPK

ComBIZ Online

ProMaster

FCAT

CalumoCBMS(DoFD)

COMCARE

Thomas Logistics

SM7

HRMS

HRFlex

DTAR/OTAR

APEX

AOD Audit

AOD Case Mgmt

SystemTimelog

eRooms

Symbion Health System

API Upload System

ChangePoint Testing System

MRS

ASSP

AWS

AFD

ASIR

ADMS

TrackerATOG Job Register

AOC Surveys

Industry PaymentsCompensation Payments

Financial Actuals

Financial Actuals

Employee Expenses / Adjustment

Journals

Salary Payments

Cash Payments

Payroll (Salary)

Cash Payments / Organisation Info

Human Resources Finance

Surveillance/Audit/Reporting/Tracking

Workflow and Online Collaboration/Service Delivery

Service Delivery

Service Delivery

Service Delivery, HR & Finance, Agreements, Permissions, Aerodromes, Participants, Aircraft

Medical Examinations

Medical Exams

Surveillance /Audits/ Reporting/ Tracking

Alcohol and Other Drugs

Surveys / Surveillance

Events/Occurrences, Aircraft, Aerodromes

Surveys/ Certifications

Examinations

Work Orders

Surveillance /Audits

Aerodromes

Aircraft

Events/Occurrences, Aircraft, Aerodromes

Defects/Events/Occurrences, Aircraft, Aerodromes

Exemptions Database

Alternative Means of

Compliance (AMOC)

Exemptions

AMOC / Exemptions

Human Resources – Flex

Time

Human Resources - Travel

Physical Inventory

Audit Data

workflow / service delivery

workflow / service delivery

Contacts – Ind, Org’s Contacts – Ind, Org’s

ExaminationsMedical

Examinations

Search and Rescue

Surveys

Human Resources – Time

Aircraft Equipment Finances

MMELBaseline/Minimal Equipment

Medical

PAWS

Retain

Details of Operators

Incidents

Applications / Permissions

Trending

Workflow

MAAT

Permissions / Change of Status

Permissions / Change of Status

Service Log

Alternative Means of Compliance (AMOC)

Dangerous Goods

Dangerous Goods

Content

Inventory

FTTO

FTNS

Individual Flight Data

Organisational Flight Data

Human Resources – Time

Aircraft

Individuals/ARNS

Payments

HR - TimeCash Receipts

Reconcile Invoice against

Flown Hours

Surveys / Surveillance

Enterprise Data Warehouse

CASA Internet

Airports

Landings/ Take Offs

Data

Mandatory

Core

Corporate Support

External

Business Process

Surveillance/Audit/

Reporting/Tracking

Bank Data File

PAYG payments, Salary

payments, and Superannuation

payments.

External Superannuation

Companies

Cash Payments

Superannuation Contributions

Suppliers

Remittance Advice

300+ Access Databases

Contacts

Airspace

Organisational

Human Resources

Aircraft

Permissions Info Asset Register

(inventory) System Interfaces map

“Science is organized

knowledge. Wisdom is

organized life.”






QW007

ORG007

PROC005

PROC003

STRAT007

STRAT002

STRAT005

CUL002

ORG003

ORG006

POL001

POL002

ORG002

SYS004

STRAT015

SYS008

QW004

CUL003

ORG008

QW006

ORG010

PROC014

PROC002

POL003

SYS010

SYS009

SYS002 SYS003

QW005

QW002

SYS006

ORG009

QW003

QW001

SYS007

SYS001

PROC010

PROC008

PROC004

PROC001

PROC011

POL004

POL005

PROC007

PROC009

PROC013

“B” is for “Benchmark”

“One day Alice came to a fork in the

road and saw a Cheshire cat in a

tree. Which road do I take? she

asked. Where do you want to go?

was his response. I don't know,

Alice answered. Then, said the cat, it

doesn't matter.”






“B” is for “Big Data”

• Three, four, five, six “Vees”?!

• A lot of data (Tb/day)

• Streaming data (monitoring, flow-of-control and

alerting analytics)

• Inference from semi-structured data (Twitter,

Facebook)

• Synthesise insight from millions of pages of text

• Programmatic analysis for specific scenarios (hard in

SQL)

• A disruptive catalyst to put information at the top of

the organisational agenda

• Not just about the data! Business scenarios are key

• Beware the Vendors!






“C” is for “Cloud”

14

… but security, privacy considerations are heightened.

In principle, it’s just another place to store data….






“C” is for compliance

• Freedom of Information Act 1982 (Cth)

• Freedom of Information Amendment (Reform) Act 2010

(Cth)

• Privacy Act 1988 (Cth)

• Privacy Amendment (Private Sector) Act 2000

• Privacy Amendment Act 2012 (Cth)

• Privacy Amendments (Privacy Alerts) Bill 2013 (Cth)

• State Records Act 1998 (NSW)

• Government Information (Public Access) Act 2009 (NSW)

• Privacy & Personal Information Protection Act 1998

(NSW)

• Health Records & Information Privacy Act 2002 (NSW)

• NSW Government Guide To Labelling Sensitive

Information 2011 (NSW Financial & Services)

15

But is “compliance” a motivator?

“All I want is compliance with

my wishes, after reasonable

discussion.”






“C” is for Collaboration Culture

16

• A general willingness to share

information

• Co-operative, communicative &

collegiate OR control, coercion

& criticism?

• The “whose data is this?” cue

• Call-to-action?

• Accountability &

measurement?

“Respond intelligently even

to unintelligent treatment.”






“D” is for Data Models

Metadata

Repository

Master Data Repositories UNSW Core Systems

Information

Asset

Register

Physical Instantiations

Physical Layer

Logical Layer (Transition)

Analytical

DB Models

Cubes

Conceptual Layer (Business)

Physical

Messages

Formats

DWHDB

HRDB

Student Admin

etc...

Operational

DB Models

Reference models

Data Subject Areas

Data Entities

Data Attributes

Information

Concepts

Business Content Business Rules Data

Business Data

Element

Domain Values

Endorsed Standards

for Content

Business Constraints

Business Measures

Master data models

Classification Entity

Hierarchies

Mappings

Business Rules

Definitions

Business Constraints

Business Measures

Core

SystemsMDM

Me

tad

ata

Ma

na

ge

me

nt P

rocess

Info

rma

tion M

od

el M

an

ag

em

en

t Pro

ce

ss

Info

rmatio

n A

lliance

s: D

ata

Ow

ne

rsh

ip &

Ste

wa

rdship

Pro

cess

MDM Processes

Related Data Governance Processes

Application Logical Data

ModelsLogical

Message

SchemasMDM Data

Model

Systems Data Models

SOA/EP

MessagesG/L

Application Logical Data

ModelsLogical

Message

Schemas

Analytical

DB Models

Cubes

Physical

Messages

Formats

Operational

DB Models

Business Glossary

Conceptual Model: Groupings & Relationships

Data Elements, Definitions,

Aliases, and Security

Data Domains

Enterprise Information Model

“Do not quench your

inspiration and your

imagination; do not

become the slave of your

model.”






Information

Disconnect

Careless data entry

& lack of validation

Teams use different

IT systems

?

Organisations

change rapidly

Teams have different ways

of reporting data

Month Region

Multiple codes exist

for the same thing

Data is in different

Formats

Overlapping subsets

in different places

Multiple, inconsistent

master data

“D” is for Data Quality

“Get your facts first,

then you can distort

them as you please.”






Summary: Data Governance increases utility

& value of information

Information System

Data Quality Management(Profiling, root-case analysis, issues tracking & resolution)

Data Modelling(Consistent, inter-operable data structures & semantic meaning)

Information Requirements & Business Analysis

(Identification & traceability of business definitions & rules)

Information Asset Register (Catalogue of data holdings)

Information SystemInformation System

Information System(s)

Data Set






What data should be stored, and how

long should we keep it?

Sponsored by Freddie Mercury

“The bigger the better;

in everything.”






“E” is for “Everything”

• All of the data, all of the time?

• Granular, forensic history

• Modern data management & analytics solutions can make “all

of the data, all of the time” a reality

• The bigger challenge is that the business community is not

analytically skilled enough to navigate the data and draw

meaning from it…






“E” is for “Expectations”

• Finding data that makes an impact

• Having data for the problem at hand

• Trusting the data to guide your

decision

• Justifying pre-determined answers

• Setting inappropriate goals

• Not having the right data tools

• Not thinking about value

“Two things are infinite.

The universe and human

stupidity. …and I’m not so

sure about the universe.”






“F” is for “Forethought”

• Think about both current and future demand

– Cf. Google philosophy to “keep everything”

– Every click, every font change…

• Tiered storage technology for different scenarios

– Fast-access disk for computational processing (TB)

– Drill-through near-line disk (TB/PB)

– Off-line active store (optical, tape) for archive (PB/EB)

• Information Lifecycle Management

– Copy/replication management

– Test data management

– Auditability of data movements between environments

– e.g. IBM Optim, Oracle ILM, Informatica ILM, Hadoop Project

Falcon

• Integration/coupling of HDFS & EDW/BI?

• Data Requirements Management…

“Forethought we may have,

undoubtedly, but not

foresight.”






Summary: Data Requirements Management -

two modes of operation

Source new data; Collection & Integration;

Preparation & Quality.

Demand-oriented Inbound requests for specific requirements

Data Factory

(“push”)

Product-based

delivery (“pull”)

Need both “push” and “pull” modes

for evidence-based decision-making

Plus metadata management, info asset management,

data management, process management practices..






Information Management Excellence

sponsored by Carl Sagan

“I try not to think with my

gut. If I‘m serious about

understanding the world,

thinking with anything

besides my brain, as

tempting as that might be,

is likely to get me into

trouble.”






“G” is for Governance

• “I don’t know what you’re going to do with my data once

you have it.”

• “If I give you my data, you might then ask me to do some

extra work to meet your additional requirements.”

• “You may not interpret the data in the same way that I do.”

• “I’m an expert in this area, you’re not. The data is too

complex for you to understand.”

• “It’s too difficult to get the data out of the system and I’d

need help from I.T.”

• “I don’t have the budget to pay for your requirements.”

• “I’d like to help but I’m just far too busy.”

• “I know there are flaws in the data, but it’s good enough

for my needs. You might criticize me for the errors.”

• “Management may ask additional questions and hold me

to account for the work I’m doing”.

26

“I’m not interested in

preserving the status quo;

I want to overthrow it.”






Data Governance: overcoming “I object!”

27

• Control, structure, discipline &

compliance? OR Advocacy service &

information broker?

• Intimate understanding of business goals

& processes

• Engagement, diagnosis & facilitation

• Understand & articulate the meaning of

data, in context

• Coach, mentor and advocate

• Highly visible point-of-access

• Self-service Information Portal

• Conduit, communicate & co-ordinate

• Leadership & direction

• “Info as a Product”

“The art of government is to

make two-thirds of a nation

pay all it possibly can for the

benefit of the other third.”






Conclusion: Information Excellence EIM Framework:

Enterprise Information Management Framework describes each aspect of an organisations

information management state, provides a baseline of maturity against best practice and a

framework of business transformation to your aspirational information management state.

Provides linkage and balance between business,/IT, and human/technical aspects of EIM.

Information

Governance

Information Security

Information Asset

Mgmt

Metadata

Ownership &

Stewardship

Information and IM

Strategy and

Planning

Information and IM

Quality Mgmt

Information Asset

Classification

Intellectual Property

Reporting design AnalyticsInformation

Security Policy

and Governance

Asset

Management

Human Resources

Security

Management

Knowledge Transfer

Data Mining

Data WarehousingBusiness Intelligence

Information IM

Workforce

Management

Information and IM

Risk Management

Registration

Data Modelling

Data management

Data Integration

Data Cleansing

Data Capture

Data Migration

Data De-duplication

Record Keeping

Knowledge Management

Information Asset Access and Use

Management

Privacy Publishing

CopyrightPhysical and

Environmental

Management

Communications

and Operations

Management

Information

Security Incident

Management

Access

Management

Information

system

acquisition,

development and

maintenance

management

Compliance

Management

Information and IM

Policy, Principles

and Architecture

Information and IM

Governance

Processes

Meta Knowledge

Search and Discovery

ExchangePricing

Licensing and

Rights

Management

Assess and

Accessibility

Redress MechanismsData Quality and

Integrity

Data Conversion

& Transformation

Record Management Archiving Conservation and

Preservation

Record Creation

and Capture

Digital Continuity

Collection Management

Retrieval and Access

Retention and Disposal

Business

Continuity

Enterprise Information Model IM Solutions and Technology IM Policies

Organisation and People Data Governance Information Culture

IM Processes

Business Processes

DB Models

Definitions, Derivations, Decision Rules, Execution Rules

IM Governance Process

IM Stewardship Process

Technical MetaData Management

Logical Model

ETL SpecsReport

DefinitionsSemantic Specs

Data

Marts

ETL Cubes

Semantic

Layer

Sta

ndard

Report L

ibra

ry

ETLOperational

SystemStaging Warehouse

Conceptual Model

Logical Model

Physical Model

Capture & Formalise

Requirements

& Rules

Impact

Assessment

& Implementation

Metadata

Lineage

Impact

Etc.

Metadata

Collection

Asset Alignment/Mgt

Architecture Changes

Architecture

Mgt

A holistic, data-centric approach to Information Management & Data Governance,

addressing both human and technical factors in both Business and IT domains






Intellectual curiosity

Skeptical scrutiny

Critical thinking

http://www.informationaction.blogspot.com.au/

@Alan_D_Duncan


