Quality assurance activities Quality assurance activities at EUROSTATat EUROSTAT

CCSA ConferenceHelsinki, 6-7 May 2010

Martina Hahn, Eurostat

204/21/23

Structure of the presentation

Code of Practice

Institutional environment

Process quality

Output quality

ESS Quality Architecture

3

European Statistics Code of Practice

4

Content of the Code of Practice

3 headings – 15 principles – 77 indicators

Institutional environmentProfessional independence, Mandate for data collection, Adequacy of Resources, Quality commitment, Statistical confidentiality, Impartiality and objectivity

Statistical processesSound methodology, Appropriate statistical procedures, Non-excessive burden on respondents, Cost effectiveness

Statistical outputRelevance, Accuracy and reliability, Timeliness and punctuality, Coherence and comparability, Accessibility and clarity

5

Institutional environment

ESS-wide peer reviews during 2006-08– to monitor the implementation of the Code of Practice– to boost quality, enhance credibility– to share ESS best practices and transfer knowledge

Establishment of the European Statistical Governance Advisory Board (ESGAB):– to provide an independent overview of the implementation of the

Code of Practice by Eurostat– to advise Eurostat and the ESSC on updating and communicating

the Code to users and data providers Establishment of general guidelines (e.g. confidentiality

manual)

6

7

ESS Quality Coaching

Two quality managers work together to address specific improvement actions

Based on experience, dialogue and information exchange

Agreement on commitments from both sides

Joint working – not doctor/patient relation

17 coaching pairs took part

Coaching exercise completed at the end of 2009

Appropriate where there are clear standards and guidelines

8

Process and output quality: the Eurostat Quality Assurance Framework

A system of coordinated methods and tools to ensure a sustainable level of quality of processes and outputs in line with the Code of Practice:

– product/ output quality requirements are explicitely documented– processes are defined and made known to all staff– the correct implementation of the processes is monitored– users are informed on the quality of the products and possible

deficits– a procedure to guarantee that the necessary improvement

measures are planned, implemented and evaluated

9

The Eurostat Quality Assurance Framework

1010

The Eurostat Quality Assurance Framework

User Requirement Standards

III. CONFORMITY Evaluation function

Quality assessments

Rolling reviews Cost/benefit 5YP

Best practices

II. EVALUATION CVD steps | | | | | | Quality diagram

Improvement actionsQuality

Barometer

AMP

I. DOCUMENTATION MEASUREMENT

Process description

Quality reports

SDMX

Production processess Statistical products User perception

Eurostat Institutional/Legal environment

Labelling

ASSESSMENT CHECKLIST

User satisfaction survey

1111

Process Quality Assessments (1)C

on

cep

tua

l fra

me

wo

rk (

2)

IT conditions (11) – Management, planning and legislation (12) – Staff, work conditions and competence (13)

User needs(3)

Data collection

(4)

Validation

Country level (5)International level (6)

Confidentiality (7)

Dissemination (9)

Documentation (8)

Follow-up (10)

Common tool: The Process Quality Assessment Checklist (based on the International ISPAC)

Method: chronologically examining the process

Outputs:– Assessment Diagram– Summary Assessment report with improvement actions– Highlight of a good practice

1212

Process Quality Assessments (2)

Each process is matched with an assessment category

Self-Assessment

Prod. unit: fills in Checklist, produces outputs

B1: provides support on demand

Supported Self-Assessment

Prod. unit: in control of assessment,participates in all steps

B1: pre-fills the Checklist, produces the outputs

Peer reviewSame as

Supported Self-Assessment with one or more peers (int/ext)

taking part in all steps

Rolling reviewA comprehensive reviewbased on the checklist

implemented by ext.contractor, involving partner & user surveys

and cost-benefit analysis

Process 1

Characteristics:- Periodicity- Legal Basis- Output- ESTAT intervention-availability of quality report

Process 4Process 2 Process 3 Process 5

13

Process Quality Assessments Implementation Plan: Principles

Minimize burden for units

– Test of the approach in advance

– Support from the Quality unit

– Flexibility

Build on existing information

– Quality reports

– Process analysis

Integrate QA with other horizontal activities

14

Process Quality Assessment Outputs (1)

Summary Assessment Report

15

Process Quality Assessment Outputs (2)

Coherence General coherence

Timeliness Timeliness of final publication

Accuracy Overall accuracy

Relevance User satisfaction

Accessibility and clarityOverall quality of metadata

Comparability

Comparabilityover time

Comparabilityacross countries

0

1

2

3

4

5

Assessment Diagram

1616

The Eurostat Quality Assurance Framework

User Requirement Standards

III. CONFORMITY Evaluation function

Quality assessments

Rolling reviews Cost/benefit 5YP

Best practices

II. EVALUATION CVD steps | | | | | | Quality diagram

Improvement actionsQuality

Barometer

AMP

I. DOCUMENTATION MEASUREMENT

Process description

Quality reports

SDMX

Production processess Statistical products User perception

Eurostat Institutional/Legal environment

Labelling

ASSESSMENT CHECKLIST

User satisfaction survey

17

ESS Standard & Handbook for Quality Reports (1)

Provide (mainly producer oriented) recommendations / guidelines for the preparation of comprehensive quality reports for statistical processes and their outputs by distinguishing between 6 types of statistical processes:

– Sample survey– Census– Statistical process using administrative sources– Statistical process involving multiple data sources– Price or other economic index process– Statistical compilation

18

ESS Standard & Handbook for Quality Reports (2)

Promote harmonized quality reporting across statistical processes and Member States => facilitate comparisons

Are applicable to National Statistical Offices and Eurostat in their roles as producers of statistics

Also include 21 quality and performance indicators Handbook: is more detailed, with lots of examples and

explanations, describes the quality concepts, explains how the different quality documents relate to each other and is aligned with the 15 Principles of the Code of Practice

19

ESS Quality Declaration (national TQM framework)

ESS Quality architecture

ESS Statistical Law

ESS level national / Eurostat level

national Statistical Law

National QAF:Description of practices

and procedures for implementing the CoP

QAF:ESS standards

for implementing

the CoP