Upload
dangkhanh
View
216
Download
2
Embed Size (px)
Citation preview
Standards and Guidelines
Volume 1
Quality in Statistics
Quality Assurance & Audit Section Version 1.0 January 2006
Volume 1: Quality in Statistics 3
TABLE OF CONTENTS
1 INTRODUCTION ................................................................................................. 4
2 OFFICE POLICY ................................................................................................. 4
3 QUALITY IN STATISTICS ................................................................................... 5
3.1 Defining Quality ....................................................................................................................................... 5
3.2 The ESS Quality Framework ................................................................................................................... 6
3.3 Quality Control and Auditing ................................................................................................................... 6
4 STATISTICAL QUALITY STANDARDS AND GUIDELINES ............................... 7
4.1 Planning for Statistics Production ............................................................................................................ 7
4.2 Classifications and Standards ................................................................................................................... 7
4.3 Survey Frame ........................................................................................................................................... 8
4.4 Sampling Procedures ................................................................................................................................ 9
4.5 Questionnaire Design ............................................................................................................................... 9
4.6 Measures to Reduce Non-Response ....................................................................................................... 10
4.7 Data Editing ........................................................................................................................................... 11
4.8 Data Quality Evaluation (including Macro-Editing) .............................................................................. 11
4.9 Seasonal Adjustment .............................................................................................................................. 12
4.10 Statistical Confidentiality ....................................................................................................................... 13
4.11 Presentation and Dissemination ............................................................................................................. 14
4.12 Use of Administrative Data .................................................................................................................... 14
4.13 Documentation and Metadata ................................................................................................................. 15
4.14 Revisions to Published Data ................................................................................................................... 15
REFERENCES ......................................................................................................... 17
APPENDICES .......................................................................................................... 18
A. European Statistics Code of Practice (2005) .......................................................................................... 18
B. UN Fundamental Principles of Official Statistics .................................................................................. 23
C. Recommendations of LEG on Quality ................................................................................................... 24
Volume 1: Quality in Statistics 4
1 Introduction Good quality information is essential for informed public debate and decision-making. As the
provider of official national statistics it is vital that the output of the Office is of the highest quality.
Several National Statistical Institutes (NSIs) have advanced their reputations through the
implementation of effective quality assurance procedures. The issue of quality assurance was
addressed in the Deloitte & Touche Report to the National Statistics Board – Review of
Organisational Performance and Capability of the Central Statistics Office – May 1997. The report
recommended the establishment of quality assurance and internal audit functions. This
recommendation was a High Level Goal in the Statement of Strategy 2001-2003 and the Quality
Assurance & Audit section was duly set up in 2002.
This document, Quality in Statistics, mainly draws on guidelines developed by other NSIs1,2
and
adapted to fit the Office’s statistical system. The document has a number of objectives and uses:
To introduce and define what is meant by quality in the realm of official statistics (Section 3)
To further awareness of the importance of quality in the work of the Office
To help ensure adherence to principles to which the Office subscribes (see Section 2)
To form the basis of the Office’s quality assurance system
To identify approaches to improve the overall quality of statistical outputs
To set out in a clear and unambiguous way the high level rules under which the Office
carries out, or aims to carry out, its work. These can be seen as goals to which all engaged in
the work of the Office must aspire.
Quality in Statistics represents a visible symbol of the Office’s commitment to quality.
Overall the purpose of the document is to be of assistance to those working in the Office and
particularly those who are directly responsible for the data we produce.
2 Office Policy The Office subscribes to the European Statistics Code of Practice (see Appendix A) and the UN
Fundamental Principles of Official Statistics (see Appendix B). Eurostat’s increasing quality
requirements are an ongoing consideration for the Office.
Customer Service has been set as a key competency for all staff in the Office. Quality is at the core of
customer service and is a challenge for all.
This document is to be regarded as Office policy. It contains an agreed set of ways and means of
ensuring that standards are met. It provides information on good practice and lists guidelines which
should be followed in the work of the Office. Implementation of these guidelines, where relevant, is a
requirement for each business area. However, it is accepted that the professional judgement by a
relevant staff member in particular situations and circumstances may result in the guidelines not being
fully implemented.
For a summary of the Deloitte & Touche report’s recommendations see: Corporate Documents\CSO\General
information\1997 Deloitte & Touche Report on the CSO - Summary of Recommendations Made and
Management Response
Volume 1: Quality in Statistics 5
3 Quality in Statistics
3.1 Defining Quality
Until fairly recently the quality of statistical output has traditionally been viewed in terms of accuracy.
However, quality as employed in other activities has generally included broader interpretations. These
tend to take in concepts such as ‘fitness for use’, ‘meeting user need’ and ‘customer satisfaction’.
Such concepts are clearly appropriate for official statistics and highlight the shortcomings of accuracy
alone as a measure of overall quality. Quality initiatives in the European Statistical System (ESS)
have reflected this need for a broader interpretation. The International Standards Office (ISO)
approach is considered particularly appropriate, to both statistical products and statistical services.
They define quality as ‘the totality of features or characteristics of a product or service that bear on its
ability to satisfy stated or implied needs of customers’.
Arising from the broader interpretations of quality is a need to define the elements that impact on
quality and that can be used to characterise quality. There is no universally agreed list of the
characteristics which define quality, however there is considerable overlap in the approaches adopted
by various NSIs. A lot of effort has gone into standardising quality concepts within the ESS3.
In February 2005, the ESS Statistical Programme Committee (SPC) unanimously endorsed the
European Statistics Code of Practice. The Code of Practice has the dual purpose of:
- Improving trust and confidence in the independence, integrity and accountability of both National
Statistical Authorities and Eurostat, and in the credibility and quality of the statistics they
produce and disseminate
- Promoting the application of best international statistical principles, methods and practices by all
producers of European Statistics to enhance their quality.
The code is a self-regulatory instrument consisting of 15 principles grouped into three sections
addressing respectively the institutional environment, statistical processes and statistical outputs as
follows: -
European Statistics Code of Practice: 15 Principles
Institutional Environment Statistical Processes Statistical Output
1. Professional Independence 7. Sound Methodology 11. Relevance
2. Mandate for Data Collection 8. Appropriate Statistical Procedures 12. Accuracy and Reliability
3. Adequacy of Resources 9. Non-Excessive Burden on
Respondents
13. Timeliness and Punctuality
4. Quality Commitment 10. Cost Effectiveness 14. Coherence and Comparability
5. Statistical Confidentiality 15. Accessibility and Clarity
6. Impartiality and Objectivity
A full explanation of the Code of Practice and indicators of good practice for each of the 15 Principles
are given in Appendix A.
The ESS comprises Eurostat and the statistical offices, ministries, agencies and central banks that
collect official statistics in EU Member States, Iceland, Norway and Liechtenstein.
Volume 1: Quality in Statistics 6
3.2 The ESS Quality Framework
The principle risk for any NSI is that it should face a loss of credibility by users. Consequently, the
last decade has seen considerable effort spent on assuring and evaluating the quality of the statistics
produced. Much of the work has been undertaken co-operatively between NSIs in the ESS. In 1998
Quality of Statistics was the theme of the annual conference of Presidents and Directors-General of
the European NSIs. This led in 1999 to the establishment of a Leadership Group (LEG) on Quality.
The LEG was given a broad remit to:
Establish a framework for considering quality issues
Identify key elements to be considered
Obtain information on the status of these elements in the ESS
Demonstrate with examples how improvements in NSIs and in the ESS could be made
Propose future actions for the ESS.
The output of the LEG consisted of a summary report (see SMD\Quality\Summary Report from the
Leadership Group (LEG) on Quality) containing 22 recommendations (see Appendix C).
Several of the recommendations have been acted on and quality related requirements specified in
Council Regulations now apply to certain statistics required by Eurostat. Such statistics include the
Labour Force Survey, Short-term Business Statistics, Structural Business Statistics and Labour Costs
Statistics
All NSIs within the ESS have signed up to the LEG report, therefore the Office is committed to its
recommendations.
3.3 Quality Control and Auditing
Internal audit forms part of any quality control system. Quality Assurance & Audit Section will
conduct audits to monitor the implementation of quality standards and to evaluate systems throughout
the Office. Such audits might result in reports with recommendations to management with
responsibility and authority for the matter in question.
It is intended to issue, separately, standards and guidelines in relation to non-statistical (including
financial) aspects of the Office’s work. Auditing of internal financial systems will be undertaken.
Requirements arising from the Report of the Working Group on the Accountability of Secretaries
General and Accounting Officers (Mullarkey Report) will continue to be addressed.
Volume 1: Quality in Statistics 7
4 Statistical Quality Standards and Guidelines
4.1 Planning for Statistics Production
The planning process for a new statistical activity or the redesign of an ongoing activity should
include the definition of broad objectives, a targeted user population and the key questions or issues to
which analysis will be directed. In order to translate this initial planning into actual production,
objectives and uses should be stated precisely to help ensure that the new or redesigned activity will
meet specific user requirements.
The Office has set up a Project Office and has adopted project management as a core standard for the
way the Office conducts its business. The principles of project management apply to all projects
regardless of size. The document Project Management Guidance for the CSO (see Corporate
Documents\Strategic Planning\Project Management\Documents\Project Management Guidance for
the CSO) explains how project management is effectively applied in the CSO.
Guidelines
4.1.1 Focus analysis of user needs on finding the most cost-effective solution for both the short and
long term.
4.1.2 Develop survey objectives in partnership with important users and stakeholders through, for
example, Liaison Group meetings. Establish and maintain relationships with users in order to
enhance the relevance of the information produced and as part of marketing products and
services.
4.1.3 In determining the extent to which a survey will meet user needs, seek a reasonable trade-off
between these needs and the budget, response burden and confidentiality considerations.
Although the Office may have little discretion where a legal requirement is in place, in other
cases look at alternative methodological approaches, frequencies, geographical details, etc.
with a view to arriving at an optimum solution.
4.1.4 Review ongoing statistical activities at regular intervals. Statistical activities need to evolve,
adapt and innovate to keep pace with the demands of the users they serve.
4.1.5 Apply project management to the statistical activity. State clearly the scope of the project and
agree a detailed project plan including budgets and resource allocations.
4.2 Classifications and Standards
The Classifications and Related Standards system (CARS) (see Corporate Documents\CSO\Home
Pages\Classifications & Standards Homepage) is the Office's system for storing and accessing
classifications. The CARS database is the Office's central repository for all classifications,
concordances and coding indexes.
The purpose of CARS is to:
Use database technology to provide centralised classification storage, maintenance and access
facilities for classification data that are used both in the development and processing of
surveys and in the subsequent analysis and evaluation of the data
Help reduce the time and resources required when developing surveys and to contribute to
improved data quality by supporting the use of standard classifications
Facilitate the comparison and analysis of data by storing concordances.
Volume 1: Quality in Statistics 8
Guidelines
4.2.1 All classifications used in the Office must reside on CARS. The responsibility for ensuring
this rests with the Office's business areas.
4.2.2 The official Office policy for CARS (see Corporate Documents\IT\Systems\Policy for Using
CARS) must be followed in all matters relating to classifications.
4.3 Survey Frame
A survey frame is any list or register that delimits, identifies, and allows access to the elements of the
target population. The target population is the set of elements about which information is wanted and
estimates are required. The extent to which a survey frame includes all the elements of the target
population is referred to as coverage. Practical considerations may dictate that some units be excluded
(e.g. companies with less than five employees, institutionalised individuals) from some frames.
The survey frame should conform to the target population and contain minimal undercoverage and
overcoverage (e.g. duplication). Frame creation, use, maintenance and monitoring should be
implemented within operational and cost constraints.
Characteristics of the frame units (e.g. classification, contact, address, size) should be of high quality
because of their use in stratification, collection, follow-up, estimation, record linkage, quality
assessment and analysis.
Guidelines
4.3.1 In designing business surveys, or in the redesign of existing ones, the Office’s Business
Register should be used to construct the appropriate survey frame.
4.3.2 Where possible, use the same frame for surveys with the same target population, to avoid
inconsistencies and to reduce costs of frame maintenance and evaluation.
4.3.3 Incorporate procedures to eliminate duplication and to update for births, deaths, out-of-scope
units and changes in characteristics.
4.3.4 Monitor the frame quality by periodically assessing its coverage.
4.3.5 For area frames, implement map checks to ensure clear and non-overlapping delineation of
the geographic areas used in the sampling design (e.g. through field checks or the use of other
map sources).
4.3.6 For statistics production from administrative sources, determine and monitor coverage
through contact with the source manager. Where influence on the frame is possible, negotiate
required changes with the source manager.
4.3.7 Whenever necessary, adjust the statistical results or use supplementary data to offset coverage
differences between the frame and the target population.
4.3.8 Include descriptions of the target population, frame and coverage in the survey
documentation.
Volume 1: Quality in Statistics 9
4.4 Sampling Procedures
Sampling is the selection of a set of units from a survey frame. This set of units is referred to as the
sample. The choice of sampling method has a direct impact on the data quality. It is influenced by
many factors, including the desired level of precision of the information to be produced, the
availability of appropriate frames, the availability of suitable stratification variables, the estimation
methods that will be used and the available budgets.
The intention is to gather useful information from the sampled units to allow inferences about the
target population.
Guidelines
4.4.1 When determining sample size, take into account the required levels of precision needed for
the survey estimates, the type of design and estimator to be used, the availability of auxiliary
information, as well as both sampling factors (e.g. stratification) and non-sampling factors
(e.g. non-response).
4.4.2 For highly skewed populations, include in the survey a stratum of large units that will be
sampled with certainty.
4.4.3 In determining sample allocation for stratified samples, account for expected rates of
misclassification of units in the frame.
4.4.4 For periodic surveys that use designs in which the sample size grows as the population
increases, develop a method to keep the sample size stable.
4.4.5 For periodic surveys, if efficient estimates of change are required or if response burden is a
concern, use a rotation sampling scheme that replaces part of the sample in each period.
4.4.6 For periodic surveys develop procedures to monitor the quality of the sample design over
time. Set up an update strategy for selective redesign of strata that have suffered serious
deterioration.
4.5 Questionnaire Design
A questionnaire is a set of questions designed to collect information from a respondent. A
questionnaire may be interviewer-administered or respondent-completed, using paper methods of data
collection or electronic modes of completion. Questionnaires play a central role in the data collection
process. They have a major impact on data quality, respondent behaviour, interviewer performance
and respondent relations.
The design of questionnaires takes into account the statistical requirements of data users,
administrative requirements of the survey organisation, and the requirements for data processing, as
well as the nature and characteristics of the respondent population.
Guidelines
4.5.1 Questionnaires in periodic surveys should be evaluated regularly.
4.5.2 Use words and concepts that have the same meanings for both respondents and the
questionnaire designers. In the case of business surveys, choose questions, time reference
periods and response categories that are compatible with the respondent's record-keeping
practices.
Volume 1: Quality in Statistics 10
4.5.3 In the introduction to all questionnaires:
Provide the title or subject of the survey
Explain the purpose of the survey
Request the respondent’s co-operation
Indicate the authority under which the survey is taken, and what confidentiality protection
arrangements are in place.
4.5.4 Ensure that the value of providing information is made very clear to respondents. In addition,
the importance of completing the questionnaire and how the survey data will be used must be
highlighted.
4.5.5 The opening questions should be applicable to all respondents, be easy and interesting to
complete, and establish that the respondent is a member of the target population.
4.5.6 Questionnaires that are to be administered in person or over the telephone must be made
interviewer-friendly as well as respondent-friendly.
4.5.7 Ensure that the instructions to respondents and or interviewers are short, clear, and easy to
find. Provide definitions at the beginning of the questionnaire or in specific questions, as
required.
4.5.8 Ensure that time reference periods and units of response are clear to the respondent, specify
“include” or “exclude” in the questions themselves and not in separate instructions.
4.5.9 Ensure that response categories are mutually exclusive and exhaustive.
4.5.10 Provide titles or headings for each section of the questionnaire, and include instructions and
answer spaces that facilitate accurate answering of the questions.
4.6 Measures to Reduce Non-Response
Non-response has two effects on results: one contributing to bias of estimates when non-respondents
differ from respondents in the characteristics measured; the other contributing to a decrease in the
accuracy of the survey estimates resulting from the smaller effective sample size.
The degree to which response is pursued is subject to budget and time constraints and the risk of non-
response bias. Adjustments are subsequently made to data to compensate for non-response (e.g.
weighting adjustments or imputation).
An effective respondent relations programme and a well-designed questionnaire are critical elements
in maximising response.
Guidelines
4.6.1 Establish and maintain good relationships with respondents.
4.6.2 Ensure interviewers are fully trained in interviewing techniques etc.
4.6.3 When operational constraints permit, follow-up the non-respondents either as a complete
enumeration or on a sub-sample basis.
4.6.4 Prioritise follow-up activities. For example, in business surveys, follow-up large or influential
units first.
Volume 1: Quality in Statistics 11
4.6.5 Record and monitor reasons for non-response (e.g. refusal, non-contact, temporarily absent,
technical problem).
4.6.6 Use an appropriate method of imputation to compensate for non-response. Only as a last
resort should grossing factor adjustment be used.
4.6.7 Where applicable, ensure that those being surveyed are aware that the survey is statutory.
4.7 Data Editing
The goals of editing are to:
Provide the basis for future improvement of the survey vehicle
Provide information about the quality of the survey data
Tidy up the data.
It may be that a disproportionate amount of resources is concentrated on the third objective of
‘cleaning up the data’. A danger is that learning from editing processes may play an undeserved,
secondary role.
While it is recognised that fatal errors (e.g. invalid or inconsistent entries) should be removed from
the data sets in order to maintain credibility and to facilitate further automated data processing and
analysis, caution should be exercised against the overuse of query edits (those pointing to
questionable records that may potentially be in error).
Guidelines
4.7.1 Ensure that all edits are internally consistent (i.e. not self-contradictory).
4.7.2 Reapply edits to units to which corrections were made to ensure that no further errors were
introduced directly or indirectly.
4.7.3 Perform edit checks for missing values, invalid values, etc. as quickly and as expediently as
possible in the processing cycle.
4.7.4 Rationalise 'query' editing (i.e. checks for apparent errors or inconsistencies), and find an
appropriate balance between error detection and cost.
4.7.5 Consider editing to be an integral part of the data collection process in its role of gathering
intelligence about the process. Use editing to:
Sharpen definitions
Evaluate the quality of the data
Identify non-sampling error sources
Serve as a basis of future improvement of the whole survey process.
4.8 Data Quality Evaluation (including Macro-Editing)
Data quality evaluation refers to the process of evaluating the final statistical output in the light of the
original objective of the statistical activity, in terms of the data’s accuracy or reliability. Such
information allows users to make more informed interpretations of the survey results, and can be used
by the Office to improve surveys.
Volume 1: Quality in Statistics 12
Two general types of data quality evaluation can be distinguished:
Macro editing or quality validation is the process of reviewing the data before official release
to ensure that grossly erroneous data are not released, or to identify data of marginal quality.
Sources of error studies generally provide quantitative information on specific sources of
error in the data. While timeliness is still important, the results of these studies often are only
available after the official release of the data.
Guidelines
4.8.1 Make planning of data quality evaluations part of the overall statistical process, as the
information needed to conduct such evaluations often must be collected during the process
itself.
4.8.2 Involve users of the results, whether they are external or internal, in setting the objectives for
the data quality evaluation programme. Where circumstances permit, also involve them in the
evaluation process itself.
4.8.3 The following macro editing and quality validation methods should be used:
Checks of consistency with external sources of data, for example from other surveys or
from previous instances of the same survey
Internal consistency checks, for example calculation of ratios that are known to lie within
certain bounds (sex ratios, average value of commodities, etc.)
Unit-by-unit reviews of the largest contributors to aggregate estimates, typically the case
in business surveys
Debriefings with staff involved in the collection and processing of the data.
4.8.4 The following sources of error should be evaluated:
Coverage errors, which consist of omissions, erroneous inclusions, and duplications in
the frame used to conduct the survey
Non-response errors, which occur when the survey fails to get a full response
Measurement errors, which occur when the response received differs from the ‘true’
value, and can be caused by the respondent, the interviewer, the questionnaire, the mode
of collection, or the respondent’s record-keeping system
Processing errors, which can occur at the subsequent steps of data editing, coding,
capture, imputation and tabulation
Sampling errors, which occur when the results of the survey are based on a sample rather
than the entire population.
4.9 Seasonal Adjustment
Seasonal adjustment consists of estimating seasonal factors and applying them to a time series to
remove the seasonal variations. These variations represent the composite effect of climatic and
institutional factors that repeat with a certain regularity within the year.
Many series are published in seasonally adjusted form to reveal the underlying trend movements and
to help data analysis. Seasonally adjusted series comprise not only the trend but also the irregular
component; consequently, they only give an approximate idea of the underlying trend movements. To
eliminate the irregular component the seasonally adjusted series may be further smoothed and the
Volume 1: Quality in Statistics 13
trend derived. In some cases the trend estimates may be considered for publication if the seasonally
adjusted series is considered unsuitable.
The Statistical Methods and Development (SMD) Seasonal Adjustment Policy document (see
SMD\Quality\Corporate Seasonal Adjustment Policy) sets out the procedures to be followed for the
seasonal adjustment of data in the Office.
Guidelines
4.9.1 All time series which exhibit evidence of seasonality and for which the underlying seasonality
can be identified reliably should be seasonally adjusted.
4.9.2 Before publishing a seasonally adjusted time series for the first time, conduct a thorough
seasonal adjustment analysis to assess if the seasonality is identifiable.
4.9.3 Follow the procedures set out in the SMD Seasonal Adjustment Policy document.
4.9.4 For seasonal adjustment, use concurrent seasonal factors. These are the factors obtained using
all recent values of the series. This is will give rise to frequent, mostly small, revisions to
seasonally adjusted series. A definite policy on the publication of the revisions should be used
and clearly explained to users.
4.9.5 For aggregate series resulting from the addition or subtraction of component series, seasonally
adjust only those component series that contain identifiable seasonality, and leave the others
unadjusted. Seasonally adjust the aggregate series directly (i.e. do not use the aggregate of the
component seasonally adjusted series). Inform users that small discrepancies will arise in the
seasonally adjusted estimates of aggregate series.
4.9.6 Wherever seasonally adjusted figures pertaining to the same economic activity are published,
co-ordinate the seasonal adjustment options applied by the areas involved.
4.10 Statistical Confidentiality
The Statistics Act, 1993 (see Corporate Documents\CSO\Home Pages\CSO Policies Home Page)
provides that all information collected by the Office can be used only for statistical purposes and that
(subject to stated exceptions) any information which can be related to an identifiable person or
undertaking cannot be disseminated, shown or communicated to any person or body.
The requirements of the Statistics Act, 1993 are additional to:
The obligations on the staff of the CSO under the Official Secrets Act, 1963 not to make
unauthorised communications, directly or indirectly, about matters which come to their
knowledge in the course of their official duties
The obligations on the CSO under the Data Protection Act, 1988 regarding the handling of
personal data.
Guidelines
4.10.1 The Office's Code of Practice on Confidentiality (see Corporate Documents\CSO\Home
Pages\CSO Policies Home Page|CSO Statistical Confidentiality Code of Practice) should be
followed at all times.
Volume 1: Quality in Statistics 14
4.11 Presentation and Dissemination
Presentation and dissemination are among the final stages in the statistical production process. While
presentation and dissemination procedures may not affect the accuracy of the statistical data, the
perception of quality by the users may be strongly influenced.
The forms of presentation and dissemination should facilitate access and interpretation by users.
Tables should have clear headings, identifiable units and logical layout. Charts should convey a clear
and accurate representation of the phenomenon being studied.
Guidelines
4.11.1 In official statistical releases and reports a suitable analysis highlighting the more important
and interesting features of the published statistics should be provided. This should be based
on the statistical results available and should never contain any political or similar
judgements. In general the larger and more infrequent the release or report the more detailed
the analysis should be.
4.11.2 Guidelines for tables:
Every table should be labelled to clearly identify the content
The table layout must be clear and easy to follow
All units of measurement should be displayed clearly
Where a comparison between numbers is required, these should be listed in columns to
facilitate reading
Ensure all rounding to significant digits is mathematically correct
Ensure that footnotes are clearly marked and that the text is clear and readable.
4.11.3 Guidelines for charts:
The chart title must explain what phenomenon is represented and the time periods
covered.
All axes must be clearly labelled and include the units of measurement.
Legends, labels and tick marks should all be clear and readable.
All elements on the chart should be identified.
Any apparent discrepancy should be highlighted and explained.
4.11.4 The document Write Well, Write Clearly (see Corporate Documents\Dissemination
\Information Section\Write Well, Write Clearly - CSO Usage and House Style (1996)) is the
official policy on written content and must be followed for all releases and publications.
4.12 Use of Administrative Data
The term administrative records refers to data collected for the purpose of carrying out various
programmes, for example, income tax collection.
Administrative records present a number of advantages. Since they already exist, costs of direct data
collection and further burden on respondents are avoided. They are usually available for the complete
universe, and hence, they are most of the time not constrained by sampling error limitations. Most
importantly, they can be used in numerous ways in the production of statistical outputs. Examples of
their uses include:
The creation and maintenance of frames
Volume 1: Quality in Statistics 15
The complete or partial (via record linkage) replacement of statistical collection
Editing, imputation and weighting of data from statistical collection
The evaluation of statistical outputs.
Guidelines
4.12.1 Actively investigate and assess all potential sources of administrative data. Bear in mind that
even partially complete and/or partially inaccurate administrative data may still prove useful
in areas such as the reduction of the response burden in surveys and the improvement of
survey results.
4.12.2 Set up an edit and imputation procedure or a weight adjustment procedure to deal with non-
response.
4.12.3 The confidentiality implications of the publication of information from administrative records
must always be borne in mind. Although the Statistics Act, 1993 provides the CSO with the
authority to access administrative records for statistical purposes, the use may not have been
foreseen by the original suppliers of information.
4.12.4 Maintain continuing liaison with the provider of administrative records.
4.12.5 Understand and document concepts, definitions and procedures underlying the collection of
the administrative data.
4.12.6 Implement continuous or periodic assessment of incoming data quality.
4.13 Documentation and Metadata
Documentation refers to the collection of material that provides a description of the activity. It should
include the concepts, definitions, metadata, methodology, and an outline of the production processes
used.
The Business Process Improvement Inquiry (BPI) (see Corporate Discussion\ITSIP\BPI\ Business
Process Improvement Project Report) was a means of documenting, at a high level and in a standard
way, the inputs, methodologies, processes and outputs for every survey in the Office.
Guidelines
4.13.1 Business areas must ensure that a Business Process Improvement (BPI) methodology
questionnaire has been completed and is kept up to date for every survey or statistical process.
4.13.2 The metadata held on the Databank and, in future on the new Data Management System, must
be reviewed and updated whenever a change is made to any part of the statistical activity.
4.14 Revisions to Published Data
Revisions to published data can, and do, occur for many reasons. It is important that all users of
statistics are at all times made fully aware of the revision policy relating to these statistics.
Common sources of revisions to statistics are:
The availability of additional information (e.g. late survey responses; a new period’s data
available for the calculation of seasonal factors)
The receipt of amended information (e.g. as a result of the response to a query)
Changes resulting from additional and more detailed data editing.
Volume 1: Quality in Statistics 16
Occasionally revisions can arise from a change in methodology when it is deemed best to recompile
previous period’s data using the new methodology.
Guidelines
4.14.1 Produce, for users, a clear and concise data revisions policy for all published statistics.
4.14.2 When publishing data that is likely to be subsequently revised (e.g. preliminary estimates)
always indicate this to the user.
4.14.3 Describe the main reasons why particular statistics are subject to revision.
4.14.4 For established statistical data series inform users of the frequency and size of past revisions.
Volume 1: Quality in Statistics 17
References 1Statistics Canada, October 1998, Statistics Canada Quality Guidelines, 3
rd edition
2Statistics Finland, 2002, Quality Guidelines for Official Statistics
3Eurostat, 2003, Assessment of quality in statistics, Methodological documents – Definition of Quality in
Statistics, Doc. Eurostat/A4/Quality/03/General/Definition
Volume 1: Quality in Statistics 18
Appendices
A. European Statistics Code of Practice (2005)
Institutional Environment Institutional and organisational factors have a significant influence on the effectiveness and credibility
of a statistical authority producing and disseminating European Statistics. The relevant issues are
professional independence, mandate for data collection, adequacy of resources, quality commitment,
statistical confidentiality, impartiality and objectivity.
Principle 1: Professional Independence - The professional independence of statistical authorities
from other policy, regulatory or administrative departments and bodies, as well as from private sector
operators, ensures the credibility of European Statistics.
Indicators
– The independence of the statistical authority from political and other external interference
in producing and disseminating official statistics is specified in law.
– The head of the statistical authority has sufficiently high hierarchical standing to ensure
senior level access to policy authorities and administrative public bodies. He/she should be
of the highest professional calibre.
– The head of the statistical authority and, where appropriate, the heads of its statistical
bodies have responsibility for ensuring that European Statistics are produced and
disseminated in an independent manner.
– The head of the statistical authority and, where appropriate, the heads of its statistical
bodies have the sole responsibility for deciding on statistical methods, standards and
procedures, and on the content and timing of statistical releases.
– The statistical work programmes are published and periodic reports describe progress
made.
– Statistical releases are clearly distinguished and issued separately from political/policy
statements.
– The statistical authority, when appropriate, comments publicly on statistical issues,
including criticisms and misuses of official statistics.
Principle 2: Mandate for Data Collection - Statistical authorities must have a clear legal mandate
to collect information for European statistical purposes. Administrations, enterprises and households,
and the public at large may be compelled by law to allow access to or deliver data for European
statistical purposes at the request of statistical authorities.
Indicators
– The mandate to collect information for the production and dissemination of official
statistics is specified in law.
– The statistical authority is allowed by national legislation to use administrative records for
statistical purposes.
– On the basis of a legal act, the statistical authority may compel response to statistical
surveys.
Principle 3: Adequacy of Resources - The resources available to statistical authorities must be
sufficient to meet European statistics requirements.
Indicators
Volume 1: Quality in Statistics 19
– Staff, financial, and computing resources, adequate both in magnitude and in quality, are
available to meet current European statistics needs.
– The scope, detail and cost of European statistics are commensurate with needs.
– Procedures exist to assess and justify demands for new European statistics against their
cost.
– Procedures exist to assess the continuing need for all European statistics, to see if any can
be discontinued or curtailed to free up resources.
Principle 4: Quality Commitment - All ESS members commit themselves to work and co-operate
according to the principles fixed in the Quality Declaration of the European Statistical System.
Indicators
– Product quality is regularly monitored according to the ESS quality components.
– Processes are in place to monitor the quality of the collection, processing and
dissemination of statistics.
– Processes are in place to deal with quality considerations, including trade-offs within
quality, and to guide planning for existing and emerging surveys.
– Quality guidelines are documented and staff are well trained. These guidelines are spelled
out in writing and made known to the public.
– There is a regular and thorough review of the key statistical outputs using external experts
where appropriate.
Principle 5: Statistical Confidentiality - The privacy of data providers (households, enterprises,
administrations and other respondents), the confidentiality of the information they provide and its use
only for statistical purposes must be absolutely guaranteed.
Indicators
– Statistical confidentiality is guaranteed in law.
– Statistical authority staff sign legal confidentiality commitments on appointment.
– Substantial penalties are prescribed for any wilful breaches of statistical confidentiality.
– Instructions and guidelines are provided on the protection of statistical confidentiality in
the production and dissemination processes. These guidelines are spelled out in writing
and made known to the public.
– Physical and technological provisions are in place to protect the security and integrity of
statistical databases.
– Strict protocols apply to external users accessing statistical microdata for research
purposes.
Principle 6: Impartiality and Objectivity - Statistical authorities must produce and disseminate
European statistics respecting scientific independence and in an objective, professional and
transparent manner in which all users are treated equitably.
Indicators
– Statistics are compiled on an objective basis determined by statistical considerations.
– Choices of sources and statistical techniques are informed by statistical considerations.
– Errors discovered in published statistics are corrected at the earliest possible date and
publicised.
– Information on the methods and procedures used by the statistical authority are publicly
available.
– Statistical release dates and times are pre-announced.
Volume 1: Quality in Statistics 20
– All users have equal access to statistical releases at the same time and any privileged pre-
release access to any outside user is limited, controlled and publicised. In the event that
leaks occur, pre-release arrangements should be revised so as to ensure impartiality.
– Statistical releases and statements made in Press Conferences are objective and non-
partisan.
Statistical Processes European and other international standards, guidelines and good practices must be fully observed in
the processes used by the statistical authorities to organise, collect, process and disseminate official
statistics. The credibility of the statistics is enhanced by a reputation for good management and
efficiency. The relevant aspects are sound methodology, appropriate statistical procedures, non-
excessive burden on respondents and cost effectiveness.
Principle 7: Sound Methodology - Sound methodology must underpin quality statistics. This
requires adequate tools, procedures and expertise.
Indicators
– The overall methodological framework of the statistical authority follows European and
other international standards, guidelines, and good practices.
– Procedures are in place to ensure that standard concepts, definitions and classifications are
consistently applied throughout the statistical authority.
– The business register and the frame for population surveys are regularly evaluated and
adjusted if necessary in order to ensure high quality.
– Detailed concordance exists between national classifications and sectorisation systems and
the corresponding European systems.
– Graduates in the relevant academic disciplines are recruited.
– Staff attend international relevant training courses and conferences, and liaise with
statistician colleagues at international level in order to learn from the best and to improve
their expertise.
– Co-operation with the scientific community to improve methodology is organised and
external reviews assess the quality and effectiveness of the methods implemented and
promote better tools, when feasible.
Principle 8: Appropriate Statistical Procedures – Appropriate statistical procedures, implemented
from data collection to data validation, must underpin quality statistics.
Indicators
– Where European statistics are based on administrative data, the definitions and concepts
used for the administrative purpose must be a good approximation to those required for
statistical purposes.
– In case of statistical surveys, questionnaires are systematically tested prior to the data
collection.
– Survey designs, sample selections, and sample weights are well based and regularly
reviewed, revised or updated as required.
– Field operations, data entry, and coding are routinely monitored and revised as required.
– Appropriate editing and imputation computer systems are used and regularly reviewed,
revised or updated as required.
– Revisions follow standard, well-established and transparent procedures.
Volume 1: Quality in Statistics 21
Principle 9: Non-Excessive Burden on Respondents - The reporting burden should be
proportionate to the needs of the users and should not be excessive for respondents. The statistical
authority monitors the response burden and sets targets for its reduction over time.
Indicators
– The range and detail of European statistics demands is limited to what is absolutely
necessary.
– The reporting burden is spread as widely as possible over survey populations through
appropriate sampling techniques.
– The information sought from businesses is, as far as possible, readily available from their
accounts and electronic means are used where possible to facilitate its return.
– Best estimates and approximations are accepted when exact details are not readily
available.
– Administrative sources are used whenever possible to avoid duplicating requests for
information.
– Data sharing within statistical authorities is generalised in order to avoid multiplication of
surveys.
Principle 10: Cost Effectiveness - Resources must be effectively used.
Indicators
– Internal and independent external measures monitor the statistical authority’s use of
resources.
– Routine clerical operations (e.g. data capture, coding, validation) are automated to the
extent possible.
– The productivity potential of information and communications technology is being
optimised for data collection, processing and dissemination.
– Proactive efforts are being made to improve the statistical potential of administrative
records and avoid costly direct surveys.
Statistical Output Available statistics must meet users’ needs. Statistics comply with the European quality standards and
serve the needs of European institutions, governments, research institutions, business concerns and the
public generally. The important issues concern the extent to which the statistics are relevant, accurate
and reliable, timely, coherent, comparable across regions and countries, and readily accessible by
users.
Principle 11: Relevance - European statistics must meet the needs of users.
Indicators
– Processes are in place to consult users, monitor the relevance and practical utility of
existing statistics in meeting their needs, and advise on their emerging needs and priorities.
– Priority needs are being met and reflected in the work programme.
– User satisfaction surveys are undertaken periodically.
Principle 12: Accuracy and Reliability - European statistics must accurately and reliably portray
reality.
Indicators
– Source data, intermediate results and statistical outputs are assessed and validated.
– Sampling errors and non-sampling errors are measured and systematically documented
according to the framework of the ESS quality components.
Volume 1: Quality in Statistics 22
– Studies and analyses of revisions are carried out routinely and used internally to inform
statistical processes.
Principle 13: Timeliness and Punctuality - European statistics must be disseminated in a timely and
punctual manner.
Indicators
– Timeliness meets the highest European and international dissemination standards.
– A standard daily time is set for the release of European statistics.
– Periodicity of European statistics takes into account user requirements as much as
possible.
– Any divergence from the dissemination time schedule is publicised in advance, explained
and a new release date set.
– Preliminary results of acceptable aggregate quality can be disseminated when considered
useful.
Principle 14: Coherence and Comparability - European statistics should be consistent internally,
over time and comparable between regions and countries; it should be possible to combine and make
joint use of related data from different sources.
Indicators
– Statistics are internally coherent and consistent (e.g. arithmetic and accounting identities
observed).
– Statistics are coherent or reconcilable over a reasonable period of time.
– Statistics are compiled on the basis of common standards with respect to scope,
definitions, units and classifications in the different surveys and sources.
– Statistics from the different surveys and sources are compared and reconciled.
– Cross-national comparability of the data is ensured through periodical exchanges between
the European Statistical System and other statistical systems; methodological studies are
carried out in close co-operation between the Member States and Eurostat.
Principle 15: Accessibility and Clarity – European statistics should be presented in a clear and
understandable form, disseminated in a suitable and convenient manner, available and accessible on
an impartial basis with supporting metadata and guidance.
Indicators
– Statistics are presented in a form that facilitates proper interpretation and meaningful
comparisons.
– Dissemination services use modern information and communication technology and, if
appropriate, traditional hard copy.
– Custom-designed analyses are provided when feasible and are made public.
– Access to microdata can be allowed for research purposes. This access is subject to strict
protocols.
– Metadata are documented according to standardised metadata systems.
– Users are kept informed on the methodology of statistical processes and the quality of
statistical outputs with respect to the ESS quality criteria.
Volume 1: Quality in Statistics 23
B. UN Fundamental Principles of Official Statistics
Adopted by The United Nations Statistical Commission, in its Special Session of 11-15 April 1994.
Principle 1. Official statistics provide an indispensable element in the information system of a
society, serving the government, the economy and the public with data about the economic,
demographic, social and environmental situation. To this end, official statistics that meet the test of
practical utility are to be compiled and made available on an impartial basis by official statistical
agencies to honour citizens’ entitlement to public information.
Principle 2. To retain trust in official statistics, the statistical agencies need to decide according to
strictly professional considerations, including scientific principles and professional ethics, on the
methods and procedures for the collection, processing, storage and presentation of statistical data.
Principle 3. To facilitate a correct interpretation of the data, the statistical agencies are to present
information according to scientific standards on the sources, methods and procedures of the statistics.
Principle 4. The statistical agencies are entitled to comment on erroneous interpretation and misuse
of statistics.
Principle 5. Data for statistical purposes may be drawn from all types of sources, be they statistical
surveys or administrative records. Statistical agencies are to choose the source with regard to quality,
timeliness, costs and the burden on respondents.
Principle 6. Individual data collected by statistical agencies for statistical compilation, whether they
refer to natural or legal persons, are to be strictly confidential and used exclusively for statistical
purposes.
Principle 7. The laws, regulations and measures under which the statistical systems operate are to be
made public.
Principle 8. Coordination among statistical agencies within countries is essential to achieve
consistency and efficiency in the statistical system.
Principle 9. The use by statistical agencies in each country of international concepts, classifications
and methods promotes the consistency and efficiency of statistical systems at all official levels.
Principle 10. Bilateral and multilateral co-operation in statistics contributes to the improvement of
systems of official statistics in all countries.
Volume 1: Quality in Statistics 24
C. Recommendations of LEG on Quality
Recommendation no. 1: Each NSI should report product quality according to the ESS quality
dimensions and sub-dimensions.
Recommendation no. 2: The measurability of each ESS quality dimension and sub-dimension should
be improved.
Recommendation no. 3: Process measurements are vital for all improvement work. A handbook on
the identification of key process variables, their measurement, and measurement analysis should be
developed.
Recommendation no. 4: All organisations in the ESS should adopt a systematic approach to quality
improvement. ESS members should use the EFQM excellence model as a basis for their improvement
work except for those already using a similar model.
Recommendation no.5: NSIs should strive to improve their relationships with data suppliers, and
research should be conducted on how data suppliers perceive their task. A special emphasis should be
placed on issues that involve a decrease of the respondent burden and enhance suppliers’ awareness of
the role of statistics in society.
Recommendation no. 6: ESS members should develop service level agreements for their main
programmes.
Recommendation no. 7: A development project regarding the design, implementation and analysis of
customer satisfaction surveys should be initiated.
Recommendation no. 8: Each ESS member should provide a report regarding the present status of its
user – producer dialogue including descriptions of any user involvement in the planning process.
Good practices in promoting user awareness of quality problems should be collected and made
available to ESS members.
Recommendation no. 9: An in-depth analysis of the most important ESS strengths and weaknesses
should be conducted. An action programme should be developed based on the findings of this
analysis.
Recommendation no. 10: NSIs should develop CBMs for their most common processes. A handbook
for developing CBMs covering construction, dissemination, implementation and revision of CBMs
should be developed. Existing and relevant CBMs should be collected and distributed in the ESS.
Recommendation no. 11: A set of recommended practices for statistics production should be
developed. The work should start by developing recommended practices for a few areas followed by a
test of their feasibility in the ESS.
Recommendation no. 12: ESS members should use the list of current good information management
and dissemination practices compiled by the LEG and consider actions for internal use.
Recommendation no. 13: The user needs of the current ESS information system should be reviewed
and Eurostat’s current database expanded accordingly. Guidelines regarding the future management
of the information system should be developed.
Recommendation no. 14: A biennial conference covering any methodological and quality-related
topics of relevance to the ESS should be organised.
Recommendation no. 15: A generic checklist should be developed for a simple self-assessment
programme for survey managers in the ESS.
Recommendation no. 16: The methods for auditing on different levels and for different purposes
such as internal, external, one point in time, continuing or rolling, rapid, and more extensive (such as
EFQM assessment) should be reviewed and recommendations should be provided to the ESS.
Volume 1: Quality in Statistics 25
Recommendation no. 17: ESS members should study staff perception. One way to do this is to
conduct staff perception surveys.
Recommendation no. 18: ESS members should analyse their documentation status in a report. The
report should include an action plan with clear priorities for improvement and a timetable.
Recommendation no. 19: Each ESS member should make publicly available documents describing
its mission statement, dissemination policy and quality policy.
Recommendation no. 20: All staff should be trained in quality work with different types of training
programmes for different types of staff. Each ESS member should develop a training programme.
Training on a European level should be enhanced.
Recommendation no. 21: A biennial quality award in official statistics should be established. The
award could be given to a single improvement project team, for an innovative idea, to a well-
performing ESS organisation or to a statistical programme team.
Recommendation no. 22: There is a need to establish a LEG Implementation Group that coordinates
the activities generated by recommendations approved by the SPC.