Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
ECMWF COPERNICUS REPORT
Copernicus Climate Change Service
State of data rescue assessments
Issued by: University of Bern / Stefan Brönnimmann
Date: 03/01/2018
Ref: C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx
Official ref. number service contract: 2017/C3S_3S311a_Lot1_UBERN/SC1
This document has been produced in the context of the Copernicus Climate Change Service (C3S).
The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts,
operator of C3S on behalf of the European Union (Delegation Agreement signed on 11/11/2014). All information in this
document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose.
The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission
and the European Centre for Medium-Range Weather Forecasts has no liability in respect of this document, which is merely
representing the authors view.
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 3 of 25
Contributors
MET OFFICE 1. Paul van der Linden
2. Rob Allan
UNIVERSITY OF BERN 1. Stefan Brönnimmann
2. Yuri Brugnara
UNIVERSITY ROVIRA I VIRGILI 1. Manola Brunet
UNIVERSITY OF COLORADO 1. Gilbert Compo
IEDRO 1. Richard Crouthamel
UNIVERSITY OF EAST ANGLIA 1. Phil Jones
2. Clive Wilkinson
MÉTÉO-FRANCE 1. Sylvie Jourdain
UNIVERSITY OF GIESSEN 1. Jürg Luterbacher
KNMI 1. Peter Siegmund
UNIVERSITY OF LISBON 1. Maria Antonia Valente
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 4 of 25
Table of Contents
1. Introduction 5
2. State of data rescue assessments 6
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 5 of 25
1. Introduction
This document describes the current state of data rescue assessments, as prepared for the journal
Geoscience Data.
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 6 of 25
2. State of data rescue assessments
A roadmap to climate data rescue services (C3S Data Rescue Service: Deliverable
1.3.1-2017)
As prepared for Geoscience Data Journal
Abstract. The scientific environment surrounding climate change adaptation, mitigation and risk
management is evolving rapidly. Quantitative approaches such as risk mapping or impact modelling rely on
past meteorological data with daily or sub-daily resolution, a large fraction of which have never been
digitised. Here we argue that climate data rescue must be seen as a continuous, coordinated long-term
effort. Technical developments (e.g., data assimilation), new scientific questions (e.g., process understanding
of extreme events) and new social (e.g., risk assessment, health) or economic services (e.g., new renewable
energy sources, agriculture, tourism, etc.) render data valuable that hitherto were not considered so. This
continuous effort is currently undertaken by projects of various sizes, structure, funding and staffing, as well
as by dedicated programs within many national weather services. These activities are often not sufficiently
coordinated at international level and will benefit considerably from the data rescue services being
established within the Copernicus Climate Change Service (C3S) (https://climate.copernicus.eu/). In this
paper, we discuss the state of climate data rescue, the requirements for data rescue services, plus the
challenges, and opportunities presented to them.
1 Introduction
The past few years have seen considerable progress in our scientific understanding of climate change and
related consequences (IPCC, 2013). Adaptation and climate risk management are rapidly developing fields,
that increasingly use numerical models and rely on quantitative high resolution, high-quality and continuous
data sources. With this, comes the need for more and longer data series than are currently available. In fact,
the amount of information that could be made available, but only exists in hard copy form, is substantial. As
a consequence of the growing need for data to support climate change research, climate data rescue has
seen a marked revival over the past years. This resurgence has been a result of climate data rescue being
pursued as a community effort, whose success depends not only on the individual groups, projects and
initiatives involved, but on maintaining and growing a coherent and strong community to lead it forward.
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 7 of 25
In the following we would like to posit three premises that are discussed in the remainder of the paper.
(i) Climate data rescue should not be approached as a ’one-off’ task but seen as a continuous activity. The
increased demand for historical weather data is at least partly due to new techniques, which have permitted
the reconstruction of past global- to regional-scale weather patterns through the assimilation of such
observations in ways that were not previously possible. Data assimilation approaches allow making use of
only sparse input data to obtain useful new data products (Compo et al., 2006, 2011). Several historical
dynamical weather reconstructions or reanalyses have been produced in the last few years (Compo et al.,
2011; Poli et al., 2016, Laloyaux et al., 2017), demonstrating that large-scale weather reconstruction is
generally possible and can be improved by more observations from additional data rescue efforts. These long
historical reanalyses primarily rely on sub-daily surface pressure observations, which traditionally have
received less weight or interest by historical climatologists. This shows that new techniques (data
assimilation), new scientific questions (e.g., process understanding of extreme events) and new social (e.g.,
risk assessment) or economic issues (e.g., new renewable energy sources) render data valuable that hitherto
were not considered so (e.g., barograph strips). This requires going back to original archives and data
repositories, for the recovery, imaging and digitising of their relevant historical records as well as quality
control. Thus, considering meteorological and climate data rescue as a continuous long term effort that
changes with new techniques and new demands require an environment that can sustain continuity.
(ii) Although it is a global, continuous effort, it has been argued that data rescue is best performed by many
small, individual projects (see Brunet and Jones, 2011). Each of these projects would establish its own links to
archives (e.g., Veale et al., 2017), libraries, but also science and end users. At the same time, through close
collaborations among these groups, the networks are shared where possible. Also, many of the projects have
a local connection, which is important as information is often local-to-regional scale. The experience of the
last 10 or so years has also shown that a big, sustainable, multinational data rescue project funded by major
international bodies, such as the World Meteorological Organization (WMO), Global Framework for Climate
Services (GFCS), Global Climate Observing System (GCOS), Group on Earth Observations (GEO), Joint
Technical Commission for Oceanography and Marine Meteorology (JCOMM), and the like, is not likely to
happen. Nevertheless, what has been demonstrated by the Atmospheric Circulation Reconstructions over
the Earth (ACRE) initiative (Allan et al., 2011a; Allan et al., 2011b; Allan et al., 2016) is that an international
umbrella linking together various local, regional and global, projects, programs, institutions and organizations
into a coherent community, effectively ‘pooling’ its resources, personnel and funds, with various regional
data rescue foci under it (e.g. Williamson et al., 2015; Williamson et al., 2017a; Williamson et al., 2017b), is a
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 8 of 25
viable role model for ongoing data rescue activities. Climate data rescue therefore requires maintaining an
active, vibrant community.
(iii) To sustain such a community effort requires various undertakings at different levels. While at the level of
the individual research groups, serving aids such as best practice guides, metadatabases, and software tools
can save efforts and lead to better coherency, there is also the level of community at large, of national
weather services and funding bodies and of international organizations. Topics including scientific merit,
ownership and data policies concern all levels from the WMO down to the individual researcher.
In this paper, representing Deliverable 1.3.1-2017 of the Copernicus Climate Change Service 311a Lot1 for
Collection and Processing of In Situ Observations - Data Rescue (C3S Data Rescue Service), we provide a
roadmap for meteorological and climate data rescue services that addresses these three premises. A
roadmap for the storage and management of the meteorological data over land has been outlined elsewhere
(Thorne et al., 2017). For marine data, details of the latest release of the ongoing international marine
meteorological data repository, the International Comprehensive Ocean-Atmosphere DataSet (ICOADS)
version 3, are provided by Freeman et al. (2017).
We first discuss how climate data rescue currently works (an annex to this paper, which will be updated
annually, provides an inventory of global activities). Then we discuss the three premises specified above,
from which follow requirements for climate data rescue services, embedded within a coherent, ‘end-to-end’
IT infrastructure encompassing an international data portal, data registry, tools and techniques, that are now
being established by the C3S Data Rescue Service (DRS).
2 Climate Data Rescue
Climate data rescue mostly concerns data that pre-date the age of electronic data acquisition. In developing
countries, weather observations are still performed manually and require digitisation. Although important
terrestrial surface and upper air data series have been digitised and processed (e.g., Alcoforado et al., 2012;
Camuffo and Bertolin, 2012; Camuffo et al., 2013; Domínguez-Castro et al., 2014; Brunet et al., 2014;
Slonosky, 2014; Ashcroft et al., 2014; Stickler et al., 2014; to give only very few examples) a huge fraction has
still not yet been digitised. This also applies to the historical marine observations from exploration, naval,
postal, merchant and passenger ships.
Overall, the fraction of yet to be digitised data is difficult to quantify. Data sources include information
contained in the charts of thermographs, barographs, and the like (Fig. 1), while the great bulk of digitised
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 9 of 25
climate data, be it from marine or terrestrial sources, has been keyed from hard copy readings in one
tabulated form or another. Sources of historical weather observations from ship log books have almost
exclusively focused on the holdings from the major European colonial powers (United Kingdom, The
Netherlands, Spain, Germany - and are far from exhausted) and the United States. However, recent
discoveries in various archives and museums in Norway and Finland have recovered some 53K images of log
books, with at least another 10K log books from the latter country yet to be imaged. Archives in countries
such as Sweden and Denmark have yet to be ‘touched’, and those in France and Portugal are in need of
greater attention. Also a wide range of repositories, such as the Ottoman and Venetian archives, are as yet
little known to climate scientists (e.g., Luterbacher et al. 2012). It is very important to note that many of
these logs cover exploration and commercial/mercantile voyages into the ‘data sparse’ oceans of the South
Hemisphere middle to higher latitudes (see Figure 1 in Freeman et al., 2017). The above examples only begin
to illustrate the huge amount of non-digitised climate information that can be found in archives around the
world.
Most developed countries already have the infrastructure (organization of the archives, formation of the
personnel, etc.) to efficiently carry out a national data rescue program, although they rarely allocate enough
resources for a full-time implementation. In developing countries, the WMO and other organizations have
been funding numerous programs in recent years to reorganize archives, buy digitisation hardware and
software, and train the personnel, avoiding the loss of many documents to improper storage. WMO also
publishes guidelines for best practices in the digitisation of climate data (WMO, 2016) and supervises an
International Data Rescue (I-DARE) Portal (https://idare-portal.org) where actual and potential data rescue
activities can be reported (Siegmund, 2014).
https://idare-portal.org/
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 10 of 25
Fig. 1. Barograph strip chart from a French station. Strip charts are a precious but often neglected source of
high-resolution observations. Image credit: Météo-France.
2.1 Current status of data rescue
Who is doing the climate data rescue work presently? One of the tasks of the C3S Data Rescue Service is to
answer this question, in order to improve coordination between different projects and avoid repetition of
work. We launched an online survey in late 2017, targeted at the coordinators of data rescue projects, an
exercise that will be repeated every year to compile an inventory of activities as completely as possible. The
respondents were representative of all six WMO regions, although the majority (60%) were from Europe.
International meetings (including ACRE workshops), the scientific literature and the web were additional
sources of information for the inventory of data rescue activities in the year 2017, which are given in the
Appendix and will be used to update the WMO I-DARE portal and the C3S DRS portal.
Data rescue is usually approached on three different time scales: i) efforts of short duration (weeks or
months), where the data to be rescued is determined by the specific needs of one-off activities (such as an
undergraduate thesis or a research article) and carried out by a single person or research group; ii) projects
with limited duration, usually 1-3 years, that are funded by regional, national or international institutions
(such as the European Commission) to achieve pre-determined goals that are usually tightly linked with
research activities involving different institutes; iii) long-term endeavours, typically carried out by public
national/regional weather services or through voluntary work (e.g., citizen science) and international
‘grassroots’ data community initiatives (e.g., ACRE).
Projects and one-off activities can set priorities that better suit the needs of climate research and leave clear
traces of what is done by publishing reports or journal articles; moreover, in most cases the rescued data are
immediately made public without restrictions. However, their short durations mean the loss of specific
know-how that will cost time to be acquired by a new project. Long-term endeavours allow a more efficient
and thorough data rescue, but they can be hampered by lack of funds or personnel. Moreover, in the case of
national or regional weather services, it is sometimes difficult to obtain detailed information on what has
been done and the data may be subjected to limitations in their distribution. In fact, there exists a certain
amount of data that have already been digitised years or even decades ago, but which remains inaccessible
to most of the scientific community. In some cases data owners may not even be aware that their data can
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 11 of 25
be useful to others, or they do not know any platform where they can share them. It is not uncommon for
rescued data or metadata to be stored on private computers until they are eventually forgotten or lost
(particularly for one-off activities). This not only applies to keyed data. For instance, during the WMO DARE1
program in the 1990s, nearly 2 million pages of historical surface hydro-meteorological data from 48 African
countries was imaged and placed on nearly 100,000 microfiche sheets. These sheets, originally held by the
Belgian National Meteorological Service, were turned over to the African Centre of Meteorological
Application for Development (ACMAD; located in Niamey, Niger) in the mid-2000s to be held in safe keeping
until funding was available to digitize these imaged data. Unfortunately, although placing imagery onto
microfiche and microfilm was state-of-the-art technology at the time, scientists soon realized that unless
those microfiche were kept in environmentally controlled storage areas, the images would soon fade. For
more than a decade, these microfiche sheets have been exposed to temperatures approaching 40 degrees
Celsius. Although a digitisation project was eventually funded by the US Agency for International
Development in 2013, this allowed the scanning of only a small fraction of the microfiche sheets. At this
point in time, neither ACMAD nor the WMO is any closer to rescuing and digitising these historic surface data
than when the DARE1 program started. This example also shows how data rescue can be negatively affected
by the postponement of funding, as paper and film degrade over time. An issue with this project is that many
of the African countries destroyed the original paper records, as they were told that the information was
now safe.
Nevertheless, several data rescue activities have delivered, and continue to deliver, data to the scientific
community. A new focus for the data rescue community has seen the linking together of historical
climatologists with impact researchers, (e.g., Allan et al., 2011a; Allan et al., 2011b; Allan et al., 2016;
Williamson et al., 2015; Williamson et al., 2017a; Williamson et al., 2017b), at the centre of which is the
previously mentioned international ACRE initiative (Fig. 2). ACRE undertakes and facilitates historical global
surface terrestrial and marine weather data recovery, imaging and digitisation, feeding these data into open-
access international repositories responsible for such material, seeing that they provide the best quality and
quantity of surface weather observations for assimilation into all reanalyses, and ensuring that reanalyses
outputs are freely available and feed seamlessly into the climate science, climate applications and services,
impacts, risks and extremes communities.
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 12 of 25
Fig. 2. The interplay between the international ACRE initiative (including the locations of its various annual
international workshops), linked data rescue projects, and C3S DRS regional data rescue activities.
2.2 What is rescued?
Most data rescue activities are focused on data from surface stations on land (Fig. 3, top). This is the only
category that can provide long (centennial) records of the same geographical location and has, therefore,
obvious value in climate change research. Marine data rescue is important to improve reanalyses and can
provide critical observations for remote regions or where land data are insufficient (tropics, southern
hemisphere, polar regions), particularly in early colonial times. Reanalyses also benefit from upper-air data
(Stickler et al., 2014), which is also instrumental in understanding climate change (e.g., Thorne et al., 2011;
Seidel et al., 2011). In the polar regions, observations involving ice surfaces and extents are also essential
(e.g., Turner and Comiso, 2017).
Temperature, precipitation and pressure are the three core variables in climate data rescue (Fig. 3, middle).
While temperature and precipitation are often directly analysed, pressure is usually assimilated into
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 13 of 25
reanalyses or used for other types of weather reconstruction (e.g., Luterbacher et al., 2002; Schwander et al.,
2017). Other variables, previously considered “minor”, are now also being rescued regularly. Wind and
radiation, in particular, have received increasingly high attention over the last decade in response to the
observed “global stilling” (Vautard et al., 2010) and “global brightening” (Wild, 2012) phenomena. These
variables are also particularly important to the emerging renewable energy sector (see e.g. C3S ECEM and
C4ENERGY, http://climate.copernicus.eu/ecem-european-climatic-energy-mixes,
http://climate.copernicus.eu/clim4energy-service-providing-climate-change-indicators-tailored-energy-
sector).
Awareness of the importance of the single observations is widespread in the scientific community, therefore
data rescue is no longer limited to daily or monthly means (Fig. 3, bottom). One third of the respondents to
the survey were dealing with hourly or finer resolutions (e.g., thermographs).
The survey also included an open question on what assistance is needed. Nearly all of the respondents asked
for resources for data keying. In fact, many data rescue projects can only afford the imaging of documents
and leave the transcription of the data to future initiatives. It is easy to see why citizen science and
volunteers play an increasingly important role for climate data rescue.
http://climate.copernicus.eu/ecem-european-climatic-energy-mixeshttp://climate.copernicus.eu/clim4energy-service-providing-climate-change-indicators-tailored-energy-sectorhttp://climate.copernicus.eu/clim4energy-service-providing-climate-change-indicators-tailored-energy-sector
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 14 of 25
Fig. 3. Survey results (based on 30 responses).
3 Premises on climate data rescue
(i) Climate Data Rescue as a Continuous Effort
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 15 of 25
Climate data have been rescued and compiled in many efforts over the years. Already early 19th century
scientists such as Alexander von Humboldt, Wilhelm Brandes, Heinrich Dove, Matthew Maury, James Pollard
Espy and others tried to compile global meteorological data sets (e.g., Dove, 1844). Their goals were to
establish global climatologies of temperature, weather conditions over the oceans and air pressure. Later,
scientists such as Felix Exner and Sir Gilbert Walker compiled global data sets in order to study typical
variations on a seasonal and multi-seasonal scales. In the 20th century, data were compiled to study climate
trends, fluctuations and changes.
While these efforts built typically on previous work, this is not always the case. For instance, none of these
past applications required sub-daily surface pressure data, which today is the most important input for long
reanalyses (Compo et al., 2006). Similarly, in the past, monthly or seasonal values could be used to respond
to most research questions, while nowadays extreme events have a greater priority, implying a need for daily
or even hourly values. Therefore, archive work is still necessary to provide the input. Perhaps future long
reanalysis efforts will make use of wind data over land, cloud motion, or rainfall data. In addition to
anticipating the future needs, we should establish an environment in which returning to the archive is made
easy. Likewise, some variables may contain important information on other properties of interest, such as
aerosol information that can possibly be derived from sunshine duration records (Sanchez-Romero et al.,
2014). The numerous spectroscopic data (solar and stellar spectra) that were obtained in the early 20th
century might eventually also be used for atmospheric science, once suitable methods have been developed,
but digitising the data now would be extremely costly. Thus, technical changes (new methods) render data
valuable that were not hitherto seen as so.
These technical changes are sometimes related to scientific questions, for which historical information is
desired (e.g., the role of aerosols or clouds, for the examples above). Of course, scientific questions (such as
improving process understanding of extreme events) also steer the demand for data products. The scientific
demand is also related closely to societal demands. The interest in new renewable energy sources might also
trigger interest in wind and cloud cover information and in sunshine data records, which may not have been
digitised in previous efforts. Going back to the archives or to the imaged data sheets will often be necessary,
as not all future needs can be anticipated. Thus, climate data rescue must be seen as a continuous long term
effort that changes with new techniques and new demands.
(ii) A Distributed Approach to Climate Data Rescue
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 16 of 25
Even if demand and techniques change, would it not make sense to raise funds (millions) and do it “once and
for all”? Unlike launching a satellite of installing a super-computer, data rescue is labour intensive. It requires
training many persons, some of whom would have to come from other fields and whose knowledge would
be lost afterwards. Large masses of data can be rescued in citizen science projects, especially if they are well
defined, such as oldweather.org (Fig. 4). This is suitable for some of the sources that are well structured.
However, many other sources require dedicated scientific work and are not suitable for such efforts.
More importantly, even if the digitising aspect (keying, OCR) is done efficiently, the preceding (imaging,
preparing for online data rescue) as well as the following steps (controlling, documenting, formatting, etc.)
are typically more time consuming and require expertise that is beyond most the scope of citizen scientists.
Data rescue comprises a much wider range of task skills than scanning millions of pages and typing numbers
from sheets.
Online digital archives offered by an increasing number of institutions, from the local science societies to the
large libraries facilitate the access to manuscript climate data, and the trend towards digital humanities will
further allow exploiting synergies (use of data bases etc.). These are places and institutions where “once and
for all” efforts are undertaken.
Our inventory shows that data rescue projects have very different sizes and targets; they are driven by
specific data needs. Therefore, weather and climate data rescue is a distributed approach, which however
makes use of citizen science approaches, imaged library holdings and data bases of digital humanities. Not
only is climate data rescue dealing with spatial data, which often requires knowledge on the location, access
to a local archive, but also the expertise and knowledge is distributed.
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 17 of 25
Fig. 4. Screenshot of oldweather.org (new Whaling Chapter). Citizen science projects such as oldweather.org
have provided millions of valuable ship data, but cannot “do the job”. Only together with research projects,
efforts of national weather services and projects supported by development agencies and foundations can
the body of climate data be rescued.
(iii) Efforts must target projects, the scientific community, and science administration
For a distributed approach to work, it must not only be beneficial to society, but also attractive for scientists.
For individual research project to engage in weather and climate data rescue, the threshold must be low. For
the community it is important that the individual projects are guided towards delivering data in a common
format to data repositories. The C3S Data Rescue Service plays an important role in both aspects, as is
outlined in the following Section.
Other factors that are important to make data rescue attractive for scientists, but are beyond the control of
C3S DRS, concern the community at large, national weather services, funding bodies and international
organizations. Here, the C3S Data Rescue Service linkages to ACRE and its associations with WMO, GCOS,
JCOMM and the like will need to come into play.
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 18 of 25
Science requires an open, traceable data policy. During the past ten to twenty years, a lot has been achieved
towards this aim. Data sets can obtain digital object identifiers and funding bodies are more willing to fund
data rescue projects than in years past. In the last few years, several data journals have appeared that cater
for the needs of the data rescue (and other) communities. Most of them require the data to be deposited at
an approved public repository, with a digital object identifier. All of them are Open Access. Efforts on
weather and climate data rescue are still often published in regular meteorological and climatological
journals, which however mostly do not have a specific data policy. The fact that data journals have impact
factors that are similar or even higher than that of climate journals can be taken as an indication that the
merit of data rescue is higher than a decade ago.
Data policies do, of course, not only concern the journal, but national policies as well as WMO
recommendations. At the Seventeenth World Meteorological Congress in Geneva 2015, the WMO adopted
its policy for the international exchange of climate data and products to support the implementation of the
GFCS as Resolution 60. It states that GFCS relevant data and products from specified sources (such as World
Data Centres) are made accessible among members on a free and unrestricted basis.
4 Climate Data Rescue Services
The functioning of the C3S Data Rescue Service from a user perspective is shown in Fig. 5. On every step from
the idea of a project to data submission, C3S DRS provides support, ranging from data bases, to software
user support, community building, best practice reports, and capacity building. The first steps are data bases
with existing projects (I-DARE) and metadata. Here, metadatabases, where crucial information can be
retrieved (existing metadata on the series, links to archive holdings, contact persons, in the best cases links
to already imaged data sheets). The annual overview of data rescue activities is another example; it might
help to trigger or better embed activities.
Once digitising is undertaken, several tools may facilitate the work surrounding the digitising process.
Researchers should find answers to specific technical questions, e.g. how to digitize thermograph, barograph
or pluviograph charts. Software tools that can help in every phase of digitisation, from file naming of the
images to the quality control of data and metadata, will be provided on a free license. Projects targeting
working with volunteers will find outreach material helpful. Moreover, an online facility for data keying will
be developed. Equally important are capacity building workshops, since data rescue projects often target
developing countries. Following best practice guidelines may save time and efforts and at the same time
might contribute to increasing consistency across projects.
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 19 of 25
In addition to documents, the C3S Data Rescue Service in conjunction with ACRE provides a network of
interested scientists. Projects can then easily be embedded within an international community, with
communication (blogs, newsletters), workshops, session at important meetings etc. Most important in this
context are links to archives, museums and libraries, interdisciplinary collaborations, and links to end users in
order to anticipate future needs. In this way, climate data rescue services can make it attractive for a project
to engage more widely, and the project in turn enlarges the expertise of the community, and builds
knowledge and ‘confidence’ in weather and climate disciplines and their wider value to society.
Apart from the political side of data policies, the scientific side and practical aspects of data management
need consideration. Climate data rescue services need to be closely linked with data holdings (e.g., Menne et
al., 2012; Rennie et al., 2014; Cram et al., 2015; Freeman et al., 2017) and global repositories within C3S
(Thorne et al., 2017) and offer these links as services to projects. In that respect, the C3S Data Rescue Service
should channel data rescue projects towards common data holdings.
Fig. 5. Schematic that summarizes all phases of a data rescue project, together with the assistance that will
be provided by the C3S Data Rescue Service (red background).
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 20 of 25
Conclusions
Historical weather and climate data are relevant for climate sciences as well as for society in the context of
climate change adaptation and mitigation, and risk assessment. Large amounts of meteorological data that
could be helpful have not yet been digitised. To promote data rescue, climate data rescue services are now
being developed under C3S. In this paper we outline a roadmap towards successful climate data rescue
services based on three premises, from which follow requirements for such a service. We posit that data
rescue must be seen as a continuous, long-term activity. Changes in the demand (e.g., for analysing past
extreme events with respect to future changes in extremes) and changes in the technical possibilities (e.g.,
use surface pressure to obtain a complete description of the atmospheric state at high temporal and spatial
scale) lead to a re-valuation of historical data and thus the potential value of rescued data. Such changes will
also occur in the future: Demand will change (e.g., focus of solar and wind data for planning renewable
energy generation, or stronger focus on climate impact data) and technical possibilities will change (e.g.,
assimilation of further variables). We also postulate that a distributed approach is most suitable to the
problem of rescuing climate data, and hence the task will be shared by many research projects and efforts
from weather services and other institutions. Society is thus best served with an active data rescue
community that provides expertise on procedures, infrastructure, tools, and metadatabases. Furthermore, to
maintain this community, ensuring scientific merit and an open data policy will make it attractive for
research projects to engage in the process. Data rescue services operate as facilitators for individual projects
and collectively push towards these aims.
Over the last years, data rescue efforts have led to new data sets, and the importance of data rescue work
has been more and more appreciated. These efforts must be sustained, and climate data rescue services
under the C3S Data Rescue Service will contribute.
Acknowledgements. The work was founded by the EC via Copernicus Climate Change. In addition to being
supported by funding from the EU Copernicus Climate Change Service (C3S), Prof. Rob Allan also
acknowledges the University of Southern Queensland, Toowoomba, Australia and the Centre for Maritime
Historical Studies, University of Exeter, UK where he is an Adjunct and Honorary Professor respectively.
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 21 of 25
References:
Alcoforado MJ, Vaquero JM, Trigo RM, Taborda JP. 2012. Early Portuguese meteorological measurements
(18th century). Climate of the Past 8: 353–371, doi: 10.5194/cp-8-353-2012.
Allan R, Brohan P, Compo GP, Stone R, Luterbacher J, Brönnimann S. 2011a. The international Atmospheric
Circulation Reconstructions over the Earth (ACRE) initiative. Bulletin of the American Meteorological Society
92: 1421–1425.
Allan R, Compo G, Carton J. 2011b. Recovery of global surface weather observations for historical reanalyses
and international users. Eos, Transactions, American Geophysical Union 92: 154,
doi:10.1029/2011EO180008.
Allan R, Endfield G, Damodaran V, Adamson G, Hannaford M, Carroll F, Macdonald N, Groom N, Jones J,
Williamson F, Hendy E. 2016. Toward integrated historical climate research: the example of Atmospheric
Circulation Reconstructions over the Earth. Wiley Interdisciplinary Reviews: Climate Change 7: 164–174.
Ashcroft L, Gergis J, Karoly DJ. 2014. A historical climate dataset for southeastern Australia, 1788–1859.
Geoscience Data Journal 1: 158–178, doi:10.1002/gdj3.19
Brunet M, Jones P. 2011. Data rescue initiatives: bringing historical climate data into the 21st century.
Climate Research 47: 29–40, doi:10.3354/cr00960.
Brunet M, Gilabert A, Jones P, Efthymiadis D. 2014. A historical surface climate dataset from station
observations in Mediterranean North Africa and Middle East areas. Geoscience Data Journal 1: 121–128,
doi:10.1002/gdj3.12.
Camuffo D, Bertolin C. 2012. The earliest temperature observations in the world: the Medici Network (1654–
1670). Climatic Change 111: 335–363.
Camuffo D, Bertolin C, Diodato N, Cocheo C, Barriendos M, Dominguez-Castro F, Garnier E, Alcoforado MJ,
Nunes MF. 2013. Western Mediterranean precipitation over the last 300 years from instrumental
observations. Climatic Change 117: 85–101.
Compo GP, Whitaker JS, Sardeshmukh PD. 2006. Feasibility of a 100-year reanalysis using only surface
pressure data. Bulletin of the American Meteorological Society 87: 175–190.
Compo GP, Whitaker JS, Sardeshmukh PD, Matsui N, Allan RJ, Yin X, Gleason BE, Vose RS, Rutledge G,
Bessemoulin P, Brönnimann S, Brunet M, Crouthamel RI, Grant AN, Groisman PY, Jones PD, Kruk M, Kruger
AC, Marshall GJ, Maugeri M, Mok HY, Nordli Ø, Ross TF, Trigo RM, Wang X, Woodruff SD, Worley SJ. 2011.
The Twentieth Century Reanalysis Project. Quarterly Journal of the Royal Meteorological Society 137: 1–28.
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 22 of 25
Cram TA, Compo GP, Yin X, Allan RJ, McColl C, Vose RS, Whitaker JS, Matsui N, Ashcroft L, Auchmann R,
Bessemoulin P, Brandsma T, Brohan P, Brunet M, Comeaux J, Crouthamel R, Gleason BE, Groisman PY,
Hersbach H, Jones PD, Jonsson T, Jourdain S, Kelly G, Knapp KR, Kruger A, Kubota H, Lentini G, Lorrey A, Lott
N, Lubker SJ, Luterbacher J, Marshall GJ, Maugeri M, Mock CJ, Mok HY, Nordli O, Rodwell MJ, Ross TF,
Schuster D, Srnec L, Valente MA, Vizi Z, Wang XL, Westcott N, Woollen JS, Worley SJ. 2015. The International
Surface Pressure Databank version 2. Geoscience Data Journal 2: 31–46, doi: 10.1002/gdj3.25.
Domínguez-Castro F, Vaquero JM, Rodrigo FS, Farrona AMM, Gallego MC, García-Herrera R, Barriendos M,
Sanchez-Lorenzo A. 2014. Early Spanish meteorological records (1780–1850). International Journal of
Climatology 34: 593–603.
Dove, HW. 1844. Über die nicht periodischen Änderungen der Temperaturverteilung auf der Oberfläche der
Erde im Zeitraume von 1782 bis 1842. Reimer: Berlin, Germany.
Freeman E, Woodruff SD, Worley SJ, Lubker SJ, Kent EC, Angel WE, Berry DI, Brohan P, Eastman R, Gates L,
Gloeden W, Ji Z, Lawrimore J, Rayner NA, Rosenhagen G, Smith SR. 2017. ICOADS Release 3.0: a major
update to the historical marine climate record. International Journal of Climatology 37: 2211–2232.
IPCC. 2013. Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth
Assessment Report of the Intergovernmental Panel on Climate Change, Stocker TF, Qin D, Plattner G-K,
Tignor M, Allen SK, Boschung J, Nauels A, Xia Y, Bex V, Midgley PM (eds.). Cambridge University Press:
Cambridge, UK and New York, NY, USA; 1535 pp.
Laloyaux L, De Boisseson E, Dahlgren P. 2017. CERA-20C: An Earth system approach to climate reanalysis.
ECMWF Newsletter 150: 25–30.
Luterbacher J, Xoplaki E, Dietrich D, Rickli R, Jacobeit J, Beck C, Gyalistras D, Schmutz C, Wanner H. 2002.
Reconstruction of sea level pressure fields over the Eastern North Atlantic and Europe back to 1500. Climate
Dynamics 18: 545–561.
Luterbacher J, García-Herrera R, Akcer-On S, Allan R, Alvarez-Castro MC, Benito G, Booth J, Büntgen U,
Cagatay N, Colombaroli D, Davis B, Esper J, Felis T, Fleitmann D, Frank D, Gallego D, Garcia-Bustamante E,
Glaser R, González-Rouco JF, Goosse H, Kiefer T, Macklin MG, Manning S, Montagna P, Newman L, Power
MJ, Rath V, Ribera P, Riemann D, Roberts N, Sicre M, Silenzi S, Tinner W, Valero-Garces B, van der Schrier
G, Tzedakis C, Vannière B, Vogt S, Wanner H, Werner JP, Willett G, Williams MH, Xoplaki E, Zerefos CS,
Zorita E. 2012. A review of 2000 years of paleoclimatic evidence in the Mediterranean. In: The Climate of
the Mediterranean region: from the past to the future, Lionello P (Ed.). Elsevier: Amsterdam, The
Netherlands; 87–185.
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 23 of 25
Menne MJ, Durre I, Vose RS, Gleason BE, Houston TG. 2012. An overview of the global historical climatology
network-daily database. Journal of Atmospheric and Oceanic Technology 29: 897–910.
Poli P, Hersbach H, Dee DP, Berrisford P, Simmons AJ, Vitart F, Laloyaux P, Tan DG, Peubey C, Thépaut JN,
Trémolet Y. 2016. ERA-20C: An atmospheric reanalysis of the twentieth century. Journal of Climate 29:
4083–4097.
Rennie JJ, Lawrimore JH, Gleason B, Thorne PW, Morice CM, Menne MJ, Williams CN, Gambi de Almeida W,
Christy J, Flannery M, Ishihara K, Kamiguchi K, Klein-Tank AMG, Mhanda A, Lister D, Razuvaev V, Renom M,
Rusticucci M, Tandy J, Worley S, Venema V, Angel W, Brunet M, Dattore B, Diamond H, Lazzara M, Le Blancq
F, Luterbacher J, Mächel H, Revadekar J, Vose R, Yin X. 2014. The international surface temperature
initiative global land surface databank: monthly temperature data release description and methods,
Geoscience Data Journal 1: 75–102, doi:10.1002/gdj3.8.
Sanchez-Romero A, Sanchez-Lorenzo A, Calbó J, González JA, Azorin-Molina C. 2014. The signal of aerosol-
induced changes in sunshine duration records: A review of the evidence, Journal of Geophysical Research
119: 4657–4673.
Schwander M, Brönnimann S, Delaygue G, Rohrer M, Auchmann R, Brugnara Y. 2017. Reconstruction of
Central European daily weather types back to 1763. International Journal of Climatology 37: 30–44.
Seidel DJ, Gillett NP, Lanzante JR, Shine KP, Thorne PW. 2011. Stratospheric temperature trends: Our
evolving understanding. Wiley Interdisciplinary Reviews: Climate Change 2: 592–616.
Siegmund P. 2014. ‘White paper on an International Data Rescue Portal (I-DARE)’. 16pp. WMO: Geneva,
Switzerland.
Slonosky V. 2014. Historical climate observations in Canada: 18th and 19th century daily temperature from
the St. Lawrence Valley, Quebec. Geoscience Data Journal 1: 103–120, doi:10.1002/gdj3.11.
Stickler A, Brönnimann S, Valente MA, Bethke J, Sterin A, Jourdain S, Roucaute E, Vasquez MV, Reyes DA,
Allan R, Dee D. 2014. ERA-CLIM: Historical Surface and Upper-Air Data for Future Reanalyses. Bulletin of the
American Meteorological Society 95: 1419–1430.
Thorne PW, Lanzante JR, Peterson TC, Seidel DJ, Shine KP. 2011. Tropospheric temperature trends: history of
an ongoing controversy. Wiley Interdisciplinary Reviews: Climate Change 2: 66–88.
Thorne PW, Allan RJ, Ashcroft L, Brohan P, Dunn RJH, Menne MJ, Pearce PR, Picas J, Willett KM, Benoy M,
Brönnimann S, Canziani PO, Coll J, Crouthamel R, Compo GP, Cuppett D, Curley M, Duffy C, Gillespie I,
Guijarro J, Jourdain S, Kent EC, Kubota H, Legg TP, Li Q, Matsumoto J, Murphy C, Rayner NA, Rennie JJ,
Rustemeier E, Slivinski LC, Slonosky V, Squintu A, Tinz B, Valente MA, Walsh S, Wang XL, Westcott N, Wood
K, Woodruff SD, Worley SJ. 2017. Towards an integrated set of surface meteorological observations for
Copernicus Climate Change Service
C3S_DC3S311a_Lot1.3.1_2017_State of data rescue assessments_v2.docx Page 24 of 25
climate science and applications. Bulletin of the American Meteorological Society (early online), doi:
10.1175/BAMS-D-16-0165.1.
Turner J, Comiso J. 2017. Solve Antarctica’s sea-ice puzzle. Nature 547: 275–277.
Vautard R, Cattiaux J, Yiou P, Thépaut JN, Ciais P. 2010. Northern Hemisphere atmospheric stilling partly
attributed to an increase in surface roughness. Nature Geoscience 3: 756–761.
Veale L, Endfield G, Davies S, Macdonald N, Naylor S, Royer M-J, Bowen J, Tyler-Jones R, Jones C. 2017.
Dealing with the deluge of historical weather data: the example of the TEMPEST database. Geo: Geography
and Environment 4: e00039, doi: 10.1002/geo2.39.
Wild M. 2012. Enlightening global dimming and brightening. Bulletin of the American Meteorological Society
93: 27–37.
Williamson F, Allan R, Switzer AD, Chan JCL, Wasson RJ, D'Arrigo R, Gartner R. 2015. New Directions in
Hydro-Climatic Histories: Observational Data Recovery, Proxy Records and the Atmospheric Circulation
Reconstructions over the Earth (ACRE) initiative in Southeast Asia. Geoscience Letters 2: 2,
doi:10.1186/s40562-015-0018-z.
Williamson F, Ren G, Allan R. 2017a. The Atmospheric Circulation Reconstructions over the Earth (ACRE)
initiative ACRE China workshop: Recovery, digitisation, and analysis of pre-mid-twentieth century climate
observational data in East Asia workshop on 23–24 August, Beijing, China. Earth and Space Science 4: 40–
43.
Williamson F, Allan R, Ren G, Lee TC, Lui WH, Kubota H, Matsumoto J, Luterbacher L, Wilkinson C, Wood K.
2017b. Collating Historic Weather Observations for the East Asian Region: Challenges, Solutions, and
Reanalyses. Advances in Atmospheric Sciences (Special CSSP China Issue), in press.
WMO. 2016. Guidelines on Best Practices for Climate Data Rescue. WMO-No. 1182, 30pp. WMO: Geneva,
Switzerland.
Copernicus Climate Change Service
climate.copernicus.eu copernicus.eu ecmwf.int
ECMWF - Shinfield Park, Reading RG2 9AX, UK
Contact: [email protected]
1. Introduction2. State of data rescue assessments