11
NOAA’s Climate Database Modernization Program- Data Rescue Activities
Program Manager- Tom Ross
22
Climate Database Modernization Program
• ←
The Climate Database Modernization Program (CDMP) goal is to preserve and make major climate and environmental data available via the World Wide Web
The CDMP supports NOAA’s data stewardship through all of its activities
33
Climate Data Base ModernizationOutcomes & Benefits to the Nation
Outcomes• Improved access and digital conversion of data stored on aging paper, film & obsolete digital media• Provides partnerships and jobs through private sector companies in 4 states
Benefits to the NationHuman Dimension: Reduced loss of life & impacts due to environmental events
Business Dimension: Enhanced opportunities and socio-economic gains by considering climate information in decisions
Policy Dimension: More effective government policy by appropriately using climate information in decisions
Tenth year of the program (began 2000)
44
Over 11 terabytes of climate data now digitized‣ 53 million weather and environmental images online‣ Hundreds of millions of records digitized now online ‣ International data access and rescue activities‣ 86 current NOAA climate/environmental rescue projects
July 1st, 1842 hourly weather data from Washington, DC, imaged and
digitized through the CDMP Program
Imaged Records Example:Glacial Pairs – Muir Glacier, Alaska
1941 2004
Climate Database Modernization Program: 2000-2009Climate Database Modernization Program: 2000-2009
Keying and Imaging the data increases data accessibility and data integration- Work must be done by CDMP contractors in KY, MD, WV
55
CDMP Proposal Process has NOAA Working Together on Data Rescue and Recovery
The proposals judged against the following criteria:
1. Supports NOAA’s Strategic Goals2. Contribution to improved data access & rescue3. Value to climate community4 General merit to overall program5. Cost effectiveness6. Ease of digitization by the contractors
Call for papers- Issued Each July/AugustData Access Workshop – Held November each Year
60 Proposals were submitted for FY 2009, 53 in 2006 but delayed until FY 2008 funding
Climate Database Modernization Program
66
NCDC ArchivesPaper Archives Room
77
CDMP Digitization Partners
IMC handles paper scanning, COOP scanning from microfiche, and priority scanning
HOV Services handles bound records, publications, and fragile records
SourceCorp handles incoming records, COOP/SAO keying projects
88
35mm & 16mm Film 125,129 Rolls
Microfiche860K fiche containing 51 million pages
Manuscript / Autograph*55 Million Pages stored in 120,000 boxes
* Located at Asheville; additional paper records located at the Federal Records Center in Georgia that will be inventoried and prioritized for digitization
Percent digitized
(Keyed or imaged)
50%(55 million)
1.7%(2,105 reels)
1.0%(8,600 fiche)
Non-Digital Data Archive
99
Contents of the Archives: Micrographics
1010
Fragile, bound document scanningCDMP contractors have specialized equipment to scan difficult media while preserving its physical state.
1111
Large Document ScanningIn addition to standard document sizes, contractors can also handle large paper records, such as maps and charts.
1212
What’s being digitized at present?“High Volume” Projects, 2009
SWO daily forms: 897,961 images 3,592 per day
SWO 1001 monthly forms: 19,368
documents 77 per day
COOP microfiche: 369,625 images 1,479 per day
1313
Typical Camera and Imaging Setup for International Projects
High tech digital cameras and camera stands are set up in a well lit environment- sponsored by NOAA/NWS
Technicians are trained to image the records using the camera equipment they check for quality, exposure and completeness
Digital jpeg images are stored in the camera’s hard drive then transferred to the computer
Images on the computer are then written to CD-ROM or DVD and sent to NCDC for storage and keying
Keying format developed at NCDC based on data captured from host country
Data must be keyed in U.S. by CDMP contractors
End to end process can take 1-3 years depending on complexity and amount of data to be imaged/keyed
A technical leader responsible for each step in the process at each international location is critical !
International Activities
Climate Database Modernization Program
1414
NOAA & the Uruguayan Meteorological Service have entered into a cooperative project on data rescue In coordination with the U.S. Embassy. The U.S. National Oceanic and Atmospheric Administration (NOAA) donated digital equipment used for the project.
72,000 images received from the Uruguayan Meteorological Service and the Uruguayan Navy
7 synoptic stations (4 civilian and 3 military) keyed with observations as early as the 1930’s up to 2005 for a total of 9 million keystrokes.
Total imaging/keying for on-going project involves data from over 30+ stations totaling nearly 1,000,000 records
Keying data for Punta Arenas, Chile data record begins in 1896
Integrate data into NCDC global databases
International Activities-Uruguay and Chile 2004-2008
Climate Database Modernization Program
1515
CDMP African Upper Air (Pibal) Project
32,000
30,000
12,600
5,400
51,500
3,500 20,000Kenya
Malawi
Mozambique
Niger
Senegal
Tanzania
Zambia
Over 150,000 records imaged under CDMP
1616
Making Data User Friendly and Accessible
The index on the CD-ROM will be the main point for searching for related data.
The index page provides links to all available data types (both images of recorded data and actual keyed data) associated with the CD-ROM
• Station:67423 Day/Mo/Year Time:07/11/1970 04Z• Keyed Image ID: CDMP06MA\MA0001\0009.jpg• Source Image ID: .\MA0001\1\P6160006.jpg • • COMPUTED OBSERVED• TIME HGHT Azimuth Elev DIR SPD DIR SPD• MINS FEET DEG DEG DEG KTS DEG KTS
• > 0 0 130.0 5.0• 1 500 327.2 21.2 147 13 0 0• 2 1000 337.9 22.4 170 12 0 0• 3 1500 239.7 24.9 165 8 0 0 • 4 2000 342.7 26.1 174 9 0 0• 5 2500 337.4 25.5 140 12 0 0• Note: > Marks the Beginning of each Record in the Launch
• The highlighted .jpg has a link connecting it to the original image from which the data was keyed
The data above is an upper air pibal, due to the limited wind data available a keying format was developed to key the azimuth/elevation data along with the surface wind data, found in the header section. The wind data, to the middle right of the image, we found, was often determined by the average of multiple levels of wind data.
1717
Climate Database Modernization Program
1818
Climate Database Modernization Program
One of the earliest forms with detailed hourly data
1919
Selected NOAA ProjectsClimate Database Modernization Program
Hurricane Celia 1962- Hurricane Wallet Project
2020
Electronic Hurricane Wallet Project
Tropical Cyclone 'Storm Wallet' Electronic Archive
After the dissipation of every tropical cyclone occurring in the Atlantic and eastern north Pacific basins, all of the data and relevant materials related to that cyclone are collected by the NHC staff. The materials are placed in a "storm wallet" which currently takes the form of an expandable binder, or series of binders. These storm wallets have proven to be extremely useful in the post-analysis of many tropical cyclones, both near-term and in some cases, decades later.
The procedure for storing this data dates back to well before the routine use of computers in the office environment. In the Atlantic, the wallet series begins in 1958 and proceeds continuously through the present. In the eastern north Pacific, wallets begin in 1988, the year in which operational responsibility for that basin was assumed by NHC.
NHC Central Pacific is developing the same “Storm Wallet” archive
2121
Electronic Hurricane Wallet ProjectHurricane Camille 1969
2222
Typhoon/Hurricane Database Keying task
New Delhi Tropical Cyclone Keying Project
2323
Typhoon/Hurricane Database Keying task
Tropical cyclone best track position and intensity from New Delhi, India need to be keyed. Photocopies will be provided of data provided by New Delhi. The data are primarily made up of tables of cyclone position and intensity.
Format
The data are requested in a spreadsheet format (e.g., MS Excel). This format was chosen since the data originates as tables. The data should be on one worksheet beginning with the earliest data and working chronologically downward with one blank row between cyclones. The column names are only required to be present once at the top of the worksheet.
Columns
The following columns are requested. They are present in most tables. In the event they are absent, the column should be left blank. It is imperative that the data for each storm have the same columns to facilitate inputting the data into analysis programs. The columns are:
Date
Time (UTC)
Latitude (lat.)
Longitude (lon.)
CI No [or “T. No.”]
Central Pressure (hPa) [or “E.C.P.”]
Maximum Surface Wind (kt) [or “E.M.S.” or “E. Max”]
Pressure Drop (hPa) [ or “Δ P” ]
Grade (text)
Outermost closed isobar (hPa)
Diameter / Size of outermost close isobar
Remarks
2424
Typhoon/Hurricane Database Keying task
Caveats
Times should be made consistent as HHMM (e.g., some only report the hour (18) which should be adjusted by zero-filling (1800)).
Landfall is often noted by text in the table taking up an entire line. When keying, this should be moved to the “Remarks” column for the time referenced.
Example: “The system crossed the Orissa coast … around 1500 UTC” should be put in the remarks for the 1500 UTC observation
There is often text following a table describing how the system dissipated. This should be recorded in the Remark column for the last reported position of the storm.
Tables which span a page should be included in one table in the keyed file.
Table numbers and caption are not required.
Damage estimates and other information above the table caption should not be keyed.
In the tables, the date is recorded only on the first observation of the day, with other cells left blank. It is requested that no blank cells be left in the date row, but be filled with the actual date by entering the date in each row.
Systems which are over land do not generally have intensity estimates and are often denoted with “over land” in some columns. In these cases, the text “over land” should be entered in the remarks section with affected cells left blank.
If some columns are missing (e.g., the ΔP column in 1996), it should be left blank.
2525
• WWII Royal Navy Ship logs from 1938-1947 were microfilmed and scanned at the British National Archives. The records have been keyed by CDMP and will be QA/QC’d then merged into the International Comprehensive Ocean Atmosphere Data Set (ICOADS).
•Expanded to East India Company Logbooks in 2008- Joint UK Met Office Project
Climate Database Modernization Program
International Activities
2626
EIC visualization visualization by Philip Brohan, UK Met Office - Hadley Center
http://imma.appspot.com/interfaces/imma_edit_test.html
2727
Voyage of the EIC ‘William Fairlie’ from the UK to India to China and back to UK, 1824-1825
2828
Voyage of the EIC ‘Windham’ from the UK to China and back to UK, 1816-1817
2929
Voyage of the EIC ‘Astell’ from the UK to India and back to UK, 1824
3030
Remarks from voyage of the EIC ‘Thomas Coutts’ 1818. Although the remarks are in regards to a nasty winter North Atlantic extra-tropical cyclone, additional logbooks from this collection may contain similar entries that may give tropical cyclone information for the Indian Ocean (from remarks and/or from barometric pressures).Notice the pressure from the Barraud’s barometer increasing 0.89” (30+ mb) in 4 hours.
3131
The Foreign Data LibraryThe Vast Unknown…
3232
The Foreign Data LibraryThe Vast Unknown…
Both recent and archived publications are well-indexed
The problem is with the actual data in the library
Larry Nicodemus is working to create a better inventory and also to identify data that should be keyed
3333
The Foreign Data LibraryThe Vast Unknown…
The best index we have for our foreign data:
3434
Summary
Current efforts Surface weather observations
Cooperative observations
Tropical Publications-Based on Conference Input !
Next few years
Publications for customer service and IPS
Charts
Historic journals
Down the road
Barograms, thermograms, radar
Saving data only available on film
3535
• Password secure system access to Data Cabinets with various data (observations, pubs, climate journals, manuals)
•Login http://edads.imcwv.com/default.aspx
• e-mail: [email protected] for login and password
•Mention ‘Tropical Cyclone Project”
•Other NOAA Web Sites (e.g. NOAA Central Library)
Climate Database Modernization Program
3636
Climate Database Modernization Program
Questions ?
Contact Info
NOAA/ Climate Database Modernization Program
151 Patton Avenue- Room 516-D
Asheville, NC 28801
Tom Ross- Program Manager
828 271 4499
http://www.ncdc.noaa.gov/oa/climate/cdmp/cdmp.html