Getting Your Data Ready for Archiving

Getting Your Environmental Data Ready for ArchivingJohn Porter

1Why this presentation?The forms data take for analysis are often different than the forms data take for archival storageSpreadsheets are widely used for simple analysesBut they have poor archival qualities Different versions over time are not compatibleFormulas are hard to capture or displayThey allow (encourage) users to structure data in ways that are hard to use with other softwareOur goal with archived data is to store the data in ways that it can be used in automated ways, with minimal human intervention

2Data that can be automatedBelow is a picture of a data spreadsheet that could NOT be easily automated.. Why not?

3Ugly DataProblemsDates are not stored consistentlySometimes date is stored with a label (e.g., Date:5/23/2005) sometimes in its own cell (10/2/2005)Values are labeled inconsistentlySometimes Conductivity Top others conductivity_topFor Salinity sometimes two cells are used for top and bottom, in others they are combined in one cellData coding is inconsistentSometimes YSI_Model_30, sometimes YSI Model 30Tide State is sometimes a text description, sometimes a numberThe order of values in the mini-table for a given sampling date are differentMeter Type comes first in the 5/23 table and second in the 10/2 table

4Ugly DataAdditional problemsConfusion between numbers and textFor most software 39% or