The Data Documentation Initiative (DDI)

Ron NakaoSocial Science Data and Software (SSDS)Stanford University Libraries

With input from Gretchen Gano, Sanda Ionescu, Jim Jacobs, Nancy McGovern, Wendy Thomas, Mary Vardigan

Presented to the DLF Fall Forum 2007 - Philadelphia, PA

A Metadata Specification for Social Science Data

Presentation Overview

What is the DDI? What is the DDI Alliance?

A taste of the DDI specification

Futures

What is the DDI?

“The Data Documentation Initiative (DDI) is an effort to establish an international XML-based standard for the content, presentation, transport, and preservation of documentation for datasets in the social and behavioral sciences.”

What is the DDI Alliance?

Host Institutions Member Institutions Organization Structure

Director Steering Committee Expert Committee Working Groups

DDI Alliance Host Institutions and Associations

Inter-University Consortium for Political and Social Research (ICPSR) - b.1962

Roper Center for Public Opinion Research - b.1946+

Council of European Social Science Data Archives (CESSDA) - b.1976

International Federation of Data Organizations (IFDO) - b.1977

International Association of Social Science Information, Service, and Technology (IASSIST) - b.1974+

DDI Alliance Member Institutions (30) University of Alberta, Canada University of California, Berkeley --

Computer-Assisted Survey Methods Program and UCDATA

University of California, California Digital Library

Centre for Survey Research and Methodology (ZUMA)

Centro De Investigaciones Sociologicas (CIS), Spain

DDI Alliance Member Institutions (30) CEPS/INSTEAD -- Luxembourg Danish Data Archive Data Archiving and Networked Services

(DANS), The Netherlands Emory University Finnish Social Science Data Archive German Socio-Economic Panel Study (SOEP) University of Guelph, Canada Harvard-MIT Data Center

DDI Alliance Member Institutions (30) Inter-university Consortium for Political and

Social Research (ICPSR) Massachusetts Institute of Technology (MIT) University of Minnesota National Opinion Research Center (NORC) Norwegian Social Science Data Service (NSD) Open Data Foundation, Tucson, Arizona Princeton University Roper Center Stanford University

DDI Alliance Member Institutions (30) University of Surrey, United Kingdom Swedish Social Science Data Service (SSD) Swiss Data Archive for the Social Sciences

(SIDOS) United Kingdom Data Archive (UKDA) University of Wisconsin World Bank, Development Data Group

(DECDG) Yale University Zentralarchiv fuer Empirische

Sozialforschung, University of Koeln

DDI Alliance Structure

Comparative Data/Families of Datasets

Qualitative Data

Outreach and Usability

Instrument Documentation

Aggregate, Geography & Time

Expert Committee(Voting members

+ observers)

Steering Committee

Director

Controlled Vocabularies

Technical Implementers Committee

What is the DDI specification?

A word about social science data & codebooks

DDI 1 & 2 Data Life Cycle & DDI 3

Here’s some Data…

00100 1 D10 999900492410047015004978300502370051084005298200546990055641005775900577850058717005927900623000064131006485900687180071076007231300750330076553007973100805520097635013970301658930190775022718602340220264434032536200000007629200100 1A D10 999900339520034403003477900350550035313003574000363940037030003771100384210039055003991900402370040842004116900420870042662004290100430550041492004069900408050041092004452300447540049672005035900481050047333004909200000007629200100 1B D10 999900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003414000690900101130011237001140500109040010822001131500121357629200100 1C D10 999900015600001658000167200017130001777000189100018450001880000227800023130002557000325000035700004159000379500039400004197000541900063730005748000648800067260007697000689600068670007441001082900147780020606002197300193337629200100 1D D10 999900137290010954001333200134690013994001535100164600016731001777000170510017105001611000184930019130001989500226910024217002399300256050029313003254400330210045432008137501041590122425015459301602350185673024298200000007629200100 1 SS0 999900492410047015004978300502370051084005298200546990055641005775900577850058717005927900623000064131006485900687180071076007231300750330076553007973100805520097635012867901527990158142018555501999050227600026785400000007629200100 1A SS0 999900339520034403003477900350550035313003574000363940037030003771100384210039055003991900402370040842004116900420870042662004290100430550041492004069900408050041092004101200412200041175004113300410920040739004041600000007629200100 1B SS1 999900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000034142006363400931480093149009314900931480093150009314700931507629200100 1C SS1 999900156030016579001671500171290017774001891400184480018796002277700231280025573003249800357020041587003795000394040041972005418500637250057480006488300672570076968006351400632470061681008844401262400177361018088701483947629200100 1D SS0 99990013729001095400133320013469001399400153510016460001673100177700017051001710500161100018493001913000198950022691002421700239930025605002931300325440033021004543200749520095939010148401262630136874015981002000350000000762920010334 I62 99990000000000000000000000001672000176100018370001953000206400021630002133000223500023900002510000268100028700003131000344700037390004022000430200046920005100000548900061410006972000794100088590010000001150100129980000000762920010334 X51 9999000000000000000000000000000000000530000043000006300000570000048-000014000004800000690000050000006800000700000091000010100000850000076000007000000910000087000007600001190000135000013900001160000129000015000001300000000762920010664 62 99990000000000000000000000000000000000000033120003350000339500034970003634000378800039130004027000413200042810004455000465900048880005137000535400055910005878000623000065980006982000765100088210010000001110200123440000000762920010664 X51 99990000000000000000000000000000000000000000000000011000001300000300000039000004200000330000029000002600000360000041000004600000490000051000004200000440000051000006000000590000058000009600001530000134000011000001120000000762920010770 D32 99990005345000546300056610007738000744500075540007777000848500094540010150000966300102550011459001197600125180013743001539600167120018360001927300215240024664002837200317350037681005242500773110079612009066101027780000000762920010771 D32 99990006061000595700059030008174000805100076720007966000891700098470010822001013200107230012016001257500132760014460001621900176160019389002034200226530025817002971100332080038920005361100785560081405009230401057750000000762920010774 D12 99990000000000000000000000000000000000000040390003979000398400040870004181000406200040230004073000409000040700004118000420200042660004358000436700043350004483000473000049890005426000667100091760010000001016200111000000000762920010775 D12 99990000000000000000038320004818000467700043400004288000432900044440004575000427400041560004197000417300041300004181000427000043230004388000438500043420004435000477100050060005398000657900092430010000001010600110010000000762920010776AXD61 19750000000000000000000000000000000000000000000000000000000000000000000570000051900005120000512000049200004820000512000054700005360000553000051800005140000555000057500005420000618000095200012160001000000112300013530001289762920010776IAD61 19750000206000020300002430000277000020300001660000159000015800001700000251000017100001450000153000014200001450000415000028600001030000091000009900000970000164000018300002210000362000047000014610001000000056400003950000381762920010776IAZ 2 9999000042300004160000498000056700004170000341000032600003240000348000051500003500000297000031400002910000298000085000005870000212000018600002030000198000033700003750000453000074300009630002996000205000011570000810000078176292

And Here’s the Codebook

A digital Codebook (pdf)

Evolution of the DDI

Concept of DDI and definition of needs grew out of the data archival community

1995 - DDI efforts initiated by ICPSR 1997 - XML DTD released 2000 - DDI 1.0 released 2003 - DDI 2.0 released - DDI Alliance formed 2007 - DDI 3.0 Candidate Draft Release 2008 - DDI 3.0 Final Release

DDI: Early Development

2000 – DDI 1.0 Simple survey Archival data formats Microdata only

2003 – DDI 2.0 Aggregate data (based on matrix

structure) Added geographic material to aid

geographic search systems and GIS users

DDI versions 1 & 2

Document Description Study Description Data Files Description Variable Description Other Study-Related Materials

DDI 3: The Data Life Cycle

Capturing the Data Life Cycle Study Unit

- Research question - Funding - Concepts - Background research

Study Unit Data Collection

- Instrument - Data collection process - Questionnaire

Capturing the Data Life Cycle

Study Unit Data Collection Logical Product

- Intellectual content of data - Relationship to questions and concepts- Relationship to processing (recodes, weighting, derivations, imputations)

Capturing the Data Life Cycle

Study Unit Data Collection Logical Product Physical Data Product

- Describes the structure (microdata, tabular,aggregate, Ncube…)

Capturing the Data Life Cycle

Study Unit Data Collection Logical Product Physical Data Product Physical instance

- Each describes a single data file (e.g., Census data by state...each state is an instance)

Capturing the Data Life Cycle

Study Unit Data Collection Logical Product Physical Data Product Physical instance “Instance” (METS-inspired)

-An instance module “wraps” the other modules. Like a table of contents to a group of studies and files and modules it brings everything together.

Capturing the Data Life Cycle

Study Unit Data Collection Logical Product Physical Data Product Physical instance “Instance” Archive

- Each archive can add its own local information with an archive module.

Capturing the Data Life Cycle

Group module

- Describe concepts, questions, and variables that occur in several studies.- Describe a series (e.g., CPS, Eurobarometer) - Describe a collection of studies (not a series) and identify the common comparable concepts, questions and variables.

Capturing the Data Life Cycle

Group module Comparative module

-The Comparative module contains information for comparing concepts, questions, and variables between or among Study Units that have been housed in a Group.

Capturing the Data Life Cycle

<?xml version="1.0" encoding="UTF-8"?><r:Coverage xmlns:r="ddi:reusable:0_1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xhtml="http://www.w3.org/1999/xhtml" xsi:schemaLocation="ddi:reusable:0_1 Schemas/reusable.xsd"><r:SpatialCoverage><r:Identification><r:ID>GEOCOV</r:ID><r:IdentifyingAgency>MPC</r:IdentifyingAgency><r:Version>1.0</r:Version></r:Identification><r:BoundingBox> <r:WestLongitude>-177.1</r:WestLongitude> <r:EastLongitude>-61.48</r:EastLongitude>

<r:SouthLatitude>+13.71</r:SouthLatitude> <r:NorthLatitude>+76.63</r:NorthLatitude></r:BoundingBox><r:Description translated="false" translatable="true"><xhtml:p>United States, Region, Division, State, County, County Subdivision, Place, Tract/Block Numbering Area within Place/Remainder within County Subdivision.</xhtml:p></r:Description><r:SpatialObject>Polygon</r:SpatialObject><r:GeographicStructure><r:Geography><r:Identification><r:ID>G001</r:ID>

DDI 3.0 Geography Example

DDI - User Community Data archives and libraries world-

wide (e.g., ICPSR, CESSDA) Health Canada Statistics Canada World Bank WHO (World Health Surveys) Gallup-Europe Metadata Management Toolkit

(IHSN)

International Household Survey Network (IHSN) To coordinate and improve survey

collecting operations in developing countries

Developed to support the survey collection activities of the International Household Survey Network (IHSN)

Sponsors: 18 organizations, such as ILO, UNESCO, World Bank, UNICEF, WHO, UNDP, Eurostat

Goal: improve the quality of collected data and encourage more dissemination and long-term preservation

100% DDI compliant

Futures Continued development of DDI Outreach, train, promote Expand Alliance membership Foster tools development Build ties & interoperability with

other metadata specifications Funding ISO Standard status

That’s all folks! Thanks!

http://www.ddialliance.org/