Introduction to DDI with a Technical Perspective
7th Workshop of German Panel Surveys
01.02.2013
Thomas Bosch
M.Sc. (TUM)PhD student
boschthomas.blogspot.comGESIS - Leibniz Institute for the Social Sciences
Outline
2
• MISSY
• DDI-L Main Structures
• DDIInstance
• Comparison
• StudyUnit
• ConceptualComponent
• LogicalProduct
• DataCollection
Outline
• PhysicalDataProduct
• PhysicalInstance
• DDIProfile
• Colectica
Outline
4
• MISSY
• DDI-L Main Structures
• DDIInstance
• Comparison
• StudyUnit
• ConceptualComponent
• LogicalProduct
• DataCollection
General Information About Missy
• Missy (Microdata Information System)
• Provides detailed information about individual datasets
• Contains metadata of microdata relevant for the social sciences
• Facilitates data usage for research
• Currently consists of microcensus survey• Largest household survey in Europe
• statistics about• general population in Germany
• situation about employment market
(occupation, professional education, income, legal insurance, …)
5
General Information About Missy
• May be split in two parts• Missy Web (end-user front-end part)
• Missy Editor for documentation (back-end part)
• Consists of approx. 500 Variables & Questions per year
• Captures 25 years, since 1973
6
Missy 3
• That’s what it’s all about! The “next generation Missy”
• Integration of further surveys, e.g. EU-SILC, EU-LFS, …
• Implementation of Missy Editor as a Web-Application
7
Use-Cases in Missy
• General Information about survey 'microcensus'
• Variables by thematic classification and year
• List of variables by year
• List of generated variables by year
• Show details of a variable with statistics
• Variable-Time Matrix• List of variables by thematic classification and year (selectable)
• Questionnaire Catalogue
• In future: browse by survey and country!8
9
Use-Cases in Missy
• General Information about survey 'microcensus'
• Variables by thematic classification and year
• List of variables by year
• List of generated variables by year
• Show details of a variable with statistics
• Variable-Time Matrix• List of variables by thematic classification and year (selectable)
• Questionnaire Catalogue
• In future: browse by survey and country!10
11
Use-Cases in Missy
• General Information about survey 'microcensus'
• Variables by thematic classification and year
• List of variables by year
• List of generated variables by year
• Show details of a variable with statistics
• Variable-Time Matrix• List of variables by thematic classification and year (selectable)
• Questionnaire Catalogue
• In future: browse by survey and country!12
13
Use-Cases in Missy
• General Information about survey 'microcensus'
• Variables by thematic classification and year
• List of variables by year
• List of generated variables by year
• Show details of a variable with statistics
• Variable-Time Matrix• List of variables by thematic classification and year (selectable)
• Questionnaire Catalogue
• In future: browse by survey and country!14
Use-Cases in Missy
• General Information about survey 'microcensus'
• Variables by thematic classification and year
• List of variables by year
• List of generated variables by year
• Show details of a variable with statistics
• Variable-Time Matrix• List of variables by thematic classification and year (selectable)
• Questionnaire Catalogue
• In future: browse by survey and country!15
16
Use-Cases in Missy
• General Information about survey 'microcensus'
• Variables by thematic classification and year
• List of variables by year
• List of generated variables by year
• Show details of a variable with statistics
• Variable-Time Matrix• List of variables by thematic classification and year (selectable)
• Questionnaire Catalogue
• In future: browse by survey and country!17
18
Use-Cases in Missy
• General Information about survey 'microcensus'
• Variables by thematic classification and year
• List of variables by year
• List of generated variables by year
• Show details of a variable with statistics
• Variable-Time Matrix• List of variables by thematic classification and year (selectable)
• Questionnaire Catalogue
• In future: browse by survey and country!19
20
21
Flow Diagram
UML Class Diagrams
22
Outline
23
• MISSY
• DDI-L Main Structures
• DDIInstance
• Comparison
• StudyUnit
• ConceptualComponent
• LogicalProduct
• DataCollection
24
25
26
Modules
Outline
27
• MISSY
• DDI-L Main Structures
• DDIInstance
• Comparison
• StudyUnit
• ConceptualComponent
• LogicalProduct
• DataCollection
28
DDIInstance
DDIInstance
29
30
ResourcePackage
ResourcePackage
31
Allows packaging of any maintainable item as a resource item
Structure to publish non-study-specific materials for reuse
ResourcePackage
32
Any module (except: StudyUnit, Group, LocalHoldingPackage)
33
Any Scheme (Banks)
Internal or external references to schemes and elements within schemes
ResourcePackage
Group
34
Group
35Groups allow inheritance
Group
36
Organization
37
Organization
38
Outline
39
• MISSY
• DDI-L Main Structures
• DDIInstance
• Comparison
• StudyUnit
• ConceptualComponent
• LogicalProduct
• DataCollection
Comparison (DDI 3.1)
1. ResourcePackage
2. Group
3. Comparison
40
Comparison (MISSY)
41
42
Comparison (DDI 3.1)
1. ResourcePackage
2. Group
3. Comparison
43
References to same contents of ResourcePackages
44
Comparison (DDI 3.1)
1. ResourcePackage
2. Group
3. Comparison
45
46
label = 'variable age'
label = 'age'
Group (study and variable information)
--> variables in this group are comparable
--> inherit
--> overwrite variable information
+ easy to understand
- Implementation
- changes --> reorganize groups
Comparison (DDI 3.1)
1. ResourcePackage
2. Group
3. Comparison
47
Comparison
48
Comparison
Comparison
VariableMap
CategoryMap
CodeMap
ConceptMap
QuestionMap
UniverseMap
Why not more? good understanding of these elements
49
50
Variable-Age Variable-AgeGroup
codes/categories codes categories
20 1 20 – 29
… … …
29 1 20 – 29
30 2 30 – 39
… … …
67 6 60 – 67
Comparison (Example)
VariableMap
Variable-Age --> Variable-AgeGroup
commonality, difference
51
CodeMap
commonality, difference
52
Variable-Age Variable-AgeGroup
20 1
… …
29 1
30 2
… …
67 6
CategoryMap
commonality, difference
53
Variable-Age Variable-AgeGroup
20 20 – 29
… …
29 20 – 29
30 30 – 39
… …
67 60 – 69
Outline
54
• MISSY
• DDI-L Main Structures
• DDIInstance
• Comparison
• StudyUnit
• ConceptualComponent
• LogicalProduct
• DataCollection
55
StudyUnit
56
Coverage
Coverage
57
TemporalCoverage
58
TopicalCoverage
59
SpatialCoverage
60
SpatialCoverage
61
TopicalCoverage
62
TemporalCoverage
63
64
Citation
Outline
65
• MISSY
• DDI-L Main Structures
• DDIInstance
• Comparison
• StudyUnit
• ConceptualComponent
• LogicalProduct
• DataCollection
ConceptualComponent
66
ConceptualComponent
ConceptualComponent
67
ConceptScheme
68
ConceptScheme
69
Concept
70
Thematic Classification (MISSY)
71
Concept
72
Universe
73
Universe
74
Outline
75
• MISSY
• DDI-L Main Structures
• DDIInstance
• Comparison
• StudyUnit
• ConceptualComponent
• LogicalProduct
• DataCollection
76
LogicalProduct
77
LogicalProduct
78
CodeScheme
CodeScheme (MISSY)
79
CodeScheme
80
Code
81
Code (MISSY)
82
Code
83
84
CategoryScheme
CategoryScheme (MISSY)
85
CategoryScheme
86
Category
87
MISSY (Category)
88
Category
89
90
VariableScheme
VariableScheme (MISSY)
91
VariableScheme
92
Variable
93
Variable (MISSY)
94
Variable
95
Representation
96
Representation
97
Outline
98
• MISSY
• DDI-L Main Structures
• DDIInstance
• Comparison
• StudyUnit
• ConceptualComponent
• LogicalProduct
• DataCollection
99
DataCollection
DataCollection
100
101
QuestionScheme
QuestionScheme
102
103
QuestionItem
QuestionItem (MISSY)
104
QuestionItem
105
106
Response Domain
Response Domain
107
108
Instrument
Instrument (SOEP)
109http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf
Instrument
110
111
ControlConstruct
ControlConstructScheme
112
ControlConstruct
113
Sequence
114
Sequence (bahavioral)
115
Sequence (SOEP)
116http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf
Sequence (structural)
117
118
StatementItem
Explanatory background related to questions
StatementItem (behavioral)
119
StatementItem (SOEP)
120http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf
StatementItem (structural)
121
122
QuestionConstruct
QuestionConstruct (behavioral)
123
QuestionConstruct (SOEP)
124http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf
QuestionConstruct (structural)
125
126
IfThenElse
IfThenElse (behavioral)
127
IfThenElse (SOEP)
128http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf
IfThenElse (structural)
129
130
Loop
Loop (behavioral)
131
Loop (SOEP)
132http://www.diw.de/documents/dokumentenarchiv/17/diw_01.c.394158.de/soepfrabo_haushalt_2011.pdf
Loop (structural)
133
134
RepeatWhile
RepeatWhile (behavioral)
135
RepeatWhile (structural)
136
137
RepeatUntil
RepeatUntil (behavioral)
138
RepeatUntil (structural)
139
Outline
• PhysicalDataProduct
• PhysicalInstance
• DDIProfile
• Colectica
141
PhysicalDataProduct
PhysicalDataProduct
142
• Physical layout of data that do not change when physical instance data files are merged or split
• Physical Data Structure
Outline
• PhysicalDataProduct
• PhysicalInstance
• DDIProfile
• Colectica
144
PhysicalInstance
145
PhysicalInstance
• Physical instances of physical
data products
• Represents data files
Outline
• PhysicalDataProduct
• PhysicalInstance
• DDIProfile
• Colectica
147
DDIProfile
148
DDIProfile
149
DDIProfile
Describes which DDI-L elements you use in your institution’s DDI flavor
Outline
• PhysicalDataProduct
• PhysicalInstance
• DDIProfile
• Colectica
Colectica
151
• Commercial software tool
• http://colectica.com/
• Multiple versions for individuals, departments, and institutions
• Multiple videos on the website for getting started
Colectica Reader For Free
152
Colectica (Navigation)
153
Colectica (Variables)
154
Colectica (Variables)
155
Colectica (Questions)
156
Colectica (Codes & Categories)
157
Feel free to ask any unsolved questions!
158
Thomas Bosch
M.Sc. (TUM)PhD student
boschthomas.blogspot.comGESIS - Leibniz Institute for the Social Sciences