Data Management Seminar, 8-11th July 2008, Hamburg
Data Entry OverviewData Entry Overviewand Changes from the Field and Changes from the Field
TrialTrial
Data Management Seminar, 8-11th July 2008, Hamburg2
The ICCS MS Data Entry Process
Data Entry Part 1– Entering School Data– Entering Teacher Data– Entering Student Regional Module Data
Data Entry Part 2– Entering Student Questionaire Data
Data Entry Part 3– Entering Student Achievement Data
Data Management Seminar, 8-11th July 2008, Hamburg3
Table of Content
1. Changes in Data Entry Process since Field Trial
2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring8. The Process of (Reliability) Scoring
Data Management Seminar, 8-11th July 2008, Hamburg4
Changes in Hierarchical ID System
School level– 8001, 8002, 8003...
Teacher level– 800101, 800102...
805201, 805202...
Student level– 80010101, 80010102...
80520101, 80520102...
School level– 1001, 1002, 1003...
Teacher level– 100101, 100102...
105201, 105202...
Student level– 10010101, 10010102...
10520101, 10520102...
Field trial Main Survey
Data Management Seminar, 8-11th July 2008, Hamburg5
One form of Student Questionnaire
Student
Questionnaire
Form A
Fie
ld t
rial
Student
Questionnaire
Form B
Student
Questionnaire
Form C
Main
Su
rvey
Student
Questionnaire
Data Management Seminar, 8-11th July 2008, Hamburg6
One form of Student Questionnaire
Codebook has:– One form of Student Questionnaire inside– No IDQUEST variable
Thus...
Data Management Seminar, 8-11th July 2008, Hamburg7
Introduction of Coding Module (ISG)
Fie
ld t
rial
Data Management Seminar, 8-11th July 2008, Hamburg8
Main
Su
rvey
Introduction of Coding Module (ISG)
Data Management Seminar, 8-11th July 2008, Hamburg9
Advantages– Entering the Occupational Data using
WinDEM– More convenient for the Data Entry Staff– Better coordination between the Puncher
and the Coder– Higher Data Entry speed– Less errors
Introduction of Coding Module (ISG)
Data Management Seminar, 8-11th July 2008, Hamburg10
Number of Achievement Booklets
Fie
ld t
rial
Main
Su
rvey
6 bookletsReliability booklets – only 1, 2, and 3
7 bookletsReliability booklets –1 through 7
Data Management Seminar, 8-11th July 2008, Hamburg1111
Table of Content
1. Changes in Data Entry Process since Field Trial
2. Organizing the Data Entry Process 3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring
Data Management Seminar, 8-11th July 2008, Hamburg12
Data Documentation
Data
Organizing the Data Entry Process
Instruments/ Forms
InternationalCodebooks
WinW3S File
Data Entry Rules
...
...
DEUnit
Data Manager
12
Data Management Seminar, 8-11th July 2008, Hamburg1313
Table of Content
1. Changes in Data Entry Process since Field Trial
2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring
Data Management Seminar, 8-11th July 2008, Hamburg14
Data Manager’s Responsibilities
Data EntrySoftware
Source ofInformation
Hardware
Software
Adaptation ofCodebooks
Verification
Documentation
Instruments
TrackingForms
Scoring Material
Organize:
Training in: Organize: Perform:
Send Out
MaterialsData Entry
StaffEquipment Data Files
Data Manager
14
Data Management Seminar, 8-11th July 2008, Hamburg1515
Table of Content
1. Changes in Data Entry Process since Field Trial
2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring
Data Management Seminar, 8-11th July 2008, Hamburg16
From ICCS International Study Centrehttp://iccs.acer.edu.au/International codebooks
- Questionnaire codebooks - Achievement and reliability codebooks
Survey Operations Procedures (Unit 3) Windows Within-School Sampling
Software (WinW3S)Windows Data Entry Manager (WinDEM)
program IEA Survey System
Necessary Material for Data Entry
16
Data Management Seminar, 8-11th July 2008, Hamburg17
Important documents for data entry– Survey Operations Procedures (Unit 3)
Copy relevant parts of documents for key punchersCopies of survey tracking forms should be available for key punchers
Prepare Key Punchers for Data Entry
17
Data Management Seminar, 8-11th July 2008, Hamburg18
Planning for Required Resources
Staff for scoring and data entry depends on…– How much material?– How much time?– How much experience?– How much money?– How many persons available?
Computer requirements– IBM-compatible (Pentium processor or higher) – OS: Windows XP/Vista– Office 97 or higher
Ask countries that participated in previous IEA studies for their
experiences!
18
Data Management Seminar, 8-11th July 2008, Hamburg19
Amount of MaterialsScoring: Scorer
7 Achievement Booklets =
~ 3750 Student Achievement Test Booklets
~ 700 Student Achievement Test Booklets for Reliability Scoring
~ 150 School Questionnaires
~ 2250 Teacher Questionnaires
~ 3750 Student Questionnaires
~ 3750 Regional Module Questionnaires
~ 3750 Student Achievement Test Booklets
~ 700 Reliability Scoring Sheets
Data Entry: Key Punchers
19
Data Management Seminar, 8-11th July 2008, Hamburg20
Rece
ivin
g M
ate
rial
InstrumentPreparation
Test
Adm
inis
trati
on
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Su
bm
issi
on
Sampling and Preparing
Test AdministrationScoring
Data Entryand Verification
20
Data Management Seminar, 8-11th July 2008, Hamburg2222
Table of Content
1. Changes in Data Entry Process since Field Trial
2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material 6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring
Data Management Seminar, 8-11th July 2008, Hamburg23
Control Receipts of Test Material
Record return date on ICCS MS School Sampling Form
23
Contact schools that have not returned testing material or where the material is incomplete!
Data Management Seminar, 8-11th July 2008, Hamburg24
Control Receipts of Test MaterialCheck that all survey tracking forms have been returned and completed correctly
...
...
24
Check participation statusparticipation status on tracking forms!
Contact schools that have not returned tracking forms or returned incomplete!
Data Management Seminar, 8-11th July 2008, Hamburg25
Control Receipts of Test MaterialCheck that all booklets/ questionnaires have been returned
Verify identification codesidentification codes on instruments!
Contact schools that have not returned all testing instruments!
25
Data Management Seminar, 8-11th July 2008, Hamburg2626
Table of Content
1. Changes in Data Entry Process since Field Trial
2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material6. Organize Material for Data Entry 7. Organize Material for Scoring 8. The Process of (Reliability) Scoring
Data Management Seminar, 8-11th July 2008, Hamburg27
OrganizeMaterial
for Data Entry
Rece
ivin
g M
ate
rial
27
Data Management Seminar, 8-11th July 2008, Hamburg28
Separate questionnaires and tracking forms from booklets!Sort and forward…
Organize Material for Data Entry and Scoring
Forms + Questionnaires to
Data Entry Staff
Booklets to
Scorers
...
...
28
Data Management Seminar, 8-11th July 2008, Hamburg
Forms + Questionnaires to
Data Entry Staff
...
Sorting Forms and Questionnaires
29
Data Management Seminar, 8-11th July 2008, Hamburg30
1 0 0 1 0 1 0 1
1 0 0 1
1 0 0 1 0 1
1 0 0 1 0 1 1 0 0 1
0 1 0 1
0 1
1 0 0 1 0 1
1 0 0 1
Hierarchical ID SystemIDSCHOOL: 4 digits school
IDCLASS: 6 digits school + class
IDSTUD: 8 digits school + class + student
IDTEACH: 6 digits school + teacher
Unique identifier in ICG file
Unique identifier in xSG, xSE/xSL, xSA, xSR file
Unique identifier in ITG file
1 0 0 1
30
Data Management Seminar, 8-11th July 2008, Hamburg31
1004
Arrange Instruments for Data Entry
School Questionnaires
Sort by IDSCHOOL
...
31
1003
1002
1001
Data Management Seminar, 8-11th July 2008, Hamburg32
100102
Teacher Questionnaires
Sort by IDTEACH
...
Arrange Instruments for Data Entry
32
100101
School 1001
100202
100201
School 1002
Data Management Seminar, 8-11th July 2008, Hamburg33
...
Sort by IDSTUD
Student Questionnaires
Arrange Instruments for Data Entry
33
10010103
10010102
10010101
IDSCHOOL 1001IDCLASS 100101
10010203
10010202
10010201
IDSCHOOL 1001IDCLASS 100102
10020103
10020102
10020101
IDSCHOOL 1002IDCLASS
100201
10020203
10020202
10020201
IDSCHOOL 1002IDCLASS
100202
Data Management Seminar, 8-11th July 2008, Hamburg34
...
Sort by IDSTUD
Regional Module Student Questionnaires
Arrange Instruments for Data Entry
34
10010103
10010102
10010101
IDSCHOOL 1001IDCLASS 100101
10010203
10010202
10010201
IDSCHOOL 1001IDCLASS 100102
10020103
10020102
10020101
IDSCHOOL 1002IDCLASS
100201
10020203
10020202
10020201
IDSCHOOL 1002IDCLASS
100202
Data Management Seminar, 8-11th July 2008, Hamburg3535
Table of Content
1. Changes in Data Entry Process since Field Trial
2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material6. Organize Material for Data Entry7. Organize Material for Scoring 8. The Process of (Reliability) Scoring
Data Management Seminar, 8-11th July 2008, Hamburg36
OrganizeMaterial
for Data Entry
OrganizeMaterial
for Scoring
Rece
ivin
g M
ate
rial
36
Data Management Seminar, 8-11th July 2008, Hamburg37
Sorting Booklets for ScoringTwo Teams for Scoring: A and B
Booklets to
Scorers
Scoring Team A
Scoring Team B
37
Data Management Seminar, 8-11th July 2008, Hamburg38
...
1,2,...,6,7
Steps for Booklet Arrangement
Step 1: Sort by Booklet Type
38
Data Management Seminar, 8-11th July 2008, Hamburg39
Arrange Test Booklets for Scoring
Step 1: Sort by Booklet Type
Booklet
1
Booklet
1
Booklet
1
Booklet
1
Booklet
2
Booklet
2
Booklet
2
Booklet
2
Booklet
5
Booklet
5
Booklet
5
Booklet
6
Booklet
6
Booklet
6
Booklet
6
Booklet
7
~536 ~536 ~536
~3750
~536
...
...
...
Data Management Seminar, 8-11th July 2008, Hamburg40
Step 2: Divide into Two Sets ...
Set A, Set B
Step 1: Sort by Booklet Type
Steps for Booklet Arrangement
Note: From now on the example in this presentation will focus on
booklet 1 only!
Data Management Seminar, 8-11th July 2008, Hamburg41
Arrange Test Booklets for Scoring
Step 2: Divide into Two Sets
Booklet
1
Booklet
1
Booklet
1
Booklet
1~268
~268
Booklet
1
Set A
Booklet
1
Set A
Booklet
1
Set A
Booklet
1
Set A
Booklet
1
Set B
Booklet
1
Set B
Booklet
1
Set B
Booklet
1
Set B
~536
Scoring Team A
Scoring Team B
~50%
~50%
Data Management Seminar, 8-11th July 2008, Hamburg42
Step 3: Divide into Batches ...
Batch 1, Batch 2, ...
Step 1: Sort by Booklet Type
Step 2: Divide into Two Sets
Steps for Booklet Arrangement
Data Management Seminar, 8-11th July 2008, Hamburg43
Arrange Test Booklets for Scoring
Step 3: Divide into Batches
~268
Booklet
1R
Set A
Booklet
1
Set A
Booklet
1R
Set A
Booklet
1
Set A~ 25
Booklet
1R
Set A
Booklet
1
Set A
Scoring Team A
~ 25
Booklet
1R
Set A
Booklet
1
Set A
~ 25
Booklet
1R
Set A
Booklet
1
Set A~ 25
Booklet
1R
Set B
Booklet
1
Set B
~ 25
Booklet
1R
Set B
Booklet
1
Set B
~268
Booklet
1R
Set B
Booklet
1
Set B
Booklet
1R
Set B
Booklet
1
Set B
Scoring Team B
~ 25
Booklet
1R
Set B
Booklet
1
Set B
... ...
Data Management Seminar, 8-11th July 2008, Hamburg44
Step 4: Separate “R” Booklets
Step 3: Divide into Batches
Step 1: Sort by Booklet Type
Step 2: Divide into Two Sets
Steps for Booklet Arrangement
Data Management Seminar, 8-11th July 2008, Hamburg45
Arrange Test Booklets for Scoring
~ 5
Booklet
1R
Set A
Scoring Team A
Scoring Team B
~20
Booklet
1
Set A
Booklet
1R
Set A
Booklet
1
Set A
~25
Booklet
1R
Set A
Booklet
1
Set A
Booklet
1R
Set A
Booklet
1
Set A
...
Step 4: Separate “R” Booklets
Data Management Seminar, 8-11th July 2008, Hamburg46
Arrange Test Booklets for Scoring
Scoring Team A
Booklet
1R
Set A
~ 5
As Main Scorers
As Reliability Scorers
Scoring Team B
~20
Booklet
1
Set A
Booklet
1R
Set A
Booklet
1
Set A
~20
Booklet
1
Set A
Booklet
1R
Set A
Booklet
1
Set A
...
~20
Booklet
1
Set A
Booklet
1R
Set A
Booklet
2
Set A
~20
Booklet
1
Set A
Booklet
1R
Set A
Booklet
2
Set A
...
...
Booklet
1R
Set A
~ 5
...
Booklet
2R
Set A
~5
Booklet
2R
Set A
~5
...
...
Step 4: Separate “R” Booklets
Data Management Seminar, 8-11th July 2008, Hamburg4747
Table of Content
1. Changes in Data Entry Process since Field Trial
2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material6. Organize Material for Data Entry7. Organize Material for Scoring8. The Process of (Reliability) Scoring
Data Management Seminar, 8-11th July 2008, Hamburg48
Scoring
InstrumentPreparation
Test
Adm
inis
trati
on
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Su
bm
issi
on
Data Entryand Verification
ScoringSampling
and PreparingTest Administration
Data Management Seminar, 8-11th July 2008, Hamburg50
Step 6: Reliability Scoring
Step 5: Main Scoring (I)Reliabilit
y
Scoring
Sheet
Step 7: Return “R” Booklets
Step 8: Main Scoring (II)
Step 4: Separate “R” Booklets
Step 3: Divide into Batches
Step 1: Sort by Booklet Type
Step 2: Divide into Two Sets
Steps for Booklet Arrangement
Data Management Seminar, 8-11th July 2008, Hamburg51
Main ScoringStep 5: Main Scoring (I)
202
~218
Booklet
1
Set A
Booklet
1R
Set A
Booklet
1
Set A
~218
Booklet
1
Set A
Booklet
1R
Set A
Booklet
1
Set B
Scoring Team A
Scoring Team B
Data Management Seminar, 8-11th July 2008, Hamburg52
Reliability ScoringStep 6: Reliability Scoring
202
Scoring Team A
Scoring Team B
Booklet
1R
Set B
~50
Reliability
Scoring
Sheet
1R Set B
~50
Booklet
1R
Set A
~50
Reliability
Scoring
Sheet
1R Set A
~50
Data Management Seminar, 8-11th July 2008, Hamburg53
101
Booklet
1R
Set B
Reliability Scoring
101
1011001010
3974
2
Scoring Team A
Reliability
Scoring
Sheet
1R Set B
Data Management Seminar, 8-11th July 2008, Hamburg54
Return „R“ Booklets for Main Scoring
Step 7: Return “R” Booklets
Sco
rin
g T
eam
A
Sco
rin
g T
eam
B
Booklet
1R
Set A
~50
Reliability
Scoring
Sheet
1R Set A
~50~218
Booklet
1
Set A
Booklet
1R
Set A
Booklet
1
Set A
Booklet
1R
Set B
~50
Reliability
Scoring
Sheet
1R Set B
~50~218
Booklet
1
Set A
Booklet
1R
Set A
Booklet
1
Set B
Data Management Seminar, 8-11th July 2008, Hamburg55
Main ScoringStep 8: Main Scoring (II)
202
Scoring Team A
Scoring Team B
Booklet
1R
Set B
~50
Booklet
1R
Set A
~50
Data Management Seminar, 8-11th July 2008, Hamburg56
Scoring Team B
101
Documenting Main Scorer
202
Booklet
1R
Set B
202
1
Data Management Seminar, 8-11th July 2008, Hamburg5757
Table of Content
1. Changes in Data Entry Process since Field Trial
2. Organizing the Data Entry Process3. Data Manager’s Responsibilities 4. Necessary Material for Data Entry 5. Control Receipts of Survey Material6. Organize Material for Data Entry7. Organize Material for Scoring 8. The Process of (Reliability) Scoring
Data Management Seminar, 8-11th July 2008, Hamburg58
Data Entryand Verification
Data Entryand Verification
InstrumentPreparation
Test
Adm
inis
trati
on
Codebook Adaptation
Rece
ivin
g M
ate
rial
Data
Su
bm
issi
on
ScoringSampling
and PreparingTest Administration
Data Management Seminar, 8-11th July 2008, Hamburg59
CompleteStudent
Information
EnterBackground Data
EnterAchievement and
Reliability Data
Data Entryand Verification