1st NRC Meeting, 16-19 October 2006, Amsterdam 1 ICCS 2009 Field Operations

1 1st NRC Meeting, 16-19 October 2006, Amsterdam

ICCS 2009 Field Operations


Field Operations Overview

National Context Questionnaire

Item Pilot

Field Trial Operations Overview

Online Data Collection

Support Materials

Field Trial Timeline

Next Steps

Contact Information


National Context Questionnaire

Language will be English only

NRCs will have to gather information from several sources

Data for National Context Survey will be collected online

NRCs can login as often as required

Questionnaire will be online end of 2006/beginning of 2007

– NRCs will receive email with individual access information


Item Pilot

Submit Items to ACER via email at [email protected]

Review Items and commenting on study website http://iccs.acer.edu.au

End of reviewing process mid January 2007


Field Trial Operations Overview


InstrumentPreparation

Sampling and Preparing

Survey Administration

Data Entryand Verification

SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Adapt Questionnaires & DocumentAdaptations

TranslationVerification

Layout Verification

TranslateQuestionnaires& Verifiy within

Country

Correct &Assemble

Questionnaires

CorrectQuestionnaires(if necessary)

Print

Send Out


NRCIEA Sec./ISC/DPCtime

Time planning before administration

Layout Verification

Implement changes &assemble Instruments

Translation Verification

Translate Instruments

2 weeks

4 weeks

3 weeks

2 weeks

4 weeks

Start data collection15

weeks

Instruments Send Out

Printing


due dateNRC/ISC/IEAtime


Layout Verification

Implement Changes &Assemble Instruments

Translation Verification


4 weeks

3 weeks

2 weeks

4 weeks

Instruments Send Out 2. July 2007

30. July 2007

27. August 2007

17. September 2007

1. October 2007

15. October 2007Start data collection

Print Instruments2 weeks 15. October 2007


Meanwhile…






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Contact Schoolsand get

within SchoolInformation

Perform Within-School

Sampling

Prepare List of Schools

and send it to DPC

Recruit and Train Test Administrators

DPC provides list of sampled

Schools

Recruit and Train Scorers

Translate School Coordinator Manual

Translate Test Administrator Manual


School Contact

Send sampling frame (list of schools) to DPC

Sampling of schools will be done by DPC preferably

List of sampled schools will be provided by DPC

Contact schools, ensure participation

Identify liaison person (School Coordinator)

Provide School Coordinator Manual

Record status for each sampled school in WinW3S

Adaptation and translation of School Coordinator and Test Administrator Manual (no verification on international level, modifications to some extend possible)


Within-School Sampling (Students)

Two levels: students and teachers

Identification of classes on Class Listing Forms

– Class name

– Number of students

Random within-school sample of classes

Identification of students within sampled classes on Student Listing Forms

Send list of sampled students to School Coordinator on Student Tracking Form

WinW3S serves as survey administration tool


Within-School Sampling (Teachers)

Identification of eligible teachers on Teacher Listing Forms

Within-school random sample of teachers

Send Teacher Tracking Form to School Coordinator


Within-School Sampling

WinW3S will support NRCs with

– Listing of classes / teachers / students

– Sampling of classes / teachers

– Assignment of forms

– Survey administration

– Tracking of participation

Will be subject to Data Management Seminar


Assigning Questionnaires

School Questionnaire

Teacher

– Every Teacher will receive one questionnaire

Student

– Booklet

– Questionnaire (perception and background part in one form)

– Regional Modules instrument(s)

WinW3S will assign forms and provide labels for questionnaires


NRCSchool

Time Planning before Administration

time

Contact Schools

Decision to participate

Send Listing Forms

List Students/Teachers

Sample Teachers

Send Tracking Forms

Send Questionnaires

Start data collection

Name School Coordinator

Sample Classes

List Classes


School ContactInstrumentstime

Time Planning before Administration

Contact Schools

Send Listing Forms

Sample Teachers

Send Tracking Forms

Send Questionnaires

Implement changes & assemble Quest.


Printing

Data collection

Sample Classes






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Adapt Codebooks To National Questionnaires

Document Adaptations inNational Adaptaion Forms






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Monitor Test Administrators

Ad hoc Support

Arrange for Material Receipt

Train Scorers

Train Data Entry Staff






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Organize Material for (direct) Data Entry

Organize Material for Scoring and Reliability Scoring

Check if Information and Material is complete

Get back to School

if necessary

Enter Participation Information






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Conduct and Monitor Scoring

Conduct and Monitor

Reliability Scoring

Organize Material forData Entry


Scoring

All open ended items have to be scored according to scoring guide

Reliability scoring for open ended items for a certain amount of booklets

Subject to Scoring Training Seminar in 2007






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Complete Participation Information

Verify Data

Enter Data


Data Entry

Data entry software (WinDEM) and international codebooks for the FT are provided by the DPC in time

National staff should be trained on data entry, even if all or parts are being outsourced

Computer workstations need to be set up

National Centers enter and verify data from paper questionnaires

Adapt codebooks to reflect national conditions (e.g. ISCED levels)






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Set of Final Instruments

National Codebooksand Documentation (NAF)

VerifiedData sets

TrackingForms

Reliability Coding Data






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Online Data Collection


Online Data Collection (ODC)

Used for all questionnaires that National Centers should answer

– National Context Questionnaire

– Survey Activities Questionnaire

International Option for

– School Questionnaire

– Teacher Questionnaire

Countries that decide to use ODC in Main Study are obliged to conduct Field Trial in a split half design

Ensure fall-back to paper questionnaire

True mixed mode design

Goal: reduce response burden

– Ask schools for preferred mode (in Main Study)

– Mode assignment for all questionnaires of one type possible


Codebook Adaptation


Sampling and PreparingAdministration

Data Entry and

Verification

Ad

min

istr

ati

on

Rece

ivin

g M

ate

rial

Data

Sen

doutCodebook Adaptation

Prepare ODC

Monit

or

OD

C


School contactInstrumentstime


Contact Schools

Sample Students / Teachers

Send Tracking Forms

Send Questionnaires

Implement changes & assemble Quest.


Printing

Data collection

PrintingPrepareODC

Send Quest./Cover Letters

Send Listing Forms

Sample Classes


Online Data Collection (ODC) - Facts

Based on concept of cultures

All languages supported (character sets, text directions,…)

Participants receive “cover letter”

– Label with login information provided by WinW3S

– Culture specific URL

– Personal username and password

– Sent to school coordinator for distribution

Participants can log in and out as often as desired

Same administration period as paper questionnaires


Support Material


Support Material

Survey Operations Manual (FT)– Part 1: Survey Operations Overview, Dates and Activities– Part 2: Survey Administration

School Coordinator ManualTest Administrator ManualWithin School Sampling Software (WinW3S) + ManualIEA SurveySystem Designer + ManualNational Quality Control Monitor Manual

– Part 3: Data EntryScoring GuideCodebooksWindows Data Entry Manager (WinDEM) + Manual

Distribution of all materials and software packagesthrough study website only


FT Timeline


Field Trial Timeline

Receive instruments & NAF by 2 July 2007

Submit translated questionnaires with NAF v1 for international translation verification by 30 July 2007

Receive verified instruments by 27 August 2007

Submit for layout verification by 17 September 2007

Start printing and distribution of questionnaires by 1 October 2007

Proposed FT administration period (subject to country conditions):15 October – 16 November 2007

Submit data to DPC by end of 2007


Next Steps


Next Steps

Send items to ISC via email to: [email protected]

Review and comment on items at http://iccs.acer.edu.auuntil Mid January 2007

Complete National Context Questionnaire beginning of 2007

Set up team at national center and plan for– Staff– Translation– School Contact– Test Administrators– Scorers– Data Management (Data Entry and Verification)

NRC meeting in June 2007

Data Management Seminar in July 2007

Be prepared to receive Field Trial instruments in July 2007


Contact Information


Contact Information

Field Operations / Data Management:

IEA Data Processing Center

Falk Brese Michael Jung

[email protected]


Thanks for your attention!

Documents

1st NRC Meeting, 16-19 October 2006, Amsterdam 1 ICCS 2009 Field Operations