Data Management Seminar, 9-12th July 2007, Hamburg 11 ICCS 2009 – On-line Data Collection A General Introduction

1 Data Management Seminar, 9-12th July 2007, Hamburg1

ICCS 2009 – On-line Data Collection

A General Introduction

2 Data Management Seminar, 9-12th July 2007, Hamburg

Content

History

Potential

Key Features of the IEA SurveySystem

Integration with Survey Operations

Guidelines

Restrictions

Timeline


History Of Project

Interest for On-line Data Collection from within and outside IEA in 2003/2004 (SITES, PISA CBAS, PISA 2009).

1st implementation in the IEA SITES 2006 (Second Information Technology in Education Study), which had an near-ideal match between research domains and methodology.

Currently used for OECD TALIS.


History Of Project

Staged development and implementation in SITES:– Technical try-out in ~10 countries (Spring 2005)– Split-sample design in field trial in 17 countries

(Summer 2005)– FT analysis showed

Feasibility: ODC methodology and procedures work in the context of an international, educational survey Validity: no contraindication that the two modes yield comparable data, thus allowing the implementation of both modes in and across countries.

Problems were identified with respect to drop-out and reliable administration of fallback paper questionnaires.


Potential

On-line Data Collection (ODC) provides an alternative channel for respondents.

One-step printing, postage, data collection and entry!

Options not available on paper.

Reduced workload for data entry and preparing datasets at National Center!

For returned data processing steps are collapsed or eliminated and it requires little or no verification as particular sources of error are eliminated.

But, this means added complexity and increased workload for NRCs and IEA DPC. More tasks go on in sequence and in parallel.


SurveySystem - Key Features

Hierarchical, structure oriented survey model that keeps all information, including text passages, translations, and adaptations in files that can be easily send by email.

Three application components:

– ‘Designer’ to flexibly define survey components (e.g., questionnaire sections, questions etc.), translate text passages and preview;

– ‘Web’ for respondents around the world to complete their personalized questionnaire;

– ‘Monitor’ to audit and track participation in real-time.


SurveySystem - Key Features

Based on the concept of “cultures”, for example ‘fr-CH’ stands for: French in Switzerland.

Hosted and maintained by IEA DPC.


‘Designer’

Off-line application that runs on Windows PCs in National Centers (no internet connection required).

Works from English survey definition files received from IEA DPC.

Role based: ‘Administrator’, ‘Manager’ and ‘Translator’.

Used to create, delete, disable and edit survey components and their properties.

Allows for translation of all text passages and “system resources” (e.g., ‘Next’-button or validation messages).

Integrated preview system (built-in web server).


‘Designer’ Screenshot


‘Web’ – “Respondent-friendly Design”

Information collected on specific content domains, but we are not assessing computer literacy!

Web interface and usage has to be simple.

Standard features: login, save-and-resume, page-by-page navigation.

Accessible from any place at any time. No setups. No downloads.

Includes a Table of Contents with visual feedback about which questions have been answered or not


‘Web’ – Technical Details

Works with ‘thin client’ (internet browser) with minimum requirements:

– 800x600 screen size, 256 colours

– HTML 4.0 (standard since 1998) for tables

– ‘Dir’ attribute and bi-directional Unicode support

– Cascading Style Sheets v1 support

Supported browsers that meet these requirements (~99,9%):

– Internet Explorer 5.0+, Netscape 5.0+, Mozilla/Firefox 1.0+, Safari 1.0, Opera 7.0+, all regardless of platform

No “fancy” technologies like cookies, JavaScript, or pop-ups required!


‘Web’ – Login Screenshot


‘Web’ – Matrix Question Screenshot


‘Web’ – Table of Contents Screenshot


Real-time on-line accessible information about participation and completion status

Provides summary statistics (participation rates)

Filter can be set to highlight recent actvities, e.g. respondents who logged in the last two days

‘Monitor’


‘Monitor’ Screenshot






Implications for Survey Operations

Before and during survey administration, additional tasks are necessary in parallel and sequence to the paper instrument preparation and administration.

The following slides will give you a visual, side-by-side and detailed overview of these steps.


‘Paper’ administrationwith questionnaires only.

Use Teacher Listing/Tracking Forms.

Use National Adaptation Forms.

SurveySystem not used.

Manually adapt WinDEM codebooks to reflect national adaptations.

Use WinW3S labels.

Entire data entry with WinDEM.

Track participation with WinW3S based on Teacher Tracking Forms.

‘Paper’ and ‘On-line’ administration (split design in FT) with questionnaires AND cover letters.

Use Teacher Listing/Tracking Forms.

Use National Adaptation Forms.

Convert and adapt questionnaires with SurveySystem AFTER translation/layout verification.

Export codebooks (no alterations allowed after export!).

Use WinW3S labels for both modes.

Enter ‘paper’ data only with WinDEM.(‘on-line’ data merged at IEA DPC)

Track participation with WinW3S, based on Teacher Tracking Forms and ‘Monitor’ reports.

Mixed Paper/On-linePaper only


InstrumentPreparation

Sampling and Preparing

Survey Administration

Data Entryand Verification

SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring

Done by SurveySystem.

Convert for ODC after process is complete

for paper instruments.

Determine and track administration mode.

Monitor participation on-line

and follow-up.

No data to enter, just participation

status.

Send “cover letters” instead of

questionnaires.

No data to send. It’s at the IEA DPC already.


Integration – Sampling

School sampling is not affected in terms of coverage and selection

Within-School Sampling– WinW3S allows you to assign the

questionnaire type (paper or on-line) per default for principals and/or teachers

– WinW3S allows you to track the questionnaire type for individuals that require the other type (e.g., a school that was supposed to fill all questionnaires on-line, but in which some teachers requested paper questionnaires - FT only on-line to paper)


Integration – Instruments

Instrument Preparation– International definition files prepared by IEA DPC.– National adaptations to structure within

countries.– “Cut and Paste” conversion AFTER paper

translation, within country and international translation verification, and layout verification.


Integration – Instruments

Instrument Verification– Within country content comparison and optical

check (final paper versions vs. on-line)– NRCs send localized definitions to IEA DPC– Final optical / logical checks against paper and

NAF– Approval or request for corrections from IEA DPC– IEA DPC sets up survey for the Internet.

Codebook Adaptation– WinDEM codebooks are automatically created!


Integration – Administration

On-line respondents, usually entire schools, receive “cover letters” instead of questionnaires.

Same WinW3S labels used.

Credentials created with WinW3S are distributed to respondents together with a culture-specific and typo-safe login address, for instance: http://www.iccs2009.net/odc/71792

Administration period applies to both modes.


Integration – Data Entry and Verification

Trivial: No need to enter data!

Track participation in WinW3S from downloadable ‘Monitor’ reports (“returned, data available” vs. “no data available”) and teacher tracking forms (for other participation indicators, like “left school permanently”).

‘On-line’ data merged with ‘paper’ data from WinDEM at IEA DPC.

Verification: Done at the IEA DPC for the FT and MS.


Guidelines

Overall goal: highest possible response rates, both on questionnaire as well as variable level.

Decision and responsibility of usage lies with NRCs, who are generally free to decide about type and scope of implementation, i.e. as the default mode, for principals and/or teachers, or for specific schools or individuals.

Individual restrictions in terms of computer/Internet penetration, poor on-line response rates in previous surveys or other obstacles should be taken into account.

Eventually, the results of the field trial in terms of response rate and drop-out should become the basis for decision.


Restrictions

While participation is optional, participation in the FT is mandatory if you want to use ODC in the main run.

On-line data collection is administered with the IEA SurveySystem ONLY. The IEA DPC can not manage the verification and data conversion for other systems.


Restrictions

On-line mode may become the default mode in your country for the MS, but it is MANDATORY to provide paper questionnaires to individuals/schools without Internet access, equipment, or that simply refuse.

For the MS National Centers should therefore check with schools whether the on-line mode is acceptable for them BEFORE send-out of materials. In other words, school coordinators should not need to ask for paper questionnaires if they prefer them. Enquiries should take place during the first contact/notification of schools or together with the send-out of teacher listing forms.


Field Trial Timeline

Documentation of software is part of the ICCS 2009 Survey Operations Procedures Manual Unit 3.

International definition files distributes together with SurveySystem software (and if applicable a short list of deviations from the paper version) on August 20, 2007 or earlier.

“Copy-and-Paste” conversion may start AFTER translation AND layout verification for paper questionnaires is completed, starting October 15, 2007.


Field Trial Timeline

Submit zipped definition files to IEA DPC within one week after layout verification approval, latest by October 22, 2007.

Allow at least one week for review and paper/on-line comparison at IEA DPC.

Begin paper and on-line administration from November 5, 2007.


Summary

Participation is optional and should be guided by the level of confidence that participants have with the mode.

FT participation is mandatory and vital to collect experience for the main data collection implementation.

Fall-back paper questionnaires are mandatory (ODC can not be the exclusive mode for the time being).


Summary

More tasks going on in sequence and parallel in terms of preparation and monitoring.

Challenge to assign and track the correct questionnaire mode (on-line or paper) to individuals in schools. Some may agree to participate on-line, but others will require paper versions.

Advantages gained from reduced printing and shipping costs, less workload during data entry and preparation.

Software and hosting will be provided by the IEA DPC.


Thank you for your attention!

Documents

Data Management Seminar, 9-12th July 2007, Hamburg 11 ICCS 2009 – On-line Data Collection A General Introduction