Mario AlbisinniElisa MarzilliFederica Pintaldi

Cognitive Test related to Labour Force Survey ad-Hoc Module on Migrants and their

Immediate Descendants

The European Conference on Quality in Official Statistics

Rome. 9-11 July 2008

Introduction: Labour Force SurveyThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The Labour Force Survey (LFS) began on quarterly basis in 1959

It is the biggest and most complex sample survey - 75.000 households / 170.000 individuals

- average sampling rate: 1/300- statistics for small geographical areas (NUTS II)

The LFS is a survey of households that provides information about people's employment status and conditions

Official source for employment and unemployment

Many Eu Regulations

Introduction: Labour Force SurveyThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

In order to match EUROSTAT standards. on 2002 ISTAT developed a new Labour Force Survey

It replaced the previous old survey in 2004

The main changes concern both content and technical points of view:

- the survey has become continuous- from a PAPI technique to a mixed technique CAPI-

CATI- a new questionnaire- new information

Since 2005 we have started to diffuse information about non nationals

Introduction: Labour Force SurveyThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

Ad hoc modules: specific items every year

2000 Transition from school to working life

2001 Length and patterns of working time

2002 Employment of disabled people

2003 Lifelong learning

2004 Work organization and working time

2005 Reconciliation between work and family life

2006 Transition from work into retirement

2007 Accidents at work and work-related health problems

2008 Labour market situation of migrants

LFS: Ad-hoc Module 2008The European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The aim is:

complete identification of the population of migrants and their immediate descendants

comparable data at European level

comparison between migrants (and their descendants) and nationals about main characteristics of labour market

analysis of the factors affecting integration in the labour market

The European Conference on Quality in Official Statistics

Rome. 9-11 July 2008

In order to improve the quality of survey data for migrants by decreasing language barriers we conducted a cognitive test

LFS: Ad-hoc Module 2008

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

the cognitive test is a method to reduce non-sample errors and improve the quality of the data

the cognitive interview identifies the aspects of the question that lead to errors in the answers

the cognitive interview decomposes and codifies the thoughts of the interviewees when they answer a

questionnaire

the purpose is to stimulate the respondents to suggest the difficulties encountered in answering questions

The European Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The cognitive process can be drawn up into 4 steps:

1. Comprehension: the respondent tries to understand the question and its terms

2. Retrieval of relevant information from the memory: the respondent retrieves and selects the information in

order to be able to answer the question

3. Decision-making (or judgment) process: the respondent reflects on what is the correct answer and

about personal experience to share

4. Answer process: the respondent decides on the answer to be given and communicates it to the interviewer

Cognitive Test

AM 2008: Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

Methodology

An interviewer accompanied by one observer conducted a face-to-face qualitative interview

Both the interviewer and the observer had a trace they had to follow and which allowed them not to go

off the point

Each interview was registered in order to be able to listen to it again and transcribe it

A standardized scheme was compiled to evaluate the interviewee’s level of question comprehension

The European Conference on Quality in Official Statistics

Rome. 9-11 July 2008

Sample

We interviewed a selected group of 30 migrants by sex. citizenship and

professional status

The sample has been divided in two:

- first 15 interviews: test questionnaire

Cognitive Test

some questions were re-formulated because of problems emerged

- second 15 interviews: test new questions

The European Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The questionnaire contains 3 kind of questions:

Questions to be tested: read exactly the way they are written on the questionnaire to evaluate whether the terms are correct and comprehensible

Probe questions: asked after the questions to be tested allow to examine more in-depth the meaning of the answers given

Questions for the observer: allow to survey the non-verbal reactions of the communication process and suggest what kind of information should be written on for each question

The interview

Moreover the probe questions can be integrated with other questions the interviewer may freely elaborate regarding aspects that emerge during the interview

Cognitive Test

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The probe questions concern:

problems relative to the question comprehension/ interpretation

re-wording of the question by the interviewer

certainty of the answer given

strategy used for retrieving past information

opinion on a specific aspect discovered during the interview

reason for choosing a specific answer and the difficulties encountered in formulating it

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The cognitive test questionnaire: exampleQUESTIONNAIRE QUESTIONS PROBE QUESTIONS

D1. In which year did you come to live in Italy for the first time? -        Do you understand “to live in Italy”?

-          Year___________  

-        Is there a difference between “live” and “reside” in Italy?

-       What does “residence” make you think of?

D9. Which is the main reason for coming to Italy?

-       What were you thinking of when answering the question?

1|_| To work -       Has that reason changed over time?

2|_| To study -       What does “main reason” make you think of?

3|_| For family reason (reunion. transfer) -       The reason you mentioned. to what year does it refer to?

4|_| For marriage. affective relationship Read the modalities

5|_| To ask for political asylum

-       Are the answers provided clear? Is it easy to choose an answer?

996|_| Other (specify_____________) -        Is there an answer you would like to add?

997|_| Does not know -       Could this question be annoying? In your own words…

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The first step was to transcribe the dialogues of the interviews and notes of the observer.

For each question. we indicated:

answer given respondent’s difficulties observer’s comments

The analysis

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

We compiled a standardized scheme to evaluate the interviewee’s level of comprehension:

interviewee’s question

answer modality

pertinence of the answer

The analysis

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The interviewee’s question comprehension:

A1 – Did the interviewee ask to repeat the question?

1 – yes

2 – no

A2 – Did the interviewee understand the question immediately?

1 – yes

2 – no

A3 – How difficult was for the interviewee answering the question?

1 – a lot

2 – some

3 – not much

4 – none at all

A4 – Did the interviewee understand the time reference?

1 – yes

2 – no

3 – not present (the question did not have a time reference)

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The interviewee’s question comprehension:

A5 – Did some words of the question result not very comprehensible?

1 - yes

2 - no

A5a – (if no) State the not very comprehensible words

A6 – Other question comprehension problems?

1 - yes. question too long

2 - yes. complex sentence

3 - yes. other problems (________________________________________________________)

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The answer modality:

A7 – Were the answer modalities clear?

1 - yes. very clear

2 – yes. rather clear

3 – no

4 – not present (the question did not provide for answer modalities)

A8 – Were the answer modalities exhaustive?

1 – yes

2 – no3 – not present (the question did not provide for answer modalities)

A9 – How difficult was for the interviewee choosing an answer modality?

1 – a lot

2 – some

3 – not much

4 – none at all

5 – not present (the question did not provide for answer modalities)

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The pertinence of the question with the interviewee’s condition :

A10 – Through probe questions did the interviewee’s initial answer change?

1 – yes. it changed

2 – no. the answer was always the same

A11 – How pertinent was the interviewee’s answer to his/her condition?

1 – a lot

2 – enough

3 – not a lot

4 – not at all

5 – I do not know (difficult to understand whether the interviewee’s answer is correct)

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

for each question. the combination of the information from the content analysis has allowed

reconstructing a comprehension index

it was created adding the “positive comprehension” answers and dividing these per total number of

answers

The index moves from 0 to 100 (max comprehension)

answers with “positive comprehension” total answers

Comprehension Index = *100

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

In relation to the comprehension index of wording questions we considered positive “comprehension answer”:

Did the interviewee understand the question immediately? [Modality: yes]

How many difficulty was for interviewee answering the question? [Modality: not much/not at all]

Did the interviewee understand the time reference? [Modality: yes]

Did some words of the question result not very comprehensible? [Modality: no]

Other question comprehension problems? [Modality: no]

Cognitive TestThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

In relation to the comprehension index of modalities of answer we considered positive “comprehension answer”:

Were the answer modalities clear? [Modality: yes]

Were the answer modalities exhaustive? [Modality: yes]

How difficulty was for the interviewee have to choose an answer modality? [Modality: not much/not at all]

Cognitive Test: results The European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

Questions

Comprehension Index of wording

question

Comprehension Index of modalities of

answer

1st group 2nd group 1st group 2nd group

Year of arrival in Italy 94.7 82.7 Open

Age at arrival in Italy 72.2 94.7 Dichotomised

Main reason for coming to Italy 76.8 91.7 78.6 91.1

Coming for work 75.0 94.4 80.0 96.3

Help received to find a job 57.6 76.1 81.0 93.3

Services used during the first two years 50.7 81.0 76.9 95.6

Type of permit 81.5 93.3 100.0 95.6

Permit that allows to work 83.7 100.0 Dichotomised

Improve Italian language for adequate job 46.7 53.3 88.4 86.7

Recognition of educational degree 70.0 86.5 86.7 97.4

Knowledge of Italian language 75.0 98.2 83.3 92.0

Cognitive Test: results The European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

The new wordings improved the comprehension levels: the average comprehension level of

wording question arises from 71% of the 1st group to 87% of 2nd group

Cognitive Test: resultsThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

Problem Type of correction

The structure of the question is complex

New wording of the questionDecomposition of the question

The terms used are not clear WordingThe answer modalities are not

pertinent or exhaustiveElimination/addition of some

modalities

The question is superfluous or not easy to understand

Elimination of the question

Two close questions do not appear logically consequential

Change in order of the questions

The main problems that could arise when using cognitive test and how they can be corrected

Cognitive Test: some examplesThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

Which is the main reason for coming to Italy? 1|_|To work 2|_|To study

3|_|For family reasons (reunion. transfer) 4|_|For marriage. affective relationship

5|_|To ask for political asylum996|_|Other (specify____________)

997|_|Does not know

PROBLEMS:

The answer modalities were neither clear nor exhaustive. The modality “to work”. produced some uncertainties for those who

came to Italy to search for a job. R: to create myself a future. I cannot say for work because I did not have a

work contract; I came to Italy to find a job

The reason changes in the timeR: To work. Well. at first not to stay in Italy. My objective was to go to

Spain and continue to study. Then. I had to stay in Italy because I did not have much money anymore; I found a job and stayed…

Cognitive Test: some examplesThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

Which is the main reason for coming to Italy? 1|_|To work 2|_|To study

3|_|For family reasons (reunion. transfer) 4|_|For marriage. affective relationship

5|_|To ask for political asylum996|_|Other (specify____________)

997|_|Does not know

CORRECTIONS: the first modality was divided in “look for a job” and “to work” the verb was put to the past tense to focus the attention on

the reason for migrating rather than on the reasons that subsequently came to create

a new modality was added “short stay in Italy to go to another country” to survey when Italy is a transitory destination rather

than the reason for migrating

Cognitive Test: some example of resultsThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

In the 2 years subsequent to your arrival in Italy. did you use one of the following services supplied by public or private institutions/associations:

1|_| Assistance for searching a job 2|_| Professional training (including the one on the workplace)

……

PROBLEMS: the structure of the question resulted to be complex and the time

reference was not always understood.I: is it clear in the question that we refer only to first two years? R: No. I

understood the whole period

CORRECTIONS: the wording of the question was simplified

During the first 2 years you lived in Italy. did you use one of the following services:

Cognitive Test: some example of resultsThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

Do you need to know better Italian to find a job that is more adequate to your skills and/or previous working experiences?

1|_|Yes 2|_|No

PROBLEMS: the phrase structure resulted complex: interviewees asked several

times to repeat the question. three terms were not very comprehensible: “Italian”. “skills” and “adequate”

most of the interviewees answered only the first part of the question (knowledge of the Italian language) without considering the second part of

the question (work more adequate to one’s own skills and/or previous working experiences)

R: what does “Italian” mean? The Italian language or Italian as person?”R: “adequate” though is difficult …I would say “better”…

R: to persons who do not understand well Italian. the question is too long. At first. one is not sure of what it is all about

R: Instead of “adequate”. I would say “more right for the things you can do”

Cognitive Test: some example of resultsThe European

Conference on Quality in Official Statistics

Rome. 9-11 July 2008

Do you have a job close to your studies and competences? 1|_|Yes

2|_|No

To have a job close to your studies and competences. do you need to improve the knowledge of the Italian language?

1|_|Yes 2|_|No

CORRECTIONS: the question was divided into two parts

in the first part. respondents are asked if they have an adequate job if “no” we asked if the difficulty is because of poor knowledge of the

Italian language

Ad-hoc Module 2008: Cognitive Test

Conclusion

after a second cycle of interviews the questionnaire becomes definitive

the cognitive test has been particularly useful in order to set-up of the ahm 2008

we reduce the non-sample error that bias the accuracy of data

Rome. 9-11 July 2008

Cognitive test highlights hidden errors

The European Conference on Quality in Official Statistics

Rome. 9-11 July 2008

THANK YOU A LOT!