Cognitive Interviews

For Business surveys

Marcel Levesque

QDRC

Statistics Canada

Structure of the presentation

Cognitive interviewing defined. How is an interview constructed? How is an interview conducted? Cognitive interviewing applied to business

surveys. Examples of issues tested in the UES survey. Conclusion.

Cognitive interviewing defined

A process to explore:a respondent’s understanding of questions their strategies to retrieve the information

requested by a survey their judgment of the adequacy of the

retrieved information their ability to report the required information

How is the interview constructed?

Meeting with survey manager to discuss the issues to be testedo The appropriateness of questions,

concepts and terminology

o Respondents` willingness and ability to respond to survey questions

o Respondents’ understanding of the questions and what to report

o Respondents’ use of external sources of information such as financial or administrative records and their need to consult other individuals who can provide the information requested

o The compatibility of questions and response categories with respondents` record keeping practices

o Difficulties respondents may encounter in retrieving information and completing the questionnaire

Designing an interview guide

o Based on the issues discussed with the survey manager, a guide is designed to test the questions and issues identified.

o Comprehension, information retrieval, judgment and communication are usually issues to be explored.

Comprehension

o Difficulties encountered by a respondent in understanding a question may be explored:

o Instruction content - inaccurate or conflicting instructions, instructions separate from the question

o Question wording - ambiguous terms, critical definitions missing

o Question structure – several questions in one, Question /answer mismatch

o Question content - complex topic, topic carried over to another section

o Navigational instructions - inaccurate instructions (move to the wrong place), confusing flow

Information retrieval

o Difficulties encountered by a respondent in gathering the requested information may be explored:o Multiple sources - information may be

distributed in multiple sections /departments in the organization

o Record retrieval - records in multiple sources,

record access issues

o Memory retrieval- reference period too long, recall problems and telescoping errors.

Judgment

o Difficulties encountered by a respondent in assessing the relevance of the retrieved information to the data requested may be explored:

o Data incompatible with existing recordso Need to evaluate and synthesize multiple sources of

informationo Potentially sensitive information, strategic or proprietary

issues may prevent revealing certain information

Communication

o Difficulties encountered by a respondent in reporting the response to a question may be explored:

o Missing response categorieso Mismatch with technical languageo Response terminology – critical definitions may

be missing

How is the interview conducted?

o Respondents for cognitive interviews are usually recruited from Statistics Canada’s Business Register or from another frame of establishments provided by the industry-specific program areas.

o Respondents are identified based on the project team’s specifications, designed to ensure a selection of business establishments for each industry sector to be tested.

o Appointments are made with the person within the business establishment who usually completes questionnaires for Statistics Canada.

o The ideal respondent is the person in the business who is most knowledgeable about the data requested, who has access to this data and who also has the authority to release it.

o Respondents are provided with an explanation of the purpose of the cognitive interview with emphasis on the fact that Statistics Canada is consulting with them to obtain their feedback and opinions.

o The importance of their participation is emphasized and they are given assurances of privacy.

o A consultant from the QDRC conducts a one-on-one interview at the respondent’s place of business and observers from the client industry sector take notes during the interview.

o The interviewer walks through the questionnaire with the respondent and discusses issues as they arise.

o The interview may be recorded with the respondent’s acceptance.

Cognitive interviewing applied to business surveyso The Unified Enterprise Survey (UES) began in

1997 with a pilot of a small number of industries and has continued to the present time where it now encompasses the majority of business surveys conducted by Statistics Canada.

o The UES is basically a single questionnaire with a variant for each industry sector. There is a set of core questions asked of every industry and industry specific questions for each different sector.

o Core questions cover such topics as the reference period covered by the survey (financial year of the establishment), revenues and expenses, personnel, sales according to types of clients and their location and, if appropriate, questions on international transactions.

o Industry specific questions refer to the main commercial activity of the establishment and industry characteristics, namely sales activity specific to the sector.

o The purpose behind UES is to develop a common approach with concepts harmonized across all industries to:

o facilitate data collection and data captureo reduce and better manage response burden o to increase response and data quality.

Examples of issues tested in the UES

Comprehension: Question structure

One of the difficulties encountered with the 2005 survey was in relation to a mismatch between the question and one of the answer categories.

F- Industry characteristics – automotive repairs and maintenanceSales of goods and services

1. Repairs and maintenance – labour costs

________________________________________

2. Repairs and maintenance- parts and accessories

used for repairs and services _______________________________________

3. Sales of merchandise, parts and accessories for

resale in their original state

________________________________________

o This section referred to sales of goods and services, but the first answer category asked to record labour costs. Consequently, respondents were confused.

o The intention behind this question was to obtain the sales value of labour costs billed to the client but respondents failed to understand the question in this way. The difficulty here was a mismatch between question and answer, which led to confusion.

Recommendation

o Replace the expression

‘labour costs’

by

‘sales resulting from labour costs billed to clients’.

Comprehension: Question wording

o Another difficulty encountered was in relation to the question wording in the French version of the questionnaire, resulting from the translation from English to French.

E- expenses

1. Cost of employment and personnel agencies

___________________________________

Coûts des bureaux de placements et des

services de location de personnel ______________________________

o The expression ‘personnel agencies’ translated into ‘services de location de personnel’ did not make sense to the French respondents since the word ‘location’ in French means rental or leasing. Respondents did not understand the meaning of the expression ‘services de location de personnel’. As a result, many respondents did not know what costs to report in this column.

Recommendation

o Since ‘bureaux de placement’ was a more common expression:

o leave out the reference to ‘services de location de personnel’.

o include only this category in the question

Retrieval: multiple sources and record retrieval

2006 survey on entertainment and leisure

Some respondents were involved in the health fitness industry offering a variety of services such as a spa, massage, tennis courts, pool and exercise rooms. Some of the expense items listed in the questionnaire did not correspond to their books.

E- expenses

1. Wages and salaries

________________________________

5. Contract workers ___________________________

Item 1 in the questionnaire asked for salaries in Canadian dollars.

Item 5 asked for the dollar value attributed to contract workers.

Some employees are hired on contract but the costs are listed under salaries in item 1.

Respondents would not have reported any dollar value in item 5 for contract workers and there was no way for them to indicate that these expenses were all under item 1 on salaries. Consequently we would have ended up with an unanswered question without knowing why.

Since expenses are part of the core questions applicable to all industry sectors, it is quite difficult to have answer categories correspond to all industry sectors. RecommendationAdd a space for ‘comments’ under the section on expenses to allow respondents the possibility of explaining the items they are unable to provide as listed.

Judgmento Potentially sensitive information, strategic or

proprietary issues may prevent respondents from revealing certain information.

2005 survey on service industries

o Some respondents involved in “real estate, leasing and property management” said they would not provide information on revenues because their company had a policy on confidentiality which would not allow the release of such information. They did, however, mention that they would be willing to provide estimates.

Recommendation

Provide revenue categories to encourage respondents to report information on revenues they would not give if exact figures were requested.

Communication

o One of the difficulties encountered in the communication phase of the process for the 2006 Service industry survey was related to response categories.

F- Sales according to product type Architectural products

1. Residential construction projects

______________________________

2. Office construction projects ______________________________

3. Construction projects for retail

businesses and restaurants

______________________________

4. Construction projects for hotels

_____________________________

5. Construction projects for hospitals ______________________________

6. Construction projects for educational institutions

______________________________

7. Construction projects for industrial

buildings

_______________________________

Where would you report construction projects for airports?

Recommendation

Add significant categories identified by respondents i.e. construction projects for airports.

Conclusion

In a business environment, a respondent’s main preoccupation is the success of his business. Few respondents will consider spending time researching information if it is not readily available or spending time trying to understand questions that may not be clear.

As indicated in this presentation, many issues may contribute to response burden and have a negative impact on the quality of the results of a survey.

Cognitive interviewing:o provides us with a greater

understanding of some of these issueso allows us to improve a questionnaireo allows us to increase the clarity of the

questions, reduce response burden and result in increased data quality.