Effectiveness of Monetary Incentives and Other Stimuli Across Establishment Survey

Populations

ICES III 2007

Montreal, Quebec Canada 4.30.07

Danna Moore and Mike Ollinger

(1) Why is this study important?

(2) Gaps in Establishment Survey Literature

(3) Provide suggestions for implementing establishment surveys

Objective

Establishment Respondents: What type of respondent are they?

Household Respondente.g. small farmer, small business owner

Large business or Org Respondent Multiple locations Gate keepers Record system

Can vary along the continuum

Why is this important?

Questionnaire variation Converting Mandatory reporting R’s Vary over time Risk/Difficulty -- cash incentives Population based sample sizes Experimental design Cost/Benefit Generalize

Stimuli Tested

• Cash Incentives

• Cash plus special postage/package

• Multiple modes

• Mode sequencing

• Mode preferences

• Visual Design— color background vs. none• Answer boxes stand out

Some Answers Towards the Big Question

Do incentives help or hinder in obtaining survey responses from businesses?

• Crosses types of establishments and industries.

• Experiment based • Population based• Random assignment

Understanding Why People ParticipateSeveral theories (Dillman, 1978; Gouldner, 1960; Biner and Kidd, 1994; Groves

et al., 1999).• Social exchange

Leverage Saliency Theory—Groves et al. (1999)Decision to participate is a series of interactive additive

factors.

Some are survey specific and others are person specific.

Incentives are viewed as an inducement used to compensate for absence of some factors (i.e., saliency or sense of duty).

1999 POQ vol 64

Leverage Saliency TheoryOf Survey Participation

Features of Establishment Surveys That Often Lead to Survey Errors

1. One person selected to

represent

establishment.

2. Burden increases as

they answer as a

representative.

3. There is a respondent

questionnaire

interaction.

4.Respondent’s characteristics in relation to the establishment influences their ability to respond.

5. intermediary between the questionnaire and the characteristics of the record system.

6. Organizational environment

7. Extenuating survey situation

• Businesses differ across industries by size, structure, and organizational environment.

• Each survey may have a situation or

circumstance that impedes contact.

• These differences and circumstances often

influence how well a survey request can

penetrate an establishment

Factors Influencing Response

Gaps

• Lack of monetary incentive studies:

Few experimental treatments for definitive comparisons and outcomes.

Few comparisons of cash versus “cash like” (e.g. checks, ATM cards) studies—could be especially important for government.

No empirical demonstrations of effectiveness of incentives across types of industries and firm size.

Few comparing effects of various size incentives. Not much on effectiveness of survey modes and mode

sequencing Response attributable to survey mode sequencing.

2001 HMO Survey Physicians N=1474

43%

57% 63% 66% 64%

80%70%

0%

20%

40%

60%

80%

100%

1Control T1: FC, FC T2:Priority No, FC $10 T3: Priority $10, FC T4; Fc, Priority $10T5 Priority $10,FC T6: Priority $10 2x 2004 Priority $10 2x

***

**

2003 USDA Nationwide Meat Manufacturers N=1,705 Response rates achieved by experimental treatment group

44%

23%

47% 45%

56%

67%70%

57%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Pilot-- Redmeat Mf.

Pilot--PoultryMf.

Full-- RedmeatMf

Full Poultry Mf

Control: 2x FC No $ Trt 2: 2x priority Trt 3: 2x Priority $5

******

2006 Snake River Grain Warehouse and Shipper Survey Response by Experimental Group (n=424 elevators)

58%

76%

64%63%

49%

10%

57%

72%

43%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Graincompanies

Port Shippers Grain Elevators

Control Trt 1: 2X FC No $ Trt 2: FC $5 , FC No $ Trt 3: 2x FC $5

Chi SQ 6.7P < .01

***

N.S.

N.S.

2007 Evaluation of WA Plastic Pesticide Container Use/Recycling Control versus Treatment, N=1,986

35%

52%

22%

42%

0%

10%

20%

30%

40%

50%

60%

CommercialApplicators/Dealers Licensed

for Pesticides

Licensed Producers

Control Trt1: No incentive, FC Trt 2: $5 incentive, Priority

Chi Sq. 15.86***P<.001

Chi Sq. 33.5 ***P< .0001

2006 Oregon Business Environmental Management Survey Type and Experimental Treatment Group Initial sample n=1964

3%

15%

51% 51% 52%47%

28% 25%

8%

18%19%

33%

23% 24%

46% 42% 43%40%

42%41%

0%

10%

20%

30%

40%

50%

60%

Construction: n=394

Manufacturing n=752

Transportation n=343

Accomodation n=475

All Response

Ctrl T1: FC 2x Trt 2: Priority No $ 2x Trt 3: Priority $5, FC Trt 4: Priority $5 2x

***

**

** *** ***

**

***

34%

53%

0%

10%

20%

30%

40%

50%

60%

Percentage

1Control t1: FC 2x No Incentive Ttrt 2: FC $5 2x

2006 Trade Adjustment Assistance Survey, All Qualifying Industries N=6,429Exp. Incentive Treatment vs. Control ---Completion Rates

Sig. Chi Sq. 51.07

P<.001 ***

***

Trade Adjustment Assistance (TAA) 2006 Survey of Program Nationwide – Fisheries N=5,592

29%

58%

31%

48%41%

66%

35%

54%

41%

62%

48%54%

65%

45%

59%

70%

0%

10%

20%

30%

40%

50%

60%

70%

Catfish WASalmon

AKSalmon

TXShrimp

LAShrimp

ALShrimp

FLShrimp

GA-Shrimp

MSShrimp

NCShrimp

SCShrimp

Control T1: FC 2x Trt 2: $5, Priority 2x

**

**

******

***

Trade Adjustment Assistance (TAA) 2006 Survey of Program Nationwide – Commodities

N=837

73%

88%

59%

77%

59%

74% 76%

62%

0%

20%

40%

60%

80%

100%

Grapes CA Olives ID-Potatoes Blueberries Lychees

Control T1: FC 2x Trt 2: $5, Priority 2x

NSNSNS

2%1%

3%

69%

13%12%

Mail--No incentive Mail--$5 priority Web -- No incentive, FC

Web--$5 priority Tel.-No incentive, FC Tel-$5 priority

TAA Survey—Percentage of Completes Associated with Incentive Experimental Assignment and Survey Mode

Mail Completes

Telephone Completes

Web Completes

Suggestions for implementing establishment surveys

Effective Practice:

• Contact respondents multiple times

• Contact respondents in multiple modes.

• Allow for survey mode preference.

• Design surveys that reduce burden • short, conditional branching, ease

• Use leverages

Visual Design Effect – FARW Commercial Dealers N=1600

Background Shading w/ Visible Answer Boxes

Does it make a difference?Is there an interaction effect w/ Incentives?

34%

48%

36%

56%

0%

10%

20%

30%

40%

50%

60%

1

Bkg. Color No $5Bkg. Color $5

No Bkg Color No $5No Bkg Color $5

***

***

Visual Design TestFARW Growers With Pesticide License

**

24%

44%

21%

39%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

1

Bkg. Color No $5

Bkg. Color $5

No Bkg Color No$5

No Bkg Color $5

******

***

• All 3 survey modes generated completes

• Exper. treatment (Cash incentive & priority mail) stimulated more responses in all 3 survey modes—large interaction effect.

• Offering web as an alternative option garnered 15%

• Telephone last 3% – still effective

Influential CircumstancesSaliency: Topic interest area of business emphasis for entity. High personal interest for respondent. High level of public or political concern

Role of survey sponsor. Regulator or source of certification. Mandatory reporting. Source of program $$ or sponsorship

Response Burden Complexity, length, multiple reports

What was learned from experimental trials

• Token cash incentives were effective across types of establishment and

industry populations.

• 2-day Priority mail was more effective than first class mail.

• Priority mail alone just slightly better than First class

• Cash incentives combined with priority mail was synergistic.

• Mixed mode strategies are very helpful and work.

• Respondents may have mode preferences.

• Establishment population characteristics and the selected respondent characteristics need to be considered jointly in explaining response.

• The survey circumstances and situation impact establishment response.