By:Brian Blake, Ph.D.

Jennifer VedderErin Sonenstein

Consumer-Industrial Research ProgramCleveland State University

Vanguard ConferenceNational College of Natural Medicine

Portland, Oregon

January 11, 2011

Evaluating Survey Evidence:A Tale of Two Surveys

Too often we hear…

“A survey usually raises more questions than it answers!”

“Be careful that a survey doesn’t lead you in a wrong direction!”

Too little do we listen…

“A goal without a plan is just a wish.”- A. Saint-Exupery, 1900-1944

“You designed this research backwards!!!”

Five Key Questions

1. Are we listening to the right people?2. Are we listening to enough of the right

people?3. Can we trust what they say?4. Do we understand what they tell us?5. Does the information provide guidance for

action?

Two Study Designs

“Gen-Pop” ND Survey

“Gen-Pop”Descriptive and Prescriptive

Online survey of Oregon residents to:Identify the public’s familiarity with NDs Gauge the public’s image of NDs relative to other

health care providers (HCPs), broadly definedUnearth the distinct segments that compose the

Oregon population, the segments being sectors that differ in regard to their orientations toward medical care

Estimate the demand for naturopathic medicine (NM) and how it changes from one segment to another

Calculate the “targetability” of each segment, i.e., the relative contribution of each segment to the spread of NM in Oregon

Devise a marketing/communication/service distribution strategy based on the above to increase usage of NDs and NM care in Oregon.

“Gen-Pop”Discover parameters that can be effectively

brought to the attention of Oregon legislators and health insurance carriers that can provide a rough gauge of:

Unmet health care needs in OregonPublic satisfaction and particular difficulties

experienced with health care coverageReadiness to turn to NDs as a Primary Care

Physician

ND SurveyMainly Descriptive

Online survey of the nation’s ND community to:Profile the ND community in America in regard to a

variety of parameters:o Engagement in particular professional activitieso Professional goals and interestso Provision of alternative treatment modalitieso Form of reimbursemento Involvement in a primary care roleo Reliance upon various forms of medications in

prescriptionso Income from NM and other income streamso Demographic characteristicso Licensures and certifications beyond the ND

ND SurveyChart the diversity of NDs in the above regards

and identify sectors or “types” of NDs that compose the contemporary ND community in America.

Assess the needs and priorities of each ND type/sector in order to learn how to serve their distinct interests.

Identify graduates of ND programs who are not presently involved in NM. Who are they? Why are they no longer active?

In conjunction with information from the Oregon General Population survey explore the viability of mutually beneficial referral networks with non-ND healthcare providers.

Five Key Questions

1. Are we listening to the right people?2. Are we listening to enough of the right

people?3. Can we trust what they say?4. Do we understand what they tell us?5. Does the information provide guidance for

action?

Q1: Are we listening to the right people?

“Right people” = Representative sample of population

So, how do we obtain a representative sample?

Six steps to a representative sample…

For a representative sample…

1. Specify population clearly. Who is in? Who is out? Gen-Pop… adults (+18 years old) in Oregon who

have received care from a health care provider in the previous 2 years (from January 1, 2009)

ND Survey… persons who graduated in the past ten years (since 2001) with an ND degree from an accredited US college and/or who are current members of the American Association of Naturopathic Physicians (AANP) or a state level ND organization (e.g., OANP). Does not include current students working toward an ND degree. Includes ND degree holders not presently active in NM

For a representative sample…2. Devise the “sampling frame,” a

population listing with reasonable coverage current and is error free Gen-Pop - members of an online panel residing in

Oregon previously screened for age and medical care based on vendor records and confirmed by questions in survey. Like e-Rewards

Evaluation - practical, not perfect- coverage OK, but suspect- current and clean

• ND Survey - listing of alumni records of five major ND colleges in US, membership list of AANP, membership listing of all state level ND organizations, compiled-cleaned by HRI

For a representative sample…3. Draw probability sampleHere a “stratified random sample”• Gen-Pop - sample proportionate to

population of Oregon counties (most recent US Census estimate)

• ND Survey - sample drawn proportionate to the ND population in each of 50 states (% calculated from sampling frame)

For a representative sample…4. Design to obtain a high response rate

Dillman reviewFinding in ancient crypt

High rate essentials

Procedure Gen-Pop ND Survey

1. Pre-notification

2. Personalized cover letter

3. Guaranteed anonymity

4. Incentive

5. “Noble cause”

6. Credible sponsor ?

7. Accessible contact person

8. Fairly easy to answer questionnaire

9. Multiple (3) waves

For a representative sample…5. Design so that subsample weighting of

obtained data is possibleGen-Pop - double weighting

by county to make data set proportional to county’s population (Census based)

by those few demographics (e.g., age) empirically shown in study to be predictive of readiness to see an ND (Census based)

ND Survey - single weightingby state (based on percent of sample frame)

Five Key Questions

1. Are we listening to the right people?

2. Are we listening to enough of the right people?

3. Can we trust what they say?4. Do we understand what they tell us?5. Does the information provide guidance for

action?

Q2: Are we listening to enough of the right people?

A. Standard formulae based on:• desired precision, e.g., ±3%, ±2% band • desired level of confidence that population

characteristic is within band, often 95%• variability in characteristic of interest

• more variability, larger sample needed• typically assume maximum• population vs. sector

• number/size of subpopulations to be analyzed “stand alone,” males vs. females

• projected response rate, e.g., if project 50% response, send out double needed sample size

B. Gen-PopGiven: population =3.8 million (adjusted from

3,421,399)

qualified (had care) = 70%, response rate = 20%

variability = pop. max (50-50), seg mod (70-30)

For OR whole need 600

precision = +/- 4%

confidence = 95%

send out = 3,000 For typical (of 5) segment with average 150 per

segment

precision = =/- 7%

confidence = 95%

send out = 3750

C. ND SurveyGiven:

For population (US) as a whole = 6,000, response= 50%, 90% qualify

Variability = pop max (50-50), seg mod (70-30)

precision = ±3%confidence = 95%

For typical (of 5) segment precision = ±6% confidence = 90%

D. Recommend

Gen-Popneed 750 (assuming 5 segments)send out 3750 (assuming 20%

response) possible “booster” sample if

familiarity with NDs lowND Survey

needed - 750send out - 1,500

Five Key Questions

1. Are we listening to the right people?2. Are we listening to enough of the right

people?

3. Can we trust what they say?4. Do we understand what they tell us?5. Does the information provide guidance for

action?

Q3: Can we trust what they say?Eliminate “bad respondents,” for example

overstatementssuspicious patterns of answers

Eliminate bad items for “illogical” answers based on relationships (correlations) among items

For Gen-Pophealth care provider (HCP) familiarity with licensed

phentologist What they say versus what they do. Stated Impact

vs. Predicted Impact for HCP characteristics

Q3: Can we trust what they say?

EvaluationMinimal checks for NDPractical approach for Gen-Pop

Five Key Questions

1. Are we listening to the right people?2. Are we listening to enough of the right

people?3. Can we trust what they say?

4. Do we understand what they tell us?

5. Does the information provide guidance for action?

Q4: Do we understand what they tell us?

How to interpret a given questione.g., suppose numerous NDs say they place high priority

upon time to educating the public about health care issues? Why do so? Good for business? Desire to advance NM

care? Enjoy being in limelight?Use of mathematically derived dimensions that

“look underneath” overt answers in both Gen-Pop and ND Surveys

Example of dimensions in image of NDs and other HCPs in Gen-Pop

First, the HCPs considered Second, the characteristics Then, the dimensions

HCPs Included A conventional medical doctor (MD degree) Naturopathic doctor (ND degree) Licensed physicians assistant (PA) for primary care Licensed practical nurse/licensed nurse practitioner/registered

nurse for primary care Licensed midwife/Dula Psychiatrist (MD) Psychologist (PhD/PsyD) or other licensed mental health

professional Chiropractor (DC degree) Physical therapist (PT degree) Certified massage therapist Licensed acupuncturist Registered dietician Yoga, other mediation/controlled movement trainer

How well are particular HCPs described by each characteristic? (Scale 1-5)

Characteristic ND MD Rest

Prescribes principally natural healing methods (such as medicines from plant products) 4.1 1.5 2.0Prescribes only medications that have had significant scientific research and have been approved by the Food and Drug Administration

…Emphasizes preventative medicine to avoid ailments before they occur 4.1 2.1 3.1Spends time getting to know me as a patient 4.2 1.9 2.5

…Uses conventional healing methods (for example: acupuncture, herbal remedies) 4.3 1.2 3.1Makes it simple to schedule appointments that fit into my schedule 3.4 1.2 3.9

…