Survey Dilemmas:Determining Sample Size Improving Response Rate Measuring Satisfaction

Daniel R. Judd, Ph.D., M.P.A.Judd Research

2009 NACEP Member Survey 

% Very Important

1-Developing strong school partnerships 74%2-How concurrent enrollment compares to other accelerated learning options 71%3-Finding the resources needed to build a strong program 58%

4-How to conduct a student survey; how to raise the response rate to surveys. 56%5-The process of becoming accredited—what

to expect 54%

Why do Surveys?

Collect data from CEP stakeholders to

Measure for regulatory requirements, e.g., NACEP accreditation standards

Monitor quality outcomes for improvement

Example: Retention of CEP students

CEPs are being asked to Quantify their

Contribution to the Sponsoring Institution

Does greater involvement in CE increase the likelihood that students will attend the Sponsor Institution?

Credits earned n USU

1-6 credits 60 20%

7-13 credits 52 29%

14-24 credits 41 46%

25-63 credits 47 49%

Comparison Current Status

Study n Work

Vol. Svc

2 yr Coll.

Other4 yr. USU

5-yr. 196 7% 0% 10% 29% 55%

1-yr. 200 7% 16% 13% 30% 35%

How do I Conduct a Survey? Plan in established ethical context – OIR/IRB

Formulate research questions - latent variablesexample: Does CEP affect student development?

Simultaneously, develop instrument & get list Compose tracking strategy – submit to IRB Pilot test instrument and tracking strategy Send out a request for responses – follow-up... Collect and process data Perform analysis – develop a report

AAPOR PROFESSIONAL ETHICS We shall not knowingly make interpretations

of research results that are inconsistent with the data available, nor shall we tacitly permit such interpretations.

We shall not knowingly imply that interpretations should be accorded greater confidence than the data actually warrant.

American Association for Public Opinion Research, Code of Professional Ethics and Practices.

How Many Respondents Do You Really Need?

First Dilemma:

Determining sample size is a

very important issue:Samples that are too large may

waste time, resources, and money, while samples that are

too small may lead to inaccurate results.

Sample Size Formula

Sample size = z 2 * (p) * (1-p) c 2

Where:z = z value (e.g. 1.96 for 95% confidence level) p = percentage picking a choice, expressed as decimal (.5 used for max. )c = confidence interval, expressed as decimal 

(e.g., .04 = ±4)

Reduced n because of homogeneity:p = 30%

n=1.96² x .3(1-.3) .05²

n=3.8416 x .21.0025

n =.8068 .0025

n =322.72 ~ 323

p = 80%

n=1.96² x .8(1-.8) .05²

n =3.8416 x .16.0025

n =.61465 .0025

n =245.86 ~ 246

Count Valid Percent

NA 28 2%

Strongly Disagree 13 1%

Disagree 78 6%

Agree 798 57%

Strongly Agree 488 35%

Total Agree 1,286 92%

Total 1,406 100%

I was better prepared

Calculating a Minimum Sample Determine confidence level and interval

95% confidence level confidence interval is the plus-or-minus range, if

question asked of entire population—usually ±5%

Use Internet: sample size calculator surveysystem.com researchinfo.com

Second Dilemma

How Important is the Response Rate?

Dilemma Response rate: the number of eligible sample

units that cooperate in a survey -- has historically been central to survey research in the United States. . . . but the relationship between response rates and survey quality has become much less clear.

 

Results that show the least bias have turned out, in some cases, to come from surveys with less than optimal response rates. http://www.aapor.org/Response_Rates_An_Overview.htm

NACEP Member Survey - March 2009 Note about Methodology: A notice sent to

all 255 members of the Partnership Listserv at the end of February 2009, directed people to a survey in SurveyMonkey. Reminders were sent 1 week, 2 weeks, and 3 weeks . . .

Response rate: 183 of the 255 members responded, for a 72% response rate.

What works? Grades!! UU grants early access to grades

for completion of online survey = 90% rr

Follow-up. . . by telephone and/or postcard

Parents? (next)

What have you found that works?

Can Parents Answer for Students?

Ranking of Decision Factors - Identical Results

Students Parents

1 Education for better job 1

2 Affordable tuition 2

3 Quality program for chosen career 3

Concurrent Enrollment as a Vehicle for Recruitment & Retention, 2008

Third Dilemma

What is satisfaction? The gap between expectation and experience

Why measure satisfaction? Word of mouth recommendation is powerful.

Who is it best to ask? Students, parents, faculty, HS counselors

Q8. How do you rate the value of NACEP?

Answer Options Count PercentNot very valuable 2 1%

Somewhat valuable 27 15%

Valuable 61 34%

Very valuable 92 51%

Total 182 100%

Comparison Credit and Satisfaction 2006-07 1-Yr. Follow-up StudyUSU credits earned n Excellent Good Fair Poor

1-6 credits60

55% 35%8%

2%

7-13 credits52

52% 40% 2% 6%

14-24 credits41 71% 29% 0 0

25-62 credits47 83% 17% 0 0

Thanks

Complete the Evaluation!