Upload
charleen-stone
View
217
Download
2
Tags:
Embed Size (px)
Citation preview
HomeTown Competitiveness:Conducting Phase I Assessment
Jennifer Bott, PhD, SPHRAssociate Professor of Management Miller College of Business Ball State University
Agenda
•Background on HTC Assessment•Goals of Assessment•Types of Assessments
▫Focus Groups and interviews▫Designing surveys▫Archival data analysis
•Basic analyses and decision making•Resources
Background on Assessment
•Phase I Assessment phase typically lasts up to four months▫Community conducts local assessments of
development history, context, issues and assets
▫Create customized strategic opportunities•Leads into implementation of the
strategic development plan in Phase II
Assessment Goals
•Determine perspectives of community stakeholders on chosen pillar within the community: ▫Entrepreneurship▫Charitable Assets▫Youth Engagement▫Leadership
Assessment Outcome
•End product of the assessment:▫Collection of data that direct community
team toward path of action▫Summary of current thinking in community
on current status of selected pillar•Launching assessment with community
should engage them more in the process, outcome and implementation of the strategic plan
Types of Assessments
•Traditional measures of assessment:▫Focus groups▫Interviews▫Surveys (telephonic, online, mail, intercept)
•Other opportunities:▫Re-analysis of previously-collected data
Types of Data
•Two primary types of data emerge from assessments:▫Qualitative
Verbal responses to open or closed-ended questions
▫Quantitative Numeric responses to (usually) closed-ended
questions
Choice of Assessment(s)• Assessment method should:–Provide most insight into questions at hand– Involve stakeholders–Adhere to following criteria:• Reliable• Valid• Cost-effective• Feasible
• Combinations of assessment methods may be most beneficial
Focus Groups
•Characteristics:▫7 – 10 members▫Share some relationship to topic of interest▫Represent divergent opinions on topic and
typically represent different stakeholders•Repeated several times with different
groups of people•Requires collection of qualitative data
Topics for Focus Groups
•Strategic planning•Needs assessment•Current state analysis•Program design•Perspectives on a singular issue
Uses of Focus Groups
•Focus groups are especially useful:▫Exploratory, preliminary study▫Gap in understanding or communication▫Uncover data related to complex behavior▫Discover interactive synergies between
participants▫Uncover additional information leading to a
large-scale study effort▫High value placed on open-ended data
collection
When NOT to Use Focus Groups•Focus groups are not recommended for:
▫Emotionally-charged subject▫Researcher/HTC team not in control of the
environment or message▫Team expects statistical summaries from
data▫Is not economical to achieve data collection
needed▫Confidentiality can’t be guaranteed
Outcomes of Focus Groups
•Categorized data from focus groups can be used:▫As stand-alone, qualitative information▫As starting point for development of a
quantitative measure
Designing Successful Focus Groups• Create “safe” environment–May require use of a neutral facilitator
• Create a plan for focus group that details:–Purpose –Who to include • Participants• Number of sessions
– Incentives for participation (if necessary)–Location
Focus Group Questions• Categories of questions:– Opening • Everyone answers quickly (10 - 20 seconds)
– Introductory • General topic of discussion, foster conversation
– Transition • Approach the focal questions
– Key • Drive the study – 2 to 5 questions that require
most attention– Ending • Bring closure and allow for reflection
Focus Group Questions (cont’d)
• In most cases, focus group questions are open-ended– “What are our greatest weaknesses in
youth engagement?”• Closed-ended questions are not
appropriate for sparking discussion– “Would you participate in a training
success?”• Avoid asking why in response to a
participant’s answer
Analyzing Focus Group Data
•Determining analysis strategy:▫Least time intensive = Memory based
Report is created immediately based on facilitator’s memory
▫Most time intensive = Transcript based Combines field notes with transcript to write
final report▫Most common = Note based
Relies on field notes to create focus group report
Focus Group Report
•The focus group report summarizes:▫Participants▫Setting▫Key questions▫Themes that emerged from key question
responses▫Limitations▫Recommendations
Conducting Interviews
•Interviews can also be used as assessment tools▫One-on-one or panel (2 or more) ▫Collect qualitative data▫Can be used as singular collection method
or in tandem with focus groups or surveys
Uses of Interviews
•Appropriate when:▫Group perspectives are not valuable▫Topic is too sensitive to discuss in group
setting▫Want to collect limited information (e.g.,
only want to speak to a few individuals)▫Focus groups are not feasible
Uses of Surveys
•Surveys are appropriate when:▫Know the parameters of the content to be
assessed▫Have identified a sample that will provide
meaningful data▫Need quantitative figures (rather than
qualitative descriptions)•Can be coupled easily with qualitative
data collection
Considerations with Survey Administration and Design•Decision points on survey design:
▫Sample Who will respond? How many?
▫Administration mode Telephone Intercept (in-person) Snail mail Internet
▫Necessary response rates
Survey Questionnaire Development•Six basic steps:
▫Decide what information to collect▫Decide what type of questionnaire should
be used▫Develop first draft▫Revise questionnaire▫Pilot test▫Edit questionnaire and specify methods of
collection
Question Writing Guidelines
•Characteristics of “good” items:▫Simple, direct, familiar language▫Clear and specific questions▫Do not use leading, loaded, or double-
barreled questions▫Use short statements (20 words or less)▫Edit for readability
Designing the Scale
•Most common response options:▫Agreement▫Frequency▫Evaluation
•Number of response categories ▫3 to 5
•Neutral point – good or bad idea?
Evaluate Wording
•Which is a better question?▫Age?▫What was your age on your last birthday?
•Is there a problem with this item?▫What do you like best about this
neighborhood? (We’re interested in anything, like houses, the people, the parks, or whatever.)
▫Do you favor or oppose gun control legislation?
Item Stem Creation
•Agreement items – Declarative▫“I am uncomfortable around strangers.”
•Frequency items – Events, behaviors▫“You exercise strenuously enough to raise your
heart rate.”•Evaluation items – Persons, places, things
▫“How well your favorite sports team played last week.”
Designing the Survey
•Some simple rules for the ordering and design of questions:▫Keep items together if they use the same
responses▫Don’t overfill each page with questions▫Avoid including too many different
response options•Pilot test with individuals similar to those
being sampled
Importance of Instructions•Give direction for response for those
unfamiliar with format•Create a common frame of reference
▫“The following questions concern peoples’ opinions and beliefs about jobs and careers. These questions refer to jobs in general and not the job you presently have or a particular job you once had.” (Spector, 1992)
Using Archival Data
•In some cases, data has already been collected regarding your pillar of interest▫Ask groups that are associated with your
pillar (e.g., Chamber of Commerce, local schools, philanthropic groups)
▫Can be either quantitative (surveys) or qualitative (interviews, news coverage, etc.)
Possible Analyses with Archival Data•Content analysis•Assessment of trends•Analysis of communications
Limitations
•Archival data has many advantages and disadvantages▫Advantages:
Data is already collected (cheap, easy to use) Data may already be analyzed
▫Disadvantages: Can’t be tailored to fit needs Inability to determine the reliability of the
methods used to collect data
Conclusion of the Assessment
•Use the assessment data to identify “key findings” about the pillar assessment▫A key finding is a full sentence used to
summarize learning. Including a statistics strengthens this finding. “73% of high school students surveyed stated
that they would return to our community if opportunities exist.”
Conclusion (cont’d)
•Key findings are used to drive the development of the problems to be solved at the end of Phase I.
Tangible Outcomes of Assessment•A written report should be provided to
individuals who participated in the survey▫Details overall findings (at the group level
only)▫May provide information about uses of the
assessment Depends on confidentiality of the process
▫Can hopefully meet a need of the sample
Moving to Decision Making
•Totality of evidence from both qualitative and quantitative assessment efforts should be examined▫Looking for evidence that supports teams’
beliefs or uncovers new, unexpected information
▫Evidence should create a logical next step in assessment or implementation
Resources to Aid in the Assessment Process•Center for Business and Economic
Research at Ball State University▫Partnered with Building Better
Communities•Professional consultants•Other HTC Communities
Resources (cont’d)
•Books:▫Survey Research Methods (4th ed.), Floyd
Fowler (Sage Publications, $39.95)▫Focus Groups: A Practical Guide for
Applied Research, Krueger and Casey (Sage Publications, $45.32)
Links to Nebraska Surveys
•http://www.zoomerang.com/Shared/SharedResultsSurveyResultsPage.aspx?ID=L22Z2S57GK54
•http://www.zoomerang.com/Shared/SharedResultsSurveyResultsPage.aspx?ID=L23KD72VDQVC
•http://www.zoomerang.com/Shared/SharedResultsSurveyResultsPage.aspx?ID=L23KDC4HWP6J