View
216
Download
3
Category
Tags:
Preview:
Citation preview
Student Life Assessment Project
Module 4: Analyzing & Reporting Assessment Results
© 2013 Christie Cruise-Harper, PhD All Rights Reserved
The purpose of this module is to assist you with analyzing the data you gathered for the program/service you assessed in the 2013 – 2014 academic year.
Student Life Assessment Project
Department Program
Office of Multicultural Programs Multicultural Scholars Program/Dean’s Award Program
Personal Counseling Mandated Substance Abuse Assessment Program
Health and Wellness HEROs Program
Campus Ministry and Community Service
KLILV Sophomore Colloquium
Student Involvement CAB/MSG/CSI
Athletics SAAC and Champs
Residence Life Resident Assistant Program
Student Life/Associate Dean of Students
Habitat for Humanity
Student Life Assessment Project
In module 1 you developed learning outcomes for your program/service using Bloom’s Taxonomy as a guide.
Module 2 allowed you to strengthen those learning outcomes and guided you through the process of choosing learning activities.
Module 3 assisted you with choosing appropriate assessment tools/methods for the learning outcomes and learning activities you established.
This module will help you with analyzing and reporting the assessment data.
Student Life Assessment Project
Learning Outcome
What should your students be able
to do?
Learning Activity
What activity will help your
students achieve the learning
outcome?
AssessmentHow will you know whether students have achieved the learning outcome?
AnalysisWhat will you do
with all the information you collected from
your assessment plan?
Module 1 Module 2 Module 3 Module 4
Student Life Assessment Project
Assessment results must be analyzed to determine if student learning outcomes were met.
Data is analyzed for context, understanding and to draw conclusions.
Analysis of data gives the information meaning.
Taken from Academic Program Assessment: Tools & Techniques for Program Improvement
Analyzing Assessment Data
Determining how to organize, synthesize, interrelate, compare and present the assessment results are all part of analyzing the data.
Assessment data can be compared to findings from previous assessments, baseline data and existing criteria.
Taken from Academic Program Assessment: Tools & Techniques for Program Improvement
Analyzing Assessment Data
Quantitative◦ Also known as “empirical research”◦ Refers to any research based on something that
can be accurately and precisely measured.◦ Collects numerical data in order to explain or
predict a particular phenomena.
Taken from University of Wisconsin – Madison’s Ebling Library & R. Ouyang’s Basic Inquiry of Quantitative Research
For more information visit: http://researchguides.ebling.library.wisc.edu/content.php?pid=325126&sid=2940225
Types of Data
Qualitative◦ Refers to any research based on something that
cannot be accurately and precisely measured.◦ Collects narrative data to gain insights into a
particular phenomena.
Taken from University of Wisconsin – Madison’s Ebling Library & R. Ouyang’s Basic Inquiry of Quantitative Research
For more information visit: http://researchguides.ebling.library.wisc.edu/content.php?pid=325126&sid=2940225
Types of Data
There are four types of quantitative research methods:◦ Descriptive: collecting data for hypotheses testing*◦ Correlational: determining whether and to what degree a
relationship exists◦ Cause-Comparative: establishing the cause-effect
relationship◦ Experimental: establishes the cause-effect relationship,
but manipulates the cause
See notes section for more detailed information.
Taken from R. Ouyang’s Basic Inquiry of Quantitative Research
Types of Quantitative Research
The following are common methods of data collection in quantitative research:◦ Surveys and Questionnaires◦ Structured Interviews◦ Observation or Interaction Analysis◦ Secondary Data or Content Analysis◦ Experiments
See notes section for more detailed information.
Methods of Data Collection
In quantitative research there are two ways in which data are analyzed:
Descriptive Statistics◦ Procedures used to describe a given collection of
data.◦ The purpose is to describe the sample at hand-the
collection of cases that we have examined. Inferential Statistics
◦ Procedures that let us generalize our findings beyond the particular sample at hand to the larger population represented by that sample.
Taken from Diekhoff, G.M.(1996). Basic Statistics for the Social and Behavioral Sciences.
Data Analysis
Most Student Life assessment projects do not seek to generalize its findings to the entire Maryville University student body. Because our goal is to learn about the sample at hand, descriptive statistics will be the focus of the quantitative data analysis for module 4.
There are three types of descriptive statistics to provide you with an overview of your data:◦ Central Tendency Measures*◦ Variability Measures◦ Frequency and Percentages*
Descriptive Statistics
More on descriptive statistics later. Before beginning any data analysis, you
must first identify the level of measurement associated with your quantitative data. There are four levels of measurement:◦ Nominal◦ Ordinal◦ Interval◦ Ratio (Scale)
Levels of Measurement
Nominal: basic classification data; do not have meaningful numbers attached to them, but are broader categories
Ordinal: have numbers attached to them and the numbers are in a certain order, but there are not equal intervals between the numbers
Interval: have equal intervals between the numbers; the distance between attributes have meaning
Ratio: have equal intervals between the numbers; there is an absolute zero that is meaningful
Taken from: http://www.uni.edu/commstudies/researchmethods/chapterfour1.html
Levels of Measurement
Once you have decided on your data collection method, decided on the level of measurement for your variables, and collected the data, you are ready to begin analyzing the data. There are two software programs I recommend and they are available on campus:
Qualtrics SPSS (Statistical Package for the Social
Sciences)
Data Analysis and Software
The following quantitative data analysis procedures are used to describe the data and can be done in Qualtrics and SPSS:◦ Data Tabulation (Frequency Distributions, Percentiles)
◦ Descriptive Data (Central Tendency)
◦ Data Disaggregation
◦ Moderate and Advanced Analytical Methods
For a detailed description of these analyses visit:http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/ ORChapters 1-3 in Diekhoff’ s Basic Statistics for the Social and Behavioral Sciences
Data Analysis and Software
Surveys and questionnaires can be developed in Qualtrics and the data can be analyzed within the software. There are tutorials to assist you with analyzing your data.
Crosstabs: http://qualtrics.com/university/researchsuite/reporting/cross-tabs/about-cross-tabulations/
Understanding Statistics: http://qualtrics.com/university/researchsuite/reporting/cross-tabs/understanding-statistics/
Qualtrics
SPSS◦ Below are several links to resources to assist you
with using SPSS. http://www.slideshare.net/sspink/seminar-on-spss http://www.youtube.com/watch?v=eTHvlEzS7qQ http://
www.youtube.com/watch?v=HT0Skh2UP1U&feature=related
http://calcnet.mth.cmich.edu/org/spss/toc.htm
Data Analysis and Software
In addition to a narrative about your data analysis, quantitative results are presented in the following ways:◦ Charts◦ Graphs◦ Tables
The following websites provide examples, in APA, of how quantitative results are presented:
https://owl.english.purdue.edu/owl/resource/560/19/ https://owl.english.purdue.edu/owl/resource/560/20/
Reporting Quantitative Results
Tables and graphs can be created in Qualtrics. For more information visit: ◦ http
://qualtrics.com/university/researchsuite/reporting/reporting-beta/tables/#AboutTables
◦ http://qualtrics.com/university/researchsuite/reporting/reporting-beta/graphs/
Graphs can also be created in SPSS. For directions visit:◦ http://julius.csscr.washington.edu/pdf/spss.pdf◦ http://
academic.udayton.edu/gregelvers/psy216/spss/graphs.htm◦ http://
www.ats.ucla.edu/stat/spss/seminars/SPSSGraphics/spssgraph.htm
Reporting Quantitative Results
It is important to achieve empathic understanding to comprehend the participant’s experience with a minimum of distortion or bias;
The researcher must attempt to recognize their own personal prejudices, stereotypes, myths, assumptions and other thoughts or feelings that may cloud the perception of other people’s experiences;
Knowledge of other’s experience cannot be assumed regardless of familiarity with their subcultural landscape; and
Do not expect participants to hold the same values as you.
M. Ely, M. Anzul, T. Friedman, D. Garner, A.M. Steinmetz in Doing Qualitative Research: Circles within Circles
Before Analyzing Qualitative Data…
Qualitative Research Methodologies◦ Basic Interpretive Qualitative Study*◦ Grounded Theory◦ Phenomenology◦ Case Study◦ Ethnography◦ Postmodern Research◦ Critical Qualitative Research◦ Narrative Analysis
For more information visit: http://www.fctl.ucf.edu/researchandscholarship/sotl/creatingsotlprojects/implementingmanaging/qualitativeresearchtypes.php
Types of Qualitative Research
Participant Observation Interviewing (formal & informal) Focus Groups Document Analysis Logs (notes/reflections from observations
and interviews) Audio and videotaping On-going data analysis
M. Ely, M. Anzul, T. Friedman, D. Garner, A.M. Steinmetz in Doing Qualitative Research: Circles within Circles
Methods of Data Collection
Coding is the process of organizing data into chunks or segments before making meaning of the information.
Coding involves taking text data or pictures gathered during data collection, segmenting it into categories, and labeling those categories with a term, often a term used by the actual participant.
Taken from: Creswell, J.W.(2009). Qualitative, Quantitative and Mixed Methods Approach, chapter 9, Qualitative Procedures.
Coding and Categorizing
Begin the coding process by first reviewing your learning outcomes as a reminder of what you are assessing. Your coding scheme will be based on your learning outcomes.◦ For example: Undergraduate students who
participate in the Multicultural Scholars Program will be able to describe their talents, strengths and social group memberships.
Coding involves assigning a word, phrase, number or symbol to each coding category.◦ For example: describe talents and strengths
Coding Qualitative Data
Codes can be pre-set or emergent. You should have both.◦ Pre-set: A list of codes created in advance by the
researcher based on the research question, learning outcomes, or conceptual framework.
◦ Emergent: Ideas, concepts, actions and meanings that come up from reading and analyzing the data that are not in the pre-set codes.
Coding will serve as a system to help you to organize your data.
For more information and an example of coding visit: http://programeval.ucdavis.edu/documents/Tips_Tools_18_2012.pdf
Coding Qualitative Data
Once you have gone through all documents and coded them, they can now be gathered into families of codes or categories.
Materials are sorted by these categories, identifying similar phrases, patterns and relationships.
As you code and categorize the data, look for the interrelationships among categories.
Sorted materials are examined to isolate meaningful patterns.
Identified patterns are used to create themes.
Creating Categories
It is best to start any report of qualitative results with an overview of how data were processed and coded.
The results are presented as “findings”. The findings are organized by themes with
substantial evidence that links to the themes included in the findings (i.e. quotes from participants).
Reporting Qualitative Results
Closing the loop - using the assessment results for program improvement.
Closing the Loop
Assessment Plan
Data Collection
Data Analysis
Assessment Report
Identify Changes
Implement Changes
The findings from your quantitative and/or qualitative research will yield rich information that will be included in your assessment report.
These findings will provide you with insights into what’s working with your program/service and areas of improvement.
From these findings you will make recommendations for improvement of your program/service.
Closing the Loop
Recommendation:
Action step(s) What action steps must be completed to implement the recommendation?
Estimated implementation date When does the program expect to begin to implement the action steps?
Estimated completion date When does the program expect the recommendation to be fully implemented and/or achieved?
Person(s) responsible Who will take responsibility for seeing that the actions steps are implemented?
Expected outcome What is the expected impact/outcome the recommendation will have on the program if it is implemented?
Estimated cost(s) What is the estimated cost of implementing the recommendation?
Status update What progress has been made towards achieving the recommendation?
Program ImprovementYou may find it helpful to create a program improvement plan to transform the recommendations made into actions for improvement.
Accreditation Planning and budgeting Maryville University requirements Student Life requirements Program promotion/marketing Recruitment/retention initiatives Publications Conference presentations Student development opportunities Professional development opportunities Grant applications
Taken from: Academic Program Assessment: Tools & Techniques for Program Improvement
Other Uses of Assessment Results
Academic Program Assessment: Tools & Techniques for Program Improvement.(2013). SUNY.
Analyzing Assessment Data.(2013). http://www.sunyorange.edu/assessmentapa/docs/AnalyzingandUtilizingAssessmentData.pdf. State University of New York (SUNY) Orange County Community College.
Basic Inquiry of Quantitative Research.(n.d.). http://ksumail.kennesaw.edu/~rouyang/ED-research/details.htm. Kennesaw State University.
Center for Evaluation and Research.(2012). http://programeval.ucdavis.edu/documents/Tips_Tools_18_2012.pdf. University of California at Davis.
References
Creswell, J.W.(2009). Qualitative, Quantitative and Mixed Methods Approach, chapter 9, Qualitative Procedures.
Diekhoff, G.M.(1996). Basic Statistics for the Social and Behavioral Sciences.
Differences Between Quantitative and Qualitative Research. (2013). http://researchguides.ebling.library.wisc.edu/content.php?pid=325126&sid=2940225. University of Wisconsin-Madison-Health Sciences Ebling Library
Ely, M., Anzul, M., Friedman, T., Garner, D., Steinmetz, A.M.(1991). Doing qualitative research: Circles within circles. RoutledgeFalmer: New York.
References
Faculty Center for Teaching and Learning.(2013). Types of qualitative research: Explained within a SOTL framework. http://www.fctl.ucf.edu/researchandscholarship/sotl/creatingsotlprojects/implementingmanaging/qualitativeresearchtypes.php. University of Central Florida.
Gathering Data and Assessing Results.(2013). http://nnlm.gov/evaluation/guide/stage 5.pdf. National Network of Libraries of Medicine.
The Pell Institute and Pathways to College Network.(2013). Evaluation Toolkit. http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/
University of Northern Iowa.(2013). Communication Studies – Research Methods Chapter Four. http://www.uni.edu/commstudies/researchmethods/chapterfour1.html
References
Recommended