Upload
educause
View
217
Download
0
Embed Size (px)
Citation preview
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 1/77
Measuring the Effectiveness of Online andBlended Programs
An ELI and Quality Matters webinar
April 8, 2015
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 2/77
What is evidence of impact?
Why do we need it?
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 3/77
Providing evidence to:
persuade
plan & set goals
scale
develop consensus
know your work makes a difference
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 4/77
“Does OL/BL work?”
Do the students learn “better”?
Do the faculty teach “better”?
Are the students more engaged?
Are the student evaluations “better”?
Is it “efficient”?
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 5/77
Brown, Diaz, “Seeking Evidence of Impact: Opportunities and Needs, EDUCAUSEReview, Sept/Oct 2011, pp. 42-54
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 6/77
how do we get to “what works?”
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 7/77
Answers via methodsAnswers and methods
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 8/77
how do we get there?
Research question1
Research plan2
Execute3
Communication to stakeholders4
here’s what works
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 9/77
tools
methodologies
rubrics
frameworks
data types
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 10/77
Challenges
afterthought syndrome
recalcitrant data
lack of expertise
poverty
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 11/77
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 12/77
ELI/QM: Measuring the Effectiveness of Online/Blended Programs
Kay Shattuck April 8, 2015
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 13/77
A continuous improvement model for
assuring the quality of online courses andonline components of blended courses
through a collegial review process.
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 14/77
• QM tools and processes for online learning quality
assurance
• Evidence of QM impact on learner success
•Research tools for providing meaningful evidence
Today’s Takeaways
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 15/77
How much do you know about QM and the continuousimprovement processes for effective online/blendedprograms?
• I am very knowledgeable and/or actively engaged in
them
• I’ve investigated and understand the basics about QMtools and processes, but do not actively use them
• I know little or nothing
Poll
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 16/77
A validated RubricTM of quality standards and a
process of continuous improvement at course,program, institutional levels.
QM Rubric is central; however, QM is more than “a Rubric”
About continuous, rigorous application of improvement based
on impact evidence
QM: Rubric and Processes
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 17/77
• Eight general standards (https://www.qualitymatters.org/rubric)
Course Overview and Introduction
Learning Objectives (Competencies)
Assessment and Measurement
Instructional Materials
Course Activities and Learner Interaction
Course Technology
Learner Support
Accessibility and Usability
• 43 specific standards Alignment of critical components to ensure students achieve desired
learning outcome
• Each specific standard detailed for application in annotations
Validated QM Standards of Quality
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 18/77
QM Rubric Designed for Application
PeerFeedback
QM Rubric sets a threshold
Student
Learning
Course Design
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 19/77
Application: Impact Starts with a Course Review…
QM Peer
Reviewer
QM Peer
Reviewer
QM Peer
Reviewer
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 20/77
Student Impact: All of the Instructor’s Courses
123
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 21/77
Student Impact: Peer Reviewers’ Courses (carry over impact)
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 22/77
Whole is Greater Than Sum of
Parts
QM Rubric
ProfessionalDevelopment
Peer CourseReviews
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 23/77
Design-Based Programs & Research
• Specific application of educational theory &research
• QM Standards of quality
•QM Process of peer review as a component of aninstitution’s quality assurance efforts
• Evidence seeking from results of application
for continuous improvement
• QM focused-research
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 24/77
What is the impact on X of a course redesigned to meet QM Standards?
√ What exactly is X? What evidence is already available about X’s impact?
√ What is the context? Which review format is to be used? Official QM review?Informal review?
√ How is the sample to be gathered? What is the study size? How representative
are the results?
√ What exactly is the “unit” counted/observed/documented? How is it counted,etc.? What is the time frame of the study?
√ Which and how are details of the specific sample (for example, students, course-
types, and instructors) obtained and analyzed in relationship to their possibleinfluence?
√ Pre-and post-measures allow you to gauge the extent and significance of courseimprovement (QM review scores not a meaningful measure without pre/post )
Improved Course Impact Example
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 25/77
Consider the Impact of the Whole
Dot Potter
Sometimes we can’t see the forest for the trees
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 26/77
PollPlease identify the two most important shareholders for
evidence of effectiveness/impact of your online/blendedprograms?
• Yourself
• Your colleagues
• Your department
• Your institution’s academic head
• Your students
• Members of a governing board
• Members of an accrediting body• The institution’s marketing department for community
awareness
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 27/77
Think About/Make Notes
What evidence do those
shareholders want?
(We’ll return to this)
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 28/77
QM-related research from the field
What do we know now?
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 29/77
Positive Impact of Rubric
• Providing easy-to-use guide or self-assessment tool by designers &instructors
• Establishing shared language/concepts by teams (Greenberg, 2011)
• Linking discussion, development, and implementation of moreeffective online learning for new online instructors (Ward, 2011)
• QM standards have a high correlation with TPACK (TechnologicalPedagogical Content Knowledge) (Ward, 2011)
• High correlation with technological affordances (Bose, 2012)
• Actions, abilities, and possibilities offered within online learning environmentswhen using technology (p. 193). Refers to e-learning tools properties andinteraction of learner
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 30/77
Learner Satisfaction Studies
•Less confused, increased satisfaction
(Finley, 2005)
• Satisfaction ↑ in QM peer-reviewed courses/institutions(Aman, 2009)
• Students (Knowles & Kalata, 2010)
• Students still like lectures; even when there is more teacherinteraction online (Knapp & Paull, 2013)
• Student perceptions are not significantly impacted whencourse design is guided by QM standards (Parscal, Frey, & Lucas, 2011)
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 31/77
Learner Voice
• Students valued design elements identified in Rubric (Iyengar,
2006) and missed them when absent (Mott, 2006)
• Courses should be designed to meet needs of all learners,including those who might need assistive technologies (Bowen &
Bartoletti, 2009)
• Learners correlate quality design with QM standards• Quantitative, survey research
• N=3,160; 22 states; 31 institutions (Ralston-Berg, 2011, 2014)
• Research on the Impact of QM for Students projects (DropThought& Ruffalo Noel Levitz)
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 32/77
Student Learning
++ Engagement when activities met QM (Runyon, 2006)
↑ Higher grades on discussion board activities (Hall 2010)
↑↑Major assignment/final exam scores, as well as
overall course grades statistically significant (Swan & colleagues,
2010, 2011, 2014)
A-D course grades ↑ F course grades ↓ over 5 yrs.(Harkness, 2014)
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 33/77
Students’ Motivation & Self -Efficacy
Findability matters! (Simunich, Robins, & Kelly, 2012)
Motivation & self-efficacy positively correlated
Controlled study
• Eye-tracking & talk aloud (cognitive overload)
• Motivation & self-efficacy instrument
Readiness factors matter, however (Geiger, Morris, & Subocz, 2013)
Which SmarterMeasuresTM readiness factors correlate with learning
Controlled study
• QMed courses
• QMed, expert instructor• Strong LMS support
Only typing speed/accuracy and reading rate/recall had a statistically significantcorrelation to student course retention & course grade.
Question raised: What was the instructor’s role in mediating issues?
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 34/77
Faculty Culture
• Faculty beliefs regarding QM participation: Study it, don’t just
rely on hearsay & myths (Altman, Schwegler, & Bunkowski, 2014)
Heard faculty concerns about voluntary, unofficial QM PRprocess
Theoretical framework: Theory of planned behavior
Quantitative analysis → Qualitative lens (mixed)
Surveyed participants and non-participants
• Both: positive attitudes; likely promotion help; not likely toinfringe on academic freedom
• Nonparticipants more likely to think process would be effortful &time consuming
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 35/77
Organizational Impact
• Carryover effect (diffusion of treatment) to non-reviewed courses(Aman, 2009, p. 107)
• Informal sharing among faculty (Strickland & Alarcon, 2010)
• Impacting institutional course development (Parscal, Frey, & Lucas, 2011;
Harkness, 2014)
• Valuable recognition: “Expecting QM certification to solve allstudent success alone is unrealistic” (Miner, 2014, p. 106)
• Development & dissemination of accessibility policy template (Frey,
Kearns, & King, 2011)
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 36/77
Back to Your Shareholders? What Evidence Do They Want?
Improved grades?
Improved retention?
Improved course persistence/completion?
Improved satisfaction?
“Other” (post specifics in chat)?
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 37/77
Persistence/Completion/RetentionComplications of quick study
• Without controls:• No apparent difference in completion rate; however, positive
comments (Loser & Trabandt, 2006)
• Increased satisfaction, but could find no relationship with coursecompletion in one semester study via instructors’ self -report (no
evidence data tracking) (Aman, 2009)
*****************************************
• Taking a step at controls:
• 2 QMed courses completion/same instructors• Consistently higher completion rates (95.5; 95%) than average
rate for other online courses over multiple (6; 11) semesters (Dietz-
Uhler, Fisher, & Han, 2007)
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 38/77
Persistence/Completion/Retention: Results of Longitudinal Study
University of D.C. mappedProfessional development participation, QM courses
Applications into LMS technologies
QMed courses
Findings: Student outcomes pre/post QM
2007-2012N =1,570
Withdrawals ↓23.53 Pass class grades A-D ↑19.74
Failing class grades F ↓66.66
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 39/77
Research Issues and Realities• Too late to ignore possible QM influence in the “before” courses
Must detail the before for evidence of afterNot just recognizing the variables, but including in data analyses
• Definitions, definitions, definitionsQM implementation levels; persistence/retention
Exactly which input and which output variable
• Sample size, sample size, sample size & needed Time!Challenge to get participants, initially & completion
• QM has no direct access to institutional implementation dataAssist in designing
Inter-institutional study
• Expanding theoretical frameworks
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 40/77
•QM Research Pages
• https://www.qualitymatters.org/research
• https://www.qualitymatters.org/node/1866/download/What%20we%27re
%20learning%20paper_FINAL_May%2018,%202012_Dec2012ks.pdf
• QM Research Library• https://www.qmprogram.org/qmresources/research/
• Toolkit for Designing QM-focused Research
• https://softchalkcloud.com/lesson/serve/irsu5Mk1P8WFR4/html
Resources
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 41/77
©2015 MarylandOnline, Inc.
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 42/77
POLL QUESTION
Please respond "yes" or "no" to each of the following
1. We have the mandate to conduct measurement of impact (e.g.,
tasked by campus or organizational leadership)
2. We have the resources to conduct measurement of impact (e.g.,human resources, technical capacity, workload, etc.)
3. We have an audience that is receptive to our reports on the
measurement of impact
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 43/77
• Join ELI as we Seek Evidence of Impact
–Catalyze new collaborations to advance evidence-basedapplications of learning innovation
– Encourage the teaching and learning community to explore
new ways of gathering evidence
– Initiate ongoing dialogues, projects, and collaborations toenable institutions to effectively gather and share evidence
of impact
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 44/77
POLL QUESTION
Select the most difficult part of determining evidence of
impact of technology-based innovations in T&L.
1. Knowing where to begin to measure the impact of technologybased
innovations in T&L
2. Knowing which measurement and evaluation techniques are most
appropriate
3. Conducting the process of gathering evidence
4. Knowing the most effective way to analyze our evidence
5. Communicating to stakeholders the results of our analysis
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 45/77
0 20 40 60 80
Knowingwheretobegintomeasuretheimpactoftechnology-‐basedinnovationsinT&L
Knowingwhichmeasurementandevaluationtechniquesaremostappropriate
Knowingthemosteffectivewaytoanalyzeourevidence
Conductingtheprocessofgatheringevidence
Communicatingtostakeholderstheresultsofouranalysis
Please rank each of the following processes of determining evidence of impact of
technology-based innovations in T L issues in order of difficulty.
Select the 3 most important indicators you use to measure the
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 46/77
POLL QUESTION
Select the 3 most important indicators you use to measure theevidence of impact of technology-based innovations in T&L?
1. Final grades
2. Reports and testimonies from faculty and students3. Measures of student mastery; learning outcomes
4. Drop/withdrawal/incomplete rates
5. Student performance in subsequent classes
6. Graduation and retention rates
7. Measures of student and faculty engagement8. Measures of student and faculty satisfaction
9. Cost or other efficiency measures
10. Changes in faculty teaching practice
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 47/77
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 48/77
SEI Cases: http://tinyurl.com/seicases
• Probing the Inverted Classroom: A Study of
Teaching and Learning Outcomes in Engineering
and Mathematics, 2015
• Better Understanding through Data:
Completion, Motivation, and Learning inMinnesota MOOCs, 2014
• Considering Classroom Seating for Students and
Faculty in the 21st Century, 2014
• Degree Compass Course Recommendation
System, 2013
• Scaling Learning Analytics across Institutions of
Higher Education, 2013
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 49/77
SEI Cases: http://tinyurl.com/seicases
• Adoption, Pedagogy, and Analytics: Exploring TwoYears of Institutional Blog Data, 2013
• Using Laptops in the Classroom: The University ofMichigan, 2013
• Role-Differentiated Responses to Active LearningClassrooms: The University of Minnesota, 2012
• Developing a Context-Specific Measure ofTechnological Expertise: The University of
Washington, 2012• Increasing Student Success: Evaluating the
Effectiveness of U-Pace Instruction at UWM, 2012
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 50/77
Full SEI case available here.
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 51/77
Full SEI case available here.
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 52/77
Full SEI case available here.
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 53/77
Full SEI case available here.
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 54/77
Full SEI case available here.
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 55/77
Full SEI case available here.
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 56/77
Full SEI case available here.
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 57/77
SEI CASES: 2 GOALS
1. Too put a spotlight/demystify the methodology used to
uncover the impact of technological/pedagogical
innovations
2. To share outcomes of impact studies/innovation work in
teaching and learning
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 58/77
• Go to http://tinyurl.com/SEIguide to
review the SEI case guide and template
• What are 2-3 key research questions for
your teaching and learning work?• Name 2-4 things you’d most like to know
their effectiveness to support teachingand learning?
InChat
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 59/77
• Go to http://tinyurl.com/SEIguide to
review the SEI case guide and template
• What kind of data will you need?• How will you collect that data?
InChat
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 60/77
http://www.educause.edu/eli/programs/seeking-evidence-impact
Malcolm Brown, PhD
Director, EDUCAUSE Learning Initiative
Veronica Diaz, PhD
Director, Online Programs, EDUCAUSE
Associate Director, EDUCAUSE Learning Initiative
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 61/77
National Research Center for
Distance Education andTechnological Advancements
(DETA)
Tanya Joosten
Co-Director and PI
National Research Center for Distance Education and
Technological Advancement
Director, eLearning Research and Development
University of Wisconsin-Milwaukee
@tjoosten, [email protected]
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 62/77
How do we ensure all
students have access
to a quality highereducation?
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 63/77
Pathway
todegree
Online
BlendedCBE
Access through distance education
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 64/77
Ensuring quality Flickr vaxzine
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 65/77
F2F gold standard Flickr xenonbpics
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 66/77
Ensuring quality?
F2FOnline
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 67/77
Identify practices (instructional and institutional)
that impact those outcomes
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 68/77
Conduct rigorous,interdisciplinary, and
standardized research
Flicker cc
astronomyblog
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 69/77
Desired Outcomes
Access
Learning Effectiveness
Instructional Effectiveness
Satisfaction
http://uwm.edu/deta/desired-
outcomes/
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 70/77
Research questions
What are the definitions of success from students’33
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 71/77
Research questions
f f f
perspective?33
What are the different design components (content,
interactivity, assessments) that impact student learning?
29
What patterns of behaviors lead to increased student
learning for different populations?26
How can we define and measure student success beyond
traditional outcomes?
25
What support structures are critical to providing quality
access to online instruction?22
What is the currency of student learning beyond the existing
credit hours?22
What are the key components that promote a sustainable
and an effective teaching and learning ecosystem?21
POLL
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 72/77
POLLWhich three (3) research questions do you find most important toincrease access and success for students in blended and online learning,including CBE?
• What are the definitions of success from students’ perspective?
• What are the different design components (content, interactivity,assessments) that impact student learning?
• What patterns of behaviors lead to increased student learning for
different population?• How can we define and measure student success beyond traditional
outcomes?
• What support structures are critical to providing quality access toonline instruction?
• What is the currency of student learning beyond the existing credithours?
• What are the key components that promote a sustainable and aneffective teaching and learning ecosystems.
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 73/77
Framework of inquiry
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 74/77
Framework of inquiry
RQs - What are the
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 75/77
Research toolkits
Shared measures
– Student
performance is
based on numerical
representation of
grade converted to a4.0 scale received in
the course on
assessments and as
an overall grade.
different design
components (content,
interactivity, assessments)
that impact student
learning?
Go to: http://tinyurl com/whatisss
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 76/77
What is student success?
Go to: http://tinyurl.com/whatisss
Spend 10 minutes sharing yourresponse to one of the following…
What is student success for you as aneducator or researcher?
What is student success for your
institution?
Thank you!!
8/9/2019 Measuring the Effectiveness of Online/Blended Programs (262191723)
http://slidepdf.com/reader/full/measuring-the-effectiveness-of-onlineblended-programs-262191723 77/77
y
Tanya Joosten, [email protected]
Twitter | @tjoosten
http://www.uwm.edu/deta
http://www.slideshare.edu/tjoosten