Embed Size (px)
Citation preview
A Quality Scorecard for the Administration of Online Education
Programs
Kaye Shelton, Ph.D. The Sloan Consortium
April 13, 2011
Focus
Introduce the Quality Scorecard Explain the history and rationale Why use the Scorecard? How to use the interactive Scorecard Handbook as a guide for completing
the Scorecard
To develop a quality scorecard to evaluate the administration of
online education programs.
Purpose
The Need For This Study
Quality Assurance (Bates & Poole, 2003; Meyer, 2004; Sallis, 1996).
From the beginning, the quality of online education was questioned (Benson, 2007).
Standards must be clearly defined (Institute for Higher Education Policy, 1998, 2000; Khan, 2005; Lee & Dziuban, 2002; Leh & Jobin, 2002; Meyer, 2002; Onay, 2002; Shelton & Saltsman, 2005; Stella & Gnanam, 2004; Suryanarayanaravu, Srinivasacharyulu, & Mohanraj, 1995).
Tool for evaluation, benchmarking, and strategic planning.
Quality Assessment in Higher Education
Rankings in US News and World Report
National Survey of Student Engagement (NSSE)
Regional and discipline specific accreditation
Need for continuous improvement strategy Public accountability Total Quality Management Balanced Scorecard Malcolm Baldrige National Quality Award
• Most valid (Badri et al., 2006; Ruben, Russ, Smulowitz & Connaughton, 2006)
From the Literature…
Several recommended groups of standards (14 different articles and studies)
WCET, CHEA, IHEP 24 Quality Standards , Bates’ ACTION model, Sloan Consortium Five Pillars
The IHEP 24 standards was highly referenced in the literature and therefore used as a beginning list of indicators.
Methodology
Delphi Method Structured flow of information using a series of systematic
surveys and reciprocal feedback Used to gain consensus from a panel of experts
(informed judgments) • The experts were widely located throughout the United States
Results were fed back to the panel of experts in Six surveys rounds--18 weeks (total time used).
Sample (Expert Panel)
Study Population: Online Education Administrators in Higher Education
Sampling Frame: Identified experts in the field by the Sloan Consortium (gatekeeper) 76 were invited; 43 participants completed first
round 83% of the panel members had nine or more
years of experience in the administration of online education
Panel Member Experience
0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0%
5 years or less
7 years or less
9 years or less
10 or more years
9.5%
7.1%
14.3%
69.0%
Sample Distribution
Institutional Classification Type Size Total
Public (4 year) Non-profit Large 24
Public Community College (2 year) Non-profit Large 2
Private (4 year) Non-profit Large 4
Private (4 year) For-profit Large 1
Private Faith-Based (4 year) Non-profit Large 1
Public (4 year) Non-profit Medium 2
Private (4 year) Non-profit Medium 3
Private Faith-based (4 year) Non-profit Medium 3
Public (4 year) Non-profit Small 1
Private (4 year) Non-profit Small 2
Panel Participation
Delphi Round
Total Experts Enlisted
Total Experts Who Completed
the Survey Response Rate
I 44 43 97.7%
II 43 41 95.5%
III 38 33 86.8%
IV 33 30 90.9%
V 30 28 93.3%
VI 28 26 92.9%
Research Questions
1. Are the standards identified in the IHEP/NEA study in 2000 still relevant today for indicating quality in online education programs in higher education?
2. What additional standards should be included that address the current industry in 2010?
3. If additional standards are suggested, will they fall into the already identified themes or will new themes emerge?
Research Questions
4. What values will be assigned to the recommended standards that will ultimately yield a numeric scorecard for measuring quality online education programs from an online education administrator’s perspective that could also support strategic planning and program improvements?
Instrumentation
Combination of open and closed questions
The 24 quality standards identified by IHEP for Round I Open-ended for additional standards and categories.
Survey Monkey - Web-based survey tool
Five point Likert Scale: 1=Definitely Not Relevant 2=Not Relevant 3=Slightly Relevant 4=Relevant 5=Definitely Relevant.
Delphi Survey
Each survey round was open for two weeks After one week, a follow-up email was sent
Data analyzed to develop next round survey Mean scores/suggestions for additional
quality standards and standards revisions were fed back in the next survey round.
Data Analysis
Descriptive Statistics Mean Scores and Standard Deviation Mean score of 4 and above with 70% or more agreement
were included in subsequent surveys Those with mean scores of below 4 were fed back one
additional time for final disapproval or during the first few rounds.
If consensus was not achieved, 70% of response choices were fed back to panel in the next round.
Results
Are the standards identified in the IHEP/NEA study in 2000 still relevant in 2010 for indicating quality in online education programs in higher education?
A form of all 24 indicators were included in the quality scorecard (23 relevant, 1 completely revised)
22 of the 24 indicators were revised.
Results
What additional standards should be included that address the current industry in 2010?
The panel suggested a total of 80 potential quality indicators. • 45 quality indicators were approved and included in the scorecard.
Adding these 45 indicators to the 25 indicators stemming from the IHEP study yielded a total of 70 quality indicators.
Results
If additional standards are suggested, will they fall into the already identified themes or will new themes emerge?
Twenty were suggested: Three categories achieved consensus; • Only two were added to the scorecard: Technology Support (divided
from the original IHEP Institutional Support) and Student and Social Engagement.
The Instructional Design category that achieved panel consensus was combined with Course Development.
Indicators by Category and Consensus
Category Total Number of Suggested
Quality Indicators
Total Number Approved by the Panel of Experts
Percent Achieving Consensus
Institutional Support 10 4 40% Technology Support 4 3 75% Course Development and Instructional Design
11 8 72%
Teaching and Learning 6 2 33% Course Structure 12 5 42% Student Support 16 11 69% Faculty Support 6 3 50% Evaluation and Assessment
14 8 57%
Social and Student Engagement
1 1 100%
Results
What values will be assigned to the recommended standards that will ultimately yield a numeric scorecard for measuring quality online education programs from an online education administrator’s perspective that could also support strategic planning and program improvements?
Guidelines for Scoring
0 points = Not Observed. The administrator does not observe any indications of the quality standard in place.
1 point = Insufficiently Observed. The administrator has found a slight existence of the quality standard in place. Much improvement is still needed in this area.
2 points = Moderate Use. The administrator has found there to be moderate use of the quality standard. Some improvement is still needed in this area.
3 points = Meets Criteria Completely. The administrator has found that the quality standard is being fully implemented and there is no need for improvement in this area.
Guidelines for Scoring
A perfect score = 210 points. 90-99% = 189-209 - Exemplary (little improvement is
needed) 80-89% = 168-188 - Acceptable (some improvement is
recommended) 70-79% = 147-167 - Marginal (significant improvement is
needed in multiple areas) 60-69% = 126-146 - Inadequate (many areas of
improvement are needed throughout the program) 59% and below = 125 pts and below - Unacceptable
Institutional Support
Technology Support
Course Development and Instructional Design
Course Structure
Teaching and Learning Social and Student Engagement
Faculty Support
Student Support
Evaluation and Assessment
Interactive Scorecard
On the Sloan-C website Must have an institutional membership Justification notes Upload artifacts for support
Organizational chart Faculty training materials Student survey results
Sloan-C Quality Framework
Sloan-C Five Pillars of Quality
Learning Effectiveness Scale (Cost Effectiveness and Commitment)
Access Faculty Satisfaction Student Satisfaction
Why Bother?
Self-study in quality Could be reviewed by accrediting agencies
Strategic planning Continuous improvement strategies
Next Steps
Developed additional support materials that further explain each indicator
Process for benchmarking Determine minimum
scores for each category Community of practice
website (Sloan-C)
Next Steps
Community of Practice Ongoing discussion Opportunity for scorecard refinement
Benchmarking process Anonymous Information will remain private
Expert Panel Shirley Adams Matt Baker Arnold Bateman Jennifer Baysden Rhonda Blackburn Kathleen Burke Tom Cavanagh Nan Chico Nancy Coleman Sharon Davis Mary Fehr Katherine Galaitsis Gwendolyn Godard Carmen Gonzalez Jo Hamilton Kenneth Hartman Margaret Haughey Mary Hawkins Cheryl Herdklotz Lisa Holstrom Janet Ross Kendall Karen King
Jon Lenrow Frank McCluskey Darrell Naylor-Johnson Loyce Pailen Latonia Pernell Alexandra M. Pickett Patrick Pluscht Maria Puzziferro Rob Robinson George Saltsman Jodi Servatius Scott Shiffer Mike Simmons Wayne Smutz James Stenerson Helen Torres Ken Udas Marvin Van Kekerix Karen Vignare Roger Von Holzen Ginger Zillges
Institutions Represented
Abilene Christian University American Public University System Athabasca University Bellevue University Boston University California State University East Bay Central Texas College Charter Oak State College Dallas Baptist University Drexel University
Online Drexel University East Carolina University East Tennessee State University Franklin University Michigan State University New Mexico State University North Carolina Agricultural & Technical State
University Northwest Missouri State University Pace University Peirce College Pennsylvania State World Campus
Regent University Rochester Institute of Technology Rocky Mountain College of Art and Design San Antonio College Savannah College of Art and Design State University of New York Texas Tech University Texas Woman's University The George Washington University The University of Texas at Dallas UMassOnline University of Central Florida University of Cincinnati University of Maryland University College University of Massachusetts, Boston University of Nebraska-Lincoln University of North Texas University of Texas Telecampus University of Wisconsin-Madison University Wisconsin-Milwaukee Washington State University
Questions
How is this different from Quality Matters? QM is a course rubric, this examines the entire program.
Why are these quality indicators important? Because a panel of experts were brought to consensus
through the Delphi process. (43 very seasoned administrators)
Can I use the scorecard freely without constraints? The scorecard is free to use. To use the interactive scorecard on the website, you must
be a Sloan-C institutional member.
Questions
Will Sloan-C be convening special interest groups around this? Commons site
Why should Sloan-C endorse this scorecard since it doesn’t have everybody’s input? Panel of experts with most 9 or more years Inviting you to join our commons site and participate in
ongoing discussions. Why is the scorecard input based and not output?
The scorecard offers broad goals for outputs but focusing on things administrators has influence on.
Questions
Can I give the scorecard to others? Yes, share the link
• http://tinyurl.com/qualitysc Will my information be private or public?
Private, we will not share the information you provide. Will we be comparing scores?
We are working on a process that would compare them anonymously by groups of peer institutions.
How can we use the results from others to help? To see where other programs’ strengths are.
Why isn’t the score normalized (100 points)? The expert panel felt that the graduated scale of 0-3 points per
indicator was the best measure.
How can I obtain a consultation on my program? We are developing a business model now and have some
of the expert panelist that are using the scorecard and could help with evaluation.
Send requests to [email protected] What if I already have a list of standards that I
use? The scorecard should mesh with other standards easily The Sloan-C pillars of quality aligned easily.
