A Quality Scorecard for the Administration of Online Education … · Data Analysis Descriptive...

A Quality Scorecard for the Administration of Online Education

Programs

Kaye Shelton, Ph.D. The Sloan Consortium

April 13, 2011

Focus

 Introduce the Quality Scorecard  Explain the history and rationale  Why use the Scorecard?  How to use the interactive Scorecard  Handbook as a guide for completing

the Scorecard

To develop a quality scorecard to evaluate the administration of

online education programs.

Purpose

The Need For This Study

 Quality Assurance (Bates & Poole, 2003; Meyer, 2004; Sallis, 1996).

 From the beginning, the quality of online education was questioned (Benson, 2007).

 Standards must be clearly defined (Institute for Higher Education Policy, 1998, 2000; Khan, 2005; Lee & Dziuban, 2002; Leh & Jobin, 2002; Meyer, 2002; Onay, 2002; Shelton & Saltsman, 2005; Stella & Gnanam, 2004; Suryanarayanaravu, Srinivasacharyulu, & Mohanraj, 1995).

 Tool for evaluation, benchmarking, and strategic planning.

Quality Assessment in Higher Education

 Rankings in US News and World Report

 National Survey of Student Engagement (NSSE)

 Regional and discipline specific accreditation

 Need for continuous improvement strategy  Public accountability  Total Quality Management  Balanced Scorecard  Malcolm Baldrige National Quality Award

•  Most valid (Badri et al., 2006; Ruben, Russ, Smulowitz & Connaughton, 2006)

From the Literature…

 Several recommended groups of standards (14 different articles and studies)

 WCET, CHEA, IHEP 24 Quality Standards , Bates’ ACTION model, Sloan Consortium Five Pillars

 The IHEP 24 standards was highly referenced in the literature and therefore used as a beginning list of indicators.

Methodology

 Delphi Method  Structured flow of information using a series of systematic

surveys and reciprocal feedback  Used to gain consensus from a panel of experts

(informed judgments) •  The experts were widely located throughout the United States

 Results were fed back to the panel of experts in Six surveys rounds--18 weeks (total time used).

Sample (Expert Panel)

 Study Population: Online Education Administrators in Higher Education

 Sampling Frame: Identified experts in the field by the Sloan Consortium (gatekeeper)  76 were invited; 43 participants completed first

round  83% of the panel members had nine or more

years of experience in the administration of online education

Panel Member Experience

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0%

5 years or less

7 years or less

9 years or less

10 or more years

9.5%

7.1%

14.3%

69.0%

Sample Distribution

Institutional Classification Type Size Total

Public (4 year) Non-profit Large 24

Public Community College (2 year) Non-profit Large 2

Private (4 year) Non-profit Large 4

Private (4 year) For-profit Large 1

Private Faith-Based (4 year) Non-profit Large 1

Public (4 year) Non-profit Medium 2

Private (4 year) Non-profit Medium 3

Private Faith-based (4 year) Non-profit Medium 3

Public (4 year) Non-profit Small 1

Private (4 year) Non-profit Small 2

Panel Participation

Delphi Round

Total Experts Enlisted

Total Experts Who Completed

the Survey Response Rate

I 44 43 97.7%

II 43 41 95.5%

III 38 33 86.8%

IV 33 30 90.9%

V 30 28 93.3%

VI 28 26 92.9%

Research Questions

1.  Are the standards identified in the IHEP/NEA study in 2000 still relevant today for indicating quality in online education programs in higher education?

2.  What additional standards should be included that address the current industry in 2010?

3.  If additional standards are suggested, will they fall into the already identified themes or will new themes emerge?

Research Questions

4.  What values will be assigned to the recommended standards that will ultimately yield a numeric scorecard for measuring quality online education programs from an online education administrator’s perspective that could also support strategic planning and program improvements?

Instrumentation

  Combination of open and closed questions

  The 24 quality standards identified by IHEP for Round I   Open-ended for additional standards and categories.

  Survey Monkey - Web-based survey tool

  Five point Likert Scale:   1=Definitely Not Relevant   2=Not Relevant   3=Slightly Relevant   4=Relevant   5=Definitely Relevant.

Delphi Survey

 Each survey round was open for two weeks  After one week, a follow-up email was sent

 Data analyzed to develop next round survey  Mean scores/suggestions for additional

quality standards and standards revisions were fed back in the next survey round.

Data Analysis

 Descriptive Statistics  Mean Scores and Standard Deviation  Mean score of 4 and above with 70% or more agreement

were included in subsequent surveys  Those with mean scores of below 4 were fed back one

additional time for final disapproval or during the first few rounds.

  If consensus was not achieved, 70% of response choices were fed back to panel in the next round.

Results

 Are the standards identified in the IHEP/NEA study in 2000 still relevant in 2010 for indicating quality in online education programs in higher education?

 A form of all 24 indicators were included in the quality scorecard (23 relevant, 1 completely revised)

 22 of the 24 indicators were revised.

Results

 What additional standards should be included that address the current industry in 2010?

 The panel suggested a total of 80 potential quality indicators. • 45 quality indicators were approved and included in the scorecard.

 Adding these 45 indicators to the 25 indicators stemming from the IHEP study yielded a total of 70 quality indicators.

Results

  If additional standards are suggested, will they fall into the already identified themes or will new themes emerge?

 Twenty were suggested: Three categories achieved consensus; •  Only two were added to the scorecard: Technology Support (divided

from the original IHEP Institutional Support) and Student and Social Engagement.

 The Instructional Design category that achieved panel consensus was combined with Course Development.

Indicators by Category and Consensus

Category Total Number of Suggested

Quality Indicators

Total Number Approved by the Panel of Experts

Percent Achieving Consensus

Institutional Support 10 4 40% Technology Support 4 3 75% Course Development and Instructional Design

11 8 72%

Teaching and Learning 6 2 33% Course Structure 12 5 42% Student Support 16 11 69% Faculty Support 6 3 50% Evaluation and Assessment

14 8 57%

Social and Student Engagement

1 1 100%

Results

 What values will be assigned to the recommended standards that will ultimately yield a numeric scorecard for measuring quality online education programs from an online education administrator’s perspective that could also support strategic planning and program improvements?

Guidelines for Scoring

  0 points = Not Observed. The administrator does not observe any indications of the quality standard in place.

  1 point = Insufficiently Observed. The administrator has found a slight existence of the quality standard in place. Much improvement is still needed in this area.

  2 points = Moderate Use. The administrator has found there to be moderate use of the quality standard. Some improvement is still needed in this area.

  3 points = Meets Criteria Completely. The administrator has found that the quality standard is being fully implemented and there is no need for improvement in this area.

Guidelines for Scoring

 A perfect score = 210 points.  90-99% = 189-209 - Exemplary (little improvement is

needed)  80-89% = 168-188 - Acceptable (some improvement is

recommended)  70-79% = 147-167 - Marginal (significant improvement is

needed in multiple areas)  60-69% = 126-146 - Inadequate (many areas of

improvement are needed throughout the program)  59% and below = 125 pts and below - Unacceptable

Institutional Support

Technology Support

Course Development and Instructional Design

Course Structure

Teaching and Learning Social and Student Engagement

Faculty Support

Student Support

Evaluation and Assessment

Interactive Scorecard

 On the Sloan-C website  Must have an institutional membership  Justification notes  Upload artifacts for support

 Organizational chart  Faculty training materials  Student survey results

Sloan-C Quality Framework

Sloan-C Five Pillars of Quality

 Learning Effectiveness  Scale (Cost Effectiveness and Commitment)

 Access  Faculty Satisfaction  Student Satisfaction

Why Bother?

 Self-study in quality  Could be reviewed by accrediting agencies

 Strategic planning  Continuous improvement strategies

Next Steps

 Developed additional support materials that further explain each indicator

 Process for benchmarking  Determine minimum

scores for each category  Community of practice

website (Sloan-C)

Next Steps

 Community of Practice  Ongoing discussion  Opportunity for scorecard refinement

 Benchmarking process  Anonymous  Information will remain private

Expert Panel   Shirley Adams   Matt Baker   Arnold Bateman   Jennifer Baysden   Rhonda Blackburn   Kathleen Burke   Tom Cavanagh   Nan Chico   Nancy Coleman   Sharon Davis   Mary Fehr   Katherine Galaitsis   Gwendolyn Godard   Carmen Gonzalez   Jo Hamilton   Kenneth Hartman   Margaret Haughey   Mary Hawkins   Cheryl Herdklotz   Lisa Holstrom   Janet Ross Kendall   Karen King

  Jon Lenrow   Frank McCluskey   Darrell Naylor-Johnson   Loyce Pailen   Latonia Pernell   Alexandra M. Pickett   Patrick Pluscht   Maria Puzziferro   Rob Robinson   George Saltsman   Jodi Servatius   Scott Shiffer   Mike Simmons   Wayne Smutz   James Stenerson   Helen Torres   Ken Udas   Marvin Van Kekerix   Karen Vignare   Roger Von Holzen   Ginger Zillges

Institutions Represented

  Abilene Christian University   American Public University System   Athabasca University   Bellevue University   Boston University   California State University East Bay   Central Texas College   Charter Oak State College   Dallas Baptist University Drexel University

Online   Drexel University   East Carolina University   East Tennessee State University   Franklin University   Michigan State University   New Mexico State University   North Carolina Agricultural & Technical State

University   Northwest Missouri State University   Pace University   Peirce College   Pennsylvania State World Campus

  Regent University   Rochester Institute of Technology   Rocky Mountain College of Art and Design   San Antonio College   Savannah College of Art and Design   State University of New York   Texas Tech University   Texas Woman's University   The George Washington University   The University of Texas at Dallas   UMassOnline   University of Central Florida   University of Cincinnati   University of Maryland University College   University of Massachusetts, Boston   University of Nebraska-Lincoln   University of North Texas   University of Texas Telecampus   University of Wisconsin-Madison   University Wisconsin-Milwaukee   Washington State University

Questions

 How is this different from Quality Matters?  QM is a course rubric, this examines the entire program.

 Why are these quality indicators important?  Because a panel of experts were brought to consensus

through the Delphi process. (43 very seasoned administrators)

 Can I use the scorecard freely without constraints?  The scorecard is free to use.  To use the interactive scorecard on the website, you must

be a Sloan-C institutional member.

Questions

 Will Sloan-C be convening special interest groups around this?  Commons site

 Why should Sloan-C endorse this scorecard since it doesn’t have everybody’s input?  Panel of experts with most 9 or more years   Inviting you to join our commons site and participate in

ongoing discussions.  Why is the scorecard input based and not output?

 The scorecard offers broad goals for outputs but focusing on things administrators has influence on.

Questions

  Can I give the scorecard to others?   Yes, share the link

•  http://tinyurl.com/qualitysc   Will my information be private or public?

  Private, we will not share the information you provide.   Will we be comparing scores?

  We are working on a process that would compare them anonymously by groups of peer institutions.

  How can we use the results from others to help?   To see where other programs’ strengths are.

  Why isn’t the score normalized (100 points)?   The expert panel felt that the graduated scale of 0-3 points per

indicator was the best measure.

 How can I obtain a consultation on my program?  We are developing a business model now and have some

of the expert panelist that are using the scorecard and could help with evaluation.

 Send requests to kshelton@sloanconsortium.org  What if I already have a list of standards that I

use?  The scorecard should mesh with other standards easily  The Sloan-C pillars of quality aligned easily.

For further questions please contact me at

kshelton@sloanconsortium.org