Taking the Risk, Calculating the Return: Developing Successful E- Learning Partnerships Curt Bonk, Indiana University, Associate Prof cjbonkand

Taking the Risk, Calculating the Return: Developing Successful E- Learning Partnerships Curt Bonk, Indiana University, Associate Prof http://php.indiana.edu/~cjbonkand President, CourseShare.com [email protected]://CourseShare.comand Ann Hill Duin, University of Minnesota; [email protected]

Are you ready???

Primary Institutional Motives for Online Education Access (81 percent) Access (81 percent) Learning (53 percent) Learning (53 percent) Profit (29 percent) Profit (29 percent)

Key Reasons for University Investment Access to external resources (67 percent). Access to external resources (67 percent). Improved efficiency in teaching and research (63 percent). Improved efficiency in teaching and research (63 percent). Providing distance education to a potentially unlimited audience (58 percent). Providing distance education to a potentially unlimited audience (58 percent).

Less Important Reasons for University Investment Cooperation and resource sharing within the higher education community (41 percent). Cooperation and resource sharing within the higher education community (41 percent). Building partnerships with business and government (31 percent). Building partnerships with business and government (31 percent).

Some Respondent Quotes to offer equal opportunity of high quality education to students in more rural areas, we are under a mandate to increase the number of students we serve, Its a new revenue source, thats #1, and because Web-based activities are becoming ubiquitous in ALL workplaces. Students will demand Web-based courses or go somewhere else.

Facts and Projections Those participating in continuing education will rise to 100 million by 2004Those participating in continuing education will rise to 100 million by 2004 Eleven mega-universities serve learners at a cost of $350 per learner per yearEleven mega-universities serve learners at a cost of $350 per learner per year Postsecondary enrollment in web-based courses will represent 15% of total enrollment in 2002Postsecondary enrollment in web-based courses will represent 15% of total enrollment in 2002 International Data Corporation, 2001 E-learning market will be $25 billion by 2003E-learning market will be $25 billion by 2003 Merrill Lynch, 2001 By 2010, the number of corporate universities will exceed the number of traditional universitiesBy 2010, the number of corporate universities will exceed the number of traditional universities U.S. Commission on Web-based Learning, 2000

Robert Jackson, E-Learning, Corp- University Partnerships University of Tennessee, Assistant Dean of University Outreach and Director of Distance Ed & Independent Study; [email protected] Higher education has long had a rocky road creating successful partnerships with business and industry. The rise of e-learning represents a critical opportunity to all parties. Hear what separates these groups and find out what could bring them together in mutually beneficial elearning partnerships.

Distant Cousins (Robert Jackson, e-learning 2002) Historical PerspectiveHistorical Perspective Closed Architectures Separate trade shows and coordinating orgsSeparate trade shows and coordinating orgs Uneasy Truce Philosophical Differences: Training vs EDPhilosophical Differences: Training vs ED Course cost $35,000-$125,000 (mostly slick multimedia) vs $4,000-$10,000 (mostly text and static graphics) SME (part-time or temps vs. fulltime tenured professor Owned by company (work for hire) vs. owned by individual (intellectual property)

Inputs and Processes Consensus (Robert Jackson, e-learning 2002) Learning TopicCorporations ViewUniversities View Dominant elearning instructional style Self-pacedInstructor-led Interest in collaborative activities as learning tools HighHigh Preferred learning format asynchronoussynchronous Preferred live instructional metaphor Shoulder to shoulder, peer-to-peer face to face, expert to learner Use of Course Preassessments and gain scores FrequentlyRarely Curriculum Designer Training Manager Individual Faculty Instructional Technology Developer Contract Programmer Faculty or grad assistants Targeted learner community Independent Study Cohort Preferred form of elearning content management software Learning Management System (LMS) Course Management System (CMS) Interest in Learning Content Management Systems (LCMS) Moderate to high Low Interest in elearning management system uniform standards High Low to moderate

Goals & Outcomes Consensus (Robert Jackson, e-learning 2002) Learning TopicCorporations ViewUniversities View Learning purpose just in time, job enhancement focused just in case, knowledge awareness focused Target duration of learning Just enough Feast of learning Concern for dropouts Low to moderate Moderate to high Descriptive word for learning used in conversation skills or training education Assessment strategies Competency, prescriptive assessment, skill gap analysis, quantitative Mastery, performance relative to cohort, qualitative assessments by faculty Time efficiency goals Reduce time and expense Restricted by accreditation standards Length of typical learning object is measured in increments of Minutes, within learning objects Hours/Days, within courses Institutional desire for learning content modularity High Moderate to low

Common Ground Strategic Directions (Robert Jackson, e-learning 2002) AcademicAcademic Value quantified outcomes Embrace shorter courses (modularity), and by extension, elearning standards Be flexible, rethinking old campus-centric conventions Embrace standards and preassessment Corporate & Government Appreciate knowledge as a route to strategic thinking Appreciate and help change higher ed standards of seat- time. Value synchronous delivery

Making Connections Partnering in a Digital Age Ideas from Ann Hill Duin Ann Hill Duin, [email protected] 2002

An Internet gateway through which learners, employers, and learning providers are drawn together into a dynamic partnership that creates value for learners, enhances economic development, and engages institutions in meeting the lifelong learning needs of twenty-first century learners (2001, xvii).

Priorities: What outcomes should you expect?

Possible Priorities 1.Address a clear learner need 2.Leverage resources; share infrastructure 3.Respond to new markets; improve competitiveness 4.Enhance access and pedagogy of learning

US Army Choose Pricewaterhouse $453 million e-learning program$453 million e-learning program 10 companies and 29 colleges10 companies and 29 colleges 12,000 students in year 1; 10,000 more in 200212,000 students in year 1; 10,000 more in 2002 Goals: Tech savvy soliders; succeed in on the digitized battlefield, enhance retention and help soldiers achieve academic degrees.Goals: Tech savvy soliders; succeed in on the digitized battlefield, enhance retention and help soldiers achieve academic degrees. Soldiers stationed in Australia, Honduras, Kosovo, Bosnia and Herzegovina, Belgium, Japan, Egypt, the UK, Kuwait, Singapore, Germany, Korea, Macedonia, Italy and Jordan.Soldiers stationed in Australia, Honduras, Kosovo, Bosnia and Herzegovina, Belgium, Japan, Egypt, the UK, Kuwait, Singapore, Germany, Korea, Macedonia, Italy and Jordan.

Distance learner - message I currently live in Maryland and will be moving to Mason City, Iowa to be with family. I have an AA from a community college in Maryland. I wanted to know if you had a distance education program at [local community college] that I would be able to get involved in. If you do, could you email me info at or mail it to me at

Certificate: Using the Internet in Corp Training Jones International University Corp trainers retool existing content for online delivery; build an online moduleCorp trainers retool existing content for online delivery; build an online module 4 weeks, $550; also optional customized corp training certificate4 weeks, $550; also optional customized corp training certificate Use Web to enhance course content, & create a powerful, interactive learning environment for trainees.Use Web to enhance course content, & create a powerful, interactive learning environment for trainees.

Online Graduation Ceremonies with Famous Commencement Speakers

In Search of a Need? Fathom, the for-profit, online-learning provider run by Columbia University, has added corporate-training courses to its academic and cultural offerings in a bid to market itself to corporate achievers and job hunters. The move is the latest change in direction the company has made in its search for customers and investors in the competitive distance-education market.Fathom, the for-profit, online-learning provider run by Columbia University, has added corporate-training courses to its academic and cultural offerings in a bid to market itself to corporate achievers and job hunters. The move is the latest change in direction the company has made in its search for customers and investors in the competitive distance-education market.Fathom, Michael Arnone, Chronicle of HE, Feb 8, 2002

In Search of a Need? Columbia announced Wednesday that four for-profit education companies -- SmartForce, Zoologic, the Kaplan Colleges, and PrimeLearning -- are now offering mostly noncredit courses on business- related subjects through Fathom's Professional Development Learning Center The partnerships will benefit Fathom in the short run because Fathom needs the money that the courses will generate,Columbia announced Wednesday that four for-profit education companies -- SmartForce, Zoologic, the Kaplan Colleges, and PrimeLearning -- are now offering mostly noncredit courses on business- related subjects through Fathom's Professional Development Learning Center The partnerships will benefit Fathom in the short run because Fathom needs the money that the courses will generate, SmartForce,Zoologic,the Kaplan Colleges,PrimeLearning Professional Development Learning Center SmartForce,Zoologic,the Kaplan Colleges,PrimeLearning Professional Development Learning Center

University Resource Partnership 3/3/02 DSpace Archive: MIT and HP to create a LT sustainable digital repository3/3/02 DSpace Archive: MIT and HP to create a LT sustainable digital repository Instead of submitting the paper to a print commercial journal and waiting months for the results to be published, the researcher can simply pull up MITs Center of Teleportation Research Web page and instandly submit the paper and data online, for all his cohorts to review. Kendra Mayfield Wired News, College Archives Dig Deeper. http://www.wired.com/news/culture/0,1284,54229,00.htmlhttp://www.wired.com/news/culture/0,1284,54229,00.html

University Resource Partnership

Administrators and faculty members at the Massachusetts Institute of Technology are debating what could become a $100-million effort to create extensive World Wide Web pages for nearly every course the university offers. Jeffrey R. Young, March 1, 2001, The Chronicle of Higher Ed Also See: MIT Cheered from a Distance, Wired News, http://www.wired.com/news/culture/0,1284,42841,00.html University-World Resource Partnership

Cross-University Partnerships 1/23/01 The Alliance of Four1/23/01 The Alliance of Four 4 Universities Create an Alliance Against Onslight of Technology Vendors4 Universities Create an Alliance Against Onslight of Technology Vendors UC-Berkeley, Penn State Univ World Campus, Univ of Washington, and Univ of Wisconsin Learning Innovations formed an (informal) alliance to make joint technology purchases. Trying to deal with vendors who besiege universities and pick them off one at a time

State Resource Partnership

National Resource Partnership

Publishers Technology Distribution Software Developers Book Publishers Hollywood Producers Newspapers On-Line Services Cable Companies Broadcasters Telephone Cos. Computer Nets Retail Stores ISDN MPEG/DVI Photo CD HDTV QuickTime OS/2 Windows USERS National/Intl Resource Partnership Learning Object Standards

International Resource Partnership

Other Resource Partnerships 22 virtual universities to collaborate 9 universities on 4 continents collab to offer online graduate and professional development courses in Asia University of the Arctic is a consortium of 31 high latitude colleges, universities, and governments across 8 countries. First course is Introduction to Circumpolar Studies. (Feb. 15, 2002, Chronicle of HE).

Level 3 University Partnerships (Dan Carnevale, Chronicle of Higher Ed, Jan 28, 2002) Babson College MBA for IntelBabson College MBA for Intel Oregon Health and Science Univ MA in Tech Management for MicrosoftOregon Health and Science Univ MA in Tech Management for Microsoft UT-Austin Online MS in Science, Tech, and Commercialization for IBMUT-Austin Online MS in Science, Tech, and Commercialization for IBM Univ of Georgia MBA for PricewaterhouseCoopersUniv of Georgia MBA for PricewaterhouseCoopers Cenquest, President, LaVonne Reimer claims courses are more relevant to occupations (colleges target corp training and exec ed)Cenquest, President, LaVonne Reimer claims courses are more relevant to occupations (colleges target corp training and exec ed)

Not All Partnerships Work 6/9/01 Faculty wary of e-Cornell6/9/01 Faculty wary of e-Cornell 6/1/01 Company that sells Dukes online MBA program (Pensare) files for bankruptcy6/1/01 Company that sells Dukes online MBA program (Pensare) files for bankruptcy 8/7/01 UNext asks to restructure its relationshsips with CMU,Stanford, Columbia8/7/01 UNext asks to restructure its relationshsips with CMU,Stanford, Columbia 9/13/01 UNext lays off 135 employees9/13/01 UNext lays off 135 employees 12/14/01 Debating the demise of NYUonline12/14/01 Debating the demise of NYUonline 5/2/02 US Open University will close5/2/02 US Open University will close 2/20/02 SUNY-Buffalo drops online MBA2/20/02 SUNY-Buffalo drops online MBA

Reasons for Failure UNext--Difficult economic period; cited challenges of raising venture capitalUNext--Difficult economic period; cited challenges of raising venture capital US Open UInsufficient revenues; inadequate enrollments; lacking accreditation & name recUS Open UInsufficient revenues; inadequate enrollments; lacking accreditation & name rec E-Cornelldiff from mission; out of thin airE-Cornelldiff from mission; out of thin air Pensare/Dukelower interest than expected and then market tightenedPensare/Dukelower interest than expected and then market tightened NUYonlineinadequate business plan (similar problems to UMUConline, Virtual Temple, etc.)NUYonlineinadequate business plan (similar problems to UMUConline, Virtual Temple, etc.) SUNY-Buffalonot worth expense & hassleSUNY-Buffalonot worth expense & hassle Labor intensive courses, time consuming, etc.

Some Partners are Not Strong Katherine Mangan, Chronicle of HE, Feb 20, 2002 Like several other business schools whose efforts to expand online have been derailed, Buffalo was burned by an outside partner that failed to live up to its promisesThe Albany-based Institute for Entrepreneurship, which ran into financial and management problems, gave the school only about $65,000 of the $200,000 it had promised.Like several other business schools whose efforts to expand online have been derailed, Buffalo was burned by an outside partner that failed to live up to its promisesThe Albany-based Institute for Entrepreneurship, which ran into financial and management problems, gave the school only about $65,000 of the $200,000 it had promised.

IU Online MBA Homepage

Some Facts The Indiana University Kelley School of Business, for more than eighty years, has been one of the countrys premier business schools.The Indiana University Kelley School of Business, for more than eighty years, has been one of the countrys premier business schools. The undergraduate program is ranked among the top 10 in the country.The undergraduate program is ranked among the top 10 in the country. The MBA program is ranked by BusinessWeek as among the top 20.The MBA program is ranked by BusinessWeek as among the top 20. The MBA faculty have the top threeThe MBA faculty have the top three teaching faculty in the country.

IU Oncourse Partnership (now ANGEL?)

Kelley Direct Partnership with GM Joint program with Purdue School of Engineering and Continuing EducationJoint program with Purdue School of Engineering and Continuing Education Obtain MBA online after complete engineering MSObtain MBA online after complete engineering MS 48 credits normal (cut to 42 in here)48 credits normal (cut to 42 in here) March 2002 35 in programMarch 2002 35 in program 100 waiting to get in100 waiting to get in Take 1 courses/quarter (3 yr program)Take 1 courses/quarter (3 yr program)

Partnership with GM MS in Finance (20 students)MS in Finance (20 students) MS in Strategic Management (15-20 sts)MS in Strategic Management (15-20 sts) 30 credits30 credits 1 course/quarter1 course/quarter 3 year program3 year program

Partnership with United Technologies MBA, began spring 2002MBA, began spring 2002 MS in Global Supply Chain ManagementMS in Global Supply Chain Management 2 year program2 year program 30 students, 2 courses/quarter30 students, 2 courses/quarter 1 week summer residency1 week summer residency 1 week at site1 week at site

Partnership with John Deere MS in finance for financial managersMS in finance for financial managers Just beganJust began 52 credits52 credits 25 students in program25 students in program

Summary to Date 2 year program with residency2 year program with residency Started with 14 studentsStarted with 14 students Now have 240 total in MBA programsNow have 240 total in MBA programs Graduated 60 MBA students to dateGraduated 60 MBA students to date Gain students though networking, customer contacts, etc.Gain students though networking, customer contacts, etc.

1.Key component in many recruiting strategies. 2.Assists firms in retaining their most valued employees. 3.Since the program has a very small in- residence component, there is little disruption to an employees work schedule. Conclusion The Online MBA program is designed to be beneficial to the employer of a student as well:

Videoconferencing at IU

1.Members pay $10,000 per year to join 2.Full access to software and its source code 3.Documents and evaluates student achievements and learning improvements 4.Students reflect on work and instructors reflect on their quality 5.Help students transfer 6.Learning benchmarks can be established IUPUI & UCLA e-Portfolios http://eportconsortium.org

Learning to Teach with Technology Studio (Partnership of IU and PBS)

A New Type of Professional Development The Learning to Teach with Technology Studio is a Web-based professional development system offering quality instruction in quality K - 12 technology integration designed for educators.

LTTS Features Short courses (25 growing to 55+) On the web Start anytime and move at own pace Focus on technology integration in inquiry lessons and projects Problem-centered modules Practical and educational Facilitated Standards-based (ISTE NETS and national academic standards)

TICKIT: Rural Teacher Technology Integration

Overview of TICKIT In-service teacher education program Rural schools in southern Indiana Yearlong, 25 teachers from 5 schools Primarily school-based Supported by participating school systems, Arthur Vining Davis Foundations and Indiana University

Quality on the Line: Benchmarks for Success in Internet-Based Distance Ed (Blackboard & NEA, 2000) Teaching/Learning Process Student interaction with faculty is facilitated through a variety of ways.Student interaction with faculty is facilitated through a variety of ways. Feedback to student assignments and questions is provided in a timely manner.Feedback to student assignments and questions is provided in a timely manner. Each module requires students to engage themselves in analysis, synthesis, and evaluation as part of their course assignments.Each module requires students to engage themselves in analysis, synthesis, and evaluation as part of their course assignments. Course materials promote collaboration among students.Course materials promote collaboration among students. http://www.ihep.com/Pubs/PDF/Quality.pdf

Quality on the Line: Benchmarks for Success in Internet-Based Distance Ed (Blackboard & NEA, 2000) Other Benchmark Categories: Institutional Support: incentive, rewards, plansInstitutional Support: incentive, rewards, plans Course Development: processes, guidelines, teams, structures, standards, learning stylesCourse Development: processes, guidelines, teams, structures, standards, learning styles Course Structure: expectations, resourcesCourse Structure: expectations, resources Student Support: training, assistance, infoStudent Support: training, assistance, info Faculty Support: mentoring, tech supportFaculty Support: mentoring, tech support Evaluation and Assessment: review process, multiple methods, specific standardsEvaluation and Assessment: review process, multiple methods, specific standards

E-Learning: Harnessing the hype. Cohen & Payiatakis (2002, Feb). Performance Improvement, 41(7), 7-15. both instructional and graphic (design)must be compelling and engaging enough to keep the learner involved, interested, and stimulatedThe ideal future is a learning experience designed to be memorable, motivational, and magical if it is to make a lasting impact on the capabilities of the learner.

Online Courseware Development Partnerships Rather than every large higher education institution attempting to spend money to develop its own courseware platform or shell, colleges and universities should seek partnerships with courseware and other e-learning companies wherein they serve as beta test sites for new tool development efforts. They might also seek to form tool development consortia with other institutions. (Bonk, 2001) Online Teaching in an Online World.

Online Learning Pedagogy Partnerships In conjunction with the last recommendation, higher education institutions need to demand and perhaps help develop and research different types of pedagogical tools for e-learning that foster student higher-order thinking and collaboration. Once developed, online tools that target critical and creative thinking as well as teamwork online should be showcased for faculty, students, and administrators. (Bonk, 2001) Online Teaching in an Online World.

Ideas Develop a joint programDevelop a joint program Locate the best resources and bring them to learners via a learning marketspaceLocate the best resources and bring them to learners via a learning marketspace Promote collective virtual learning resources to the worldPromote collective virtual learning resources to the world Make credits transferable between programsMake credits transferable between programs Partner with private companies to assist learners with additional experiencesPartner with private companies to assist learners with additional experiences

Readiness: What factors most determine success?

Readiness The probability of success is dependent on many factorsthe basic premise is that a universitys preparation or readiness prior to a partnership initiation is the single most important contributor to success. Robinson & Daigle, 1999, p. 4. Robinson & Daigle, 1999, p. 4.

State Readiness Criteria Rosevears study of 8 virtual universitiesRosevears study of 8 virtual universities What is the states technological infrastructure?What is the states technological infrastructure? How prepared are the traditional colleges and universities to support virtual learning environments?How prepared are the traditional colleges and universities to support virtual learning environments? Do they all have equal technological capabilities?Do they all have equal technological capabilities? What is a reasonable prediction for how long it will take before the virtual university is operational?What is a reasonable prediction for how long it will take before the virtual university is operational? What are the resources gaps, and how will they be filled?What are the resources gaps, and how will they be filled?

Inter-Institutional Readiness Criteria Leadership committed to the effortLeadership committed to the effort Commitment to learner centered educationCommitment to learner centered education Climate to support partnership and changeClimate to support partnership and change Alignment of key decision makersAlignment of key decision makers Buy-in by faculty, departments, and collegesBuy-in by faculty, departments, and colleges

Readiness criteria for shared course environments Similar problemsSimilar problems Trust and respectTrust and respect Low level permissionsLow level permissions No $$ or creditNo $$ or credit Clear communicationClear communication Clear outcomesClear outcomes Delineation of tasks Matrix development Phase one prototype Course offered again Equivalent tech infrastructures In-house mentoring

What is your institutions partnership rationale?

Partnership Rationales Dave King (IHETS):Dave King (IHETS): Our rationale is that we choose partners based on learner needs and risk. The learners risk must be low. Technology is never complete until the user is satisfied. Susan Kannel (NACTEL):Susan Kannel (NACTEL): Our rationale is that we choose a college or university partner based on quality online curricula needed by industry. Bruce Chaloux (SREB):Bruce Chaloux (SREB): Our rationale is to use partnerships to level the playing field (policy development) on behalf of student access.

Partnership Rationale -- Blueprint VisionVision DescriptionDescription BeliefsBeliefs AssumptionsAssumptions OperationsOperations CommitmentCommitment Collaboration Risk Control Adaptation Return on Investment

John Chambers Partnership Rationale (criteria) Do the partners enjoy a shared vision with complementary roles?Do the partners enjoy a shared vision with complementary roles? Can the partners create short-term wins?Can the partners create short-term wins? Does the partnership create a win for all key stakeholders?Does the partnership create a win for all key stakeholders? Are the organizational cultures similar?Are the organizational cultures similar? Is there geographic proximity?Is there geographic proximity?

What is your institutions risk tolerance related to partnerships? -- legal -- financial -- experimental -- academic

Types and Levels of Risk Associated with Partnerships Commerce Alliance Minority equity investment Joint Venture Spin off Merger or acquisition LegalLowLowLowMediumHigh FinancialLowLowLowHighHigh ExperimentationLowLowMediumHighHigh AcademicMediumMediumMediumHighHigh

Consider your current partnerships. What is your potential return on investment? Learner/Citizen Faculty Campus State

Adding Value The magnitude, form, source, and distribution of that value is at the heart of relational dynamics. The perceived worth of an alliance is the ultimate determinant of, first, whether it will be created and, second, whether it will be sustained. The magnitude, form, source, and distribution of that value is at the heart of relational dynamics. The perceived worth of an alliance is the ultimate determinant of, first, whether it will be created and, second, whether it will be sustained. --James Austin, 2000, The Collaboration Challenge, 87 --James Austin, 2000, The Collaboration Challenge, 87

ROI for Iowa State University Commerce Alliance Fathom Minority investment PBS and IU Joint Venture IUPUI & UCLA Joint Venture Purdue-IU & GM Spin off Wisdom Tools & IIPI Learners / citizens HighHighHighHighMedium FacultyMediumMediumMediumMediumLow CampusLowLowMediumHighMedium StateMediumMediumMediumHighHigh

ROI: What is the return on e-learning???

Business ROI Objectives Better Efficiencies (reduced travel, instructor fees, distribution and facilities costs)Better Efficiencies (reduced travel, instructor fees, distribution and facilities costs) Greater ProfitabilityGreater Profitability Increased SalesIncreased Sales Fewer Injuries on the JobFewer Injuries on the Job Less Time off WorkLess Time off Work Faster Time to CompetencyFaster Time to Competency

ROI in Higher Education Students: opportunities, integration with work, learning on demand, inc promotion, new wealthStudents: opportunities, integration with work, learning on demand, inc promotion, new wealth Faculty: templates for dev curric, integration with real world, time, $Faculty: templates for dev curric, integration with real world, time, $ Campus: educating more, reaching local community, expanding resource base, dispersing costsCampus: educating more, reaching local community, expanding resource base, dispersing costs State: economic devel, increased business competitiveness, better resource allocationState: economic devel, increased business competitiveness, better resource allocation

Talking about ROI As a percentageAs a percentage ROI=[(Payback- Investment)/Investment]* 100 As a ratioAs a ratio ROI=Return/Investment As time to break evenAs time to break even Break even time=(Investment/Return) *Time Period

More Calculations Total Admin Costs of Former Program - Total Admin Costs of OL Program =Projected Net SavingsTotal Admin Costs of Former Program - Total Admin Costs of OL Program =Projected Net Savings Total Cost of Training/# of Students =Cost Per Student (CPS)Total Cost of Training/# of Students =Cost Per Student (CPS) Total Benefits * 100/Total Program Cost =ROI%Total Benefits * 100/Total Program Cost =ROI%

Kirkpatricks 4 Levels ReactionReaction LearningLearning BehaviorBehavior ResultsResults

Level 1 Comments. Reactions We assess our courses based on participation levels and online surveys after course completion. All of our courses are asynchronous. Typically involves Smile sheets or end- of-training evaluation forms. Easy to collect, but not always very useful. We use the Halo Survey process of asking them when the course is concluding.

Kirkpatrick Level 3: Behavior Can learner apply what learned? Might include:Can learner apply what learned? Might include: Direct observation by supervisors or coaches (Wisher, Curnow, & Drenth, 2001). Questionnaires completed by peers, supervisors, and subordinates related to work performance. On the job behaviors, automatically logged performances, or self-report data. Shepard, C. (1999b, July). Evaluating online learning. TACTIX from Fastrak Consulting. Retrieved February 10, 2002, from: http://fastrak-consulting.co.uk/tactix/Features/evaluate/eval01.htm.

Kirkpatrick Level 4: Results Often compared to return on investment (ROI)Often compared to return on investment (ROI) In e-learning, it is believed that the increased cost of course development ultimately is offset by the lesser cost of training implementationIn e-learning, it is believed that the increased cost of course development ultimately is offset by the lesser cost of training implementation A new way of training may require a new way of measuring impactA new way of training may require a new way of measuring impact

Kirkpatrick Level 4: Results Might Include:Might Include: Labor savings (e.g., reduced duplication of effort or faster access to needed information). Production increases (faster turnover of inventory, forms processed, accounts opened, etc.). Direct cost savings (e.g., reduced cost per project, lowered overhead costs, reduction of bad debts, etc.). Quality improvements (e.g., fewer accidents, less defects, etc.). Horton, W. (2001). Evaluating e-learning. Alexandria, VA: American Society for Training & Development. Of course, this assumes you have all the documents!

My Evaluation Plan

What about specific courseware or LMS and other considerations???

Online Program or Course Budget Indirect Costs: learner disk space, coordination, phone, admin training, creating student criteria, accreditation, integration with existing technology and procedures, library resources, on site orientation & tech training, faculty training, office space, suppliesIndirect Costs: learner disk space, coordination, phone, admin training, creating student criteria, accreditation, integration with existing technology and procedures, library resources, on site orientation & tech training, faculty training, office space, supplies Direct Costs: courseware, instructor, business manager, help desk, books, seat time, bandwidth and data communications, server, server back-up, course developers, postageDirect Costs: courseware, instructor, business manager, help desk, books, seat time, bandwidth and data communications, server, server back-up, course developers, postage

Online Program or Course Budget http://webpages.marshall.edu/~morgan16/onlinecosts/ [email protected] Questions to ask How large a course?How large a course? Technology fees charged?Technology fees charged? Projected growth rate?Projected growth rate? Number of courses?Number of courses? How pay for system and use?How pay for system and use? Tuition rate?Tuition rate?

Vendor Selection True commitment to learningcan define what learning is.True commitment to learningcan define what learning is. 24 x 7 support24 x 7 support PartnershipsPartnerships Any research, reports, or funded centers?Any research, reports, or funded centers? LongevityLongevity Honest talk and less hypeHonest talk and less hype

What steps in getting it work? Institutional support/White PaperInstitutional support/White Paper Identify goals, policies, assess plans, resources (hardware, software, support, people) Faculty qualifications & compensationFaculty qualifications & compensation Audience Needs: student or corporateAudience Needs: student or corporate Finding Funding & PartneringFinding Funding & Partnering Test softwareTest software usability testing system compatibility fits tech plans

How to get buy-In? Lets form a committeeat least oneLets form a committeeat least one Needs analysis, interviews, focus groupsNeeds analysis, interviews, focus groups Mini-grants, stipends, (i.,e., MONEY!!!)Mini-grants, stipends, (i.,e., MONEY!!!) Success stories, sharing, PRSuccess stories, sharing, PR Involve all stakeholders in the DM processInvolve all stakeholders in the DM process

Marketing Listed in Major Search Engines (Free in Yahoo! or pay for top listing; premium listing in GradSchools.com)Listed in Major Search Engines (Free in Yahoo! or pay for top listing; premium listing in GradSchools.com) Bulletin BoardsUsenet Newsgroups (alt.education.distance)Bulletin BoardsUsenet Newsgroups (alt.education.distance) Specific Web Sites for DomainSpecific Web Sites for Domain Mailing Listsemail to subscriborsMailing Listsemail to subscribors Internal PRInternal PR Conference Presentations and PubsConference Presentations and Pubs Make Academic Advisors AwareMake Academic Advisors Aware

How to Fail at e-Learning (Brooke Broadbent, e-learning, January 2001, p. 36-37) Thinking training, not businessThinking training, not business Promise the moon (e.g., 50% cut in costs)Promise the moon (e.g., 50% cut in costs) Outsource everything (hire consultants)Outsource everything (hire consultants) Simply make e-learning available (instead select target groups and help change)Simply make e-learning available (instead select target groups and help change) Force e-learning on resistersForce e-learning on resisters Dont evaluateDont evaluate

Not all partnerships necessarily will be long-term ones, but the commitment to partnering must be long term to counteract the competing forces working to maintain the status quo To the extent these portfolios develop will depend on the vision of the possible provided by all those involved. --Ann Hill Duin and Linda L. Baer, 2002, NLII Final Quote

Some Final Advice Or Maybe Some Questions???