Power Up Your Learning with Assessments

How Assessments Power UpYour Learning SolutionJune 11, 2015 Public

WelcomeHow Assessments “Power-up” Your Learning Solution

Rose LoisiGlobal Solution OwnerSAP Education

© 2014 SAP SE or an SAP affiliate company. All rights reserved. 3

Welcome and Agenda

Market Drivers for Assessments and the Future of Testing“Your learning solution today, and your learning solution tomorrow withassessment & testing functionality”

Why Validate Learning and Assessments to AchieveContinuous Learning Outcomes“Power up your learning solution with assessments & certification > deliveringtraining, defensibility, authoring, reporting capabilities”

Real Customer Case studies will be shared around how trustable and traceableassessments can be achieved

Closing Remarks and Q&A

Putting Assessments in Perspective. Thoughts and Considerations for aContinuous Learning Strategy

Rose LoisiGlobal Solution OwnerSAP Education

Che OsborneVice President of StrategyQuestionmark

Rose LoisiChe Osborne


What’s Driving the Market for Assessment Management

Measure and Enhance Learning� Measure knowledge, skills and attitudes –

before, during and after learning� Correct misconceptions and increase

knowledge retention

Regulatory Compliance� Ensure knowledge of essential laws,

regulations and health/safety guidelines� Demonstrate compliance and reduce risk

Qualification and Certification� Deliver reliable, defensible tests to certify

employees� Ensure sound personnel decisions

Sales Channel Verification� Ensure competence and product knowledge

of sales professionals� Increase revenues and customer

satisfaction

Technical Channel Verification� Ensure competence and expertise of

technical and service professionals� Reduce costs and improve customer

satisfaction


46%more likely to be first tomarket

Innovation Productivity Time to Market

Quality Profitability

37%greater employeeproductivity

34%better response tocustomer needs

26%greater ability to deliver“quality products”

17%more likely to be marketshare leaders

Skills for the Future

58%more prepared to meetfuture demand

Companies Who EmbraceLearning Outperform Their Competition

Bersin & Associates, 2012


Fitting Assessments within the Learning Roadmap

Save opportunity costsSAP Virtual Live Classroom TrainingScheduled Live Online

SAP Online TrainingAnytime, anywhere (eLearning, eAcademy)

Enable your workforceon demandon any SAP topicSAP Learning HubSubscription-based access to alleLearning content

Validate LearnerKnowledge, validate, tracequality and integrate tolearningIndividual certificationMulti-level and role-based exams thrutesting , assessments and exams

Accelerate end userperformance & effectivenessSAP Knowledge AccelerationOff-the-shelf end user enablementcontent

Create your own learningmaterialSAP Workforce Performance BuilderCustomized learning content

Measure solution adoptionand workforce performancelevelSAP User Experience Mgmt by KNOAVerify end user performance and knowledgeacquisition

Manage and track enterpriselearning activitiesSAP Enterprise Learning / Learning Solution/ xLSOTarget, schedule and track training

Get live hands-on trainingSAP Classroom TrainingScheduled Live Classroom

Introduction to SAP AssessmentManagement by Questionmark

Che OsborneVice [email protected] June 2015

© 2014 SAP SE or an SAP affiliate company. All rights reserved. 8Internal

Agenda

• Brief Overview of Questionmark Corporation• Questionmark in Aviation

• Assessment Solutions enabled by Questionmark• Collaborative Authoring• Blended Delivery options including:

• Secure Browser• Mobile Devices• Embedding Assessments in Blogs, Wikis, Portals and more

• Reports and Assessment Result Analytics

• Observational Assessments

• Translation Management for Multilingual Assessments

• Integrating Questionmark with other systems

• Why Questionmark OnDemand


About Questionmark

Background� Founded in 1988� Offices in US, UK, Canada; Distribution

partners in 27 countries� Questionmark provides technologies and

services that enable organizations tomeasure knowledge, skills, and attitudessecurely for certification, regulatorycompliance and improved learningoutcomes.

Customers� 1,000 current enterprise customers

worldwide� Corporate and Government (Learning/HR

function); Higher Education; AwardingBodies and Test Publishers


What can we learn from assessments?

Did theylike it?

Did theylearn it?

Can theydo it?

Can weprove it?

Trustable results are essential to answering these questions


Questionmark: A complete range of Assessment types


SAP Assessment Management

AnalyzeAbility to evaluate results

and inform stakeholderswith timely, meaningfulreports

DeliverBlended delivery to

support a broad range ofdevices and securityneeds

SAP Assessment Management, cloud editionSAP Assessment Management, on-premise edition

Create� Collaborative, secure

authoring of surveys,quizzes, tests, and exams


How assessment authoring is managed

Questions organized by topicsMultiple authors able to collaborate21 different types of questions


Wizards to create 21 item typesv *Drag and Drop

v *Essay

v *Explanation

v Fill in Blanksv *Hotspot

v *Likert

v Adobe Flash

v Adobe Captivate

v *Matching

v Knowledge Matrix

v Survey Matrixv *Multiple Choicev *Multiple Responsev *Numericv File Uploadv *Pull-down Listv Rankingv Select a Blankv *Text Matchv *True/Falsev *Yes/No

§ Enable authors to create more interesting and probing questions

§ Reveals more information during an assessment in a shorter time

§ Not offered by simpler assessment tools in LMS’s


Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

Questions betterif use application/ comprehension

Not justknowledge

Blooms Taxonomy


Multilingual Assessments

Slide 16


Right to Left Delivery

Slide17


Right to Left Delivery


Link. Launch.Embed.

Secure BrowserWeb Browser Questionmark to Go

Integration Windows / CDTest Centers

Printing andScanning

Mobile Devices

Social Networking PlatformsEmbedded in Wikis,

Blogs, and Web pages


Responsive Design

Slide20

2. Device drivers used to format assessment for device/browser

Part

icip

ant

Dev

ices

4. On-the-fly auto-sizing of content and controls

1. Senses device/Browser 3. Sends assessment to browser


Formative QuizzesCourse Evaluations

DiagnosticTests

LowStakes Tests

MediumStakesExams

HighStakesExams

Leve

lsof

Mon

itorin

gLe

vels

ofM

onito

ring

Delivery Stakes and Devices

Stakes of Assessment


Questionmark SecureOne Setting – Multiple Platforms

Slide 22

Windows

Mac

iPad

Printing

Screen capture

Task switching

Control/system keys

Right-click / context menu options

View source

Accidentally exiting exam

Prevents….


Analysis

How are our Assessments doing? How are our Participants doing?

Did the learninginvestment impact

performance?Where are the skills andknowledge gaps in my

organization?

Are my questions andtests valid and reliable?

What did participantsthink of the learning

experience?

What did participantslearn from the training or

course?

How do I coachparticipants to improve

performance?

How did one group oftest takers compare to

another?


Item Analysis: Are the questions reliable? Defensible?

Item details and statistics

Breakdown of item responsesby choice


Coaching Report: Individualized Reporting for Participantsand other Stakeholders

Configurablequestion-by-

questionreporting

Compareparticipants

to othersor to

benchmarks


Use Cases


Business drivers - key reasons our customers useassessments

Measure and Enhance Learning� Measure knowledge, skills and attitudes –

before, during and after learning� Correct misconceptions and increase

knowledge retention

Regulatory Compliance� Ensure knowledge of essential laws,

regulations and health/safety guidelines� Demonstrate compliance and reduce risk

Qualification and Certification� Deliver reliable, defensible tests to certify

employees� Ensure sound personnel decisions

Sales Channel Verification� Ensure competence and product knowledge

of sales professionals� Increase revenues and customer

satisfaction

Technical Channel Verification� Ensure competence and expertise of

technical and service professionals� Reduce costs and improve customer

satisfaction


IndustryFinancial Services / Banking

NeedsDelivers 300,000+ assessments annually for risk management and to meetannual compliance requirements

DriversNeed to centralize and standards item creation, item banking and assessmentauthoringAbility to conduct item analysis to ensure reliability and defensibility ofassessment content

Example: Regulatory compliance – Global Accountancy


Management Report ExampleA) Number of people 1,000B) Time for training if no testing (2 hours each) 2,000 hours

C) Time to take test (20 mins / test) 333 hoursD) 25% pass test and skip trainingE) 75% do compliance training 1,500 hoursF) Hours of training avoided (B – C - E) 177 hoursG) Average cost / hour $100H) Money saved (F * G) $17,700

Proving ROI: Saving money by “testing out”


Example: Sales Channel Verification - Retail

WhySalesEnsure essential product knowledge/differentiator� Increased revenue� Increased customer satisfaction

TechnicalReduce time to solve problems� Increased customer satisfaction� Reduced costs

Verify KSAs inChannel to

Market

Directemployees

Sales

Technical

Channelpartners

Sales

Technical

Who


IndustryRetail / Consumer Products

NeedsSales Channel Verification� Test knowledge of 300,000 retail sales reps

DriversScalability – compressed time-frame of product knowledge testing requiredability to deliver 40,000 assessments per hourRapid authoring to meet short time-frame product releasesBenefitsIncreased revenuesIncreased customer satisfaction

Major Electronics Retailer


Example: Learning and Development – Oil and Gas

Industry� Oil & Gas

Needs� Uniform assessment mechanism across the

organization� ensure standards are maintained� track staff progress over time� measure training effectiveness� provide training and assessment to global

workforce� Tie test results to career advancement

Key Drivers� Ability to deploy Internet-based training

essential for global company with 126,000employees in 85 different countries� Multiple tests for each stage of employee

development� Central repository for assessment results� Measurements aligned with global

standards for employee performance


Example: Aviation industry – Multiple business drivers

Slide33



Assessment for Ground Crew

Slide34



Assessment for Flight Crew

Slide35



Assessment for Front Desk

Slide36


SAP Assessment Management by Questionmark

SAP Assessment Management byQuestionmark provides the ability to:

• Create collaborative, secureauthoring of surveys, quizzes,tests, and exams

• Deliver blended delivery to supporta broad range of devices andsecurity needs

• Analyze results and informstakeholders with timely,meaningful, and defensible reports

QuizzesTest

ExamsSurveys

QuizzesTest

ExamsSurveys

QuizzesTest

ExamsSurveys


What can we learn from assessments?

Did theylike it?

Did theylearn it?

Can theydo it?

Can weprove it?

Trustable results are essential to answering these questions


Value of Assessments in the Workplace

Life, Limb and Livelihoodon the line


How assessments mitigate risk and demonstratecompliance

Regulators encourage them

Demonstrate commitment

Find problems early

Document understanding after training

Increase knowledge / reduce forgetting

Reduce cost & time for compliance training

Reduce human error


What are some of the other solutions available from Questionmark?

Slide41


Mobile Delivery of Assessments

Deliver to thedevice in your

hand

Deliver to thedevice in your

handObservationalassessmentsObservationalassessments

Mobile testcenters

Mobile testcenters



Mobile AppsAndroid

iPhone, iPadand iPod touch


Scan a QR Code With Your Mobile Device to Start anAssessment!

Benefits• Ease of access for

increased response rates• Demographic data easily

captured• Better data accuracy• Easier analysis of survey

results by demographic


Observational Assessment

Certification Process• Observational assessments can beused as critical components of thecertification process – in some casesare required by regulatory authorities.

Medical and Dental Schoolexams:• Medical and dental schools makeextensive use of observationalassessments such as the ObjectiveStructured Clinical Examination (OSCE).

Equipment operation:• Observational assessment can be usedto record and rate operators' skills,abilities, safety practices and adherenceto required procedures

Example Applications


Embedding Assessments in Wikis, blogs, portals


Benefits of Embedding in Blogs, Portals, Wiki’s

For Learners For Organizations

• Focus the learners’ attention

• Practice memory recall

• Identify what was learned

• Provide feedback to assist thelearner to:

• Correct misconceptions• Instill confidence• Point to relevant material

• Embrace 70+20+10 withconfidence

• Easily use assessmentsthroughout the learning process

• Provide seamless userexperience

• Centralized management ofassessment content and results


Launch andTrack

• AICC• SCORM

API / DataIntegration

• QMWISe• OData• xAPI• LTI• HR-XML

Connectorsfor…

• SAP• Blackboard• Moodle• SharePoint

Embedded /iFrame / Portals

• Wikis• Blogs• Portals• SharePoint

Can I integrate Questionmark with various otherEnterprise Systems?



Custom Reportsand Dashboards

MicrosoftOffice BI Tools

Slide49

Results Warehouse


Integrating with SuccessFactors Learning – AICC/SCORM


SAP LSO and SFL Customers Using Questionmark

Sanlam

• Industry• Financial Services

• Needs• Demonstrate its

compliance withgovernmentregulations andprove employeecompetence

• Key drivers• Reporting to

documentcompliance

• Blended delivery• Integrate with SAP

Human Resourcesand EnterpriseLearning

Nebraska PublicPower District

• Industry• Energy and Utilities

• Needs• Health and safety;• Meet nuclear

regulatoryrequirements

• Key Drivers• Centralize

management ofitems and exams

• Collaborative,authoring for SMEsand instructors

• Monitored(proctored) exams

• Integrate with LSO

BNSF

• Industry• Transportation

• Needs• Testing 35,000+

employees in inHazmat, RailroadSecurity, etc.

• Certify 20,000transportationemployees

• Key drivers:• Robust authoring

features• Accessibility• Security and limit

retakes• Integration with

LSO

IntermountainHealthcare

• Industry• Healthcare

• Needs• Testing and

assessment for30,000 employeesat 21 hospitals and162 clinics forregulatorycompliance, HIPAAtraining

• Key drivers:• Accommodate 80+

authors• Online delivery to

multiple locations• Timed assessments• Integrate with

SuccessFactors


Ways to Learn More

Questionmark Blog!• http://blog.questionmark.com

Questionmark Training Course• http://www.questionmark.com/training

White Papers• http://www.questionmark.com/whitepapers

Manuals, Knowledge Base and Best Practices Guides• http://www.questionmark.com/perception/help

Web Seminars• www.questionmark.com/seminars


Assessments through the Learning Process

Security of Questionmark’s OnDemand Service

Alignment, Impact and Measurement with the A-model

Embedded Assessments: Building KnowledgeChecks, Surveys and Other Assessments intoLearning Materials

How Questionmark Aids ISO 10667 Compliance

Recommended White Papers

https://www.questionmark.com/whitepapers


Ways to Evaluate

Slide54

Try Questionmark OnDemand• 30 Day Trial – Software as a Service• https://www.questionmark.com/go/try430days

Try installable version ofQuestionmark• Download – 30 day trial• https://www.questionmark.com/download


Questionmark OnDemandScalable, SaaS-based Testing and Assessment

Solution

SSAE 16 AuditedData Center

State-of-the-artdata centers

SaaS /Rapid Deployment

24/7 Monitoring Global 24/7 supportavailable 99.9+% Uptime

Scalable to meethigh-volume, peakdelivery needs

Always have thelatest version

Turnkey service - nohardware to buy andmaintain


Keep in Touch

Slide 56 blog.questionmark.com

www.questionmark.com/newsletter

twitter.com/questionmark

linkedin.com Look for Questionmark under “Groups” and join

youtube.com/questionmarkvideo

flickr.com/questionmarkpictures

facebook.com Become a “fan” for news, updates and more


• Confidently use data to connect Learning Outcomes to Strategy

• Tools (and learning materials) available for SMEs• Help harvesting content and produce valid/reliable tests

• Question Banks• Easily organize thousands of questions

• Responsive Design• Work with the devices and browsers that your users use

• Multi-lingual user interfaces• Use worldwide

• Translation Management• Easily manage multi-lingual assessments

• Joined up data model• Helpful report for user• Assessments are proven to be defensible

Why Questionmark?

Q&A Session


Powering-up Your Assessments and LearningMaking it a Strategic Priority

SAP Insider Article

© 2014 SAP SE or an SAP affiliate company. All rights reserved.

Thank you