Volume 6, Issue 46 Closing the Gaps in LMS Reporting ... · The Bersin & Associates High-Impact Measurement Framework4 (see Figure 2) ... General business measures or HR measures

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

Introduction: The Problem with LMS Reporting

The learning management system (LMS) continues to be the core

technological foundation underpinning the efforts of learning and

development (L&D) in organizations today. Yet many LMS owners

are dissatisfied with their purchase. A leading reason for this

unhappiness is the seeming inability of these systems to meet the

reporting needs of buying organizations. Perception or reality, this

gap in reporting capability has been a remarkably consistent and

persistent problem. As shown in Figure 1, it has been one of the top

concerns on the part of LMS owners for as long as Bersin & Associates

has been following this space.1

The need for comprehensive, easy-to-use reporting and analytics2 tools in an LMS is easy to understand. Training organizations are under increasing pressure to demonstrate the business impact of learning, which brings measurement issues to the forefront. At a basic level, organizations need to report on the operational metrics that go hand-in-hand with the business of L&D, including learning opportunities offered and completed, learning assignments given and achieved, and in many cases training budget available and spent.

1 For more information, Learning Management Systems 2011: The Definitive Buyer’s Guide to the Global Market for Learning Management Solutions, Bersin & Associates / David Mallon, December 2010. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/lms.

2 Simply put, “reporting” is about viewing data; “analytics” is about using data to gain insights and, potentially, to draw conclusions – to tell a story based on data. For more information, Learning Management Systems 2011: The Definitive Buyer’s Guide to the Global Market for Learning Management Solutions, Bersin & Associates / David Mallon, December 2010.

August 31, 2011 Volume 6, Issue 46

Closing the Gaps in LMS Reporting: Rapid Learning Deployment and Customized Reporting Services

BERSIN & ASSOCIATES, LLC180 GRAND AVENUE

SUITE 320OAKLAND, CA 94612

(510) [email protected]

About the Authors

David Mallon,Principal Analyst

THIS MATERIAL IS LICENSED TO RAPID LEARNING DEPLOYMENT FOR DISTRIBUTION ONLY.

http://www.bersin.com/library

http://www.bersin.com/lms


BERSIN & ASSOCIATES


BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE



THIS MATERIAL IS LICENSED TO EXPERTUS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES


Closing the Gaps in LMS Reporting David Mallon | Page 2



In more enlightened organizations in which learning is seen as an

investment in strategic competitive advantage, not as simply an

expense, the complexity of LMS buyers’ reporting needs grows.

Companies today are directly investing in employees’ careers. They are

looking for employees who are learning and growing; they are using

certifications and other metrics to determine who best to reward and

promote. The ultimate goal is to find the most talented employees and

ensure they stay with the company.

Companies must manage their investment in employees, and need

highly specialized reports to track more than individual courses taken

and certifications received. For instance, reports are needed to:

• Measureanemployee’sdegreeofexpertise;

• Recordthecompletiondatesforallregulatorycompliancetraining;

• Comparehowwellemployeesaremanagingtheircareers;

• Findemployeestobuildcollocatedteams;and,

• Aidintheglobalsearchfortoptalent.

17%

18%

11%

23%

20%

25%

31%

27%

38%

45%

8%

14%

14%

19%

20%

28%

32%

32%

39%

39%

0% 10% 20% 30% 40% 50%

System Stability and Performance under Workload

Making Content Work Correctly

IT Implementation and Security Issues

Provider Service and Support

Administration

Adoption

Ease of Use

Integration

Customization

Reporting

2011 2009

Figure 1: Top 3 Challenges with Current LMS

Source: Bersin & Associates, 2010.

Training

organizations are

under increasing

pressure to

demonstrate the

business impact

of learning,

which brings

measurement

issues to the

forefront.

KEY POINT


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





As stated, the drivers behind these reporting needs are readily apparent

yet, in case after case, LMS buyers feel as though their systems have “let

them down” when it comes time to generating actionable3 business

intelligence. The question is, why? Perhaps more importantly, what can

the LMS owner do about it?

In this report, we will examine the reasons why reporting and analytics

continue to be ongoing problems for the LMS, review strategies

available to LMS owners for mitigating the gaps, and highlight one

provider, Rapid Learning Deployment (rapidLD), that offers one

potential solution – custom reporting services.

The Types of LMS Reporting

Before we go any further, we should establish some basic definitions

and provide an overview of where the majority of the LMS marketplace

is today with respect to reporting functionality. Understanding what

reporting and analytics functionality is supported by an LMS is best

done by first looking at the question of what L&D chooses to measure

and why.

Any organization that does learning or training eventually realizes the

need to ask the same basic questions.

• IswhatIamdoinginlearningeffective?

• Isitefficient?

• Isitwhatthebusinessneeds,wantsandexpects?

• Whatimpactdoeslearninghaveonmycompany?

3 “Actionable information” provides data that can be used to make specific business decisions. Actionable information is specific, consistent and credible.

In case after case,

LMS buyers feel

as though their

systems have

“let them down”

when it comes

time to generating

actionable business

intelligence.

KEY POINT


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





The Bersin & Associates High-Impact Measurement Framework4 (see

Figure 2) helps organizations decide how to implement a complete

measurement program. The Framework also outlines on what to focus

efforts to obtain the most actionable, credible learning measurements

in answering these four questions. Each of the nine measurement areas

shown in Figure 2 require data – much of which will likely be captured

from the LMS.

4 For more information, The Training Measurement Book: Best Practices, Proven Methodologies, and Practical Approaches, Josh Bersin / Pfeiffer, 2008. Also available at www.bersin.com.

Figure 4: Bersin & Associates High-Impact Learning Measurement Framework®


Adoption Utility AlignmentEfficiency Attainment

Satisfaction Learning

Operational Indicators or KPI’s (Key Performance Indicators)

Individual Performance Organizational Performance

Evaluative Measures (Direct Impact Measures)

Did you reach the desired audience?Did they complete orcomply as desired?Who did not comply and why?

How well do the programs solve the workforce’s particular problems? How well did it align to the specific jobrelated problems and issues?Would learners recommendthis program to their peers?

How well were program business priorities defined?How well did business units buy in on thevalue of this program relative to other investments?

How well did youmeet specificallydefined client (business user or customer) objectives?These may berevenue, time tomarket, compliance,time to complete, etc.

Indicators asked of learners and managers to gauge performance improvements.Specific operational measures identified in theperformance consulting process.

General business measures or HR measures that are already captured in the organization (i.e., engagement, retention). Special surveys can be used to determine indicators using the “wisdom of crowds.”

it?

How efficient and cost -effective was

How did it compare to other similar programs or competitive programs?How well did it use the learner’stime?

Figure 2: Bersin & Associates High-Impact Learning Measurement Framework®


http://www.bersin.com


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





Almost all LMS providers today offer basic reporting features, which

include tracking learning history (learning activity), tracking and

reporting fulfillment of preset criteria (e.g., certifications, tracks, et al),

and aggregated learning evaluation data (e.g., Kirkpatrick Level 1 –

satisfaction or Level 2 – transfer of learning).

Less widespread, advanced customized reporting and analytics features

today include customized report building and information visualization

applications, dashboards, targeted exception reporting (e.g., who has

not completed their compliance training), automatic notifications based

on customizable business logic, and the ability to compare learning

measurement data directly against business data.

Categories of LMS Reporting Functionality

In most cases, there are two types of reporting tools in relation to the

LMS today.5

• Inside-the-LMS-BoxReporting

o Standard prebuilt reports generated from the LMS

o Custom ad-hoc reports generated from a report-building tool

within the LMS

• Outside-the-LMS-BoxReporting

o Customized ad-hoc reports generated from a custom report-

building tool or platform separate from the LMS –

analytic technology.

Standard and Custom LMS Reports: Inside the Box

All LMS platforms come with a premade set of “canned” reports

standard to the system. These reports are designed to address the

most common reporting needs as communicated to the provider by its

client base over time. The LMS administrator can adjust the results of

most standard reports by making simple changes to timeframe and by

applying a limited set of data filters. Many of these reports can be quite

5 For more information, High-Impact Learning Measurement: Best Practices, Models, and Business-Driven Solutions for the Measurement and Evaluation of Corporate Training, Bersin & Associates / Josh Bersin, November 2006. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/measurement.

The Bersin

& Associates

High-Impact

Measurement

Framework is

designed to help

organizations

decide how

to implement

a complete

measurement

program.

KEY POINT

http://www.bersin.com/library

http://www.bersin.com/measurement


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





useful, but their utility is often relatively narrow in focus. Most LMS

owners quickly find the standard reports to be limiting.

As instructor-led classroom training (ILT) began to include e-learning

and other forms of learning, the LMS evolved, as well. These LMSs

graduated to more complex systems. They also started to incorporate

a “custom reporting tool” that lets the administrator or user generate

specialized reports by selecting data points to be included from a preset

list and by selecting more elaborate filtering criteria (e.g., “show me all

learners in the sales department who completed Introductory Sales Skills

in the last 90 days”). These custom reports can typically be saved for

frequent use, and set to output into a variety of formats, including PDFs

and Excel spreadsheets.

Most systems also support automatic generation of custom reports on

a recurring scheduling, and automated delivery of these reports to

administrators and stakeholders. Again, most LMS owners do find value

in the custom reports built into their systems. However, as with the

standard reports, they eventually find that these custom reports fail to

meet all needs due to the following issues.

• Noteverydesireddatafieldisanavailableselectionforareportor

available on the specific report in question.

• Noteverydesiredfilter,cross-tabordimensioncomparison

is available.

• Theoutputsarerarelyverycustomizable,requiringsubstantial

outside-the-system work to change into a format pleasing and

acceptable to the local needs and expectations of stakeholders.

• Sincethesereportsareusuallystill“built-in”tothecoreLMS

platform, there may be performance issues when running them

during times of peak LMS usage, when running them very

frequently or when there are large numbers of records to report.

While all LMS

platforms come

with a premade

set of “canned”

reports, most LMS

owners quickly

find the standard

reports to be

limiting.

KEY POINT


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





The most advanced inside-the-LMS custom reporting tools replicate

most, if not all, of a standalone report-building application, such as

those offered by Business Objects (SAP), Cognos or Microsoft. In some

cases, the advanced customer reporting tool in question is, in fact, one

of these standalone tools embedded directly inside the LMS.

Some LMS providers try to address some of the scalability and

performance problems with custom reporting tools by moving them

onto a parallel platform, so that core LMS performance and reporting

performance do not affect each other. These tools typically also support

more advanced analytics, including multidimensional looks at the data

and dynamic data access. Successful use of these tools is often still a

challenge for the LMS buyer due to:

Figure 3: Sample of Standard / Custom Inside-the-Box LMS Reports

Standard Reports

• Standardreportsforcommonlearningdatarequirements(e.g.,completions,curricula,activitytypes,status,hours)• Reportingonalllearningactivities(e.g.,ILT,e-learning,external,certifications),andtalentdatabasedon

organizationalhierarchy,talentpools,roles,careerpaths,etal• Reportingofalltestandsurveyquestionsandanswers• Reportingbycurriculum,certification,courseandobjective• Reportsoncoursewaitlistdemand• Reportingthatlinkslearningactivitytoemployeeandorganizationalperformance• Reportsofcurriculacompletionpercentages• Reportingofcertifications,licensesandcontinuingeducationunit(CEU)credits• Reportingtuitionreimbursementdatatothecompensationmodule

Custom Reports

• Reportingoflearningcostsbyresourceutilization(e.g.,instructorcosts,room,materials,andcourse)• Reportingbasedonallcompetencystructures(e.g.,jobfamilies,corevalues,leadership,careerpath,jobfunctions)

acrossorganizationalhierarchies,talentpoolsandothergroupings• Reportscustomizedforend-usersandmanagersproducedinPDF-basedformats• Reportsshowinganemployee’scomprehensivetalentprofile(includingrésumé,performanceandpotential

information)foremployeesandmanagers• Reportingofad-hocorganizationaltalentdata(e.g.,bytalentpools,organization,departments,roles,jobs,etal)• Reportsofemployeeswhoareactivelymanagingtheirlearningandcareers


A “reporting tool”

typically lets the

administrator or

users customize the

report by selecting

different filtering

criteria.

KEY POINT


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





1. Frequently insufficient documentation regarding their data models;

2. The underlying database structure for many providers can often be

remarkably unorganized (these platforms evolve over time – rarely

staying neat and orderly); and,

3. Providers may not mirror all LMS data in the parallel system.

Customized LMS Reports – Outside-the-Box Analytics

To understand the difference between a custom and customized report,

you need to first have a basic understanding of the difference between

reporting and analytics. Simply put, “reporting” is about viewing data.

“Analytics” is about using data to gain insights and, potentially, to draw

conclusions – to tell a story based on data. “Analytics” is a reporting

technology that gives users much more flexible and dynamic access to

information – it enables users to generate the customized reports

they need.

The LMS reporting tool typically lets the administrator or user customize

the report by selecting different filtering criteria. In the earlier

example, “show me all learners in the sales department who completed

Introductory Sales Skills in the last 90 days,” the report is generated by

the reporting tool when the custom filters are set for the data in the

LMS. Such a custom report produces a list of results and is usually easy

to generate in almost any LMS – yet it is still not a customized report.

An analytics tool goes much further. It takes the information in the

LMS, and organizes it into dimensions6 and measures7. Dimensions are

hierarchical, so they not only form filters – they allow you to drill up

and down, and see data dynamically. For example, in the query above

(“show me all the learners in the sales department who completed

Introductory Sales Skills in the last 90 days”), you may want to drill

down and see the number of completions for each manager within the

sales department.

Another example, you may want to “drill up” and see the number of

learners who completed this course in the U.S., as compared with their

6 Descriptive information about the audience or program is often called “dimensional” information in an analytics system.

7 “Measurement” provides the information and data, which can then be used to monitor, analyze and evaluate. A “measure” is the extent, dimension or actual data element.

An “analytics

tool” takes the

information in the

LMS, and organizes

it into dimensions

and measures

to produce a

customizable

report.

KEY POINT


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





peers in the U.K., France or Japan. A typical example of such an analysis

is for a retail, sales or manufacturing organization – for which there are

multiple locations, each with many managers and numerous employee

learners. You may wish to compare completion percentages across

groups, plants, divisions, geographies and business units. Such data will

be highly actionable; it will tell you and the line managers where within

the organization learning is not being completed. In other words, you

can customize the data based on your needs, instead of getting the

more restrictive custom reports that the LMS provides.

These examples are simple – they use the most basic measures of

adoption.8 More complex examples may include analytics and the

customized reporting of scores, number of student hours, dollars of

training consumed, number of days spent in classroom versus e-learning,

training consumption by media type (e.g., classroom, book, e-learning

and external course), consumption of external fee-based education

versus internal and much more. A powerful report is an analysis of the

utilization of a provider’s catalogue content – you may find that your

costly license of an IT or leadership provider’s content is only being

consumed by a small number of learners. Such information is highly

actionable; you could negotiate a lower price or you could pass on this

charge to the department consuming the training.

From an architectural point of view, analytics systems typically

move data into a new data structure that is often called, “the data

warehouse.” In some cases, the LMS provider will take a platform

approach and embed analytics tools from companies, like Business

Objects, Cognos or Microsoft. Some LMS users have decided (especially

if they already own an LMS) to build their own separate analytics system

that is accessible from the desktop, running on Microsoft SQL Server, or

other data warehouse technologies from Business Objects or Cognos.

The addition of the analytics tool that draws information from across

a variety of databases has the advantage of providing the users with

a customized report which answers whatever inquiry they might have

based on the data in the data warehouse. Customized LMS reports

provide the actionable information that the two other types of reports

fail to generate. The dilemma is that you get whatever customized

report you want, but at a significant cost. Adding an analytics tool

8 For more information, The Training Measurement Book: Best Practices, Proven Methodologies, and Practical Approaches, Josh Bersin / Pfeiffer, 2008.

From an

architectural point

of view, analytics

systems typically

move data into a

new data structure

that is often

called, “the data

warehouse.”

KEY POINT


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





requires significant expertise, usually dedicated resources – an available

expert who can work with a customizable SQL reporting tool (like

Crystal Reports) in his / her sleep and, in case of the data warehouse,

substantial technology investment and support.

Case in Point: Nemours Hospital – Learning to Speak Crystal

Nemours Hospital and Clinics relies on its LMS as much for

delivering training as keeping track of certifications, CEUs and

compliance with all of the rules and regulations that a hospital

must follow.

When the hospital administrators mandated new initiatives

(focusing on standards of behavior) and wanted regular reports,

the learning services department, already disappointed with the

reports it could get, knew it needed something special. Having

replaced an older LMS with a newer model within the last two

years, the supervisor of learning services at Nemours knew it

was not possible for the system to generate what the

administrators wanted.

To begin with, learning services was disappointed with the

amount of standard, out-of-the box reporting from the new

LMS. Even though all the data was in the LMS database, the

reports could only provide bits and pieces, rather than all of

what they needed in one report. There were too many fields,

with no way to turn them on or off, whether or not they were

being used. Even after a report was generated, it printed too

many pages in a font so small that it was almost impossible to

read without putting it under a microscope.

Learning services LMS reports needed serious surgery and

the learning services team was not prepared to do the work.

The team could not change the Crystal Reports produced by

the LMS. Nemours IT department offered some support with

individual reports, but was not able to integrate the LMS with

other systems. In addition, learning services was not a priority

on IT’s long list of work. Learning services needed a report that

pulled together data from a variety of locations and knew


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





that, if it was going to meet the demand for the customized

standards of behavior reports, the team needed what was

referred to as “Crystal gurus.”

Since the team did not have the staff in-house and could not

always afford to rely on a third party, it looked for someone

who could develop the reports and then publish them back into

the LMS. That way, the reports could be accessible to everyone

who had permission – both as a report on standards of behavior

and programs related to raising those standards, and also as an

ongoing assessment.

While searching for the “gurus,” the LMS vendor recommended

rapidLD to learning services. rapidLD had often partnered

with the LMS vendor to help integrate systems and generate

customized reports. Creating a customizable report and then

publishing it back to the LMS was something most companies

did not need. This required not only a deep level of knowledge

of the LMS (which rapidLD had gained from previously working

with the LMS vendor), but also a level of understanding about

Crystal Reports and the SQL database, as well as complicated

data issues lacking with other vendor companies considered by

the learning services team.

Less than two months later, the new customizable standards of

behavior reporting were up and running – and have been used

throughout the hospital and clinics. The big lesson for Nemours

learning services was that, as an LMS owner, the team needed

to spend more time on reporting and even devote a person

on staff who is technical enough to learn more about Crystal

Reports. That way, it can test new use cases9 as they arise against

the reporting function. Even though it can look forward to a

long and productive relationship with rapidLD, learning services

knows that it will still need to have someone from rapidLD who

can speak “Crystal” with the team. e

9 A “use case” provides a description of a sequence of interactions between actors, and the system necessary to complete a specific goal or function. Use cases are often co-authored by systems analysts and end-users, and are presented as a sequence of simple steps.

Case in Point: Nemours Hospital – Learning to Speak Crystal (cont’d)


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





What Is So Difficult about Reporting?

The Buyer

LMS buyers just want reports – accurate reports that are easy to get

and easy to use. In most cases, they are responding to a request from

a stakeholder, who does not know from where or how the report will

come; neither do they care. They just want actionable data.

Here is the problem. Storing data is hard – much harder than it looks to

the layperson. There is a fundamental disconnect between expectations

and what it takes to meet those expectations. The business stakeholder

wants the data provided in a certain way. That “way” translates into

certain criteria, which have implications for not just how the data

will be output, but also for how it is labeled, organized, archived and

accessed. While LMS providers can try to anticipate those criteria, the

best they can hope to do is account for the proverbial 80 percent of

cases. Even then, slight variations in expectations from stakeholders

can translate into major differences in what the LMS has to do out of

the box. So, in many respects, built-in reports are a no-win scenario for

providers, which would do well to set better expectations with their

prospects in this regard.

Instead, the LMS providers attempt to build flexibility into the system

in the form of the custom reporting tools. But flexibility comes with

a price – and the need for time, resources and business intelligence

expertise commensurate with the underlying complexity of the original

expectations. The best LMS platforms today have reporting tools that

can meet most needs in the manner desired by the business, but you

have to be fully prepared and know what you are doing to

take advantage.

The Provider

Are we saying that this is fundamentally just a buyer’s problem?

Absolutely not. Most providers have not invested nearly enough in

research and development in order to improve the reporting capabilities

of their systems. Despite being told it is a problem, this is not one that

has threatened sales. (It is far too easy to gloss over in demonstrations

and in RFP responses.) Most providers are willing to sidestep the issue

by offering access to a third-party tool, either standalone or embedded

in the system, usually touting access to a best-of-breed tool (like

Slight variations

in expectations

from stakeholders

can translate into

major differences

in what the LMS

has to do out of

the box.

KEY POINT


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





Business Objects) as a special bonus. This strategy is not really a solution,

however, since these power tools are essentially useless in the hands of

the untrained.

Even the providers which have invested substantial amounts of money

into the reporting side of their platforms have failed to account for

the user experience (e.g., how user-friendly is the tool?) or the user’s

expertise (e.g., how much does the user need to know to use the tool?).

Most providers forget that their primary user base is comprised of

learning professionals, not IT professionals.

The Data

We would be remiss in not also pointing out that the data being

tracked, recorded, accessed and reported is very complex – and

becoming more so by the day. Even small companies can be remarkably

complex in terms of the number and variations in data fields tracked

per employee. Of course, inevitably even small organizations are not

satisfied just to keep the learning data in the LMS. Integrations which

bring in data from other systems and then send it back out to still more

are the norm in most LMS implementations. Add the need to account

for complex organization structures, global issues, legal requirements,

learning standards (and the list goes on), and you begin to sense why

you are paying so much for a database to track and deliver learning.

Oh, and did we forget to mention integrated talent management?

Solving the Reporting Problem

Now that we have a sense of the issues, what can the LMS buyer do

to resolve his / her company’s reporting needs? The following sections

offer a few ideas to get started.

Educate Yourself

Spending time to raise the general level of knowledge about reporting

for yourself and the rest of the L&D team is a clear best practice. Avoid

creating problems for yourself in the future by acting as though you

fully understand the details when you might not. Everyone in the L&D

function who deals with client stakeholders should learn to do a better

job of setting requirements and managing expectations. They should

learn about the six to eight objects that most stakeholders will need

to report against (e.g., people, content, et al) and the relationships

between these objects in the system.

Everyone in the

L&D function

who deals with

client stakeholders

should learn

to do a better

job of setting

requirements

and managing

expectations.

BEST PRACTICE


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





This preparation will help when a request comes from the vice president

of compliance. With that knowledge, stakeholders can help discover

the real business issue that need to be addressed and have a better idea

about what the system can really do. The resulting conversation will

help in the requirements definition process, as well as in

managing expectations.

Be proactive. Share the responsibility for defining key reports with the

business and foster collaboration with your stakeholders with regard to

required intelligence.

Case in Point: Brinker International – Do Your Homework

Many readers of this report who have eaten at Chili’s or

Maggiano’s never knew that they were eating at one of

Brinker International’s restaurants. Brinker has more than 1,500

restaurants, and is responsible for training more than 100,000

team members in 31 countries and two territories. The company

serves up this full menu of training at the Brinker Learner Center.

The Learning Center is also responsible for keeping track of the

thousands of employees who need to stay current with their

certifications. In addition, the Center ensures that restaurants

are in full compliance with the many rules and regulations

governing the restaurant industry.

Tracking all of this activity is the LMS. The Learning Center had

problems with the system from the start, especially in the area

of reporting. For example, the out-of-the-box custom reports

came with the vendor’s logo. When the company decided that it

needed its own logo, it faced an unexpected level of complexity

and had to develop an entirely new report.

When competency, performance and talent became areas of

focus at Brinker, executives and managers began to ask for

very different levels of information from the LMS. They wanted

to know how many more hours it would take for certain

employees to complete a course or the average time for a new

program to be implemented. They were really asking about the

impact of learning on the company.


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





The answers required customized reporting from the LMS, yet

the features and functions for these reports were missing from

the LMS menu. The vendor was not willing or able to spend

time in helping to generate customized reports, and suggested

adding a new and expensive module. The IT department was

busy with other jobs; only one technical person (a business

systems analyst) in the Center knew a little about LMS reporting.

The center was pressed for time as the demand for these reports

increased. Management was looking for reports that, for

example, showed exceptions to compliance and certifications,

listed the percentage of employees trained on various new

products, and provided hard data on the average time to

performance for certain programs. Reports needed more data

than the LMS or LMS inside-the-box reports could generate.

When the senior director of learning looked into customized

reporting, he was surprised at how much more difficult it

was to generate those types of reports now than in the past.

Fortunately, he also knew about rapidLD’s reputation as the

provider which understood where the LMS data lived and how it

was connected to other databases.

Three weeks after calling rapidLD, the system for generating

the custom reports was in place. Today, the businesses tell the

Learning Center what report they need; the Center develops a

format, and the rapidLD Custom Report scales it up to the entire

enterprise and runs it on demand. The impact of learning on

the company has become an important metric for a number of

decisions, from selecting the most effective training to looking

for tomorrow’s talent. e

Staff Accordingly

Be prepared to invest in staff resources that are more technical in

nature, especially in the area of reporting and database management.

Investing in a more technically knowledgeable staff will save time and

reduce the cycles involved with a customized report request.

Case in Point: Brinker International – Do Your Homework (cont’d)


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





Understand that reporting skills are highly sought for any application.

Know that these resources are scarce and in high demand all of the

time. Build technical skills among the entire L&D team about basic

reporting concepts, as well as the other functionality of the LMS. One

person may have the responsibility for report generation, but a general

level of technical competence is needed for the advanced LMS.

Choose the LMS Wisely (if you are still looking)10

If you are still shopping for an LMS or thinking of switching, reporting

should be high on your list of shopping criteria. Find out what support

the providers offer with regard to customized reports and how you can

actively engage their communities to address future issues. Carefully

review the reporting tools that come with the LMS and test your

customized report use cases against that reporting tool to see how well

it meets all of your customized reporting needs.

Strategies for Getting Help

Inevitably, you will need to secure additional assistance. You have

several options, including:

• PartneringwithyourITdepartment;

• Purchasingathird-partyreportingplatform;and,

• Outsourcingtocustomizedreportingservices.

We will discuss the pros and cons of each of these options in the

following sections.

Partnering with Your IT Department

The LMS is not the only system with reporting needs. Your IT

department may already be well versed in supporting reporting issues

with many other enterprise systems. In fact, in many organizations

(especially larger ones), there are dedicated departments and / or

resources for business intelligence. You might be able to secure support

from such a group (or just from general IT) in the form of at least

10 For more information on selecting an LMS, please read, Learning Management Systems 2011: The Definitive Buyer’s Guide to the Global Market for Learning Management Solutions, Bersin & Associates / David Mallon, December 2010.

Build technical

skills among

the entire L&D

team about basic

reporting concepts,

as well as the

other functionality

of the LMS.

A N A LY S I S


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





internal advice and consulting, or perhaps even having this other group

do the work for you.

Benefits:

• IfyourITfunctionhastheresources,expertiseandinfrastructure,

then going this route is often the best way to get the magical

combination of business and learning data on one report, as so

many organizations are seeking. Best of all, it will likely come in

formats that the business is already familiar with and ready to use.

Potential Issues:

• Thatisalotof“ifs.”YourITdepartmentmaynothavetheexpertise

or the infrastructure. The IT department often needs to go outside

to find the person who it needs to get the work done.

• Justbecauseitisinternaldoesnotmeanitisfree;youneedtocome

with a budget. Treat IT like a preferred vendor.

• TherelationshipbetweenL&DandITistoooftennotwrittenin

stone – and IT is frequently not available to help you. Even when

there are ways to secure the department’s help, remember that you

are not its only priority. You will need to work to ensure

proper attention.

• BringingdataoutoftheLMSoftenrequiresaneedtoreplicatethe

security model from the LMS externally in order to limit data views

to only those people who should have access. It also means that you

will need the IT department to develop such logic for all reports

generated. The result is that your reporting structures will be very

unique, making it difficult for you to leverage the knowledge and

know-how from the LMS vendor or even the vendor’s

other customers.


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





Case in Point: YUM! Brands – Even Vendors Need Skilled Partners

Yum! Brands, Inc. is the world’s largest restaurant company

with more than 38,000 restaurants in more than 110 countries.

The company employs (and teaches via YUM! University) more

than one million associates worldwide. Yum! is ranked high on

the FORTUNE 500 list and generated more than $11 billion in

revenue in 2010. The company’s primary brands (KFC, Pizza Hut

and Taco Bell) are the global leaders in their food categories.

Outside the U.S., Yum! opened almost four new restaurants

each day of the year in 2010.

That level of growth demands a high degree of information

for managers and executives. In the past, YUM! University had

created all of the customized reports in-house, collaborating

with the brands and building what they agreed upon on an

ad-hoc basis. When the managers of the YUM! Brands around

the world realized that the LMS data could be associated with

other data and they could start to build their own customized

reports, YUM! University needed to integrate the other existing

databases with the LMS.

YUM! University has been working with LMSs since 2007 and

was aware of the limitations of the system. The company only

had one person on staff who was technical enough to meet

the requests for customized reports. It knew that it would need

more expert help.

The company also knew that the support it required would not

come from its LMS vendor. The use of data from outside of the

LMS was not the focus of YUM!’s LMS vendor. YUM! University

needed support from a company with a deep knowledge of

how its LMS worked. The company was already partnering with

rapidLD on technical issues related to the worldwide deployment

of the LMS. It was an easy choice to leverage rapidLD’s

knowledge to help meet the demand for customized reports.

The rapidLD solution was to put the LMS in the same data

warehouse used by the businesses around the world. Since the

brands were already familiar with the current system, this


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





solution enabled anyone to readily compare the LMS data with

other information and run only those customized reports that

were created by the IT organization.

Yum! Brands is taking advantage of the new LMS capabilities for

customized reports to help pinpoint training needs in various

groups within the company. Yum! Brands is also using its new

worldwide learning technology to give it an edge in the highly

competitive restaurant industry. e

Purchasing a Third-Party Reporting Platform

Another option is to purchase a third-party reporting platform that

generates customized reports. This approach is often used when

vendors provide third-party products or integrative solutions for

authoring tools. In this case, you would be looking for a third-party

solution for generating customized reports, which could be a parallel

system that supplements the standard and custom template

LMS reporting.

Examples of these third-party reporting platforms include Taleo and its

Taleo Learn platform; zeroedin technologies, and its impact dashboard

and impact surveyor; and, KnowledgeAdvisors with its Metrics That

Matter® platform. These companies have learning specialists who are

savvy about learning organizations and the reports they need. Their

platforms can help generate customized reports, and can also replace

an IT organization that is too busy or does not have the staff you need.

Benefits:

• Athird-partyplatformcancompensateforLMSshortcomingswhen

it comes to generating the customized reports you need.

• Theplatformcanbeeasytoimplementandessentiallybea

“turnkey” process when the third-party platform is from a learning

specialist. The third-party provider already understands who you are

and what you need.

Case in Point: YUM! Brands – Even Vendors Need Skilled Partners (cont’d)


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





• Addingathird-partyplatformcanresultinabetteruserexperience

with the LMS in the sense that it rounds out the system and provides

an easy way to get the many customized reports you need.

Potential Issues:

• Areyouwillingtopayforanothertechnologyplatforminorder

to do something that your current LMS platform should be doing?

Even though more than 39 percent of LMS users know there are

problems with reporting11, they still believe that their LMSs can

provide the customized reports they need.

• AnumberofLMSandtoolsvendorsthatmarketthird-party

reporting platforms or modules are not really separate platforms

at all. These are essentially plugins or more friendly faces for the

existing LMS reporting infrastructure. They provide additional filters

and more detailed templates. The source and focus are still the LMS

database. Essentially, you are getting better custom reports.

Outsourcing to Customized Reporting Experts

A third option for generating customized reports is to use an

outsourced reporting services provider. In this case, the provider does

more than create a one-off custom report – it provides ongoing custom

reporting support as a service.

Benefits:

• Gainaccesstothespecificbusinessintelligenceexpertiseyouwill

need now and later.

• Therelationshipwiththeprovider(ifchosenwell)islongterm,so

the provider gets to know you and can help maintain continuity

from one reporting cycle to the next. It can also better react to

changing reporting needs.

11 For more information, Learning Management Systems 2011: The Definitive Buyer’s Guide to the Global Market for Learning Management Solutions, Bersin & Associates / David Mallon, December 2010.


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





Potential Issues:

• Someoneelseisworkingwithyourdata.Someorganizations

become uncomfortable with this concept.

• Integratingwithotherbusinessdata(ifdesired)canbesomewhat

more challenging because another party is involved.

One exemplary provider of outsourced customized reporting services for

learning is rapidLD; we profile this provider in the next section.

The rapidLD Solution

Rapid Learning Deployment, LLC is a professional services company

developed to help organizations improve their organizational and

individual performance with complete, integrated and flexible learning

and talent solutions. The company’s services are designed to meet an

organization’s immediate and future needs and goals. Over a period

of 12 years, its business and technical consultants have evolved a

method that has proven successful for deploying learning and talent

management systems and generating customized reports.

The company understands that an LMS’s inability to measure outcomes

of learning activity is one of the leading reasons for the dissatisfaction

with the system. The challenges associated with configuring a business

intelligence solution and demonstrating a return on investment are

linked with the need to provide just-in-time data and customized

reports. rapidLD knows that the customized reports need to cover the

measurement of the organization’s learning and talent initiatives, as

well as the operational metrics and benchmarking of

strategic initiatives.

rapidLD’s custom reporting service does the following:

• Enhancesperformanceofintegratedcustomizedlearningand

talent reporting;

• Generateseffectiveandefficientcustomizedreportingoutcomes

and standards;

• Understandstheemergingmetricsdefinitionsandanalysisneedsfor

customized reports;

The challenges

associated with

configuring

a business

intelligence

solution and

demonstrating

a return on

investment are

linked with the

need to provide

just-in-time data

and customized

reports.

KEY POINT


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





• Sharestheevolutionofbestpracticesandprocesses;and,

• Providesaneasierwaytogenerateandviewcustomizedreports.

rapidLD understands that vendors have dropped the ball when it comes

to helping LMS owners generate customized reports. This provider has

partnered with many LMS vendors to provide this help at a level that

the vendor (and often the internal IT department) cannot offer. rapidLD

has the experts who most LMS owners are seeking when it comes to

generating customized reports. The company is not in business to sell

a customized reporting platform or reporting tool; it provides a service

that helps LMS owners get the customized reports they need efficiently

and effectively.

It is worth noting that rapidLD has become one of the leading

companies that LMS owners go to when they need a customized report

because rapidLD knows the LMS architecture and ecosystem; it has an

entire deployment framework for an LMS. The company knows the

learning business and has followed the changes that have happened in

that industry.

One of the results of all this knowledge and know-how is the

development of a customized reporting environment that it calls

rapidTM™ (Rapid Talent MeasurementTM). Part of this service includes

the dynamic ability to change the view of the data to suit your needs.

rapidLD provides a

service that helps

LMS owners get

the customized

reports they need

efficiently and

effectively.

KEY POINT


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





Changing the way you view the data has been enhanced by using

graphs and other visual data representations. rapidLD will frequently

write these kinds of reports that have unique characteristics. A

customized report, for example, can show the relationship between

an individual learner and several management levels above (say, for

example, area and regional directors). The standard LMS certification

report normally only shows the relationship between the learner and

his / her immediate manager. The customized reports produced by

rapidTM can also be viewed graphically and the graph can be inserted

into the printed customized report (see Figure 5).

Figure 4: Customized People View from an Existing LMS

Source: rapidLD, 2011.


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





Innovation at rapidLD

rapidLD is also innovating some of the newest and most forward-

looking best practices for LMS customers that are seeking customized

reports. The rapidTM™ technology is built along the following lines.

1. Unified Data Model – This approach has the learning-related data

sitting in the same place, regardless of the origin of the data.

(This could connect the LMS database with the HR and talent

management databases, or it could combine data from several

different LMS databases.) The result is that, once the databases

are “unified,” customized reports can be more quickly and easily

Figure 5: Example of Custom Report Representing Data Visually

Source: Saba, 2011.


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





generated. The Unified Data Model consolidates the data in one

place, so that you can more easily report against it.

2. Data Storage – rapidLD uses a native table approach that indexes

the data subsets to speed up the generation of customized reports.

The native table is an integration of a large number of databases

that supports a federated search12 across database fields – allowing

access to all, and treating them as a single unified database for

queries and reports.

3. Visualization of Data – Being able to see the data in real-time

is perhaps one of the most unique and valuable approaches

to generating customized reports. rapidLD has developed a

DashRibbon13 that allows the user to operationalize report results,

which are easily viewed and can be tagged with thresholds. When

the thresholds are exceeded / not met, the DashRibbon then sends

alerts or alarms to designated individuals. The DashRibbon pre-runs

customized reports that you might need, and allows you to stop the

ribbon and select a report that you find useful.

12 A “federated search” is a universal search inside and outside the LMS for all-inclusive information gathering.

13 A “DashRibbon” is an application that is a moving line of information, similar to a real-time moving stock quote, that shows pre-run customized reports which enables the user to view changes as they occur, sends alerts when thresholds set by the user are met / unmet, and can be stopped for a more detailed examination or to run a report.


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





4. Community of Practice14 – While this is perhaps the least technical

innovation, it is nonetheless one of the more important ones. It

takes into account that there are many people with the same LMS

who are looking for a customized report solution and there may

also be many people in the community who have already found an

answer. Connecting the two is a service that rapidLD has created

for its customers which it views as a community of practice. This

community can be quickly accessed by selecting the Exchange tab on

the rapidTMTM report screen.

The rapidTMTM Community of Practice (also called, rapidTMTM

Exchange) is designed to enable users of the reporting / analytics

engine to collaborate with each other. The community also includes

a Reports Store through which you can exchange custom reports

and also receive a credit if you contribute, or buy a credit if you do

not / cannot contribute.

14 A “community of practice” (or “CoP”) is often defined as a group of people who share an interest or concern about a common topic, and who deepen their knowledge in this area through ongoing interaction and relationship-building within their group. While communities often come into being spontaneously, they nonetheless require nurturing if they are to become valuable to the members and remain viable over the course of their evolution.

Source: rapidLD, 2011.

Figure 6: The rapidTM DashRibbon


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





Perhaps the most notable innovation has nothing to do with technology

but, instead, with the rapidLD approach to its products and services. The

company is giving away the base Reporting and Community package at

no cost. Knowing the LMS business as well as it does and having worked

with customers for more than 10 years have led rapidLD to the belief

that many LMS owners feel they already paid for reporting and are not

going to pay for it again.

Conclusion: You Can Get the Reporting You Need

Producing a standard or custom report from an LMS is not easy – and

generating a customized report that spans databases and requires a

high level of report-generating expertise is even harder. To get the most

from your LMS reporting, please consider the following.

• Reportingisimportant.“Youcannotmanagewhatyoucannot

measure” is a maxim that easily drives the need to more and better

LMS reporting.

• Thebasicout-of-the-boxLMSdoesnotdoeverythingitissupposed

to do. Vendors and buyers share the responsibility for this

shortcoming – vendors for not paying enough attention to the need

for customized reports and buyers for not measuring their LMS

purchase against all the use cases in which a customized report may

be required.

• L&Dorganizations,eveninlargecompanies,donotlikelyhaveall

the technical expertise or IT support they need to generate the

customized reports they require.

• Notallreportingsolutionsarethesame.Yougetdifferentresults

from different platforms and third-party providers.

• Notallthird-partyproviderssharethesameperspectiveonL&D.

They need to have worked with learning organizations and

different LMSs to really understand what a learning-focused

customized report needs to contain.


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





• Severaloptionsexistforreceivinghelpwhenyouneedtogenerate

a customized report, each with a number of advantages and

disadvantages. They include:

o Partnering with your IT department;

o Purchasing a third-party reporting platform; and,

o Outsourcing to customized reporting services.


BERSIN & ASSOCIATES






BERSIN & ASSOCIATES





The Bersin & Associates Membership ProgramThis document is part of the Bersin & Associates Research Library. Our research is provided exclusively to organizational members of the Bersin & Associates Research Program. Member organizations have access to the largest library of learning and talent management related research available. In addition, members also receive a variety of products and services to enable talent-related transformation within their organizations, including:

• Research – Access to an extensive selection of research reports, such as methodologies, process models and frameworks, and comprehensive industry studies and case studies;

• Benchmarking – These services cover a wide spectrum of HR and L&D metrics, customized by industry and company size;

• Tools – Comprehensive tools for HR and L&D professionals, including tools for benchmarking, vendor and system selection, program design, program implementation, change management and measurement;

• Analyst Support – Via telephone or email, our advisory services are supported by expert industry analysts who conduct our research;

• Strategic Advisory Services – Expert support for custom-tailored projects;

• Member Roundtables® – A place where you can connect with other peers and industry leaders to discuss and learn about the latest industry trends and best practices; and,

• IMPACT® Conference: The Business Of Talent – Attendance at special sessions of our annual, best-practices IMPACT® conference.

• Workshops – Bersin & Associates analysts and advisors conduct onsite workshops on a wide range of topics to educate, inform and inspire HR and L&D professionals and leaders.

For more information about our membership program, please visit us at www.bersin.com/membership.

http://www.bersin.com/Research/Content.aspx?id=12183#madeline

Documents

Volume 6, Issue 46 Closing the Gaps in LMS Reporting ... · The Bersin & Associates High-Impact Measurement Framework4 (see Figure 2) ... General business measures or HR measures