Executing a BUSINESS-FOCUSED MULTI-RATER ASSESSMENT TO DRIVE BUSINESS OUTCOMES.A Pharmaceutical company case study

U S I N G M U L T I - R A T E R / 3 6 0

A S S E S S M E N T S T O D R I V E

B U S I N E S S O U T C O M E S

O v e r v i e w

Multi-rater / 360 assessments are used by many organizations to provide leaders with valuable information about their job performance from a variety of sources, including managers, peers, subordinates, and customers. Each of the different rating sources contributes unique information about the leader’s performance, thus enhancing the validity of the performance assessment and helping leaders to more accurately identify their strengths and opportunities for development.

By Shane Douthitt, Ph.D., Executive Consultant Scott Mondore, Ph.D., Executive Consultant

2 © 2014 Sales Performance International, Inc

E X E C U T I N G A B U S I N E S S - F O C U S E D M U L T I - R A T E R A S S E S S M E N T

F i g u r e 1 . T h e M u l t i - R a t e r A s s e s s m e n t P r o c e s s

SALES PERFORMANCE INTERNATIONAL / / WHITE PAPERUsing Multi-Rater/360 Assessments to Drive Business Outcomes

D e f i n e

O b j e c t i v e s

D e v e l o p T o o l

& P r e l a u n c h

D e l i v e r

R e p o r t s &

F e e d b a c k

D e f i n e

C o m p e t e n c i e s

L a u n c h

A s s e s s m e n t

C o n d u c t

S y s t e m a t i c

A n a l y s i s

S t e p 1 : E s t a b l i s h a n d C o m m u n i c a t e O b j e c t i v e s / I n t e n d e d O u t c o m e s

Clearly stated, well-communicated objectives are critical to the effective implementation of any multi-rater assessment

process. In particular, it is important to communicate the business needs being met by the implementation of the multi-rater

assessment, the ways in which individuals will benefit by participating, and the impact the process will ultimately have on the

organization. During this step, the following questions should be answered:

• Which employees will participate in the process?

• How will reports be delivered and feedback facilitated?

• What competencies/behaviors will be assessed?

• What role will the managers of participants play?

• How will raters be selected?

• How will the overall process be communicated?

Key organizational stakeholders should be involved in establishing the objectives of the process and in communicating these

objectives to employees.

S t e p 2 : D e f i n e C o m p e t e n c i e s

The next step is to identify and define the competencies on which participants will be rated. Ideally, these competencies

will come directly from your organization’s validated competency model. The set of competencies selected for inclusion

on the multi-rater assessment should encompass the specific behaviors expected of individuals in their job. Examples of

competencies for sales representatives in the pharmaceutical industry include Product and Disease-State Expertise, Core

Selling Skills, and Managed Care and Reimbursement Expertise.

S t e p 3 : D e v e l o p T o o l a n d P r e - L a u n c h

At this point, it is time to develop the assessment tool (i.e., online assessment). This tool will be used to capture ratings

from multiple sources on the competencies identified in the previous step. During this step, it is also critical to begin

communicating the assessment launch to key stakeholders. Effective communications will help to clarify the purpose of the

assessment, align the initiative with broader organizational strategies, outline project timing, reduce participant anxiety, and

improve response rates.

S t e p 4 : L a u n c h A s s e s s m e n t

It is now time for participating employees to invite their raters, including managers, subordinates, peers, and customers,

to complete the assessment. To improve response rates from customers, employees should clearly communicate the

organization’s ongoing efforts to improve customer service by obtaining customer feedback and acting on their input.

Furthermore, once the assessment process is complete, employees should provide customers with a summary of the

developmental feedback received and an outline of the activities planned to address this feedback.



S t e p 5 : D e l i v e r R e p o r t s a n d F e e d b a c k

Participants should receive detailed feedback reports outlining their strengths and development needs. Reports may include

overall competency scores as well as the participant’s scores on each assessment item. To protect the anonymity of raters

(i.e., peers, subordinates, customers), scores by rater group should only be broken out for groups in which three or more

ratings were received (except for manager and self –ratings which are provided based on a single response). Participants

should use their feedback to identify areas in which growth is needed and build development plans.

S t e p 6 : C o n d u c t S y s t e m i c A n a l y s i s

Finally, the multi-rater assessment results should be analyzed at the organizational level. Using the aggregated results of

individuals across the organization, it is possible to identify the competencies that drive critical business outcomes (e.g.,

sales dollars, market share). This analysis allows organizations to better focus training and development initiatives on the

individual behaviors that contribute to desired business outcomes. This step will be illustrated with a case study.

L I N K I N G Y O U R M U L T I - R A T E R A S S E S S M E N T T O B U S I N E S S O U T C O M E S :

A C A S E S T U D Y

The sales leadership at a small pharmaceutical company wanted to invest in the individual development of its specialty sales

representatives and identify the critical skills needed to gain market share on its primary competitor. The company’s brand

possessed a 23.3% share of the market. This particular market is relatively mature and the organization’s leaders were

looking for a new commercial edge in an extremely competitive market. The following reveals the broader challenges facing

the organization:

• Need to invest in the development of specialty pharmaceutical sales representatives

• Need to identify the critical skills for its sales representatives

• Need to grow market share in a competitive, mature market

• Need better understanding of the ROI of organizational training investments

• Need to drive revenue growth

I m p l e m e n t i n g t h e 3 6 0

Organizational leaders decided to implement a development-focused 360 assessment initiative. The performance ratings

were to be used to identify strengths and development opportunities among individual sales representatives, but the results

were not linked to personnel decisions. This was done to build trust in the process and encourage sales representative

participation. To reinforce this message, individual reports were delivered solely to the sales representative such that they

owned the data and results. However, sales representatives were encouraged to share their results with their manager,

thereby allowing them to leverage the coaching process and development tools made available to them.

A s s e s s i n g S a l e s R e p r e s e n t a t i v e C o m p e t e n c i e s

The multi-rater 360 consisted of 25 skill and behavior-based items representing seven sales representative competencies:

Account Management, Core Selling Skills, Relationship Building, Consultative Selling, Product & Disease-State Expertise,

Evidence-Based Selling®, and Managed-Care Expertise. There was also an open-ended item on which managers and

physicians could comment about how the sales representative could best improve his/her performance. Each sales

representative personally invited individual physicians to complete the short on-line survey (no incentive was offered to

physicians to complete the survey). Physicians were identified by the organization based on territory market share potential.

Managers completed an assessment of the same competencies for each sales representative. Finally, each representative

completed a self-assessment of the same competencies to aid in the development planning process.


E v i d e n c e - B a s e d

S e l l i n g - S k i l l s

1


A s s e s s m e n t P a r t i c i p a t i o n

One hundred and sixteen sales representatives participated in the business-focused 360. A total of 1,266 physicians were

sent a survey invitation and more than 50% responded—a very strong physician response rate. On average, 7.5 responses

were received for each sales representative, including 5.5 physician responses, a manager response, and a

self-assessment.

S a l e s R e p r e s e n t a t i v e s a n d M a n a g e r R e s u l t s

Individual sales representatives were each provided a summary report of their assessment results. Furthermore, managers

and sales representatives attended feedback workshops designed to assist them in understanding the reports, identifying

strengths and development opportunities, building development plans, and leveraging the information available through

the organization’s existing field coaching process. Managers received a summary report for their specific territories—which

typically included 8-10 sales representatives. This report allowed individual managers to follow a similar development

planning process – one that is focused on their local sales team.

O r g a n i z a t i o n a l R e s u l t s

Based on the aggregated data set, an overall, systemic analysis was conducted. Using structural equations modeling

(a sophisticated statistical technique with many advantages over multiple regression and correlation analyses), sales

representatives’ competencies were examined as drivers of territory market share and script writing volume, the primary

business outcomes for sales representatives. Using this technique, manager and physician responses were statistically

linked to the previous six-month territory market share and script volume for each sales representative. This analysis

allowed the organization to prioritize individual behaviors that have a cause-effect impact on the desired outcome (e.g.,

market share). The results of the linkage analysis are depicted below in Figure 2.

P r o d u c t & S t a t e

D i s e a s e E x p e r t i s e

M a n a g e d C a r e

E x p e r t i s e

C o n s u l t a t i v e

S e l l i n g S k i l l s

2

3

4

M a r k e t S h a r e

X Y Z S c r i p t

W r i t i n g. 5 4 *

. 2 9 *

. 1 0 *. 1 9 *

. 1 6 *

. 6 0 *

F i g u r e 2 .

Findings and Implications:

• Evidence-Based Selling Skills are a significant, causal driver of Script Writing and Market Share.

• Evidence-Based Selling Skills, in turn, are driven by Product & Disease State Expertise, Managed Care Expertise,

and Consultative Selling Skills.

• Prioritizing initiatives that target the key drivers listed above will lead to improvements in sales performance among

XYZ Sales Representatives.



The results of the linkage analysis identified Evidence-Based Selling® skills as the only statistically significant driver

of prescription market share and script volume. The results identified Product & Disease-State Expertise, Managed

Care Expertise and Consultative Selling Skills key drivers of Evidence-Based Selling® competence in the sales force.

Furthermore, the individual behaviors comprising the Evidence-Based Selling® competency were examined in terms

of their impact on prescription market share and script volume. The items in Table 1 have the largest statistical impact

on prescription market share and script volume. The results of this analysis provide insights as to the areas in which

investments in training and development are likely to yield the greatest return.

R E T U R N O N I N V E S T M E N T

T r a i n i n g I n t e r v e n t i o n s

The systemic analysis allowed the sales organization to assess current sales representatives’ capabilities and prioritize

training & development interventions. To address key needs in the areas of Evidence-Based Selling®,T Product &

Disease-State Expertise, Managed Care Expertise and Consultative Selling Skills the organization implemented several

training interventions at the organization and individual level.

I m p a c t o f T r a i n i n g I n t e r v e n t i o n s

The 360 process was executed again 18 months later. The table below depicts the competency changes found

following the training interventions. Based on the improvements in competency ratings, it was estimated that the training

interventions resulted in a $1.4 million impact on script volume. The average script volume went from 299 to 371 in the 2

year period. In addition to the impact on the organization’s training strategy, the results of the study were used to adjust

the brand strategy of the product. Overall, the results of the 360 were found to have numerous applications and impacts

throughout the organization.

T a b l e 1 .

Priorities determined based on the magnitude of the relationship with script writing and market share.

P r i o r i t y * C o m p e t e n c y

Evidence-Based Selling Skills

Product & Disease State Expertise

Managed Care Expertise

Consultative Selling Skills

I t e m

Easily discusses concerns about bias in XYZ's clinical

studies. Accurately answers questions about clinical

study methodology

Possesses broad knowledge of the therapeutic

market. Articulates all potential effects that XYZ can

have on patients depending on therapy paradigm and

patient profile

Understands the specifics of managed care plans.

Effectively finds solutions to managed care challenges

and restrictions

Understands physicians' unique approaches to treating

the disease state and provides solutions based on

needs. Asks pertinent and insightful questions during

interactions to identify customer needs.

1

2

3

4



T a b l e 2 .

Rating Scale: 1 = Strongly Disagree; 2 = Disagree; 3 = Somewhat Disagree; 4 = Neither Agree nor Disagree; 5 = Somewhat

Agree; 6 = Agree; 7 = Strongly Agree

B E S T P R A C T I C E S I N M U L T I - R A T E R A S S E S S M E N T D E S I G N A N D E X E C U T I O N

There is no shortage of applied research examining how to effectively design and implement multi-rater assessments in

organizations. Based on our review of this research, we offer several execution-focused recommendations to help HR

practitioners implement or improve their existing multi-rater assessment processes. Additional recommendations and

details may be found in our book, Business-Focused HR: 11 Processes to Drive Results.

A s s e s s O r g a n i z a t i o n a l R e a d i n e s s a n d I d e n t i f y C h a m p i o n s

Multi-rater assessments will be most effective if implemented at an appropriate time. Therefore, before implementing

a multi-rater assessment, it is important to consider the current organizational environment/culture as well as the

organization’s performance management history. If the organization is in the middle of a stressful time – such as

during a downsizing or extensive budget cuts – then a multi-rater assessment may be perceived by employees as a

tool for determining future workforce/budget cuts and they will not fully trust or buy into the process. As a result, the

developmental intentions of the assessment will likely not be realized, and employee morale may further be harmed.

Similarly, prior efforts to implement performance management processes should be evaluated for insights into what worked

well and what could have been improved.

I n t e g r a t e w i t h E x i s t i n g P r o g r a m s

Multi-rater assessments are just one aspect of any organization’s performance management process, which may also

include such components as individual goal-setting, training/development, and team-building. To reap the greatest benefit

from multi-rater assessments, it is thus important that they be well-integrated with existing practices. For instance, multi-

rater assessment data may be used to identify gaps in managerial competencies and this information may be used to

enhance training and development practices. Similarly, multi-rater feedback may be incorporated with career development

conversations and broader succession planning initiatives. Such alignment will help to ensure swift adoption of the process

and establish it as an integral component of the organization’s overall performance management system.

C o m p e t e n c y

Account Management

Advanced-Level Selling Skills*

Consultative Selling Skills*

Product Disease State Knowledge*

Evidence-Based Selling Skills*

Managed Care Expertise

Relationship Building

Sales Representative Overall

2 0 1 1

A v e r a g e

6.65

6.55

6.59

6.65

6.56

6.64

6.70

6.61

2 0 1 2

A v e r a g e

6.64

6.59

6.47

6.51

6.34

6.42

6.64

6.53

2 0 1 2

A v e r a g e

+.01

- .04

+ .12

+ .14

+ .22

+ .22

+ .06

+ .08

S i g n i f i c a n t

C h a n g e

NO

NO

YES

YES

YES

YES

NO

NO



E n s u r e R e l i a b i l i t y a n d V a l i d i t y o f t h e I n s t r u m e n t

Multi-rater assessments are generally comprised of sets of questions or items intended to measure particular competencies

(e.g., Relationship Building, Integrity). When developing a multi-rater assessment, it is essential to analyze the extent to

which the items being used are a valid and reliable measure of each competency and whether the sets of items predict

performance outcomes. You do not need to be an expert on statistics; however, it is important to be an educated consumer

of multi-rater assessment products and ask for relevant information in the areas of:

• Content Validity

• Discriminant Validity

• Criterion-Related Validity

• Internal Reliability

Assessing the reliability and validity of your multi-rater assessment instrument will help to ensure that you have a legally

defensible instrument that measures important aspects of individual performance and predicts critical business outcomes.

C l a r i f y t h e R o l e s a n d R e s p o n s i b i l i t i e s o f T h o s e I n v o l v e d

Many individuals are involved in the design and implementation of an organization-wide multi-rater assessment process.

In addition to HR professionals and organizational leaders/managers, organizations will often partner with an external

consulting firm to design and effective process and build the assessment framework and/or tool. With so many stakeholders

and the inherent interdependence of the tasks to be executed it is essential that everyone involved understand their role in

the process. In particular, it is important to clarify the following:

• Who will be responsible for administering the process and providing performance feedback (e.g., one-on-one feedback,

facilitated group sessions)

• The role that employees themselves play in the process

• Who will deal with technical aspects of the process

• Who will be responsible for evaluating the overall multi-rater assessment process

F o c u s o n F e e d b a c k a n d D e v e l o p m e n t P l a n n i n g

HR professionals play an important role in helping employees and their managers to interpret multi-rater feedback reports

and build development plans. Training sessions may be used to help prevent employees from becoming overwhelmed by

their feedback or misinterpreting their results and misdirecting their development efforts. Depending on the size of the

organization, these sessions may take the form of one-on-one meetings, small group discussions, or training webinars.

Regardless of the session format, participants should learn the following: 1.) the business case for the multi-rater

assessment process, 2.) the overall organization results, 3.) how to interpret their feedback report, and 4.) how to build

an action plan based on their results. In addition, HR professionals should make themselves available during or after

the training session to respond to questions and concerns. Keep in mind that receiving feedback – particularly negative

feedback – can be difficult for some individuals. HR professionals and managers across the organization should thus be

prepared to deal with the reactions of feedback and help them to turn their surprise and/ or anger into acceptance.

U s e t h e M u l t i - R a t e r A s s e s s m e n t f o r t h e R i g h t P u r p o s e s

One of the more controversial issues surrounding multi-rater assessments is the question of whether to use them exclusively

for development or to also use them as the basis for personnel decisions. Research indicates that rating quality and



accuracy increase when multi-rater assessments are used purely for development purposes. When employees know that

important personnel decisions (e.g., promotions, bonuses) will be made on the basis of their assessment results, they are

likely to select more lenient raters. Additionally, raters themselves may inflate or deflate their scores depending on their

relationship with the individual being assessed. The most useful ratings will thus be obtained if the assessment results are

used only to guide individual training and development.

E v a l u a t e t h e E f f e c t i v e n e s s o f Y o u r M u l t i - R a t e r A s s e s s m e n t P r o c e s s

The effectiveness of the multi-rater assessment process should be evaluated on a regular basis. The goal of evaluation is to

determine if you are meeting the objectives established earlier and identify any aspects of the assessment process – from

assessment tools and technologies to follow-up coaching and action planning – that may be in need of revision. Evaluation

often takes the form of surveys of those who participated in the assessment process coupled with interviews and/or

observations. The most useful evaluations go beyond simply gauging satisfaction with the multi-rater assessment process to

linking the process to meaningful business outcomes (e.g., individual performance improvement, ROI).

T H E B E N E F I T S O F M U L T I - R A T E R F E E D B A C K

Multi-rater assessments have become an increasingly common HR practice due to several advantages that they offer

over traditional performance management strategies (i.e., top-down ratings). Specifically, multi-rater assessment and

feedback processes offer: 1.) greater accuracy, 2.) enhanced face validity, 3.) richer insights into individual performance,

4.) improvements in communication, and 5.) practicality and cost-effectiveness. Each of these advantages is discussed in

greater detail below.

1 . A c c u r a c y

Traditional performance management systems rely on individual managers to assess the capabilities of their employees.

Although managers’ insights into their employees’ performance are a critical component of the performance feedback

process, relying solely on managers limits the breadth of perspectives obtained and reduces the accuracy of the

performance ratings. In contrast, multi-rater assessments take into account the perspectives of several key stakeholders

(e.g., self, manager, peers, subordinates, etc.), thus providing a more complete picture of the employee’s skills, behaviors,

and performance. Additionally, ratings are generally obtained anonymously, thereby allowing raters to provide more honest

and accurate assessments of the focal individual’s performance.

2 . F a c e V a l i d i t y

Multi-rater feedback has proven to be more accepted by participants than manager evaluations alone largely because

including multiple raters and basing performance evaluations on a comprehensive competency model enhance the face

validity of the process. Employees realize that their manager is not around to personally witness them performing every

aspect of their job. Including peers, subordinates, customers, etc. in the rating process helps participants to feel that their

performance has been fairly evaluated across all aspects of their job, thus enhancing their acceptance of the feedback and

reducing the anxiety that may arise when relying solely on the manager for performance ratings. Additionally, when the

ratings from multiple sources are based on a comprehensive competency model that truly represents the employees’ job

responsibilities, the feedback becomes very relevant and actionable.

3 . R i c h e r I n s i g h t s

Feedback from multiple sources provides participants rich insights into how their skills, behaviors, and performance

are perceived by various groups of stakeholders. Such feedback provides valuable information about variations in the

participant’s behavior as well as ways in which the same behaviors may be perceived differently by each stakeholder

group. Knowledge of these variations allows participants to make more meaningful changes in their behavior and improve

perceptions across stakeholders.


4 . I m p r o v e d C o m m u n i c a t i o n

Another benefit of multi-rater feedback processes is that they create opportunities for cross-level communications about

performance. As performance feedback from multiple sources becomes ingrained in the organizational culture, participants

begin to feel more comfortable speaking openly about their performance with not only their managers but also with other key

stakeholders (e.g., subordinates, peers).

5 . P r a c t i c a l i t y a n d C o s t - E f f e c t i v e n e s s

Multi-rater feedback processes are practical, cost-effective, and can readily be implemented by almost any organization.

They provide invaluable individual- and organizational-level diagnostic information that can readily be translated into

actionable interventions. Additionally, the cost per participant is extremely low given the significant individual and

organizational applications.

C O N C L U S I O N

This whitepaper has provided a simple roadmap to help organizations move 360 feedback processes from a simple

development tool to a strategic HR process that drives business results. Multi-rater assessment processes can provide

organizations with a new strategic resource to maximize the skills and behaviors of its employees to achieve quantifiable

business results.

6201 Fairview Road, Suite 400

Charlotte, NC, USA 28210

+1.704.227.6500

www.spisales.com | [email protected]

Recruiting & HR

Executing a BUSINESS-FOCUSED MULTI-RATER ASSESSMENT TO DRIVE BUSINESS OUTCOMES.A Pharmaceutical company case study