Using Assessment Center Methodology to Accelerate ...€¦ · • Targeted leadership development seminars • Action Learning Projects linked to team assessments and team coaching

© Copyright 201035th International Congress on Assessment Center Methods, Singapore

Using Assessment Center Methodology to Accelerate Executive

MBA Leadership Development

Jeff Kudisch, Ph.D.Ralph J. Tyser Distinguished Teaching FellowManaging Director, Office of Career ServicesUniversity of Maryland, College Park, USA

Presentation to the 35th International Congress on Assessment Center MethodsOctober 20-21, 2010 - Singapore


A Quick Look at the Smith School

• 2,950 Undergraduate Students

• 970 Part-Time MBA Students

• 225 Full-Time MBA Students

• 180 Executive MBA & MS Students– U.S., China, Europe

• 100 PhD Students• Custom Executive

Education


Comprehensive School

• 150 Full-Time Faculty• 50 Part-Time Faculty• 6 Academic Departments

– Accounting and Information Assurance– Decision, Operations Information Technologies– Finance– Logistics, Business and Public Policy– Management and Organization– Marketing


Smith School Rankings• #19 Undergraduate Program (10th among public universities)

- U.S. News & Word Report, 2010

• #20 MBA Program, U.S. (5th among public business schools)- Financial Times, 2010

• #15 Part-time MBA Program- U.S. News & World Report, 2010

• #22 EMBA Program (#15 Management Skills Rank)- Wall Street Journal, 2010

• #14 World Research Ranking- Financial Times, 2010

• #3 Intellectual Capital- BusinessWeek, 2008

• #8 Top-Rated Schools by Recruiters (UMD)- Wall Street Journal, 2010 (#7 Accounting, #21 Business)


Setting the Stage:Business Education at a Crossroads

• Managers and recruiters are increasingly questioning the value and relevancy of MBA programs1

– Clear gaps between what’s being delivered in MBA curricula and what’s needed

– Newly minted MBAs lack essential leadership skills, teamwork, critical thinking, creativity and innovation, and cultural awareness

• Especially problematic given global evidence that organizations don’t have the talent to lead them into the future

1 See Rubin, R. S., & Dierdorff, E. C. (2009). How relevant is the MBA? Assessing the alignment of required curricula and required managerial competencies. Academy of Management Learning & Education, 8, 208-224. Also see Datar, S. M., Garvin, D. A., & Cullen, P. (2010). Rethinking the MBA: Business Education at a Crossroads. Boston, MA: Harvard Business School Publishing.


Fast Facts• A recent survey of 425 senior executives in North America,

Europe, and Asia showed that attracting and keeping skilled talent and developing employees into capable leaders were among the top ten business issues cited.1

• A 2005 survey of 750 CEOs indicated that 79% believed the ability to develop leaders was the most important or one of the top 5 factors in achieving a competitive advantage.2

• A recent Center for Creative Leadership (CCL) survey of 2,200 leaders from companies in the United States, India and Singapore identified that the four future key skills ranked mostimportant for future success by respondents were found to be among the weakest for today’s leaders: leading people, strategic planning, inspiring commitment, and managing change.3

1 Accenture consulting study as cited in McGee (2005, August 16th). Why promote from the outside when the talent is inside? Information Week – www.informationweek.com.; 2 see Center for Creative Leadership.; 3 see Taylor, S. (2010, February). Bridging the leadership gap. Chief Learning Officer -www.clomedia.com/business-intelligence/2010/February/2857/index.php.


Business Education at a Crossroads: The Path Forward

• Business schools must rethink accepted beliefs and assumptions and transform their curricula to better leverage hands-on exercises, experiential learning, and opportunities for self-reflection.

– Adding courses on business ethics and financial crises won’t solve the problem

• Key question: Beyond coursework, what techniques can we use to enhance students’ self-awareness and proficiency on essential competencies required for successfully leading during challenging times?


Use of ACs in Business Schools• Learning Outcome

Assessment– 1980’s AACSB partnered

with DDI

• Undergraduate-Level Efforts– Evaluating career

readiness of business majors (Riggio et al., 1997)

– Predicting early career success of business majors (Waldman & Korbar, 2004)

• Graduate-Level Efforts: Guiding Leadership Development– University of Tennessee

MBA and EMBA programs

– University of Maryland EMBA program


Here was our response for our

1st EMBA cohort…

• Data-driven executive coaching• Paper and pencil psychological

assessments• 360-degree feedback• Leadership and development

seminars• Action Learning Projects




Here is where we are now (our 10th

cohort)…

• Assessment Center-driven executive coaching

• Behavioral and paper and pencil psychological assessments

• 360-degree feedback• Targeted leadership development

seminars• Action Learning Projects linked to

team assessments and team coaching


Assessment Centers• Individuals participate in a series of

situations that simulate the real world• Behaviorally-oriented process used for a

variety of organizational/human resources purposes

• Candidates evaluated on multiple competencies across multiple techniques

• Use of multiple assessors (raters) and integration session to reduce rater bias and thus increase rating accuracy

• Effective tools for predicting future overall job performance, management potential, performance in training, and career advancement; valid across cultures.

• Entry-level to executive applications; used increasingly around the world.

Popular Simulations:• Written:

• In-basket exercise• Case analysis

• Interpersonal:• Oral Presentation• Group Discussion exercise• Interview Simulation

Pictured: Stress Interview (OSS, WWII)


Sample Organizations Using Assessment Center (AC) Methodology

a Denotes Capital One runs Executive Development Centers (top-tier) and Leadership Development Centers (next tier) – see Delahoussaye, M. (2002, March). When tomorrow comes. Training; b denotes Microsoft is using ACs as part of their selection process for Country Managers (GMs).


Purpose of the Smith SchoolAcceleration Center

The Smith School is using the AC, in conjunction with the other Executive Coaching components, to help you diagnose and enhance your leadership strengths and development needs. The data may

also be used to facilitate administrative issuesrelated to enhancing the effectiveness of your

EMBA endeavors.


Our EMBA AC Process


Interpersonal SensitivityTeam Orientation

EXPRESSION CLUSTER COGNITIVE CLUSTER

Judgment

LEADERSHIP CLUSTER

Process LeadershipPersuasion & InfluenceConfrontationInitiative

Seven Competencies Measured


Sample AC Report

Listed below is a detailed, narrative summary of your performance across the various competencies. Evaluations of each competency are supported by behavioral examples (e.g., critical incidents) of performance.

Interpersonal Sensitivity. Mr. Participant exhibited slightly below average interpersonal sensitivity in the Group Discussion Exercise. Although he acknowledged others’ ideas (e.g.., “Yeah”; “Yes, that’s fine”), his comments appeared mechanical in nature. He also displayed weak active listening skills at times (e.g., he had a tendency to “listen to respond” versus “listening to understand” - he interrupted and talked over others to assert his views rather than “learning” what was said; he looked down when others were talking), and he used an insensitive, directive tone at times (e.g., he sometimes disagreed with others in an abrupt manner - "We can do that later..."; "But it said..."; "We need to..."). Overall, rather than coming across as affiliative, his style could be best described as detached, analytical, and task-oriented.

Strengths: Judgment, Initiative, Confrontation

Proficiencies: Interpersonal Sensitivity, Persuasion & Influence

Developmental Needs: Process Leadership

Unable to Rate: Team Orientation

Executive Summary

Narrative Competency Summaries


Benefits• Positive participant reactions

– “Jolts”; high impact insights –“Ah hah” moments

• Accelerated coaching– Richer development plans

• Cohort-level needs assessment data

• Accelerated on-the-job leadership development

• Enhanced Smith Brand– Driving applications to the program


“The simulation was fantastic, would like to add additional role play simulations to the first Assessment center experience though… to really gain an insight in a persons leadership style.”

“I enjoyed the experience, and learned from it! ... This should be a very valuable part of the EMBA.”

“Only gave it a four because I wish it was more thorough. I would like to have done the full assessment center, even if it had to be broken up over 2 days (or a day and a half). Other than that, it was excellent.”

“I am able to communicate much better and more effectively with senior management. Through the executive coaching program I raised significantly my EQ by increasing my self-awareness and controlling my emotion and funned them in the appropriate manner to achieve results.”

EMBA Reactions


Signs(Indicators of Behavior)

Samples(Actual Behaviors)

1. What a personality test described: 1. What we observed in the AC:

EMBA is insightful concerning how others feel and think. He is quick to respond to social nuances, and is likely viewed by others as reasonably perceptive and observant. He is pleasant to be with and usually easy to have around.

EMBA displayed rather weak interpersonal sensitivity…On the positive side, he occasionally complimented others…On a less positive note… he frequently interrupted others before they finished speaking; instead of responding to others' comments, at times he would transition to his own ideas; although he facilitated consensus on some decisions, at times he disagreed with others’ ideas and subsequently redirected the conversation without regard for their opinions…he usually disagreed in an abrupt, brash manner… using a harsh, rather arrogant tone…Taken together the data suggest that while he can be personable, his assertive, competitive, task-oriented demeanor may prevent such behavior from consistently showing (e.g., he may come across as impatient toward others)…The data further suggest that EMBA may lack “emotional intelligence”(e.g., he demonstrated limited empathy toward others’feelings/needs and appeared to lack self-monitoring skills when interacting with others).

“Would Do” vs. “Can Do”


“Would Do” vs. “Can Do”Signs

(Indicators of Behavior)Samples

(Actual Behaviors)2. What a cognitive ability test

described:2. What we observed in the AC:

EMBA’s performance places her in the 99th percentile, meaning that her problem solving capabilities are better than 99 percent of [upper division students in four-year colleges]… When compared to a sample of mid- to upper-level managers who have completed a similar assessment battery, her score is [again] in the 99th percentile… Overall, she should have no problem at all recognizing key issues, making inferences, and coming to appropriate conclusions based on information given to her. In short, she should be an excellent problem-solver.

EMBA demonstrated reasonable, though inconsistent, judgment in the LGD… varied dramatically, from instances of excellent judgment, to instances of poor judgment …could have enhanced her performance… by doing a better job of consistently articulating the logic/rationale behind her ideas versus appearing opinionated, and by slowing down to ensure that all other team members were able to follow her thought processes. Doing so would also have had a positive impact on her ability to sway others, and would have ensured that she avoided premature closure on issues. On a final note … she could have also enhanced her performance by better leveraging financial data to support her arguments (e.g., while she had very good grasp on most issues, she confused some of the P&L data).


Impact on EMBA Admissions…“Another element of my EMBA experience and how it will help advance my ability to

execute on strategy is what I will learn through the assessment center and

executive coaching experience. The psychological evaluation and the

interpersonal approach is evidence that the program uses leading research and

knowledge to develop the intangible qualities of leadership. This bespoke and experiential approach is what I value most

out of the program, and I hope to devote as much energy as I possible can to maximize

my development in this area.”


Challenges• Cost

– As high as $10-14K for executive-level assessments

• Time• Staff Resources• Top Management

Commitment• Intellectual Capital• Demonstrating ROI


How We Overcame These Challenges

• Custom Program Success with Fortune 500 firm– Top Management Intrigue & Buy-In– Saw as a unique value proposition

• Narrowed the focus to key competencies

• Used fewer exercises• Leveraged In-house

expertise• Raised EMBA tuition


Toward ROI: Initial Research Findings

To what extent does participation in the EMBA Acceleration Center influence or shed light on executive coaching success?

• Data collected from 3 of our EMBA cohorts• 100 executives, 13 coaches• 18-month engagement (5 sessions as 2-2.5 hours each)

• Survey methodology - measured throughout the coaching engagement; solicited coach and EMBA reactions


Executive CharacteristicsWhat characteristics of an EMBA (assessee), if any, help explaindifferences in his/her success in executive coaching?

• “Behavior change” measured after session 4 and 5; based on coach ratings; 4 items: α= .92 (n =46)

• The executive…• Definitely learned and applied new behaviors throughout coaching• Acted on the majority of recommendations from previous meetings• Presents evidence of behavior change• Seems more confident about what s/he is doing on the job as it

related to areas addressed in coaching

rOAR, behavior change (time 4) = .31 p<.05rrOAROAR, behavior change (time 4) , behavior change (time 4) = .31 = .31 pp<.05<.05

rJudgment, behavior change (time 4) = .31 p<.05rrJudgmentJudgment, behavior change (time 4) , behavior change (time 4) = .31 = .31 pp<.05<.05

rInterpersonal, behavior change (time 5) = .33 p<.05rrInterpersonalInterpersonal, behavior change (time 5) , behavior change (time 5) = .33 = .33 pp<.05<.05

Limitation = Small Sample

Size

Limitation = Small Sample

Size


Executive Characteristics (cont.)

Findings consistent with past research that suggests that people with certain characteristics are more likely to accept feedback and participate in developmental activities

•c.f., judgment, personality, conscientious, goal orientation (Byham 2005; Jones & Whitmore, 1995; Kudisch & Ladd, 1997; Kudisch, Lundquist, & Smith, 2001; Poteet & Kudisch, 2003; Atchley, Smith, Hoffman, 2003)

•Further evidence that the “rich get richer”

•From a broader talent management perspective, the findings highlight the potential link between assessment center methodology and executive coaching for growing leaders (c.f., Poteet & Kudisch, 2008)

•Can we expect cross-cultural consistency in these results?


Executive Characteristics (cont.)

What EMBA (assessee) characteristics help explain differences in his/her engagement in executive coaching?

• “Engagement” rated by Executives after sessions 2-5

• Consisted of behaviors during the coaching session:

• Actively engaged in the session• Came prepared, made it a priority

• And behaviors outside the coaching session:• How hard they worked• how much time they put in• whether they accomplished what

they planned to do

3 items, average α = .75; “Since your last coaching session, how hard did you work towards your objectives laid out in coaching?”

3 items, average α = .75; “Since your last coaching session, how hard did you work towards your objectives laid out in coaching?”


Role of Cognitive Ability• Those with better reasoning and

problem solving skills are more accepting of data (e.g., feedback) and are more likely to act on it

• Those with greater cognitive skills should value the feedback provided in coaching and engage more in the coaching process

• Especially in those coaching programs that are very assessment-oriented Watson-Glaser Critical Thinking

Appraisal; 80 items, α = .75Watson-Glaser Critical Thinking Appraisal; 80 items, α = .75


1 See Ludeman & Erlandson (2004). For more see Gettman, H., Wouters, K., Stevens, C., Kudisch, J., Edinger, S., Russell, J., & Tesluk, P. (2010, April). Factors that influence executive engagement in coaching (and does it even matter?). Paper presented at the 25th annual conference of the Society of Industrial and Organizational Psychology, Atlanta, Georgia.

Interesting & Unexpected Finding• Hierarchical linear regression, controlling for coach• Executive Characteristics’ (measured at T1) effects on engagement from

T1 – T5.

• Contrary to predictions, cognitive ability was negatively related to engagement!

• T2: β= ‐.54, p=.00• T3: β= ‐.22 p=ns• T4: β= ‐.23, p=ns• T5: β= ‐.52, p=.00

• Why?• Doesn’t measure cognitive openness or flexibility• Tapping an Alpha Male profile?1

• Bright, analytical, interested in learning about business, technology, etc., but no interest in introspection or curiosity about people.1

• Why?• Doesn’t measure cognitive openness or flexibility• Tapping an Alpha Male profile?1

• Bright, analytical, interested in learning about business, technology, etc., but no interest in introspection or curiosity about people.1


Here was our response for our

1st EMBA cohort…







Here is where we are now (our 10th

cohort)…

• Assessment Center-driven executive coaching

• Behavioral and paper and pencil psychological assessments

• 360-degree feedback• Targeted leadership development

seminars• Action Learning Projects linked to

team assessments and team coaching


An Integrated EMBA & Intensive Leadership Development Program

Courses“The Right Tools”

Leadership Mastery Seminars “Targeted Skill

Development”

ActionLearningProjects“Practice and

Impact”

Assessment& Coaching

“An Integrated, Personal Leadership Development Plan”


What’s on the Smith Horizon• Using more assessment methodology

in our EMBA admissions process

• Expanding the assessment center methodology and coaching

• Using more assessment methodology in our part-time and full-time MBA programs– Importance of handling feedback in a

culturally sensitive manner given student diversity (40% FT MBA International)

• More systematically exploring our ROI– Smith Executive Coaching Consortium


Questions/Comments?

Feedback and follow-up are most welcome: [email protected]


Atchley, E. K., Smith, E. M., & Hoffman, B. J. (2003, September). Examining the relationship between performance, individual differences, and developmental activities: Getting more bang for the buck from DPACs. In T. Maurer (Chair), Where do we go from here? Accepting and applying assessment center feedback. General session presented at the 31st annual meeting of the International Congress on Assessment Center Methods, Atlanta, GA.

Byham, T. M. (2005). Factors affecting the acceptance and application of developmental feedback from an executive assessment program. Unpublished doctoral dissertation. University of Akron.

Jones, R. G., & Whitmore, M. D. (1995). Evaluating developmental assessment centers as interventions. Personnel Psychology, 48, 377-388.

Kudisch, J. D. & Ladd, R. T. (1997, April). Factors related to participants’ acceptance of developmental assessment center feedback. Paper presented at the 12th Annual Conference of the Society for Industrial and Organizational Psychology. St. Louis, MO.

Kudisch, J. D., Lundquist, C. & Smith, A. F. R. (2001, October). Reactions to “dual-purpose”assessment center feedback: What does it take to get participants to buy into and actually do something with their feedback? Paper presented at the 29th International Congress on Assessment Center Methods, Frankfurt, Germany.

References


Ludeman, K., & Erlandson, E. (2004, May). Coaching the alpha male. Harvard Business Review, 59-67.

Poteet, M. L., & Kudisch, J. D. (2003, September). Straight from the horse’s mouth: Strategies for increasing feedback acceptance. In T. Maurer (Chair), Where do we go from here? Accepting and applying assessment center feedback. General session presented at the 31st annual meeting of the International Congress on Assessment Center Methods, Atlanta, GA.

Poteet, M. L., & Kudisch, J. D. (2008, September). Using executive coaching to leverage assessment center feedback: Tips for success. Paper presented at the 34th annual meeting of the International Congress on Assessment Center Methods, Washington, DC.

References

Documents

Using Assessment Center Methodology to Accelerate ...€¦ · • Targeted leadership development seminars • Action Learning Projects linked to team assessments and team coaching