Using Simulation, 360-degree Feedback, a Balanced ... 2016 Drexel 2016 Assessment Conference September 8, 2016 Presented by: Drexel University, LeBow College of Business Jim Caruso,

Using Simulation, 360-degree Feedback, a Balanced Scorecard, and AARs to Assess Individual and

Team Performance in Different Delivery Formats

© James. V. Caruso, 2016

Drexel 2016 Assessment Conference September 8, 2016

Presented by:Drexel University, LeBow College of Business

Jim Caruso, MBA, CPA, MHRD, CPLP

2

ObjectivesDuring this session, you will learn more about:• How individual, team, and business performance can be assessed and

developed using an integrated simulation experience, and the tools used• How business acumen and leadership content can be integrated and taught• What roles the instructor can perform, how to scaffold content, and how to

develop critical thinking given the different delivery formats• Ways to apply best practices on how to integrate simulation, multi-discipline

content, instructional design resources, and various assessment tools in your setting

After completing this session, you will be able to:• Use a 360-degree feedback tool (Capsim’s TeamMATE) to assess individual and

team performance in your courses• Use After Action Reviews (AARs) in your courses to enable individuals and

teams to analyze, reflect on, and improve their performance


3

Agenda1. Introduction2. Integrated and Experiential – Corporate Contextualization3. Case Example – Successfully using a Total Enterprise Simulation

with a Graduate Management Classa. Contextb. Business Acumen Contentc. Leadership / Management Contentd. Assessment Tools

4. Delivery formatsa. 11 weeks Face-to-Faceb. 4-day Residencyc. 11 weeks Online

5. Questions


2. Integrated and Experiential –

Corporate Contextualization

4© James. V. Caruso, 2016

5

Integrating Content, Simulation, & Resources• Content – prework; scaffolding; order of content and

simulation; flipped classroom• Company-specific information – terminology, models

• Finance / Strategy / Marketing / Sales• Production / Operations• Leadership / Management / Communication /

Presentation Skills

• Simulation – total enterprise level, scenario-based• Delivery mode


6

Integrating Content, Simulation, & Resources (cont’d)

• Project-based learning, after action review (AAR), reflection, gap analysis

• Roles - instruct / facilitate / coach / consult• Considerations - Create an overwhelming scenario /

discovery learning / ‘sink or swim’

• Critical thinking• Sensitivity analysis• Engagement - Accountability & Ownership


7

Integrating Content and Simulation Possibilities

• Finance - Total Enterprise Simulation – Level 1, 2, 3

• Business Acumen (finance, strategy, marketing) - Total

Enterprise Simulation – Level 2, 3

• Business Acumen and Leadership - Total Enterprise Simulation

• Level 3, 4, 5 (with executive coaching)

• Leadership – Scenario (Influence, Change, Conflict, Strategy)

• Project Leadership – Scenario

• Business Acumen and Project Leadership – Scenario

• Sales – Scenario


Total Enterprise Simulation – Complexity Levels

8

Level 1 (59) – single-player; individual or team play

Level 2 (17) – multi-player; smaller # of products/markets

Level 3 (73) – multi-player; larger # of products/markets• E.g., Capsim – Foundation; Capstone

Level 4 (23) – multi-player; sub-divided team & decisions; negotiated transactions w/ other teams; business ecosystem - make/buy/sell decisions

• Senior executives (Executive Education); EMBA

Level 5 (7) – Level 4 + customized; more complex negotiations; executive coaching


3. Case Example –

Successfully using a Total Enterprise Simulation with a

Graduate Management Class –

a. Context


Core Intellectual and Practical Skills

Information Literacy; Communication; Critical Thinking; Self-Directed Learning; Ethical Reasoning; Technology Use

INFORMATION LITERACY - Possess the skills and knowledge to access, evaluate and use information effectively, competently, and creatively

Experiential and Applied Learning

LEADERSHIP - Develop a vision, translate that vision into shared goals, and effectively work with others to achieve these goals

Incorporates University’s Student Learning Priorities


Integrating Content, Simulation, & Resources (continuously linking)

• Graduate MBA class – Management

• Business Acumen – finance; accounting; strategy; marketing; operations

• Leadership / Management – team, conflict; feedback, communication; reflection

– TeamMATE - Team evaluation tool

• Total enterprise simulation – Capsim – Foundation– 2 Practice Rounds / 8 Competition Rounds

• Assignments

• Real-world examples

Managing the Total Enterprise


Total Enterprise Simulation - Characteristics

12

• Industry Overview

• Company

• Goals, strategies, & performance metrics

• Products / segments

• Key decisions – decision-making process

• Tradeoffs

• Sensitivity analysis

• Rounds

• Level


1. Strategy formulation / refresh

2. Planning & analysis (industry & company)

3. Make decisions (tactical)

4. After Action Review (AAR) – Part A / Group reflection

5. Individual reflection

6. Results debrief

7. After Action Review (AAR) – Part B / Group reflection

Simulation Process for each Round- Rounds – Practice (2), Competition (8)



Total Enterprise Sim Example: Capsim – Foundation

3. Case Study –

Successfully using a Total Enterprise Simulation –

b. Business Acumen Content


Finance and Accounting

Financial statements/analysis, ratio analysis, pro-forma financial statements; margin maximization

Strategy

Total enterprise impact; top management functions & responsibilities; competitive strategies; strategic planning & implementation

Marketing

Segmentation & targeting; customer loyalty; pricing

Operations

Production & operations - forecasting, capacity planning & inventory control; quality

Business Acumen


3. Case Study –


c. Leadership / Management Content


• Team – stages, structure/roles

– processes - decision-making, communication, meetings

• Managing conflict

• Giving and receiving feedback

• Developing and using influence

• Communication styles

• Communication & presentation skills

• Alignment

Reflection (team & individual) - self-awareness

• TeamMATE – individual/team development/assessment tool

• After Action Review (AAR)

Leadership / Management


3. Case Study –


d. Assessment Tools


20

Assessment Tools


Capsim

• Prework – Quiz, Rehearsal Tutorial

• Course – Success Measures; Balanced Scorecard; Analyst

Report; Star Summary Report

• Overall team performance

• TeamMATE – 360-degee feedback (individual & team)

Course

• Pre-course survey

• Assignments (8) – graded

• After Action Review (AAR)


# Assignment (I = Individual; T = Team) I / T Points

1 Pre-Course & Pre-Simulation Preparation I 20

Team Participation Information (Qualtrics Survey) I 0

2 Preliminary Goals & Strategy T 15

3 Success-Performance Metrics T 10

4 Decision Reports (3) T 30

5 Decision Making Analysis Tools T 10

6 Board Presentation Material T 10

Board Presentation Delivery T 10

Board Presentation (content, delivery, visuals, responses) I 25

7 Lessons Learned and to be Learned I 30

Company Performance T 25

TeamMATE Evaluation (Capsim) I 15

Total Points 200

8 Team Contribution-Evaluation I +/-25

Summative Assessment & Grading Tools (11-week)

3. Case Study –


d. Assessment Tools:

Success Measures


A. Financial Management GoalsSuccess Performance Metrics - Success Measures

23

1. Cumulative Profits

2. Average $Market Share

3. Avg. ROS4. Avg. Asset Turnover (ATO; T/O)5. Avg. ROA6. Avg. ROE7. Ending Stock Price8. Ending Market Capitalization

DuPont Profitability Model


• Financial – Stock Price, Profits, Leverage

• Internal Business Processes – Contribution Margin, Plant Utilization, Working Capital Days, Stock-out Costs, Inventory Carrying Costs

• Customer – Customer Buying Criteria, Customer Awareness, Customer Accessibility, Product Count, SG&A Expense

• Learning & Growth – Employee Turnover rate, Employee Productivity, TQM Material Reduction, TQM R&D Reduction, TQM Admin Cost Reduction, TQM Demand Increase

Balanced Scorecard – Strategic Alignment


Balanced Scorecard

25© James Caruso, 2016

Team Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Recap Total

Pos-sible 82 89 89 100 100 100 100 100 240 1000

An-drews 51 62 59 74 75 75 80 77 209 762Bald-win 52 41 46 59 62 50 57 61 147 575

Ches-ter 38 35 30 59 63 73 73 73 164 610

Digby 55 52 51 47 56 48 45 49 98 501

Erie 53 49 53 37 38 40 42 49 97 458

Ferris 61 52 45 45 47 43 46 43 94 476

Analyst Report – Year 8 & Cumulative


Team Mar-gins

Prof-its

Emer-gency Loan

Wor-king Cap-ital

Mar-ket

Share

Fore-cas-ting

Cust.Sat.

Pro-duc-tivity

Finl. Struc-ture

Wealth Crea-tion

Total-

Year 8

Cumul. Total

An-drews 100 100 100 50 100 100 100 92.4 0 100 842 6,474

Bald-win

66.7 100 100 50 80 100 100 63 0 100 760 5,331

Ches-ter 100 100 100 100 60 100 66.7 70.7 0 84.2 782 5,189

Digby 83.3 38.3 100 50 20 100 0 61.9 100 63.5 617 5,226

Erie 100 61.7 100 100 40 100 0 93.9 100 56.6 752 5,181

Ferris 36.7 19.7 100 50 0 100 0 49.7 100 45.7 502 4,736

Star Summary Report – Cumulative (8 years)


Team Profit Margin Emerg.Loan

Inventory Stock Price

Cumul.Total

+Contrib. Margin> 30%

$0

Met 95% of demand; Inv. Days

< 90

Increase from prior

year

Andrews 8 7 8 8 8 39Baldwin 8 6 6 0 8 28

Chester 8 6 8 4 8 34Digby 8 7 8 5 6 34

Erie 8 7 7 6 6 34

Ferris 8 5 8 5 5 31

3. Case Study –



After Action Review (AAR)


1. Where are we now?

– Current state, ‘as is’

2. Where are we going?

– Desired future state, ‘to be’, vision, goal; business issue

3. How do we get there?

– Gap; Recommendations, decisions

Gap Analysis


1. What did we intend to do?

Planned | Forecasted | Budgeted results

2. What actually happened and why?

Actual results

3. What do we need to change/improve next time?

After Action Review (AAR)


After Action Review (after each decision round)

31

1 (low)

2 3 4 5 (high)

1. Do I understand our strategy?

2. Do I agree with the actions taken to implement our strategy?

3. Do I feel that my input is valued and respected?

4. Are we leveraging the diversity of the group (experience, functional background (supporting other’s development areas), communication style)?

5. Do I feel that my team has effective decision-making processes?

Remember - self-deception leads to self-destruction


3. Case Study –



TeamMATE


• Developmental and performance-based; 360 degree feedback

• Information individuals can use to diagnose their own behaviors and overall team functionality in real time to allow for corrective, developmental action

– Individual-level Teamwork Behavior dimensions: the 4 P’s of Individual Performance - Performance … (1) Preparation; (2) Execution; (3) Monitoring; & (4) Adjustment

– Team-level Process dimensions: The 5 C’s of Team Dynamics –Confidence, Coordination, Cohesion, Cooperation, & Conflict

• Information assessors can use in evaluating individual team behavior and team performance

• The ability to correlate team dynamics scores with financial and operational results in the simulation over time, for analysis on the impact of effective team behaviors

TeamMATE Evaluation Tool – Parameters and Use


Receiving Feedback (360-degree feedback)

34

UnrecognizedWeakness

(“blind spot”)

RecognizedStrength

RecognizedWeakness

Others’ (peers’) rating of you

Your rating of self

Low (1)

Low (1)

High (5)

High (5)

UnrecognizedStrength

Which box do you focus on? Also compare to Team Average


4. Different Delivery Formats

a. 11 Weeks Face-to-Face

b. 4-day Face-to-Face Residency

c. 11 weeks online


11 Weeks (face-to-face and online)

• MBA - part-time

• MS

– Finance, Accounting, Business Analytics

– Electrical and Computer Engineering, Chemical Engineering; Television Management

• Joint – MBA and

– B.S. – Business Administration

– MS - Finance / Television Management

– Ph.D. - Electrical and Computer Engineering

• Post-masters certificate

• University employees

4-day Residency

• Incoming MBA students (first term)

– full-time

– part-time

– online

Student Profile


Course Component SummaryCONTENT SIMULATION ASSIGNMENTS & ASSESSMENTS

After Action Review - (AAR – T&I)

LECTUREDEBRIEF

DECISIONS RESULTS AARAssignment

I = IndividualT = Team

360-degree Feedback: TeamMATE (T&I; 3)

Lessons Learned (I)

Prework (I) - Finance, Survey

Strategy / Team (T)

Success Metrics (T)

Presentations – Simulation (T & I)

Prestn. – Decision Making Tools (T)

Functional Analysis - 3 (T)

Team Contribution (I)

TeamMATE

Intros, Course & Simulation Overview - Strategy

Presentation Skills

Forecasting (Ops)

Success Measures/ Metrics

Team-conflict, feedback, influence; Commcn. styles

F/S Analysis

Lessons Learned

Real World Examples

Course Summary

Portfolio Mgmt., Marketing Segmentation

Prework – F/S

Debriefs (2)

Practice Rounds (2)

Competitive Rounds (8)

Results (2)

Results (8)

Debriefs (8)

Prework – Sim (I) – Quiz…

Company Performance (T)

Communication Style (I)

Ethics


a. 11 weeks Face-to-Face


Sample Weekly Process – Thu. – Thu.Thu. - Class Fri. Sat. & Sun.

Mon. Tue. Wed. Thu - Class.

Decisions

Lecture Headlines & Results

Debrief

After Action Review – Team –

w/o Results

360-degree Feedback: TeamMATE

(I – Decisions)

Lecture

After Action Review – Indiv.

– w/ Results

360-degree Feedback: TeamMATE (T-Debrief)

360-degree Feedback:

TeamMATE (I –Reflection)

After Action Review – Team – w/ Results / Team Planning

MeetingAssignment (T) Due

Individual Analysis

Curriculum Sequence – 11 weeks face-to-facePrework Week 1 Week 2 Week 3 Week 4 Week 5

Week 6 Week 7 Week 8 Week 9 Week 10 Week 11

PR - 1

PR - 1

Lecture – 1Course,

Sim, Strategy

Results– PR1

AAR - 1

Lessons Learned

(I)

TeamMATE - 1

7- Presents.

5 - Forecasting

4 - Metrics

3 - Team

2 - Finance

8–Lessons Learn

9

10 – Course Summary

6 - Portfolio

Debrief - 1

PR - 2

AAR (T&I)

Finance, Survey, Sim (I)

Strategy / Team (T)

CR-7

PR - 2

65

4

3

2

LECTURE

8

DEBRIEF DECISIONS RESULTS

23

Decisions-1

AAR

AssignmentI = Individual

T = Team

Metrics (T)

Presentations –Simulation (T&I)

Presetn. – Decision Making Tools (T)

32

8

7

Results–CR-1 CR-2

CR-6CR-5CR-4

CR-3

CR-8

6

54

3

2

54

6

Functnl. Analysis (T) Functnl. Analysis (T)Functnl. Analysis (T)

Team Contribu -

tion (I)

TeamMATE (T&I)


b. 4-day face-to-face residency


Curriculum Sequence – 4-Day ResidencyPrework DAY 1 DAY 2 DAY 3 DAY 4 Postwork

CK 4-day slides -CONTENT

Lunch Lunch Lunch Lunch

Lecture –Course,

Sim, Team, Strategy

AAR (T&I)

Lessons Learned

(I)

TeamMATE - 1

Presentatns.

Portfolio

Finance, Metrics

Forecasting

Debrief - 1

Strategy / Team (T)

CR-7

AAR

LECTURE

8

DEBRIEF DECISIONS RESULTS

2

TeamMATE (T&I)

Decisions -1

AARAssignment


Metrics (T)

Presenta-tion

Delivery –

Simulation (T&I)

3

2 8

7

Results–CR-1

CR - 2

CR-6

CR-5

CR-4

CR-3

CR-8

6

5

4

3

25

4

6

Functnl. Analysis (T)


Course Summary

Presentation Preparation – Simulation

(T)

Team Contribu-

tion (I)



c. 11 weeks online


• Blackboard Collaborate

– Synchronous webinar (recordable)

– Team meetings

• Recorded lectures – asynchronous

– Tips – # of min., content only, no instructions or time references => reusable

–Mediasite (studio)

–Active Presenter

• Discussion Board

BlackBoard Course Tools (Online)


Sample Weekly Process – Thu. – Thu.Thu. Fri. Sat. & Sun.

Mon. Tue. & Wed. Thu.

Decisions DueRecorded Lecture

Headlines & Results

Recorded Debrief

After Action Review – Team –

w/o Results

360-degree Feedback: TeamMATE

(I – Decisions)

After Action Review – Indiv.

– w/ Results

360-degree Feedback: TeamMATE (T-Debrief)

360-degree Feedback:

TeamMATE (I –Reflection)

After Action Review – Team –w/ Results / Team Planning

Meeting

Assignment (T) Due

Individual Analysis

Live Webinar

Curriculum Sequence – 11 weeks OnlinePrework Week 1 Week 2 Week 3 Week 4 Week 5

Week 6 Week 7 Week 8 Week 9 Week 10 Week 11

PR – 1 Due

PR - 1Lecture 1

–Course,

Sim, Strategy

Results– PR1

AAR - 2

Assignments (I)–Lessons Learned

Team Contribution 2

TeamMATE - 1

7- Presents.

5 - Forecasting

4 - Metrics3 - Team

2 - Finance

Debrief 1PR – 2 Due

AAR - 1


Strategy / Team (T)

CR-7

PR - 2

AAR - 7

AAR - 6

AAR - 5AAR - 4

AAR - 3

RECORDED LECTURE

REC. DEBRIEF DECISIONS

RESULTS

TM 2TM 3

TeamMATE (T&I)

Decisions-1

AAR (T&I)Assignment


Team Cont 1

Presenta-tions –

Simulation (T&I)

Presetn. – Decision Making Tools (T)

3 Due2 Due

8 Due

7 Due

Results–CR-1

CR-2

CR-6

CR-5

CR-3

CR-8

6

5

3

5 Due

6 Due

Functnl. Analysis (T) Functnl. Analysis (T)


Webinar

Webinar

WEBINAR (rec.)

Webinar

6 - Portfolio

CR-4

4

4 Due

Webinar

Discussion Board

Discussion Board

PR - 2

Webinar

2

Metrics (T)

Residency (4-days)

• No practice rounds (team)

• Less time for content and reflection

• Immersion; different learning objectives/outcomes

• 2 TeamMATE evaluations

Online (11-weeks)

• Time needed up front to get students up to speed on sim understanding & navigation

• Discussion Board – reflection, real-life examples, simulation questions

• Team Collaboration – team & instructor communications / meetings

• Similar online group-work challenges

• Potentially less exercises / assignments

Face-to-Face (11-weeks)

• Option to use recorded lectures - reinforcement

• Flipped classroom – lectures, simulation decisions, round debriefs, real-world examples, exercises

Differences and Opportunities in 3 Formats


5. Questions

[email protected]


mailto:[email protected]

Documents

Using Simulation, 360-degree Feedback, a Balanced ... 2016 Drexel 2016 Assessment Conference September 8, 2016 Presented by: Drexel University, LeBow College of Business Jim Caruso,