Valid Research Methods for Evaluating Recognition Programs

David J. Cherrington, SPHR, DBARecognition Professionals International Conference

27 April 2010Henderson, Nevada

Why Evaluate?Significant questions managers ought to ask:

How do our employees feel about our recognition programs? Which ones, if any, are having the biggest impact?

Are supervisors adequately trained to provide meaningful recognition?

Does the involvement of top management make any difference?

Is our annual recognition banquet worth the time and money we spend on it?

How do the people who are not recognized feel about our recognition program?

Three Strategic Research Questions:

• 1. Have we chosen the best recognition programs? (Selection and Design issues)

• Informal programs• Formal programs

• 2. How well have we implemented them? (Training, Communication, and Presentation issues)

• How much do employees know about them?• How often are they being used?• How do they make employees feel?

• 3. Do our recognition programs impact employee attitudes, productivity, or profitability? (Outcomes or Results issues)

Valid Research Requires Two Things:• 1. Reliable Data – also called dependent variables or metrics

– Research Question: “How do you assess reliability?”

• 2. A Valid Design – also called the research method – When should you collect the data?

• After the program is implemented• Before and after• Several times before and after

– From whom?• Experimental group• Control group

– How much?• Total sample• Random sample• Stratified random sample

– Research Question: “How do you assess validity?”

RELIABLE DATA

Reliability

• Definition: Reliability means repeatability or consistency of measurement. Reliable data are not random numbers, subjective estimates, or capricious measures.

• Operational Definitions: Test-retest reliability Alternate forms reliability Split-halves reliability Conspect reliability (inter-rater agreement)

Dependent Variables or Metrics

Employee Attitudes• Job satisfaction• Appreciation• Company satisfaction• Organizational commitment• Intent to leave• Perceived organizational

support

Organizational Metrics Absence Rate Turnover Rate HR Expense Factor Profit per employee Revenue per employee Cost of recognition

programs as a percent of payroll

Absence Rate:

The absence rate allows organizations to track employee attendance over time or to compare their number of lost days due to absences with the rates of others in their industry and nationwide.

Absence rate = Workdays lost due to absences

# of employees x # of days

Workdays Lost Due to Absence = full days of employee absence whether paid or unpaid, excused or unexcused and includes unscheduled days for sickness, personal business, emergency, family illness or death, disciplinary suspension, and unexcused absences. For long-term absences, count only the first four days. Do not count scheduled time off for holidays, vacations, and other leave.

Illustration

What is the Absence Rate for a company that has 60 full-time employees and during the past year they had 410 total days of absence. All employees have two weeks of paid vacation during the year.

410 days of absence

60 Employees x 250 days*= 0.027 or 2.70%

*250 days = 50 weeks per year times 5 days per week

Size of company 25th % tile Median 75th %tile

1-100 employees 0.84 1.34 2.18

101-500 0.61 1.27 2.12

501-1000 1.30 1.74 3.64

Over 1000 1.48 2.32 3.05

Industry

Construction 0.14 0.57 0.90

Manufacturing 0.67 1.48 2.22

Wholesale Retail trade 0.96 1.49 1.92

Transportation 1.05 1.60 2.07

Financial 0.79 1.27 2.21

Other services 0.56 1.50 2.51

Absence Rates by Size of Company and for selected Industries

Data provided by the Utah Employers Council, 2009

HR Functions that can be costedWayne Cascio: Costing Human Resources: The Financial Impact of Behavior in

Organizations, 3rd ed., Boston: Kent Pub. Co. 1991.

• Turnover• Absenteeism and sick leave• Smoking• Employee Assistance programs• Programs to improve Attitudes• Labor Contract Costs: wages, benefits, vacations, overtime,

shift differentials, holidays, pensions• Employee Selection Tests• Assessment Centers for Selection• Dollar Value of Job performance• Utility of Training Programs.

Criteria for Evaluating Training[From: Ralph F. Catalanello and Donald L. Kirkpatrick, “Evaluating Training Programs:

The State of the Art,” Training and Development Journal, 22, 1968, 2-9. ]

1. Reactions of participants, e.g. job satisfaction, appreciation, company satisfaction, organizational commitment, engagement.

2. Learning new knowledge and skills, e.g. test scores using multiple choice, fill in the blank, matching, and essay questions.

3. Behavior changes, e.g. quantity and quality of performance, attendance, job performance.

4. Measurable Results, e.g. organizational metrics.

Sources of Data

Questionnaires and surveys Interviews Observations Performance appraisals Unobtrusive Measures

Archive data Number of hits on the awards and catalogue websites Number of supervisors who complete online training Email responses from family members to say thank-you.

EMPLOYEE ATTITUDE MEASURES

Semantic Differential Scales

Extremely Quite Slightly Neutral Slightly Quite Extremely

Appreciated Unappreciated

Efficient Inefficient

Penalized Rewarded

Satisfied Dissatisfied

Unproductive Productive

Encouraged Discouraged

Ineffective Effective

Valuable Worthless

Me at Work – How I feel Most of the Time

The average of these semantic differential scales measures a “General Affective Tone.” See, William E. Scott, Jr. “The Development of Semantic Differential Scales as Measures of ‘Morale’,” Personnel Psychology, 20 (1967): 179-198.

AppreciationLikert Scales

Strongly Disagree

Disagree Neutral Agree Strongly Agree

The company expresses sincere gratitude to its employees.

My supervisor expresses appreciation to me for what I do.

I feel personally recognized and appreciated for my contribution to this company.

When I do a good job, I am praised and recognized.

Scales created by David J. Cherrington, Brigham Young University, Provo, UT, 84602

Job SatisfactionStrongly Disagree

Disagree Neutral Agree Strongly Agree

I find real enjoyment in my work.

I consider my job rather unpleasant.*

I am often bored with my job.*

I am fairly well satisfied with my present job.

I definitely dislike my work.*

Most days I am enthusiastic about my work.

What should be done to improve job satisfaction? ___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

* The scoring of this item is reversed.

From: Brayfield, A. H., & Rothe, H. F. (1951). “An index of job satisfaction.” Journal of Applied Psychology, 35, 307-311.

EngagementStrongly Disagree

Disagree Neutral Agree Strongly Agree

Performing my job is so absorbing that I forget about everything else.

Time passes quickly when I am performing my job.

I really put my heart into my job.

My own feelings are affected by how well I perform my job.

I exert a lot of energy performing my job.

I take work home to do.

From May, D., Gilson, R., & Harter, L. (2004). “The psychological conditions of meaningfulness, safety and availability and the engagement of the human spirit at work.” Journal of Occupational and Organizational Psychology. 77, 11-37.

Organizational Commitment: Affective Commitment

Strongly Disagree

Disagree Neutral Agree Strongly Agree

I would be very happy to spend the rest of my career with this organization.

I enjoy discussing my organization with people outside it.

I feel emotionally attached to this organization.

This organization has a great deal of personal meaning for me.

I feel a strong sense of belonging to my organization.

From: Allen, N. J., & Meyer, J. P. (1990). “The measurement and antecedents of affective, continuance and normative commitment to the organization.” Journal of Occupational Psychology, 63, 1-18.

Recognition Budget per Employee

• The recognition budgets for four companies that have been recognized for outstanding recognition programs are:– A telecommunications company with 36,000

employees budgets 1.0 % of payroll– An international bank with 60,000 employees

budgets 0.7% of payroll– A healthcare company with 3,400 employees budgets

0.2 % of payroll– A bank with 23,000 employees budgets $8 mil (which

equals $348 per employee).

VALID RESEARCH DESIGNS

Research Designs~1. Post-test1. Post-test––only design (case study)only design (case study)

X OX O

2. Pretest2. Pretest––post-test comparisonpost-test comparison

O X OO X O

3. Pretest3. Pretest–post-test control group design–post-test control group design

R O XR O X OO

R OR O –– O O

4. Post-test4. Post-test–only control group design–only control group design

R R –– X OX O

RR –– –– OO

5. Solomon four-group design5. Solomon four-group design

R O X OR O X O

R OR O –– OO

RR –– X OX O

RR – – –– OO

6. Time-series design6. Time-series design

OO11 O O22 O O33 X O X O44 O O55 O O66

7. Separate sample pretest7. Separate sample pretest–post-test –post-test control group designcontrol group design

Group 1 Group 1 R O X R O X ––

R R – – X O X O

Group 2 Group 2 R O R O –– ––

R R – – –– O OKeyX = Experimental treatment or training programO = Observation or measurementR = Random assignment of participants to group/training condition

Validity

• Definition: Do the results measure what you think they are measuring? Can you interpret the results accurately?

• Two types of validity:– Internal validity: is the experimental design sound

so that the data can be properly interpreted?– External validity: can the results of this study be

generalized (or applied) to other situations?

1. Case Study Design

• X O You implement the program and then measure it. X = independent variable, e.g.

• New recognition program• Training program• New incentive program

O = dependent variable• “smile sheets” after a training program• Participation in the peer-to-peer nominations• Measures of job satisfaction or engagement

• A very weak but highly popular research design

2. Pretest-Posttest Comparison• O X O• Illustration: Changes in turnover or job satisfaction before

and after a new recognition program is initiated. ???• Problems with this design:

Mortality – some people drop out, usually the dissatisfied. Maturity – people constantly change during any period of time. History – unique external events can influence the measures. Instrument decay – measurement errors and unreliable data. Sensitizing effect of the pretest – measuring is not a neutral

event; people often behave differently after they are measured.• This is a very weak research design.

3. Pretest-Posttest Control Group Design

• R O X O• R O -- O

R = random assignment to groups X = new recognition program, such as a new kit for

supervisors to use. O = number of thank you notes sent to subordinates

• This is an excellent research design when it can be implemented.

4. Posttest Only Control Group Design

• R X O• R -- O• This design assumes that the two groups are

equivalent due to random assignment.• This is an excellent research design that is

especially appropriate when pretesting is not possible.

• Illustration: testing the effects of a new employee orientation training on participation in the peer-to-peer recognition program.

5. Solomon 4-group Design• R O X O R O – O

R - X OR - - O

• This is the ideal research design because it controls for all of the challenges to the internal validity of a research design.

• Illustration: testing the impact of a training program on “Developing a Culture of Recognition” on employee attitudes, such as empowerment, satisfaction, or appreciation.

6. Time Series Design• O O O O O O X O O O O O O• This design fails to account for the effects of history. • Otherwise, this research design is ideal for examining

systematic changes in an ongoing program. Pretest measures serve as a contrast for posttest measures.

• Illustrations: – awarding points to nominators, – increasing the points for supervisors, – increasing the percentage of employees recognized, – adding new merchandise to the catalogue,– Creating a “wall of fame”– Including a recognition column in the company’s newsletter

7. Quasi-Experimental Design• R O X O O – O

R O X O O – O

This design uses groups that are already formed and members cannot be randomly assigned.

Illustration: A new recognition program cannot be implemented to randomly selected members of a group. Therefore, randomly select some groups to start the program first, while other groups have to wait for six months before starting.

8. Path Analysis• This design utilizes a regression analysis of

multiple variables to test the relationships between them.

• The unit of analysis may be groups, such as testing whether the average satisfaction scores of employees in various divisions or departments predict levels of customer satisfaction in those divisions or departments.

• Path analysis tests the relationships in a hypothetical model, also known as structural equation modeling.

Hypothesized Recognition Model

• Do recognition programs influence employee attitudes?

• Do improved employee attitudes lead to improved employee performance?

• Does better employee performance impact the company’s bottom line?

Employee Recognition

Improved Employee Attitudes

HigherEmployee

performance

Increased Profits

Do employee recognition programs contribute to the bottom line?

Possible Metrics:Absence RateTurnover Rate

HR Expense FactorProfit per employee

Revenue per employee

“Applause” Program: ROI Linkage Analysis

Applause Participation

EmployeeSatisfaction

Customer Loyalty

Financial Return

Higher Applause participation is linked to higher

employee satisfaction

Growth in Applause = 3-4% growth in employee satisfaction

6-9% difference in employee

satisfaction = 3-4% difference in

customer satisfaction

1% change in customer

satisfaction = 3.4% change in revenue