1

THE EER: IS IT TRANSFORMATIONAL?

A PSYCHOLOGIST TURNED DIPLOMAT TAKES A CRITICAL LOOK AT THE CURRENT EER SYSTEM

Don Kilburg, Ph.D.

Transformational Diplomacy

& the EER In a key address at Georgetown University, Secretary of State Condoleezza Rice showcased “transformational diplomacy” (January 18, 2006). America needs a “diplomacy that not only reports about the world as it is, but seeks to change the world itself”, the Secretary advised. “We must transform old diplomatic institutions to serve new diplomatic purposes” and we must “prepare” and “challenge” our own diplomats with “new expertise” and “new expectations.” As both a diplomat and a psychologist, it occurred to me that if we want to advance this sort of diplomacy, we might need to take a critical look at how we formally shape ourselves as a diplomatic force: the Employee Evaluation Report, or EER. Is it transformational? To answer this question, I conducted a survey of over 600 Foreign Service employees and their experiences with the EER. I began this project after observing a common complaint about the EER system that suggests it is probably not transformational. Namely, the system is alleged to be poorly calibrated in its use of both praise and criticism. For example, many employees are shown to “walk on water”, others are “damned with faint praise”, and still others are just victims of unthinking evaluators. A system that praises and criticizes disproportionately would seem to ineffectively categorize employees, resulting in little transformation of the workforce for the better.

The temptation is to say: “Who cares? ‘Corridor reputation’ is what drives the real transformation of the workforce.” This conclusion may be valid, but it begs another question. Why spend so much time and energy on EERs if corridor reputation is what matters? Some estimate progress on other

work at our missions comes to a grinding halt for one or two months a year – simply to deal with the business of evaluating ourselves, and apparently ineffectively at that. Are EERs really worth it? Do they truly support the goals of transformational diplomacy?

An Empirical Approach to the EER

After seeing that no one had done any empirical research on the EER, I created an on-line survey to collect data on employees’ experiences with it. I wanted to know what factors go into getting a good EER and advancing in the Foreign Service. I hypothesized that one’s experience with the EER is based on much more than just his or her actual work performance. Rather it is based on the circumstances and approaches of the Rater and Reviewer, arbitrary background features of the employee, the dynamics of the EER process, and other extraneous factors. My survey consisted of 80 questions, mostly multiple-choice. The response rate was remarkably high and remarkably comprised of hires from the Diplomatic Readiness Initiative (DRI). I will focus on the DRI Generalists here.

This article is a summary of a lengthier, 20-page research report. In the full report I detail the methodology and a wider range of findings. I utilized standard survey techniques and statistical tools widely accepted in the field of psychology. If you would like a copy of the full report, please contact me at: eerstudy@yahoo.com.

Top 10 Key Findings 1) Much Seems Good about the Current EER System.

Most Generalists are satisfied with their EERs. A full 86% were either somewhat satisfied or very satisfied with the final outcomes of their EER Ratings. A full 82% of these Generalists were either somewhat

2

satisfied or very satisfied with the final outcomes of their own EER Reviews.

Further, most Generalists are satisfied with their bosses. Satisfaction with Raters ran relatively high among the Generalists. Roughly 79% liked their Raters as bosses at least “somewhat” and most liked them “very much”. Only 21% reported liking their Raters as bosses “very little”. As much as 92% liked their Raters as people at least “somewhat” and most liked them “very much”. Only 8% reported liking their bosses as people “very little”. The figures were very similar for Reviewers. 2) A Deeper Look Reveals an Unsatisfactory EER System. Most Generalists are quite dissatisfied with the EER system, despite being quite satisfied with their own Ratings and Reviews. A whopping 71% were either neutral about or dissatisfied with the current EER system. The bulk of the Generalists were either somewhat dissatisfied or very dissatisfied with the current EER system. Only 29% of the Generalists were satisfied with the current EER system. Procedural regulations for completing EERs are very poorly followed and Raters and Reviewers are not seen as being proactive. Only about half of the Generalists reported their Raters/Reviewers to be proactive in getting their EERs completed. Only 27% reported that the counseling dates on their EERs were accurate and another 27% reported their counseling dates did “not at all” correspond to any actual counseling dates. Only 44% reported getting a written counseling statement, something that is in theory supposed to document good performance as much as bad. Lastly, one in four of the Generalists reported not getting a Work Requirements Statement on time. The Generalists had many complaints about their EERs: two-thirds commented at length in the open-ended comment box at the end of the survey. The Generalists’ complaints were mainly about style, form, and timeliness. The most common complaint was that the EER was “not well-written in style/form” (32%). The second most common

complaint was that the EER was “not done as quickly as it could/should have been” (28%). Only 2% complained there was “too much criticism.” 3) An Unsatisfactory EER System has Negative Consequences.

Inaccurate counseling dates mean less satisfaction with one’s own EER and with the EER system. While the Generalists were quite satisfied with their own EER Ratings and Reviews overall, the more they reported their counseling dates as inaccurate and/or their written counseling sessions as absent, the less likely they were to be as satisfied with their Ratings, Reviews, and the EER system. Generalists are less satisfied with their EERs and the EER system when they see their bosses as low in proactiveness. The Generalists viewing their Raters and Reviewers as low in proactiveness were less satisfied all around, in terms of their own EER Ratings, EER Reviews, and the EER system in general. Generalists are less satisfied with their EERs and the EER system when they see their bosses as low in nurturant leadership. I measured the extent employees perceived their Raters as mentors or coaches “proactively nurturing professional development throughout the rating period.” In all cases, if employees viewed their Raters as low in this nurturing factor, they also reported lower levels of satisfaction with their EER Ratings, Reviews, and with the EER system as a whole. Lastly, dissatisfaction with the EER system is related to employees’ career contentment. Those who had a higher level of dissatisfaction with the current EER system had a lower level of interest in the Foreign Service as a career. If this relationship is causal, dissatisfaction with the EER system may quite possibly play a part in FS resignations. 4) Gender Matters. Females are less satisfied with their EERs. Though males and females did not differ significantly in their levels of satisfaction with the EER system, they did

3

differ in their levels of satisfaction with their own EER Ratings and Reviews. The mean satisfaction level for EER Ratings among males was 3.42; for females it was 3.22 (on a scale of 0 to 4). The mean satisfaction level for EER Reviews among males was 3.43; for females it was 3.18.

Females are less likely to be tenured on the first review. Females were reviewed for tenure an average of 1.40 times before getting tenured. In contrast males were reviewed an average of 1.28 times before getting tenured. 5) Grade Matters. Newer officers are especially dissatisfied with the EER system. Those who were relatively new to the FS (with less than nine years in) were less satisfied with the current EER system than their more experienced counterparts (with more than nine years in). That said, neither group was very satisfied with the system. The mean level of satisfaction among the new Officers was 1.7 on a satisfaction scale of 0 to 4 (where 2 is neutral), i.e. the new Officers were mainly dissatisfied. The mean level among more experienced Officers was 2.12; i.e. they were neutral. 6) Personal Liking Matters. Being liked as a person means being liked as an employee. Employees’ perceptions of how much their bosses liked them as people and how much they liked them as employees were positively correlated. Also, employees’ ratings of their bosses as people positively correlated with their ratings of their bosses as bosses. In short, there is strong evidence that people do not or cannot readily separate their feelings toward one another as people and as employees. I.e., if your boss likes you as a person, chances are he/she will like you as a subordinate, and vice versa. Further, if your subordinate likes you as a person, chances are he/she will like you as a boss, and vice versa.

Being liked as a person means getting a better EER. One’s likelihood of getting a good EER increases rapidly if one perceives he/she is liked on a personal level by his/her

boss. In this case “good” is defined by level of satisfaction one has with one’s EER. One’s likelihood of being more satisfied with one’s EER as correlated with perceiving being liked by one’s boss as a person also contributed to greater satisfaction with the EER system. In all statistical analyses of EER satisfaction, satisfaction increased as perception of being liked as a person increased. 7) Your Supervisor’s Writing Style Matters.

Rater and Reviewer writing styles are connected with employee tenure rates: the “story” format is the best. I compared satisfaction levels between groups of employees whose Raters and Reviewers had written their EERs in “list” versus “story” format. It has been said that the story format is more powerful in that it provides a chronological narrative of events, as opposed to a mere list or inventory of accomplishments. Those tenured officers who had the story format Rating had an average number of tenure reviews of 1.12, compared with their list format counterparts who were tenured on average in 1.38 reviews. A similar pattern was found for Reviews. 8) The Area for Improvement (AFI) Needs Improvement. Over a third of the Generalists (35%) did not think that their Areas for Improvement were germane to their actual performance. This would seem to imply there is a relatively large gap between Raters’ and their subordinates’ perceptions of the subordinates’ Areas for Improvement. Generalists appear disempowered or at least disinclined to respond to their AFI comments. Few chose to explicitly disagree with their Rater’s assessments: less than 2%. The bulk (40%) of the Generalists surveyed reported that they chose not to respond to the Area for Improvement at all in their EER statement, preferring to ignore it or to wait until the next EER to address it. Another 34% chose to agree with the comments in their Areas for Improvement and to grant the items as “something to work on”. The

4

remaining 25% or so chose to interpret their Areas for Improvement positively, with a “spin” or reframe of the item. How Generalists respond to AFI comments is implicated in rate of tenure: spin is most rewarded. When asked about the Area for Improvement, those who reported they “interpreted it positively, with a ‘spin’ or reframing of it” were tenured in the lowest average number of reviews (1.24), then those who “agreed with it explicitly, granting it as something to work on” (1.33), then those who “did not address it, preferring to ignore it or wait until next EER” (1.42), and finally those who “disagreed with it explicitly, offering a counterargument” (1.50).

9) EERs Are Negotiable. Requesting changes in EERs is connected to getting better EERs. It is clear from comparing EER complaints before and after requests for changes, that employees have significantly fewer complaints about EERs in the end – an estimated 30% less. The survey respondents collectively had 492 various, reported complaints about their own EER Ratings/Reviews before requesting changes. After requesting changes, the number of total complaints reported dropped to 349, a difference of 143 or 30%. In sum, it is safe to say that requesting changes from your Rater/Reviewer can dramatically reduce complaints you have about your EER, and in turn probably influence your own competitiveness vis-à-vis your peers. 10) What Generalists Want: Changes to the EER System. Most Generalists would like a 360-degree employee evaluation system considered. A whopping 92% of the Generalists reported that they would like at least a “little bit” the Department to consider changing the EER system to incorporate 360-degree evaluations. The bulk of the respondents (36%) “absolutely” wanted the Department to consider 360-degree evaluations, 25% “very much”, and 20% “a moderate amount”. Only 8% reported that they do “not at all” want the Department to

consider incorporating 360-degree evaluations. Most Generalists would like supervisors rated by subordinates. A remarkable 76% wanted to have evaluations of supervisors, by subordinates. That was the only solid agreement among the Generalists on the following 360-degree components. Fifty percent wanted to have evaluations of Americans by FSNs and/or LES employees. Forty-five percent wanted to have evaluations of same-level peers by same-level peers. Only 9% reported they wanted no additional types of evaluations within the 360-degree concept. Generalists are divided on whether quantitative measures should be added to the EER and what type. A small majority of 58% percent wanted some type of quantitative measure added. Twenty-nine percent supported scaled “grades” for employees along each of the six core competencies. Twenty-seven percent supported percentile rankings. Fourteen percent supported “within-the-person” rankings. Those who did not support quantitative measures were largely concerned about possible “grade inflation”.

Implications & Concluding Remarks Though the current EER system

results in EERs that employees are satisfied to receive, it hardly lends itself to transformational diplomacy. My research strongly confirms the hypothesis that one’s experience with the EER is based on the circumstances and approaches of the Rater and Reviewer, arbitrary background features of the employee, the dynamics of the EER process, and other extraneous factors.

I submit to you that there are structural changes that can be made to the EER system that would lead to a more transformed workforce. This would be a system wherein promising employees would advance faster than their mediocre counterparts and arbitrary factors affecting evaluation would be minimized and thereby less influential.

For starters, we need an EER system that has both qualitative and quantitative components, as well as multi-dimensional

5

perspectives on the employee. Many support “360-degree” type evaluations (and the Department has to its credit begun initiating some). One would think that supervisors and subordinates alike could better improve if they got more varied types of feedback from a wider range of employees surrounding them in the organizational structure.

Modern psychology has long held that qualitative and quantitative components of evaluation each contribute critical pieces of information that should be viewed jointly. Since quantitative components do not involve writing skills, adding one to the EER could be a useful and efficient means of addressing complaints regarding over-emphasis on Rater/Reviewer writing styles. Concerns about “grade inflation” could be mitigated by collecting data on average quantitative evaluations given by Raters/Reviewers, in order to provide context. Though we should consider adding other measures to the EER, we should also reduce overall the depth of these measures, shifting instead to more frequent, perhaps quarterly evaluations of a smaller, yet wider scale.

One way to both enhance and streamline the EER system could be to design and implement a new computer program which could be utilized by randomly selected members of a 360-degree rating panel whose members confidentially enter both qualitative and quantitative information into secured, on-line employee profiles, in a systematic fashion, orchestrated by Human Resources sections. Such a digital system would be easy to create and could advance our outdated, analog system significantly.

Regardless of whether a new system should be computerized, we might consider invoking the well-established core precepts as a foundation for additional rating components, to highlight employee strengths within the individual. Prompting questions could be taken directly from the six core precepts to stimulate quantitative, evaluative responses in order to arrive at scaled ratings for employees, within the core precepts and overall.

We might even systematically derive Areas for Improvement from the output new

scales could generate. The supervisor’s task would then not be so complicated in crafting the perfect Area for Improvement for the subordinate. It would be a matter of saying, “I see that you got your lowest 360-degree score in X core precept, and here is what I propose you do to raise that score.” Employees could then of course have the same, system-generated Areas for Improvement across EERs, as they worked to address pervasive problems and hence to better themselves.

Ultimately deciding what if anything we should do to improve the EER system depends on the answer to the question: what is our goal? Do we really want to transform our workforce to carry out the work of transformational diplomacy? Or do we want to continue shaping and reinforcing a “go along to get along” workforce? As Secretary Rice has said, “We must transform old diplomatic institutions to serve new diplomatic purposes.” I submit to you that transforming an old EER system is a key component to reaching that goal.

About the Author

Don Kilburg has been an FSO since 2003. He served in Mexico City and is moving onward to Santo Domingo with his wife Keely. He holds a doctorate in Experimental Social Psychology from DePaul University and a bachelor’s degree in Research Psychology from the University of Illinois. Before coming into the Foreign Service, he was a professor at Eastern Washington University and more recently at Saint Olaf College.