Upload
trinhthien
View
228
Download
1
Embed Size (px)
Citation preview
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 1
HR Avatar Assessment Solution
Technical Manual
August 31, 2016
www.hravatar.com 2810 Thaxton Lane Oakton, VA 22124
703-349-3070
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 2
Table of Contents
Overview ........................................................................ 8
Testing Standards .......................................................... 8
Solution Summary .......................................................... 9
Cognitive Work Simulations ........................................ 10
Previous Research ........................................................................................................................................................ 10
Content Development ................................................................................................................................................. 12
Attention to Detail ............................................................................................................................................. 12
Analytical Thinking ............................................................................................................................................ 12
Reliability ....................................................................................................................................................................... 13
Validity ........................................................................................................................................................................... 16
Fairness .......................................................................................................................................................................... 17
Attitudes, Interests and Motivations Assessment ........ 19
Previous Research ........................................................................................................................................................ 19
Content Development ................................................................................................................................................. 21
Needs Structure .................................................................................................................................................. 21
Innovative and Creative .................................................................................................................................... 21
Enjoys Problem-Solving ................................................................................................................................... 22
Competitive ......................................................................................................................................................... 22
Seeks Perfection ................................................................................................................................................. 22
Develops Relationships ..................................................................................................................................... 23
Expressive and Outgoing ................................................................................................................................. 23
Corporate Citizenship ....................................................................................................................................... 23
Exhibits a Positive Work Attitude .................................................................................................................. 23
Adaptable ............................................................................................................................................................. 23
Reliability ....................................................................................................................................................................... 25
Validity ........................................................................................................................................................................... 25
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 3
Fairness .......................................................................................................................................................................... 26
Behavioral History Survey ........................................... 29
Previous Research ........................................................................................................................................................ 29
Content Development ................................................................................................................................................. 29
Performance ........................................................................................................................................................ 30
Tenure .................................................................................................................................................................. 30
Unproductive Behavior ..................................................................................................................................... 30
Reliability ....................................................................................................................................................................... 30
Fairness .......................................................................................................................................................................... 31
Knowledge and Skills Tests ......................................... 33
Development Overview .............................................................................................................................................. 33
Sales Situation Analysis................................................ 34
Test Development ....................................................................................................................................................... 34
Reliability ....................................................................................................................................................................... 35
Fairness .......................................................................................................................................................................... 36
Typing Test .................................................................. 36
Test Development ....................................................................................................................................................... 36
Reliability ....................................................................................................................................................................... 36
Fairness .......................................................................................................................................................................... 37
Essay Test .................................................................... 37
Test Development ....................................................................................................................................................... 37
Descriptives ................................................................................................................................................................... 39
Fairness .......................................................................................................................................................................... 39
Solution Scoring ........................................................... 40
Construct Validity Evidence ...................................................................................................................................... 41
Technical Requirements .............................................. 50
Future Research ........................................................... 50
Reliability ....................................................................................................................................................................... 50
Validity ........................................................................................................................................................................... 51
Fairness .......................................................................................................................................................................... 51
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 4
Norms ............................................................................................................................................................................ 51
References .................................................................... 52
Appendix A: Summary of the HR Avatar Solutions ..... 57
Appendix B: Historical Validity Evidence for the Cognitive Scales ............................................................................ 72
Appendix C: Historical Validity Evidence for AIMS ... 73
Appendix D: Scoring Rubric for Essays ...................... 78
Appendix E: Directions for Rating Essay Tests ........ 799
Appendix F: Automated Essay Scoring System .......... 82
Appendix G: Validation Studies ................................... 85
Appendix H: Emotional Intelligence Test .................. 88
Appendix I: Work Competency Test ........................... 94
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 5
Tables Table 1. Demographic Characteristics of Sample ...................................................................................................... 15
Table 2. Participant Location ......................................................................................................................................... 15
Table 3. Descriptive Statistics and Reliability Evidence for the Cognitive Workplace Simulation Scales........ 16
Table 4. Evaluation of Cognitive Score Differences by Gender ............................................................................. 18
Table 5. Evaluation of Cognitive Score Differences by Age Group ...................................................................... 19
Table 6. Descriptive Statistics and Reliability Evidence for the AIMS Scales ....................................................... 25
Table 7. Evaluation of AIMs Score Differences by Gender .................................................................................... 26
Table 8. Evaluation of AIMs Score Differences by Ethnicity ................................................................................. 27
Table 9. Evaluation of AIMs Score Differences by Age Group ............................................................................. 27
Table 10. Evaluation of AIMs Score Differences by Race Groups: Asian and White ........................................ 28
Table 11. Evaluation of AIMs Score Differences by Race Groups: Black or African American and White .. 28
Table 12. Descriptive Statistics and Reliability Evidence for the Professional Behavioral History Survey ..... 31
Table 13. Descriptive Statistics and Reliability Evidence for the Entry-Level Behavioral History Survey ...... 31
Table 14. Evaluation of Biographical History Survey Score Differences by Gender .......................................... 32
Table 15. Evaluation of Biographical History Survey Score Differences by Ethnicity ....................................... 33
Table 16. Evaluation of Biographical History Survey Score Differences by Age Group ................................... 33
Table 17. Evaluation of Biographical History Survey Score Differences by Race Groups: Asian and White 33
Table 18. Descriptive Statistics and Reliability Evidence for the Sales Situation Analysis Assessment ........... 36
Table 19. Evaluation of Sales Situation Analysis Score Differences ....................................................................... 36
Table 20. Descriptive Statistics and Reliability Estimates for the Typing Test ..................................................... 37
Table 21. Evaluation of Essay Score Differences by Subgroup .............................................................................. 37
Table 22. Descriptive Statistics for Essay Scores ....................................................................................................... 39
Table 23. Evaluation of Essay Score Differences by Subgroup .............................................................................. 40
Table 24. Correlations between the AIMs Scales ....................................................................................................... 42
Table 25. Correlations between the Professional Behavioral History Scales......................................................... 43
Table 26. Correlations between the Entry-Level Behavioral History Scales ......................................................... 43
Table 27. Correlations between HR Avatar AIMs and Behavioral History Scales ............................................... 44
Table 28. Correlations between the Two Cognitive Workplace Simulation Scales: Attention to Detail and
Analytical Thinking .......................................................................................................................................................... 45
Table 29. Correlations between the AIMs and the Cognitive Workplace Simulation Scales ............................. 46
Table 30. Correlations between the Behavioral History Survey and the Cognitive Workplace Simulation
Scales .................................................................................................................................................................................. 48
Table 31. Compiled Validity Evidence for the Original Content of the Cognitive Work Simulation Scales .. 72
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 6
Table 32. Study 1 Results: Concurrent Validation Study for Insurance Consultants N=122-136 .................... 73
Table 33. Study 2 Results: Concurrent Validation Study for Inside Sales N=105 ............................................... 74
Table 34. Study 3 Results: Concurrent Validation Study for an Internet Services Order Processing N=84 ... 74
Table 35. Study 4 Results: Concurrent Validation Study for an Internet and Cable Sales and Service Position
N=72-93 ............................................................................................................................................................................ 74
Table 36. Study 5 Results: Concurrent Validation Study Auto Rental Sales Role N=80-92 .............................. 74
Table 37. Study 6: Concurrent Validation Study for Paramedics N=85 ................................................................ 75
Table 38. Summary of Relationships between Performance Measures and Original AIMS Scales ................... 76
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 7
Figures Figure 1. Summary of HR Avatar Solution ................................................................................................................. 10
Figure 2. Cognitive Work Simulation ........................................................................................................................... 13
Figure 3. Hypothesized Relationships between the AIMs Scales and the Big Five ............................................. 24
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 8
Overview
The HR Avatar Employment Assessment series was designed as a flexible assessment instrument for
helping corporations obtain quality hires by measuring cognitive abilities, biographical data, personality
characteristics and job knowledge related to performance and tenure in the workplace. To cater to differing
individual client needs, HR Avatar offers both individual assessments as well as complete assessment
solutions. By combining assessments and measuring multiple competencies, the assessment solutions allow
for a broader evaluation of overall candidate fit with a particular job.
This technical manual contains a summary of testing standards for the development of psychological
assessments in the workplace and an overall summary of the development of the assessment solution. There
are sections for each assessment within the solution including previous research, content development, and
reliability evidence, validity evidence, and evidence for fairness. The technical manual concludes with a
summary of HR Avatar’s research agenda.
Testing Standards
The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 2014), the Principles for the Validation and Use of Personnel Selection Procedures, (Society for Industrial Organizational Psychology, 2003), the Uniform Guidelines on Employee Selection Procedures (Equal Employment Opportunity Commission, C. S. C. U. S. D. L. U., & Equal Employment Opportunity Commission, 1978), and Testing and Assessment: An Employer’s Guide to Good Practices (Saad, Carter, Rothenberg, & Israelson, 2000) are all documents that “govern” the development and use of tests in employment settings. The developers of the HR Avatar Employment Assessment series have made, and continue to make, efforts to adhere to recommendations for test development put forth by these documents. The evidence produced in this technical manual is a summary of all of the theoretical and empirical research done on the tool to date. We anticipate frequent updates as we complete additional research. The following list summarizes the key test development standards from the above listed documents. The documents include additional standards governing test usage and we encourage test users to review and adhere to these standards.
Define the psychological construct measured by the test
Document the intended use of the test
Document how test scores should be interpreted
Demonstrate validity for the specific use of the assessment and interpretation of scores including:
o Previous research on the construct
o Previous empirical research that demonstrates the extent to which the validity evidence may
be generalizable (e.g. meta-analytic studies)
o Research supporting the link between the content of the assessment and the tasks,
knowledge, skills, and abilities required for the job (i.e., content-related validity evidence)
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 9
o Research supporting a specific response process takes place if it is inferred in the use or
interpretation of the test
o Evidence that the internal structure of the assessment conforms to theoretical expectations
o Relationships between scores and any outcomes the assessment claims to be able to predict,
such as job performance (i.e. criterion-related validity evidence)
o Evidence demonstrating that the test is statistically related to other measures in ways in
which are consistent with theoretical expectations (i.e., convergent and divergent validity
evidence)
Document sufficient details related to validity research in order to allow the reader an opportunity to
independently evaluate the quality of research
Document reliability estimates and the Standard Error of Measurement (SEM) for each score,
subscore, and combination of scores
Use a type of reliability estimate (e.g., internal consistency, test-retest) that is appropriate for the test
Document sufficient details related to reliability research in order to allow the reader an opportunity
to independently evaluate the quality of research
Design test in such a way that it allows all test takers an equal opportunity to demonstrate their
standing on the construct regardless of subgroup membership (e.g., age, disability status, ethnicity,
gender, age, or race)
Provide empirical evidence demonstrating that the construct is measured the same way across
subgroups
Solution Summary
The HR Avatar Employment Assessment series has been designed for use in identifying or screening out
those job applicants with the lowest potential for success in a given role by assessing job-relevant cognitive
abilities, personality characteristics, behavioral background, and knowledge and skills. There are currently
203 HR Avatar solutions (see Appendix A) in the HR Avatar Employment Assessment series. Each solution
has been designed for specific job roles and contains several components: one of 15 different Cognitive
Work Simulations, one of two different Attitudes, Interests, and Motivations assessment forms (AIMs), one
of two different Behavioral History Surveys, and job-specific Knowledge and Skills assessments. The 15
Cognitive Work Simulations represent 15 distinct job families and are discussed in more detail below. The
AIMs assessment and the Behavioral History Survey are both available in two forms. One form is designed
for professional positions and the other is designed for entry-level positions.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 10
Figure 1. Summary of HR Avatar Solution
The assessments are delivered via a computer or other mobile device via the internet. Applicants are sent an
email with a link to take the assessment. Once the applicant clicks on the link in the email, they are brought
to a page in their web browser. The assessment begins with an animated “host”. The host guides the
applicant through the process and provides instructions prior to the start of each assessment. The animated
host is used to provide the applicant with a more engaging and positive assessment experience. Most
solutions take about 40 minutes to complete.
Prior to implementing the solution for selecting employees, HR Avatar recommends completing a job
analysis to ensure that the competencies measured by the HR Avatar assessments are important for success
in your organization. Additionally, HR Avatar recommends that the organization conduct a pilot study to
examine the relationships between the assessments and relevant job performance criteria and to evaluate the
potential for adverse impact. This pilot study should be done prior to using the assessment for screening out
employees.
Cognitive Work Simulations
Previous Research
Those individuals with higher cognitive abilities are more equipped to solve complex problems and learn
new skills. Cognitive ability is necessary for successful performance on a broad range of tasks including
written documentation, oral communication, identifying and managing details, solving problems, reading
•Measures 1-2 specific cognitive abilities
•1 of 15 job family-specific versions
Cognitive Work Simulation
•10 personlity scales
•2 versions (professional and entry-level)
Attitudes, Interests, Motivations
•2 biographical data scales
•2 versions (professional and entry-level)
Behavioral History Survey
•1 or more of 55 job-specific knowledges tests
•Sales Situation Analysis
•Typing Test
•Essay Test
Knowledge and Skills tests
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 11
and responding to email messages, quickly learning and applying new information, quantitative
computations and analyses, and making well-reasoned decisions
The published research evidence consistently demonstrates a strong, predictive relationship between
cognitive ability, or general mental ability, and job performance and training success. In fact, several meta-
analyses have been completed, summarizing and combing the results of many research studies examining
the validity of cognitive ability (Hunter & Hunter, 1984; Schmidt & Hunter, 1998) and the results generalize
across geographical regions (Bertua, Anderson, & Salgado, 2005; Salgado & Anderson, 2003; Salgado,
Anderson, Moscoso, Bertua, & De Fruyt, 2003a; Salgado et al., 2003b). The operational validity estimates
from these meta-analytic studies tend to be in the .5-.6 range. Though cognitive ability predicts performance
at all job levels, the relationship is moderated by complexity such that the relationship is stronger for more
complex jobs (Bertua et al., 2005; Salgado et al., 2003b).
Currently, the most commonly accepted taxonomy for cognitive abilities is the CHC model (Cattell, 1941;
Horn, 1985; Carroll, 1993). The emergence of this accepted hierarchical taxonomy came from the merging
of two independent lines of research examining the taxonomy and structure of cognitive abilities (McGrew,
2008). The taxonomy posits general cognitive ability at Stratum I, broad abilities at Stratum II, and more
specific abilities at Stratum III. For the prediction of job performance, we are most interested in measuring
abilities at the second stratum as these are deemed to be broad enough to apply to multiple positions, but
specific enough to yield incremental prediction in job performance. Furthermore, recent research has
demonstrated that using specific abilities in selection decisions can reduce adverse impact without
decreasing the validity of the selection process (Kehoe, 2002; Wee, Newman, & Joseph, 2014).
The broad abilities listed in Stratum II consist of: fluid reasoning, comprehension-knowledge, short-term
memory, visual processing, auditory processing, long-term storage and retrieval, cognitive processing speed,
decision and reaction speed, reading and writing, quantitative knowledge, general (domain-specific)
knowledge, tactile abilities, kinesthetic abilities, olfactory abilities, psychomotor abilities, and psychomotor
speed (McGrew, 2008). Based on our understanding of these abilities, we believe that fluid reasoning, short-
term memory, reading and writing, quantitative knowledge, and general (domain-specific) knowledge would
be the most generalizable across jobs. However, the general (domain-specific) knowledge was determined to
be better measured by the knowledge and skills assessments. HR Avatar designed two scales for each job
family: Attention to Detail and Analytical Thinking. The development of these scales is discussed further on
the content development section, but, to summarize, Attention to Detail taps short-term memory, whereas
Analytical Thinking is more closely related to fluid reasoning and quantitative knowledge. Both scales
require that applicants read information, therefore, to some extent, the reading component of reading and
writing is also measured by both scales.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 12
Current meta-analytic research has examined the relationship of these specific abilities with overall job
performance and training performance. In a European sample, Salgado et al. (2003b) found that perceptual
ability and memory (similar to the competency measured by the Attention to Detail scale) had corrected
mean validity coefficients of .52 and .56 with job performance and .25 and .34 with training performance,
respectively. Additioanlly, they found that verbal ability and numerical ability (similar to the competency
measured by the Analytical Thinking scale) had corrected mean validity coefficients of .35 and .52 with job
performance and .44 and .48 with training perofrmance, respectively. In a UK sample, Bertua et al. (2005)
found that perceptual abilities had a corrected mean validity coefficient of .50 with job performance and .50
with training performance. Verbal ability and numerical ability had corrected mean validity coefficients
of .39 and .42 with job performance and .49 and .54 with training perofrmance, respectively.
Content Development
The Cognitive Work Simulations measure two cognitive competencies: Attention to Detail and Analytical
Thinking. Attention to Detail most represents short-term memory from the CHC model and Analytical
Thinking is a combination of fluid reasoning and quantitative reasoning.
Attention to Detail
The competency is defined by the ability to process information, recall information accurately, and identify
appropriate resources for locating specific pieces of information. This competency is important for data
entry, identifying errors in data, processing orders, working with tables, understanding policies and
procedures, and working with numerical data such as the type found in financial reports.
Analytical Thinking
The competency is defined by the ability to understand language and the numerical relationships. This
competency is characterized by the ability to process complex information, synthesize data, identify and
solve problems, and make well-reasoned decisions.
The Cognitive Work Simulation requires that candidates complete various job-related tasks in a virtual
environment designed to replicate the workplace. There are 15 versions of the Cognitive Work Simulation
and each is designed to represent scenarios common to a specific job family (listed in Table 3). Candidates
are evaluated on how they respond to avatars representing customers and colleagues and how well they
solve problems in various work-related scenarios. The realistic experience is designed to be more engaging
for applicants and to provide a better measure of ability. During a simulation, the candidate might be
required to read email, listen to voicemail and perform basic keyboard and screen navigation tasks to solve
typical business problems.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 13
Figure 2. Cognitive Work Simulation
The items used in the Cognitive Work Simulation were based on an existing library of ability items with a
history of predicting job performance in various workplace settings. Both the original items and the new
content were developed by examining the results of multiple job analyses. There are between 5 and 19 items
in each competency measured by the simulation. Responses to assessment items are scored dichotomously.
In some items, there is more than one correct response. In these situations, an applicant is awarded a point
for each correct response selected. It takes approximately 18-20 minutes to complete a simulation. The
estimates were calculated using a sample of 264 individuals.
Reliability
To evaluate the properties of the cognitive assessments as well as other assessments in the solution, HR
Avatar conducted a research study. Participants from the study were recruited via Mechanical Turk (MTurk).
MTurk is a platform operated by Amazon.com and allows the user to pay workers for completing for
individual tasks called Human Intelligence Tasks (HITs). MTurk is becoming increasingly popular for use in
research in the social sciences including research related to pre-employment testing. Many researchers have
found that MTurk provided adequate samples for the purposes of developing assessments (Buhrmester,
Kwang, & Gosling; Johnson & Borden, 2012; Minton, Gurel-Atay, Kahle, & Ring, 2013; Miller, Gentile,
Wilson, & Campbell, 2013).
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 14
HR Avatar recruited workers on MTurk to complete one of 15 versions of the solution– one for each
version of the Cognitive Work Simulation. The Cognitive Work Simulation was combined with an
appropriate AIMs assessment form, and an appropriate Behavioral History Survey Form. The Knowledge
and Skills assessments were not included in this study. Participants were allowed to take each version once.
Instructions to participants warned them that if they responded to the questions randomly, their HIT would
not be approved, and therefore they would not get paid. Workers were promised an assessment report upon
completion of the assessments. Both of these approaches should have led to more genuine responses.
Additionally, to ensure responses to the assessment were legitimate, the data used in the analyses were
restricted to those individuals who, on average, took 2.5 seconds or more to respond to items (N=675).
These individuals completed 1682 solutions. On average, participants took 2.49 solutions (SD=2.70). A
summary of the demographic information for this sample can be found in Table 1 along with 2013 estimates
(the most recent available) of the demographic breakdown of the U.S. population as reported by the U.S.
Census Bureau (2015). As compared to the U.S. population, the sample in this research contains a higher
proportion of individuals who are male, non-Hispanic and White. The average age of the sample is 33.85
years (SD=10.88) which is also slightly younger than the median age (37.5) of the U.S. population (U.S.
Census Bureau, 2015). Given the differences between the sample and the overall population, the results in
this study are to be regarded as preliminary an interpreted with caution. Table 2 provides a summary of the
geographic areas represented by the sample.
Table 3 contains the descriptive statistics for each scale and the alpha estimates of internal consistency.
Please note, based on these findings, further analyses were done at the item and distractor level. These
analyses led the developers to make substantial edits to these items and we anticipate that edits will
substantially improve the reliability of the tools. Further research evaluating the effectiveness of these
changes is forthcoming.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 15
Table 1. Demographic Characteristics of Sample
Gender N %* U.S. Only 2013 est. from U.S. Census**
Female 287 43.03 43.76 51.4
Male 380 56.97 56.24 48.6
Unknown 8
Ethnicity N %* U.S. Only 2013 est. from
U.S. Census**
Hispanic 46 7.42 6.52 17.1
Not Hispanic 574 92.58 93.48 82.9
Unknown 55
Race N %* U.S. Only 2013 est. from U.S. Census*
American Indian or Alaskan Native 3 .45 .52 .8
Asian 89 13.46 6.55 5.1
Black or African American 25 3.78 4.31 12.6
Native Hawaiian or Other Pacific Islander 5 .76 .69 .2
White 518 78.37 84.14 73.7
Two or More Races 22 3.33 3.79 3.0
Unknown 14
*Percentages calculated by summing those for whom demographic data were available.
**U.S. Census Bureau, 2013 American Community Survey
Table 2. Participant Location
Country N %
United States 589 87.26
India 54 8.00
UK 10 1.48
Other 22 3.76
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 16
Table 3. Descriptive Statistics and Reliability Evidence for the Cognitive Workplace Simulation Scales
Cognitive Simulation
Analytical Thinking
Attention to Detail
# of Items
N M SD Alpha # of Items
N M SD Alpha
Basic Entry-Level 19 171 16.68 2.08 0.70 5 171 4.20 1.05 0.53
Face-to-Face Customer Service
6 116 3.80 1.88 0.74 8 116 6.78 1.34 0.53
Administration 5 120 2.62 1.12 0.26 11 119 7.34 2.25 0.67
Business Sales 6 98 3.37 1.53 0.53 8 98 5.96 1.96 0.68
Business/Finance 7 122 3.04 1.36 0.27 6 121 3.63 1.33 0.43
Entry-Level Administration
N/A N/A N/A N/A N/A 10 100 7.38 2.32 0.74
Entry-Level Business & Finance
7 139 2.38 1.84 0.68 6 137 3.83 1.43 0.59
Entry-Level Office 26 142 20.46 4.75 0.86 13 142 9.11 2.71 0.75
First-Line Supervisor 9 102 6.42 1.59 0.47 8 102 5.84 1.85 0.68
General Office Workplace
7 100 4.04 1.63 0.42 6 99 4.67 1.40 0.68
Information Technology
7 98 4.45 1.94 0.65 7 98 5.46 1.30 0.52
Manager 10 98 5.41 1.80 0.38 5 97 3.57 1.16 0.44
Remote Customer Service
6 98 3.45 1.52 0.50 10 98 6.66 2.28 0.71
Technician 8 114 4.77 2.94 0.83 9 114 19.22 3.26 0.73
Retail Sales 10 60 4.97 2.48 0.68 7 60 7.32 1.86 0.66
Validity
As mentioned above, the simulations were developed based on an existing library of content. The original
developer has provided the results of historical validation studies done with the original content and these
are provided in Appendix B.
Validation evidence is available in two studies that demonstrate the validity of the assessments. The first is a
study with 130 managers and the second is a study with 64 managers. In the first study, the assessment
demonstrated that it predicts job performance (r=.25; p<.01), using a simple “High” or “Marginal”
categorization of job performance, and a second study demonstrated the assessment predicted performance
on a 100 point administrative performance appraisal, using a Spearman correlation (r=.25; p<.05). We
expect validity evidence to be more robust in future studies with better criterion measures. A brief summary
of the results is presented in Appendix G.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 17
Fairness
In order to evaluate how different demographic subgroups perform on the assessments, independent
samples t-tests were performed to compare mean scores of the subgroups. The results are provided in Table
4 and Table 5. Included in the tables are the descriptive statistics for each subgroup, the t-statistic (used for
evaluating whether or not the mean differences are statistically significant), and Cohen’s d. Cohen’s d is an
effect size and an indication of how large the mean differences are. For reference, according to Cohen
(1992), effect sizes of .2 are small, .5 are medium, and .8 are large. Although data were collected on ethnic
group, the sample sizes for the ‘Hispanic or Latino’ subgroup were too small for analyses (n<15). Data were
also collected for racial group. However, the sample sizes for all but the “White” subgroup were too small
for analysis (n<30). Future research will examine subgroup differences as more data become available.
For gender, there were statistically significant differences in favor of males on three scales (Entry-Level
Office: Attention to Detail, First-Line Supervisor: Analytical Thinking, and Information Technology:
Analytical Thinking). However, there does not appear to be a strong trend in males scoring higher than
females and it is possible that edits made to the items will diminish differences on the scales where
differences have been detected. For age, there were statistically significant differences on three scales
(Business Sales: Analytical Thinking, First-Line Supervisor: Attention to Detail, and Manager: Analytical
Thinking), however these were all in favor of the protected group. Score differences will be further
evaluated as more data become available on the new version of the assessments and subgroup differences
will be evaluated for ethnic and race subgroups as well.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 18
Table 4. Evaluation of Cognitive Score Differences by Gender
Cognitive Simulation Scale Female Male
n M SD n M SD t d
Basic Entry-Level Analytical Thinking 72 16.57 2.54 97 16.74 1.67 0.52 .08
Attention to Detail 72 4.07 1.18 97 4.32 0.94 1.53 .24
Face-to-Face Customer Service
Analytical Thinking 50 3.76 2.06 66 3.83 1.75 0.20 .04
Attention to Detail 50 6.68 1.27 66 6.85 1.39 0.68 .13
Administration Analytical Thinking 41 2.34 1.15 79 2.76 1.09 1.96 .38
Attention to Detail 41 7.05 2.40 78 7.48 2.17 0.99 .19
Business Sales Analytical Thinking 41 3.34 1.44 56 3.34 1.58 0.00 .00
Attention to Detail 40 6.10 2.05 53 5.83 1.91 -0.65 -.14
Business/Finance Analytical Thinking 52 2.88 1.28 69 3.12 1.38 0.98 .18
Attention to Detail 51 3.71 1.30 69 3.57 1.37 -0.57 -.10
Entry-Level Administration
Attention to Detail 43 7.35 2.50 56 7.37 2.21 0.04 .01
Entry-Level Business/Finance
Analytical Thinking 50 2.28 1.76 89 2.44 1.88 0.49 .09
Attention to Detail 50 3.92 1.31 87 3.78 1.50 -0.55 -.10
Entry-Level Office Analytical Thinking 58 19.78 4.67 83 21.02 4.73 1.54 .26
Attention to Detail 58 8.52 3.04 83 9.57 2.38 2.30* .39
First-Line Supervisor Analytical Thinking 42 6.02 1.59 59 6.69 1.57 2.10* .42
Attention to Detail 42 5.97 1.88 59 5.80 1.79 0.46 -.09
General Office Workplace
Analytical Thinking 44 4.00 1.71 55 4.04 1.57 0.12 .02
Attention to Detail 44 4.66 1.57 54 4.67 1.27 0.03 .01
Information Technology
Analytical Thinking 42 3.98 1.81 54 4.85 1.98 2.22* .46
Attention to Detail 42 5.21 1.20 54 5.65 1.38 1.64 .34
Manager Analytical Thinking 36 5.22 1.87 62 5.52 1.76 0.80 .17
Attention to Detail 35 3.40 1.09 62 3.66 1.20 1.06 .22
Remote Customer Service
Analytical Thinking 40 3.25 1.55 58 3.59 1.50 1.09 .22
Attention to Detail 40 6.88 2.23 58 6.52 2.32 -0.77 -.16
Technician Analytical Thinking 26 5.35 2.58 31 4.81 2.46 -0.81 -.21
Attention to Detail 26 7.73 1.82 31 7.23 1.69 -1.07 -.29
Retail Sales Analytical Thinking 44 4.41 3.31 69 5.03 2.69 1.09 .21
Attention to Detail 44 18.69 3.59 69 19.59 3.01 1.44 .28
Note. *p<.05
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 19
Table 5. Evaluation of Cognitive Score Differences by Age Group
Cognitive Simulation Scale 40 or Older Less than 40
n M SD n M SD t d
Basic Entry-Level Analytical Thinking 44 16.64 1.73 127 16.70 2.19 0.16 .03
Attention to Detail 44 4.20 1.07 127 4.20 1.05 0.00 .00
Face-to-Face Customer Service
Analytical Thinking 20 3.65 1.79 96 3.83 1.91 0.39 .10
Attention to Detail 20 6.85 1.35 96 6.76 1.34 -0.27 -.07
Administration Analytical Thinking 23 2.70 0.76 97 2.60 1.20 -0.38 -.09
Attention to Detail 23 7.66 2.08 96 7.26 2.29 -0.77 -.18
Business Sales Analytical Thinking 17 4.41 1.06 81 3.15 1.53 -3.23* -.86
Attention to Detail 17 6.35 1.73 77 5.87 2.00 -0.92 -.25
Business/Finance Analytical Thinking 16 2.94 1.24 106 3.06 1.38 0.33 .09
Attention to Detail 16 3.25 1.29 105 3.69 1.33 1.24 .33
Entry-Level Administration
Attention to Detail 15 7.09 2.05 85 7.43 2.37 0.52 .15
Entry-Level Business & Finance
Analytical Thinking 20 2.90 1.80 119 2.29 1.83 -1.38 -.33
Attention to Detail 20 4.05 1.23 117 3.79 1.46 -0.75 -.18
Entry-Level Office Analytical Thinking 27 20.52 4.31 115 20.44 4.87 -0.08 -.02
Attention to Detail 27 8.52 2.49 115 9.25 2.76 1.26 .27
First-Line Supervisor Analytical Thinking 22 6.88 1.08 80 6.29 1.69 -1.55 -.37
Attention to Detail 22 6.65 1.30 80 5.61 1.92 -2.39* -.58
General Office Workplace
Analytical Thinking 15 4.13 1.41 85 4.02 1.68 -0.24 -.07
Attention to Detail 15 4.87 0.92 84 4.63 1.47 -0.61 -.17
Information Technology
Analytical Thinking 21 4.14 1.62 77 4.53 2.02 0.82 .20
Attention to Detail 21 5.33 1.49 77 5.49 1.25 0.50 .12
Manager Analytical Thinking 16 6.38 1.36 82 5.22 1.82 -2.42* -.66
Attention to Detail 16 4.06 0.85 81 3.47 1.19 -1.89 -.52
Remote Customer Service
Analytical Thinking 14 3.64 1.45 84 3.42 1.54 -0.50 -.14
Attention to Detail 14 7.14 2.41 84 6.58 2.26 -0.85 -.25
Technician Analytical Thinking 13 4.85 2.34 47 5.00 2.55 0.19 .06
Attention to Detail 13 7.62 1.94 47 7.23 1.86 -0.66 -.21
Retail Sales Analytical Thinking 26 5.58 2.30 88 4.53 3.07 -1.61 -.36
Attention to Detail 26 20.20 2.44 88 18.93 3.42 -1.76 -.39
Note. *p<.05
Attitudes, Interests and Motivations Assessment
Previous Research
The Big Five framework for understanding personality is the most widely accepted taxonomy of personality
traits (Goldberg, 1992; 1993; McCrae & Costa, 1997; McCrae & John, 1992). The five factors, or traits, are
Agreeableness, Conscientiousness, Emotional Stability, Extraversion, and Openness. Agreeableness is
characterized as warmth, ability to get along with others, amd friendliness. Individuals who are
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 20
conscientiousness are characterized as being reliable, responsible, dependable, thorough, dutiful,
achievement-oriented and competent. Individuals considered Emotionally Stable are more resilient, stable,
and centered. Extraverted individuals are characterized as being sociable, outgoing, and high in energy.
Openness describes indiciduals who are “creativc, flexible, curious, and unconventional” (Judge & Ilies,
2002)
A substantial amount of research, including meta-analytic research, has demonstrated the relationship
between personality and job performance (Barrick & Mount, 1991; Bartram, 2005; Hogan & Holland, 2003;
Hurtz & Donovan, 2000; Mount & Barrick, 1995; Salgado., 1997; Tett, Jackson, & Rothstein, 1991). In a
meta-analysis of 25 years of personality and performance research drawn from 117 studies of managers,
professionals, sales people, skilled, and semi-skilled workers; researchers concluded that extroversion
predicted success in management and sales; openness to experience predicted training ability; and,
conscientiousness correlated with success in sales, management, professional, skilled and semi-skilled
positions (Barrick & Mount, 1991). The findings were similar in the European community (Salgado, 1997).
Personality traits are stable and slow to change (McCrae, Jang, Livesley, Riemann, & Angleitner, 2001). The
stability of personality traits and their relationship to job performance makes them extremely important to
measure before making a hiring decision.
Theory on personality in the workplace has advanced and researchers have demonstrated that validity
coefficients for personality assessments are stronger for those job performance criteria that can be
theoretically linked to a given personality trait. For example, Borman and Motowidlo (1993, 1997) found
support for their hypotheses that personality would have a stronger relationship with contextual rather than
task performance. The researchers defined contextual performance as those areas of performance that are
important to an organization but not explicity part of the job duties of an individual (e.g., following
organizational rules, giving additional effort when needed, providing support to other organizational
memebers, etc.). Hogan and Holland (2003) found that the Hogan Personality Inventory, a personality
assessment based on the Big Five, was substantially related to multiple measures of job performance across
a wide variety of positions. The relationships were stronger for personality traits that were theoretically
linked to the job performance measure. Bartram (2005) had similar findings, supporting the notion that
theoretically linked personality constructs and job performance domains had stronger relationships than
personality traits with overall performance. The trait-activation model proposed by Tett and Burnett (2003)
provides a theoretical model allowing for more detailed predictions regarding when a specific personality
trait might be linked to job performance.
Another approach to improving prediction with personality assessment is to measure narrower traits that are
specific facets of the Big Five (Paunonen, Rothstein, & Jackson, 1999; Schneider, Hough, & Dunnette,
1996). Researchers who argue for this approach posit that some facets of a trait may relate to a particular job
performance dimension, but that others may not - or may even be negatively related. Measuring at the broad
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 21
level may dilute the relationship between the narrower facet and the job performance dimension. For
example, achievement and reliability are both narrow facets of conscientiousness. One might expect a strong
positive relationship between reliability and retention, but there may not be a relationship between
achievement and retention. Using an assessment that measures the broad trait of conscientiousness may
under-predict retention. Researchers conducted a meta-analysis and demonstrated the superiority of using
narrow facets to predict specific job performance dimensions (Woo, Chernyshenko, Stark, & Conz, 2014).
Furthermore, many personality assessments used for pre-employment selection are actually compound traits
– or constellations of narrow facets (Schneider et. al., 1996). These constellations may come from one or
more of the broader personality traits. In fact, instruments used in both the Hogan and Holland (2003) and
Bartram (2005) meta-analyses contain measures of compound traits.
Content Development
A review of the literature on personality in the workplace allowed us to identify several competencies, or
compound traits, that are related to successful job performance for many jobs. The ten scales, their
definitions, and example items are listed below. Please note, example items are similar to, but do not reflect,
actual items from the assessment.
Needs Structure
Often, following rules and procedures is critical to successful job performance. When employees do not
follow established organizational policies, work processes may be performed incorrectly, work products may
be flawed, customer service levels may suffer, the organization may become a victim to fraudulent wage and
expense billings, and in some cases the organization may be liable for the actions of the employee. The
Needs Structure scale was designed to evaluate an applicants’ tendency to adhere to organization rules and
procedures and should be important in jobs where rule-abiding behaviors are related to job performance.
Example: I prefer to make extensive plans before I start any project.
Innovative and Creative
For many positions, Innovation and Creativity is critical to successful job performance. Individuals with
high levels of this competency should be more likely to generate new products, develop improved methods
for producing work, and develop solutions to problems.
Example: Some of my suggestions are really eccentric.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 22
Enjoys Problem-Solving
Enjoys Problem-Solving is a competency designed to predict which applicants will be more successful in
roles dealing with data, completing analyses, and conducting research. Individuals who enjoy problem
solving should be more successful in roles that require analytical thinking.
Example: I enjoy learning about how things work.
Competitive
The Competitive competency evaluates the extent to which in individual is likely to do what is necessary to
accomplish their goals – which may be of concern if the individual’s goals are not aligned with the
organization’s goals. Individuals who score highly on this competency are characterized as having concern
for outcomes rather than the feelings of coworkers.
Example: I am not above using people to get my way.
Competitive v2
The Competitive v2 competency evaluates the degree to which an individual is driven by a desire to achieve
objectives and outperform their peers. Competitiveness is the tendency to evaluate one’s performance in
comparison to others; a desire to do better than others, an enjoyment of the situations that can lead to a
clear winner; thrives in an environment where people are differentiated by accomplishments.
Unless otherwise noted, the results in this technical manual refer to Competitive scale from the original
AIMS test development. The Competitive v2 scale began being used in July 2016, and data is being collected.
The decision was made to replace the original Competitive scale due to customer complaints and upon
psychometric review of its continuing usefulness, based on research conducted through June, 2016.
Example: I get satisfaction from beating out the competition.
Seeks Perfection
When the quality of work (as opposed to pace) is important for success in a role, the Seeks Perfection
competency should predict performance. Those who score high on this competency are more likely to
double-check their work and have higher rates of accuracy. Additionally, individuals high on this
competency are more likely to be detail-oriented.
Example: My work tends to be faultless.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 23
Develops Relationships
For many roles, working in teams and with others is critical for successful performance. The Develops
Relationships scale was designed to predict which individuals are more likely to develop productive working
relationships and be successful in a team situation.
Example: I have never deliberately said anything that hurt someone's feelings.
Expressive and Outgoing
Individuals who are Expressive and Outgoing are more likely to engage with others. Being Expressive and
Outgoing is also characterized by the ability to influence and persuade others. This competency is important
for success in roles where influence is necessary such as leadership and sales positions.
Example: I tend to take control in most work situations.
Corporate Citizenship
The Corporate Citizenship competency was designed to measure applicants’ tendency to conduct
themselves ethically and responsibly in an organization. Individuals who score highly on this competency are
likely to be honest with managers and colleagues and avoid taking advantage of people and/or situations
just because it suits their self-interest.
Example: I would never use manipulation as a tactic to advance my goals.
Exhibits a Positive Work Attitude
This competency evaluates the tendency for an individual to feel positively toward their job and the
organization they work for. Individuals who score highly on this competency are characterized as feeling
satisfied with their work and are more likely to put in extra effort when needed.
Example: I volunteer for additional work.
Adaptable
The Adaptable competency measures an individual’s tendency to adjust to changes in their work
environment. Individuals who score highly on this competency thrive in fast-paced settings and respond
well to variety.
Example: I think organizational changes are fun and exciting.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 24
The figure below outlines the hypothesized relationships between the AIMs scales and the Big Five traits.
As mentioned above, the AIMs scales are compound traits and measure competencies related to multiple
narrow facets of personality and these facets may be under one or more of the five broader personality traits.
Therefore, an AIMs scale may be expected to relate, albeit moderately, to multiple Big Five traits.
Figure 3. Hypothesized Relationships between the AIMs Scales and the Big Five
The original assessment consisted of 100, Likert-type items. Each item consists of a single statement and
candidates are asked to indicate their level of agreement with the statement. Preliminary data analyses
allowed us to shorten the scales such that the assessment contains a total of 65 items. The AIMs assessment
is available in two forms. The professional form measures all ten competencies. The entry-level form
measures eight competencies as it was determined that Expressive and Outgoing and Innovative and
Creative would be less critical competencies for these roles. The time to take the professional form is
Emotional
Stability
Openness
Extraversion
Conscientiousnes
s
Agreeableness
Enjoys Problem-Solving
Innovative and Creative
Competitive
Seeks Perfection
Develops Relationships
Expressive and Outgoing
Corporate Citizenship
Exhibits a Positive Work
Attitude
Adaptable
Needs Structure
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 25
estimated to be between 6 and 7 minutes, and the time to take the entry-level form is estimated to be 5 and
6 minutes. The estimates were calculated using a sample of 402 individuals.
Reliability
The descriptive statistics and reliability estimates of the AIMs scales were estimated using the data collected
via MTurk as described in the previous section. As can be seen from the table below, the AIMs scales have
acceptable levels of internal consistency.
Table 6. Descriptive Statistics and Reliability Evidence for the AIMS Scales
Competency N M SD Alpha
Needs Structure 675 26.76 4.85 .84
Innovative & Creative 567 25.08 5.33 .85
Enjoys Problem-Solving 675 26.47 5.67 .86
Competitive 675 13.27 6.15 .86
Seeks Perfection 675 25.66 5.46 .84
Develops Relationships 675 25.97 5.36 .83
Expressive & Outgoing 567 19.71 6.22 .82
Corporate Citizenship 675 45.33 7.73 .89
Exhibits a Positive Work Attitude 675 28.15 5.14 .80
Adaptable 675 25.49 4.57 .69
Competitive v2* 318 31.08 7.21 .85
*Competitive v2 replaced Competitive in July 2016
Validity
A Confirmatory Factor Analyses (CFA) was run using R (version 3.1.2) and the “sem” package to evaluate
whether or not the internal structure of the assessment conformed to the hypothesized structure. In other
words, we assessed the extent to which the items loaded onto the particular scale they were assigned to. The
Root Mean Square Error (RMSEA), an indicator of model fit, indicates that the model was satisfactory,
RMSEA=.0675, χ2 (1970, N=567) =7057.80, p<.001. This analysis provides validation support for the
internal structure of the assessment.
As mentioned previously, the content for the AIMs scales were modified and adapted from a longer
assessment that had been used in previous validation research. The developer of the original content
provided the results of several criterion-related validity studies and tables with these results can be found in
Appendix C.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 26
Validation evidence is available in two studies that demonstrate the validity of the assessments. The first is a
study with 130 managers and the second is a study with 64 managers. In the first study, the assessment
demonstrated that it predicts job performance (r=.25; p<.01), using a simple “High” or “Marginal”
categorization of job performance, and a second study demonstrated the assessment predicted performance
on a 100 point administrative performance appraisal, using a Spearman correlation (r=.25; p<.05). We
expect validity evidence to be more robust in future studies with better criterion measures. A brief summary
of the results is presented in Appendix G.
Fairness
In order to evaluate how different demographic subgroups perform on the assessments, independent
samples t-tests were performed to compare mean scores of the subgroups (Tables 7 -11). There were too
few cases (n<22) to run analyses for the American Indian or Alaska Native, Native Hawaiian or Other
Pacific Islander, Two or More Races, and Other subgroups. There were significant mean differences on the
AIMs scales between males and females. However, all differences were relatively small or they were in favor
of the focal subgroup, females. There were no significant mean differences based on ethnicity. There were a
few significant mean differences with respect to age group, but all but one of these were in favor of the
focal subgroup, 40 and older. There were a few significant mean differences between the Asian and White
subgroups. However, individuals in this sample were from multiple countries and it’s possible that these
differences might be reflections of cultural differences. Additional research is needed to further evaluate
these differences, to calculate country-specific norms, and to evaluate subgroup differences within country.
There were no significant differences between the Black or African American subgroup and the White
subgroup.
Table 7. Evaluation of AIMs Score Differences by Gender
Scale
Female Male n M SD n M SD t d
Needs Structure 287 27.19 4.94 380 26.42 4.77 -2.03* -0.16
Innovative and Creative 240 24.62 5.47 319 25.39 5.20 1.69 0.14
Enjoys Problem Solving 287 25.30 5.84 380 27.32 5.39 4.62* 0.36
Competitive 287 12.08 5.85 380 14.08 6.19 4.23* 0.33
Seeks Perfection 287 26.20 5.46 380 25.22 5.45 -2.30* -0.18
Develops Relationships 287 26.40 5.25 380 25.63 5.42 -1.84 -0.14
Expressive and Outgoing 240 18.67 6.05 319 20.38 6.18 3.27* 0.28
Corporate Citizenship 287 47.44 6.88 380 43.79 7.97 -6.21* -0.49
Exhibits a Positive Work Attitude 287 29.10 5.04 380 27.52 5.06 -4.00* -0.31
Adaptable 287 26.01 4.73 380 25.08 4.41 -2.61* -0.20
Note. *p<.05
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 27
Table 8. Evaluation of AIMs Score Differences by Ethnicity
Hispanic or
Latino Not Hispanic or
Latino
Scale n M SD n M SD t d
Needs Structure 46 27.52 4.67 574 26.62 4.87 -1.21 -0.19
Innovative and Creative 33 24.36 5.99 485 25.17 5.31 0.84 0.15
Enjoys Problem Solving 46 26.74 5.98 574 26.51 5.63 -0.27 -0.04
Competitive 46 14.13 6.40 574 13.01 5.92 -1.23 -0.19
Seeks Perfection 46 26.33 5.61 574 25.56 5.46 -0.92 -0.14
Develops Relationships 46 25.72 5.61 574 25.97 5.37 0.30 0.05
Expressive and Outgoing 33 21.12 7.23 485 19.53 6.09 -1.43 -0.26
Corporate Citizenship 46 44.22 8.41 574 45.68 7.40 1.27 0.20
Exhibits a Positive Work Attitude 46 27.63 5.56 574 28.28 5.12 0.82 0.13
Adaptable 46 24.8 5.53 574 25.57 4.47 1.10 0.17
Note. *p<.05
Table 9. Evaluation of AIMs Score Differences by Age Group
40 and Older Less than 40
Scale n M SD n M SD t d
Needs Structure 150 27.57 5.36 524 26.54 4.67 -2.30* -0.21
Innovative and Creative 138 25.11 5.27 428 25.07 5.37 -0.08 -0.01
Enjoys Problem Solving 150 26.31 5.08 524 26.51 5.84 0.38 0.04
Competitive 150 10.38 4.02 524 14.08 6.40 6.71* 0.62
Seeks Perfection 150 25.48 6.07 524 25.72 5.28 0.47 0.04
Develops Relationships 150 26.49 4.50 524 25.83 5.58 -1.33 -0.12
Expressive and Outgoing 138 17.59 5.10 428 20.38 6.40 4.66* 0.46
Corporate Citizenship 150 48.42 5.37 524 44.45 8.08 -5.67* -0.52
Exhibits a Positive Work Attitude 150 29.79 4.83 524 27.69 5.14 -4.47* -0.41
Adaptable 150 26.25 4.63 524 25.27 4.53 -2.32* -0.22
Note. *p<.05
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 28
Table 10. Evaluation of AIMs Score Differences by Race Groups: Asian and White
Asian White
Scale n M SD n M SD t d
Needs Structure 89 27.25 4.86 518 26.66 4.80 -1.07 -0.12
Innovative and Creative 76 25.28 5.26 434 24.93 5.29 -0.53 -0.07
Enjoys Problem Solving 89 26.70 5.02 518 26.24 5.70 -0.72 -0.08
Competitive 89 18.35 7.80 518 12.38 5.31 -9.06* -1.04
Seeks Perfection 89 25.97 4.54 518 25.56 5.62 -0.65 -0.07
Develops Relationships 89 26.26 4.71 518 25.86 5.44 -0.65 -0.07
Expressive and Outgoing 76 24.21 7.23 434 18.95 5.68 -7.13* -0.89
Corporate Citizenship 89 38.60 10.27 518 46.49 6.56 9.52* 1.09
Exhibits a Positive Work Attitude 89 25.75 5.25 518 28.55 5.00 4.84* 0.56
Adaptable 89 24.72 3.59 518 25.53 4.62 1.57 0.18
Note. *p<.05
Table 11. Evaluation of AIMs Score Differences by Race Groups: Black or African American and White
Black or African
American White
Scale n M SD n M SD t d
Needs Structure 25 26.76 5.76 518 26.66 4.08 -0.10 -0.02
Innovative and Creative 19 24.47 6.52 434 24.93 5.29 0.37 0.09
Enjoys Problem Solving 25 26.40 7.07 518 26.24 5.70 -0.14 -0.03
Competitive 25 13.64 6.89 518 12.38 5.31 -1.14 -0.23
Seeks Perfection 25 26.16 5.70 518 25.56 5.62 -0.52 -0.11
Develops Relationships 25 25.04 6.17 518 25.86 5.44 0.73 0.15
Expressive and Outgoing 19 17.68 5.65 434 18.95 5.68 0.95 0.22
Corporate Citizenship 25 45.40 7.31 518 46.49 6.56 0.81 0.17
Exhibits a Positive Work Attitude 25 27.92 6.05 518 28.55 5.00 0.61 0.12
Adaptable 25 25.88 6.06 518 25.53 4.62 -0.36 -0.07
Note. *p<.05
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 29
Behavioral History Survey
Previous Research
Past behavior often predicts future behavior. Biographical data, or bio-data, assessments contain items
developed by identifying patterns associated with high productivity and low turnover. Hunter and Hunter
(1984) report the average validity coefficient for bio-data assessments to be .38. Other researchers have
estimated the validity to be .35 for supervisors (Rothstein, Schmidt, Erwin, Owens, & Sparks, 1990) and .53
for managers (Carlson, Scullen, Schmidt, Rothstein, & Erwin, 1999).
For example, Rothstein, et al., (1990) identified several bio-data factors that could be applied to many jobs.
These included things like having a pervasive feeling of self-worth and confidence; believing that he or she
works better and faster than others in his or her area of specialization; having been recognized for
accomplishments; being outgoing; being a good communicator; taking clear positions; and, feeling healthy
and satisfied with current life situations. Rothstein screened each biographical item for cross-validity then
meta-analyzed 11,000 first-line supervisors from different organizations, age levels, genders, job experience
levels and tenures. He concluded that in all cases, validity estimates for these factors were generalizable,
stable across time, and did not appear to stem from acquired skills, knowledge or abilities.
McDaniel (1989) evaluated biographical questions about school suspensions, drug use, quitting school, prior
employment experience, grades, club memberships, contacts within the legal system, and socioeconomic
status. The results successfully predicted discharge from the military for problems such as alcohol and drug
use, desertion, imprisonment, and “discreditable incidents.”
Oswald, Schmitt, Kim, Ramsay, and Gillespie (2004) reported statistically significant bio-data correlations
with 12 dimensions of college student performance: knowledge, learning, artistic ability, multicultural
sensitivity, leadership, interpersonal skills, citizenship, health, careers, adaptability, perseverance, and ethics.
Their work showed incremental validity over the traditional use of SAT and ACT with fewer differences
between subgroups than traditional admission measures.
In a similar vein, Kanfer, Crosby, and Brandt (1988) identified correlations between bio-data and tenure; and
a study of 555 real estate agents, Klimoski and Childs (1986) identified five major bio-data factors associated
with job, personal and career success. They included social orientation, economic stability, work ethic
orientation, educational achievement and interpersonal confidence.
Content Development
The Behavioral History Survey consists of three biographical history areas that generalize across most jobs:
tenure, performance, and unproductive behavior. Questions and forms were developed for both
professional and entry-level positions using a panel of experienced managers. Each bio-data item was
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 30
reviewed by an expert management panel and scored using a modified Angoff method. There are a few
versions of each form available because some of the items were relevant for some positions but irrelevant
for others. For example, an item asking about experience with an industry would not be appropriate for a
position that spans multiple industries (e.g. Administrative Assistant). Scores for each competency are
calculated by averaging across items within that competency. The use of averaging, rather than summing
across items, allows HR Avatar to compare similar scales across multiple versions of the assessment. The
entry-level form contains 14-15 items and takes approximately 2 minutes to complete. The professional
form contains 18-20 items and takes approximately 6 minutes to complete. The estimates were estimated
with two samples (N= 70 and N=371, respectively).
Performance
The Performance scale asks questions related to past performance on the job and should predict multiple
dimensions of job performance.
Example: How many times have you been promoted at work?
Tenure
The Tenure scale asks questions specifically related to tenure on previous jobs. This scale should predict
retention.
Example: What is your longest tenure with an organization?
Unproductive Behavior
The Unproductive scale includes items related to behavior that is disruptive and violates rules and/or social
norms.
Example: In the past, how many times have you violated a company policy?
Reliability
The scales in the Behavioral History Survey are relatively short – some with as few as three items.
Additionally, the scales were not expected to be internally consistent as there is no evidence that the
individual item responses should be highly correlated with one another. Therefore, a test-retest reliability
estimate was determined to be more appropriate than one of internal consistency (e.g., Cronbach’s alpha).
Descriptive statistics and reliability estimates were calculated using data collected via MTurk. This data
collection effort is summarized in the Cognitive Work Simulations section. Because participants were
allowed to take multiple versions of the solution, several participants took the Behavioral History Surveys
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 31
more than once. Test-retest reliability estimates were calculated by correlating the scale scores between the
first and second administrations of the form.
For the Professional form, the sample was restricted to those individuals who had at least two days between
administrations. The largest number of days between administrations was 25. On average, there were 11
days between administrations (SD=7.49). The descriptive statistics and test-retest reliability estimates are in
Table 12 and all reliability estimates are acceptable.
For the Entry-Level form, the sample was restricted to those individuals who had at least two days, but less
than 30 days between administrations. The largest number of days between administrations was 20. On
average, there were 7 days between administrations (SD=4.76). The descriptive statistics and test-retest
reliability estimates are in Table 13 and all reliability estimates are acceptable.
Please note, there are multiple versions of the Performance scale for both the Professional and the Entry-
Level forms as described above. The descriptive statistics, test-retest reliability estimates and fairness
analyses for this scale are estimated by averaging across various versions of the scales.
Table 12. Descriptive Statistics and Reliability Evidence for the Professional Behavioral History Survey
Competency N M SD ρxx N for
ρxx
Performance 303 2.34 0.29 .81 65
Tenure 303 2.29 0.40 .71 65
Unproductive Behavior 303 2.62 0.42 .72 65
Table 13. Descriptive Statistics and Reliability Evidence for the Entry-Level Behavioral History Survey
Competency N M SD ρxx N for
ρxx
Performance 593 2.43 0.38 .74 151
Tenure 593 2.62 0.36 .71 151
Unproductive Behavior 593 2.59 0.42 .74 151
Fairness
In order to evaluate how different demographic subgroups perform on the assessments, independent
samples t-tests were performed to compare mean scores of the subgroups (Tables 14-17). There were too
few cases (n<25) to run analyses for the American Indian or Alaska Native, Black or African American,
Native Hawaiian or Other Pacific Islander, Two or More Races, and Other subgroups. There were
significant mean differences on the Behavioral History scales between males and females. However, all
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 32
differences were relatively small or they were in favor of the focal subgroup, females. There were a few
significant mean differences based on ethnicity. Unfortunately, the sample is too small for the Hispanic and
Latino subgroup to draw any conclusions. As data become available, further research will explore these
differences at the item and response level. There were significant mean differences with respect to age group,
but all were in favor of the focal subgroup, 40 and older. There were a few significant mean differences
between the Asian and White subgroups. However, individuals in this sample were from multiple countries
and it’s possible that these differences might be reflections of cultural differences. Additional research is
needed to further evaluate these differences, to calculate country-specific norms, and to evaluate subgroup
differences within country.
Table 14. Evaluation of Biographical History Survey Score Differences by Gender
Female Male
Scale n M SD n M SD t d
Professional Performance 123 2.33 0.29 177 2.34 0.30 0.00 0.03
Professional Tenure 123 2.36 0.42 177 2.24 038 -2.58* -0.30
Professional Unproductive Behavior 123 2.71 0.35 177 2.57 0.45 -2.89* -0.34
Entry-Level Performance 240 2.33 0.37 305 2.34 0.36 0.21 0.02
Entry-Level Tenure 240 2.66 0.37 305 2.59 0.37 -2.19* -0.19
Entry-Level Unproductive Behavior 240 2.68 0.36 305 2.54 0.45 -3.93* -0.34
Note. *p<.05
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 33
Table 15. Evaluation of Biographical History Survey Score Differences by Ethnicity
Hispanic or Latino
Not Hispanic or
Latino
Scale n M SD n M SD t d
Professional Performance 25 2.36 0.30 260 2.34 0.29 -0.25 -0.05
Professional Tenure 25 2.13 0.44 260 2.31 0.39 2.18* 0.46
Professional Unproductive Behavior 25 2.61 0.43 260 2.62 0.41 0.12 0.02
Entry-Level Performance 38 2.13 0.41 467 2.33 0.35 3.37* 0.57
Entry-Level Tenure 38 2.57 0.39 467 2.64 0.35 1.18 0.20
Entry-Level Unproductive Behavior 38 2.58 0.37 467 2.61 0.40 0.45 0.08
Note. *p<.05
Table 16. Evaluation of Biographical History Survey Score Differences by Age Group
40 and Older Less than 40
Scale n M SD n M SD t d
Professional Performance 60 2.44 0.26 243 2.31 0.30 -3.14* -0.45
Professional Tenure 60 2.49 0.31 243 2.25 0.40 -4.34* -0.62
Professional Unproductive Behavior 60 2.70 0.34 243 2.61 0.43 -1.51 -0.22
Entry-Level Performance 114 2.41 0.30 439 2.31 0.38 -2.52* -0.26
Entry-Level Tenure 114 2.82 0.24 439 2.57 0.38 -6.68* -0.70
Entry-Level Unproductive Behavior 114 2.67 0.35 439 2.58 0.43 -2.06* -0.22
Note. *p<.05
Table 17. Evaluation of Biographical History Survey Score Differences by Race Groups: Asian and White
Asian White
Scale n M SD n M SD t d
Professional Performance 27 2.34 0.34 243 2.34 0.29 0.08 0.02
Professional Tenure 27 2.07 0.50 243 2.35 0.37 3.59* 0.73
Professional Unproductive Behavior 27 2.51 0.49 243 2.64 0.41 1.53 0.31
Entry-Level Performance 77 2.27 0.38 418 2.34 0.35 1.44 0.18
Entry-Level Tenure 77 2.45 0.40 418 2.67 0.34 5.07* 0.63
Entry-Level Unproductive Behavior 77 2.48 0.51 418 2.62 0.39 2.75* 0.34
Note. *p<.05
Knowledge and Skills Tests
Development Overview
Candidates need more than just the right combination of abilities, personality characteristics and
background in order to be successful in a job. Most often, specific knowledge and/or specific skills are also
required. Job knowledge tests measure job-relelevant declaritive knowledge such as technical information,
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 34
standards, and best practices as well as knowledge of specific processess and procedures. Job knowledge
tests serve as an indicator of previous job performance and serve as proximal predictors of future job
performance. Skills assessments evaluate a candidate’s ability to perform a specific task. Candidiates who
begin work with a sufficient level of knowledge and/or skill should require less training and beable to
perform better faster. Hunter and Hunter (1984) report the average validity coefficient for job knowledge
tests to be .48. In their meta-analysis, Dye, Reck, and McDaniel (2007) found the average corrected
correlation coefficient for job knowledge tests to be .45 for job performance and .47 for training.
Correlations were even higher for complex jobs and higher to the extent that the job knowledge test was
similar to the job.
HR Avatar has developed dozens of knowledge assessments. See Appendix A for a complete listing. The
knowledge assessments were developed by reviewing multiple resources to identify appropriate items. For
example, The Food Safety Fundamentals assessment was developed by reviewing, among other resources,
the USDA Safe Food Handling Fact Sheets. Each knowledge test contains an item bank and half of the
items are randomly selected to be administered to an applicant. Research on the validity, reliability, and
fairness of these assessments is forthcoming. HR Avatar has three skills assessments: Sales Situation
Analysis, Typing Test, and Essay Test. The development and research on these assessments is described in
more detail below.
Validity
Initial validation evidence is available, in a study with 130 managers, which demonstrates the validity of the
assessments. The assessment demonstrated that it predicts job performance (r=.25; p<.01), using a simple
“High” or “Marginal” categorization of job performance. We expect validity evidence to be more robust in
future studies with better criterion measures. A brief summary of the results is presented in Appendix G.
Sales Situation Analysis
Test Development
The Sales Situation Analysis is a scale that is integrated into the Business Sales Cognitive Work Simulation
(described above). In the simulation, the candidate is asked to assist and respond to customers and
colleagues and solve business-related problems. The Sales Situation Analysis scale specifically evaluates a
candidate’s ability to understand a customer’s needs and identify the most appropriate follow-up actions. In
the simulation, the candidate must read an email communication from a customer and then identify the
customer’s primary concern. Next, the candidate must identify which of several action items are most
appropriate for the specific sales situation.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 35
Reliability
The descriptive statistics and reliability estimate for the Sales Situation Analysis were estimated using the
data collected via MTurk as described in the previous section. Table 18 contains the descriptive statistics and
reliability estimate for this scale. These analyses led the developers to make substantial edits to these items
and we anticipate that edits will substantially improve the reliability of the tool. Further research evaluating
the effectiveness of these changes is forthcoming.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 36
Table 18. Descriptive Statistics and Reliability Evidence for the Sales Situation Analysis Assessment
# of
Items N M SD Alpha
Sales Situation Analysis 6 98 3.72 1.04 .22
Fairness
In order to evaluate how different demographic subgroups perform on the assessment, independent
samples t-tests were performed to compare mean scores of the subgroups. The results are provided in Table
19. Although data were collected on ethnic group, the sample size for the ‘Hispanic or Latino’ subgroup was
too small for analyses (n<15). Data were also collected for racial group. However, the sample sizes for all
but the “White” subgroup were too small for analysis (n<30). There were not significant mean differences
based on gender or age group.
Table 19. Evaluation of Sales Situation Analysis Score Differences
n M SD t d
Female 41 3.73 1.06 .09 .02
Male 56 3.71 1.05
40 and older 17 4.00 1.08 1.19 .31
Less than 40 81 3.67 1.03
Note. *p<.05
Typing Test
Test Development
The typing test consists of three typing tasks. For each task, applicants are asked to type a short passage.
Each typing task is randomly selected from a group of five passages (15 total passages). For each task, the
words per minute are calculated. This score is modified to an accuracy-adjusted words per minute by
calculating and factoring in the rate of typing errors. The typing scores are calculated by averaging the words
per minute and accuracy-adjusted words per minute across the three tasks.
Reliability
The typing test was piloted via MTurk (N=155). The majority of the sample was from the US (92.9%). The
pilot assessment consisted of five typing tasks and each typing task was randomly selected from a group of
five passages (25 total passages). Five one-way analyses of variance were completed to evaluate differences
in scores across the passages available for each item. No significant differences were found, F(4, 150)=.56,
p=.70, F(4, 150)=.27, p=.89, F(4, 150)=.79, p=.54, F(2, 121)=1.12, p=.33, F(2, 121)=.22, p=.80. Descriptive
statistics and the alpha estimate of internal consistency were estimated using the first three items from the
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 37
five item pilot assessment (see Table 20). The internal consistency of the assessment is well above accepted
standards.
Table 20. Descriptive Statistics and Reliability Estimates for the Typing Test
Typing Test N M SD alpha
3 Items 155 46.92 15.63 .97
Fairness
In order to evaluate how different demographic subgroups perform on the assessment, independent
samples t-tests were performed to compare mean scores of the subgroups (Table 21). Although data were
collected on ethnic group, the sample size for the ‘Hispanic or Latino’ subgroup was too small for analyses
(n=11). There were too few cases (n<10) to run analyses for the African American or Black, American
Indian or Alaska Native, Native Hawaiian or Other Pacific Islander, Two or More Races, and Other
subgroups. Significant mean differences were found for gender and between the Asian and White racial
subgroups. Given that this sample consisted of people piloting the assessment but not necessarily taking the
assessment in order to obtain a job, it is possible that the individuals were not attempting to perform their
best. Further research will evaluate norms using an applicant sample.
Table 21. Evaluation of Essay Score Differences by Subgroup
n M SD t d
Female 69 43.64 15.12 2.59* .42
Male 83 50.09 15.39
40 and Older 33 44.20 11.51 1.19 .26
Less than 40 120 47.85 16.50
Asian 26 41.76 18.23 2.28* .46
White 108 49.45 14.68
Note. *p<.05
Essay Test
Test Development
Written communication is a key skill in many positions. Communicating via email, writing reports, and
creating presentations all require the ability to communicate effectively. The HR Avatar Essay Test is a fast
method for evaluating an applicant’s written communication skills. The HR Avatar Essay Test consists of
one of two writing prompts. The writing prompts were designed to be general enough to provide an
opportunity for anyone to be able to write a short essay. The writing prompts are included below.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 38
1. Describe the pros and cons of working from home.
2. Describe the pros and cons of living in a big city.
Applicants are asked to write a short essay with a minimum of 100 words and are given an unlimited time to
do so.
The essays are scored using Discern, an open source, machine learning program. Discern was designed by
edX, a nonprofit organization founded by Harvard and the Massachusetts Institute of Technology (MIT)
(edX, 2015; Markoff, 2013). A YouTube video was published and provides some additional information on
how the program works: https://www.youtube.com/watch?v=zFeP678054U (Paruchuri, 2015). The system
produces a score that ranges from 0 to 100. A confidence estimate for the score is also computed. The
confidence estimate can range from 0 to 1. Scores with confidence estimates less than .10 are not considered
valid.
In order to calibrate the program, HR Avatar used MTurk to collect writing samples for the two writing
prompts (N=170 and N=163). The essays were scored on three areas: Grammar, Structure and Content, by
three independent raters. Prior to rating each essay, the raters were provided with scoring rubrics (Appendix
D) and training (Appendix E) for how to score the essays using the rubrics. A total score was calculated for
each essay by aggregating scores on Grammar, Structure, and Content and then averaging across the raters
and linearly transforming the scores to a scale of 0-100. Scores were entered into the program as the
calibration sample for Discern.
In order to evaluate the reliability of the ratings provided by the three raters, intra-class correlation
coefficients were calculated using a two-way, mixed effects model (ICC3) (Shrout & Fleiss, 1979) . The
ICC3 was chosen because it is the reliability of the average rating made by the specific raters in this study
that were of interest, and it is not necessary to generalize the reliability estimate to the population of
potential raters. There were ratings available for all three raters for 317 essays and the reliability of the
average aggregate rating was acceptable (ICC (3,3)=.74). There was also a large relationship between scores
on the two essay prompts, r(152)=.62, p<.01.
In early January 2015 additional improvements were made to the Essay scoring. First, the 196 essays that
were completed since the initial rollout were manually scored by an individual rater and re-entered into the
Discern program to further calibrate the system. Second, HR Avatar added some additional safeguards to
the automated scoring engine. The essays are truncated to 800 words. Essays with fewer than 100 words,
consisting of more than 25% of spelling errors, or more than 25% of grammar errors or style errors are
given a score of 0. An additional program was written to search the HR Avatar database for matching essay
content. The system uses a combination of three methods to determine the similarity between the essay and
existing content in the system as a way of detecting plagiarism: The Levenshtein Distance Strategy (Navarro,
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 39
2001), the Jaro distance metric (Jaro, 1989; Jaro, 1995), and the Jaro-Winkler distance metric (Winkler, 1990).
If matching content is discovered, it is assumed that the essay is plagiarized and applicants will receive a
score of 0.
Descriptives
HR Avatar has collected essays from 206 individuals who piloted the system between November 27th, 2014
and February 26th, 2015. Many of these data points are individuals piloting the system rather than applicants
applying for a job. However, based on an examination of the data, there appear to be a substantial number
of instances where it appears as if people were trying to perform well on the assessment. Therefore,
descriptive statistics have been provided in Table 22 for the essays after removing those cases in which a
“real” essay was not written (n=5). Please note that there were not significant score differences based on
which prompt was received, t(200)=1.18, p=.24, two-tailed, d-=17. The majority of the sample came from
the US and the Philippines, therefore descriptive statistics are provided at the country level for these
countries.
Table 22. Descriptive Statistics for Essay Scores
n M SD
Score 201 46.14 14.05
Confidence 201 0.73 0.14
Prompt 1 (Working from home) 103 44.99 13.00
Prompt 2 (Living in a big city) 99 47.32 15.03
Philippines 108 43.31 13.85
US 73 50.15 13.05
Fairness
In order to evaluate how different demographic subgroups perform on the assessment, independent
samples t-tests were performed to compare mean scores of the subgroups. Although data were collected on
ethnic group, the sample size for the ‘Hispanic or Latino’ subgroup was too small for analyses (n=11). There
were too few cases (n<10) to run analyses for the American Indian or Alaska Native, Native Hawaiian or
Other Pacific Islander, Two or More Races, and Other subgroups. Both the Asian and the Black or African
American subgroups were compared to the White subgroup, most often considered the reference group. As
can be seen in the table below, the only significant differences between scores were between the Asian and
The White subgroups. This finding may be related to differences found within the two primary countries
samples: The US and the Philippines. Further analyses will be completed once data allows and score
differences between racial subgroups will be estimated within country.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 40
Table 23. Evaluation of Essay Score Differences by Subgroup
n M SD t d
Female 113 45.85 13.33 .26 .04
Male 80 46.38 15.31
40 and Older 67 47.00 12.14 .58 .09
Less than 40 133 45.79 14.80
Asian 103 42.99 13.89 2.95* .49
Black or African American 25 46.88 13.64 .87 .21
White 56 49.70 13.40
Note. *p<.05
Validity
Validation evidence is available in two studies that demonstrate the validity of the assessments. The first is a
study with 130 managers and the second is a study with 64 managers. In the first study, the assessment
demonstrated that it predicts job performance (r=.25; p<.01), using a simple “High” or “Marginal”
categorization of job performance, and a second study demonstrated the assessment predicted performance
on a 100 point administrative performance appraisal, using a Spearman correlation (r=.25; p<.05). We
expect validity evidence to be more robust in future studies with better criterion measures. A brief summary
of the results is presented in Appendix G.
Solution Scoring
In addition to providing scores at the scale level, HR Avatar also provides an overall score as an indication
of overall fit between the candidate and a given job. All competencies are grouped within four broad
categories: Cognitive Abilities, Skills and Knowledge, AIMs, and Behavioral History. Each competency, or
scale, is converted to a z-score. For the Cognitive Abilities, Skills and Knowledge, and Behavioral History
categories, a category score is created by averaging z-scores within each category.
For the AIMs and Cognitive Ability categories, O*NET is used to determine which of the competencies are
relevant for a given job and to determine the appropriate weights. O*NET is an online database that
contains specific information about hundreds of occupations (Peterson, et al., 2001). The information
gathered about each job is categorized using strongly supported theoretical models about behavior in the
workplace. Additionally, the process for gathering the information and documenting it is a collaborative
effort. The O*NET Skills and Abilities Importance ratings are the average importance ratings given by at
least eight occupational analysts (Fleisher & Tsacoumis, 2012a, 2012b). The Occupational Analysts all have
two or more years of work experience, two or more years of graduate level education in a program related to
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 41
human resources or workplace psychology, and coursework in research methods and job analysis.
Occupational Analysts are provided with extensive training and detailed information about each job prior to
making ratings including job description, knowledge requirements, task descriptions and work context. The
AIMs and Cognitive scales were mapped onto the O*NET Worker Characteristics and Worker
Requirements. Weights are applied to the scales such that each scale receives a weight that is equivalent to
the proportion of its importance rating within O*NET. When a competency is listed more than once, which
is sometimes the case given that several Worker Characteristics or Worker Requirements might be mapped
to a given competency, the weight given to the competency corresponds to the highest importance rating.
For the knowledge tests, the raw scores are the percent correct. As of yet, these assessments do not have
sufficient data to estimate stable normative parameters. Therefore, a mean of .70 and an SD of .25 will be
used to estimate z-scores for applicants.
An overall z-score is calculated by computing a weighted average of the competency categories. The
following weights are assigned to each category:
Cognitive Ability competencies = 1
Skills/Knowledge competencies = 0.8
AIMs competencies = 0.7
Behavioral History competencies = 0.4
The overall z-score is transformed to a Normal Curve Equivalent (NCE) score. NCE scores have a mean of
50 and a standard deviation of 21.06 and maintain their equal-interval properties.
Construct Validity Evidence
Tables 24-30 contain the correlations between the scales in the HR Avatar Assessment Solution. Generally
speaking, the relationships conform to what would be expected. For example, within the AIMs assessment,
there is a strong negative relationship between the Competitive scales and the Exhibits a Positive Work
Attitude scale. The Enjoys the Problem Solving and Innovative and Creative scales are highly correlated
which is to be expected as they are both facets of the Openness trait of the Big Five. The relationships
between the Behavioral History scales and AIMs scales also conform to expectations. Specifically, the
Behavioral History Scales are negatively correlated with the Competitive scale and positively correlated with
Exhibits a Positive Work Attitude and Corporate Citizenship scales. Additionally, Enjoys Problem Solving is
strongly correlated with the Performance scale of the Behavioral History Survey.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 42
Table 24. Correlations between the AIMs Scales
Exhibits a Positive
Work Attitude
Corporate Citizenship
Competitive Expressive
and Outgoing
Innovative and
Creative
Seeks Perfection
Develops Relationships
Enjoys Problem-Solving
Needs Structure
Corporate Citizenship
.68** 675
Competitive -.67**
675 -.85**
675
Expressive and Outgoing
-.23** 567
-.50** 567
.49** 567
Innovative and Creative
.25** 567
.08 567
-.05 567
.41** 567
Seeks Perfection
.11** 675
.14** 675
-.08* 675
.12** 567
.40** 567
Develops Relationships
.36** 675
.24** 675
-.24** 675
.38** 567
.68** 567
.43** 675
Enjoys Problem-Solving
.26** 675
.13** 675
-.13** 675
.33** 567
.72** 567
.43** 675
.54** 675
Needs Structure
.31** 675
.30** 675
-.30** 675
.00 567
.35** 567
.64** 675
.50** 675
.42** 675
Adaptability .52**
675 .39**
675 -.39**
675 .05
567 .55**
567 .28**
675 .61**
675 .49** 675
.32 675
Note. *p<.05, **p<.01
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 43
Table 25. Correlations between the Professional Behavioral History Scales
Performance Tenure
Tenure .17**
303
Unproductive Behavior .02
303 .09
303
Note. *p<.05, **p<.01
Table 26. Correlations between the Entry-Level Behavioral History Scales
Performance Tenure
Tenure .14**
510
Unproductive Behavior .05
510 .21**
553
Note. *p<.05, **p<.01
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 44
Table 27. Correlations between HR Avatar AIMs and Behavioral History Scales
Professional Entry-Level
Performance Tenure Unproductive Behavior
Performance Tenure Unproductive Behavior
Exhibits a Positive Work Attitude
.34** 513
.11** 513
.27** 1682
.34** 1069
.11** 1169
.33** 1169
Corporate Citizenship .27**
513 .22**
513 .35**
1682 .23**
1069 .20** 1169
.38** 1169
Competitive -.25**
513 -.23** 513
-.35** 1682
-.22** 1069
-.21** 1169
-.39** 1169
Expressive and Outgoing .24**
513 -.05
513 -.14**
1460 .25**
947 -.15**
947 -.17**
947
Innovative and Creative .42**
513 .00
513 .06*
1460 .55**
1069 -.06 947
.07* 947
Seeks Perfection .23**
513 .10*
513 .17**
1682 .28**
947 -.01
1169 .15**
1169
Develops Relationships .43**
513 .12**
513 .13**
1682 .47**
1069 .01
1169 .11**
1169
Enjoys Problem-Solving .48**
513 .05
513 .03
1682 .57**
1069 -.01
1169 .06
1169
Needs Structure .15**
513 .18**
513 .31**
1682 .24**
1069 .04
1169 .25**
1169
Adaptability .35**
513 .07
513 .13**
1682 .39**
1069 .02
1169 .15**
1169
Note. *p<.05, **p<.01
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 45
Table 28. Correlations between the Two Cognitive Workplace Simulation Scales: Attention to Detail and Analytical Thinking
Simulation Correlation
/N
Administration .28**
119
Business/Finance .26**
121
Entry-Level Business & Finance
.52** 137
Remote Customer Service .59**
98
Face-to-Face Customer Service
.48** 116
General Office Workplace .28**
99
Entry-Level Office .57**
142
Information Technology .54**
98
Manager .42**
97
Retail Sales .47**
60
Business Sales .48**
94
First-Line Supervisor .43**
102
Technician .54**
114
Basic Entry-Level .51**
171
Note. *p<.05, **p<.01
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 46
Table 29. Correlations between the AIMs and the Cognitive Workplace Simulation Scales
Cognitive Simulation
Scale Exhibits a
Positive Work Attitude
Corporate Citizenship
Competitive Expressive and
Outgoing
Innovative and
Creative
Seeks Perfection
Develops Relationships
Enjoys Problem-Solving
Needs Structure
Adaptability
Administration
Analytical Thinking
-.04 120
.13 120
-.11 120
-.04 120
-.14 120
-.10 120
-.11 120
.12 120
-.05 120
-.05 120
Attention to Detail
.13 119
.37** 119
14
-.40** 119
-.19* 119
-.07 119
.01 119
.05 119
.14 119
.18* 119
.01 119
Entry-Level Administration
Attention to Detail
.14 100
.29** 100
-.24* 100
-.14 100
.04 100
.17 100
.00 100
.07 100
.11 100
.01 100
Business/ Finance
Analytical Thinking
.02 122
.11 122
-.11 122
NA NA -.27** 122
-.11 122
-.01 122
-.23** 122
-.02 122
Attention to Detail
.20* 121
.33** 121
-.33** 121
NA NA .09
121 -.12
121 .04
121 .12
121 .00
121
Entry-Level Business & Finance
Analytical Thinking
.16 139
.25** 139
-.28** 139
-.17 139
.03 139
-.09 139
.03 139
.15 139
-.11 139
.05 139
Attention to Detail
.31** 137
.40** 137
-.49** 137
-.17* 137
.05 137
.04 137
.09 137
.10 137
.09 137
.13 137
Remote Customer Service
Analytical Thinking
.22* 98
.26** 98
-.31** 98
-.11 98
.00 98
-.14 98
.02 98
.21* 98
-.13 98
.06 98
Attention to Detail
.24* 95
.35** 95
-.31** 95
-.12 95
-.02 95
-.09 95
.05 95
.06 95
-.06 95
.08 95
Face-to-Face Customer Service
Analytical Thinking
-.02 116
.17 116
-.14 116
.01 116
-.06 116
-.04 116
-.11 116
.01 116
.02 116
-.04 116
Attention to Detail
.06 116
.33** 116
-.38** 116
-.13 116
.08 116
.15 116
.01 116
.11 116
.12 116
.07 116
General Office Workplace
Analytical Thinking
.03 100
.16 100
-.15 100
NA NA 100
.03 100
-.01 100
.33** 100
.03 100
.14 100
Attention to Detail
.20* 99
.33** 99
-.31** 99
NA NA -.05
99 .02
99 .09
99 .09
99 .14
99
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 47
Cognitive Simulation
Scale Exhibits a Positive Work Attitude
Corporate Citizenship
Competitive Expressive and
Outgoing
Innovative and
Creative
Seeks Perfection
Develops Relationships
Enjoys Problem-Solving
Needs Structure
Adaptability
Entry-Level Office
Analytical Thinking
.28** 142
.41** 142
-.47** 142
-.02 142
.08 142
.08 142
.17 142
.25** 142
.10 142
.23* 142
Attention to Detail
.21* 142
.30** 142
-.34** 142
-.08 142
.04 142
-.01 142
.05 142
.18* 142
.00 142
.09 142
Information Technology
Analytical Thinking
.08 98
.05 98
-.10 98
-.01 98
.11 98
-.13 98
.00 98
.15 98
-.21* 98
.06 98
Attention to Detail
-.01 98
.07 98
-.12 98
-.07 98
.03 98
.02 98
-.07 98
.07 98
-.15 98
-.01 98
Manager
Analytical Thinking
.19 98
.07 98
-.11 98
-.02 98
.04 98
-.04 98
.05 98
.12 98
-.07 98
.00 98
Attention to Detail
.05 97
.07 97
-.14 97
-.02 97
-.02 97
-.08 97
.04 97
.08 97
-.08 97
-.05 97
Retail Sales
Analytical Thinking
.06 60
.27* 60
-.28* 60
-.16 60
.01 60
-.30* 60
-.12 60
.00 60
-.29* 60
.07 60
Attention to Detail
.22 60
.33** 60
-.36** 60
-.32 60
-.04 60
.05 60
-.05 60
.06 60
.01 60
-.10 60
Business Sales
Analytical Thinking
.04 98
.25* 98
-.18 98
-.19 98
-.20* 98
.00 98
-.06 98
.01 98
-.07 98
-.18 98
Attention to Detail
.07 94
.47** 94
-.39** 94
-.09 94
-.14 94
.06 94
.12 94
.04 94
.09 94
.02 94
First-Line Supervisor
Analytical Thinking
.27** 102
.35** 102
-.33** 102
-.06 102
.10 102
.07 102
.05 102
.27** 102
.02 102
.04 102
Attention to Detail
.21* 102
.42** 102
-.40** 102
-.20* 102
-.06 102
-.10 102
-.05 102
.03 102
-.11 102
-.01 102
Technician
Analytical Thinking
.20* 114
.38** 114
-.36** 114
-.25** 114
.15 114
.02 114
.05 114
.26** 114
.06 114
.07 114
Attention to Detail
.21* 114
.39** 114
-.36** 114
-.35** 114
-.02 114
.03 114
-.09 114
.08 114
-.02 114
-.07 114
Basic Entry-Level
Analytical Thinking
.01 171
.11 171
-.14 171
-.11 171
-.12 171
-.05 171
-.16* 171
-.13 171
-.11 171
-.09 171
Attention to Detail
.17* 171
.24** 171
-.26** 171
-.06 171
-.04 171
-.02 171
-.04 171
-.03 171
.01 171
.02 171
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 48
Table 30. Correlations between the Behavioral History Survey and the Cognitive Workplace Simulation Scales
Cognitive Simulation
Professional Entry-Level
Scale Performance Tenure Unproductive Behavior
Performance Tenure Unproductive Behavior
Administration
Analytical Thinking
.15 120
.01 120
-.11 120
NA NA NA
Attention to Detail
.23* 119
.19* 119
.25** 119
NA NA NA
Entry-Level Administration
Attention to Detail
NA NA .24*
100 .20*
100 .11
100 .17
100
Business/Finance
Analytical Thinking
NA NA .10
122 -.02
122 .16
122 .17
122
Attention to Detail
NA NA .14
121 .08
121 .10
121 .17
121
Entry-Level Business & Finance
Analytical Thinking
NA NA .01
139 .19*
139 .15
139 .11
139
Attention to Detail
NA NA .22*
137 .26**
137 .12
137 .30**
137
Remote Customer Service
Analytical Thinking
.23* 98
.27** 98
.16 98
NA NA NA
Attention to Detail
.21* 98
.28** 98
.20* 98
NA NA NA
Face-to-Face Customer Service
Analytical Thinking
NA NA -.02
116 .03
116 .08
116 .00
116
Attention to Detail
NA NA .15 .09
116 .05
116 .13
116
General Office Workplace
Analytical Thinking
NA NA .03
100 .17
100 .01
100 .11
100
Attention to Detail
NA NA .07
99 .19
99 .25*
99 .17
99
Entry-Level Office
Analytical Thinking
NA NA .11
142 .30**
142 .04
142 .10
142
Attention to Detail
NA NA .03
142 .21*
142 .13
142 .06
142
Information Analytical .17 -.09 .03 NA NA NA
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 49
Cognitive Simulation
Professional Entry-Level
Scale Performance Tenure Unproductive Behavior
Performance Tenure Unproductive Behavior
Technology Thinking 98 98 98
Attention to Detail
.06 98
-.05 98
-.01 98
NA NA NA
Manager
Analytical Thinking
.19 98
.05 98
-.01 98
NA NA NA
Attention to Detail
.17 97
.18 97
.08 97
NA NA NA
Retail Sales
Analytical Thinking
NA NA .08
60 .10
60 .01
60 .13
60
Attention to Detail
NA NA .16
60 -.03
60 .15
60 .21
60
Business Sales
Analytical Thinking
.16 98
.26 98
.07 98
NA NA NA
Attention to Detail
.21 94
.18 94
.20 94
NA NA NA
First-Line Supervisor
Analytical Thinking
NA NA .10
102 .29**
102 -.07
102 .17
102
Attention to Detail
NA NA .12
102 .17
102 .13
102 .25*
102
Technician
Analytical Thinking
NA NA .04
114 .32**
114 .13
114 .08
114
Attention to Detail
NA NA .04
114 .10
114 .20*
114 .09
114
Basic Entry-Level
Analytical Thinking
NA NA .25
171 -.07
171 .15*
171 .06
171
Attention to Detail
NA NA .07
171 .07
171 .23**
171 .13
171
Note. *p<.05, **p<.01
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 50
Technical Requirements
The HR Avatar solution is designed to be taken on a personal computer or a mobile device
including tablets and mobile phones. A high bandwidth connection is recommended, but not
required. All HR Avatar videos are compressed to less than 500kbps. Lower bitrate versions are
used for mobile devices.
The following web browsers are supported:
Internet Explorer 6 and above with Flash 9.1.115 or above
Internet Explorer 9 and above without Flash
Chrome
Firefox
Opera
Safari
The Flesch-Kincaid Reading Grade Level score is estimated to be 5.8. This indicates that an
applicant must have a reading level similar to that of a 5th or 6th grader in order to comprehend the
text in the assessment.
HR Avatar recommends that the applicant take the assessment in a quiet setting that is free from
distractions. This will allow the applicant the best opportunity for demonstrating their skills and
abilities.
Future Research
HR Avatar is committed to providing employers with high quality and legally defensible assessments
for hiring employees. To that end, HR Avatar plans to continue accumulating reliability, validity, and
fairness evidence to support the use of the solutions. The list below contains several items from our
research agenda. Please contact us, if you have any interest in partnering with HR Avatar on any of
the projects below.
Reliability
Establish the internal consistency of the knowledge tests.
Establish the test-retest reliability of the knowledge tests, the Typing Test and the Essay Test.
Establish the test-retest reliability of the composite scores for each solution.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 51
Validity
Conduct studies to establish the content-related validity of the knowledge assessments,
Typing Test and Essay Test.
Conduct a study to examine the convergent validity evidence for the Cognitive Work
Simulation by comparing scores on the Cognitive Work Simulation and other established
measures of cognitive ability.
Conduct a study to examine the convergent and divergent validity evidence for the AIMS
assessment by comparing scores on the AIMS assessment with other established measures of
personality – particularly those that measure the Big Five. The hypothesized relationships
can be found in Figure 1.
Accumulate additional criterion-related validity evidence at the individual assessment level and the
composite level by conducting multiple studies on each solution. Using these studies, conduct meta-
analyses to provide evidence for validity generalization. Validation evidence is available in two
studies that demonstrate the validity of the assessments. The first is a study with 130 managers and
the second is a study with 64 managers. In the first study, the assessment demonstrated that it
predicts job performance (r=.25; p<.01), using a simple “High” or “Marginal” categorization of job
performance, and a second study demonstrated the assessment predicted performance on a 100
point administrative performance appraisal, using a Spearman correlation (r=.25; p<.05). We expect
validity evidence to be more robust in future studies with better criterion measures. A brief summary
of the results is presented in Appendix G.
Fairness
Conduct a sensitivity review of all assessment content.
Conduct Differential Item Functioning (DIF) analyses for the items in the assessments to
determine if any of the items behave differently for subgroups as defined by race, ethnicity,
gender, and age group.
Evaluate mean score differences on each assessment and at the composite level by subgroup.
Simulate selection ratios for each group at various passing rates to estimate adverse impact
ratios.
Norms
Update estimates of global norms and estimate norms at the country and/or region level.
Compare scores across formats (PCs, tablets, mobile phones).
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 52
References
American Educational Research Association, American Psychological Association, National Council
on Measurement in Education. (2014). The Standards for Educational and Psychological Testing.
Washington DC: American Educational Research Association.
Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job performance: A
meta-analysis. Personnel Psychology, 44, 1-26.
Bartram, D. (2005). The great eight competencies: A criterion-centric approach to validation. Journal
of Applied Psychology, 90, 1185-1203.
Bertua, C., Anderson, N., & Salgado, J. F. (2005). The predeictive validity of cognitive ability tests: A
UK meta-analysis. Journal of Occupational and Organizational Psychology, 78, 387-409.
Borman, W. C., & Motowidlo, S. J. (1993). Expanding the criterion domain to include elements of
contextual performance. In N. Schmitt, & W. C. Borman, Personnel Selection in Organizations.
San Francisco: Jossey-Bass.
Borman, W. C., & Motowidlo, S. J. (1997). Task performance and contextual performance: The
meaning for personnel selection research. Human Performance, 10, 99-109.
Buhrmester, M., Kwang, T., & Gosling, S. D. (n.d.). Amazon's Mechanical Turk: A new source of
inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.
Carlson, K. D., Scullen, S. E., Schmidt, F. L., Rothstein, H., & Erwin, F. (1999). Generalizable
biographical data validity can be achieved without multi-organziational development and
keying. Personnel Psychology, 52, 731-755.
Carroll, J. (1993). Human Cognitive Abilities: A Survey of Factor Analytic Studies. New York: Cambridge
University Press.
Cattell, R. B. (1941). Some theoretical issues in adult intelligence testing. Psychological Bulletin, 38, 592.
Cohen, J. (1992). A Power Primer. Psychological Bulletin, 112, 155-159.
Dye, D. A., Reck, M., & McDaniel, M. A. (2007). The validity of job measures. International Journal of
Selection and Assessment, 1, 153-157.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 53
edX. (2015, February 25). Retrieved from discern: NY Times
Equal Employment Opportunity Commission, C. S. C. U. S. D. L. U., & Equal Employment
Opportunity Commission. (1978). Unifrom Guidelines on Employee Selection Procedures. Federal
register, 43(166), 38295-38309.
Fleisher, M. S., & Tsacoumis, S. (2012). O*NET analysit occupational skills ratings: Procedures update.
(Tech. Rep. No. FR-11-67) Alexandria, VA.: Human Resources Research Orgnaization
(HumRRO).
Fleisher, M. S., & Tsacoumis, S. (2012). O*NET analyst occupational abilities ratings: procedure update.
(Tech. Rep. No. FR-11-66). Alexandria, VA: Human Resources Research Organization
(humRRO).
Goldberg, L. R. (1992). The development of markers for the Big-Five factor structure. Psychological
Assessment, 4, 26-42.
Goldberg, L. R. (1993). The structure of phenotypic personlity traits. American Psychologist, 48, 26-34.
Hogan, J., & Holland, B. (2003). Using theory to evaluate personality and job-performance relations:
A socioanalytic perspective. Journal of Applied Psychology, 88, 100-112.
Horn, J. (1985). Handbook of Intelligence. New York: Wiley.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job
performance. Psychological Bulletin, 96, 72-98.
Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance: The Big Five revisited.
Journal of Applied Psychology, 85, 869-879.
Jaro, M. A. (1989). Advances in record linkage methodology as applied to the 1985 census of Tampa
Florida. Journal of the American Statistical ASsociation, 84, 414-420.
Jaro, M. A. (1995). Probalistic linkage of large public health data file. Statistics in Medicine, 14, 491-498.
Johnson, D. R., & Borden, L. A. (2012). Particiants at your fingertips: Using Amazon's Mechanical
Turk to increase student-faculyy collaborative research. Teaching of Psychology, 39, 245-251.
Judge, T. A., & Ilies, R. (2002). Relationship of personality to performance motivation: A meta-
analytic review. Journal of Applied Psychology, 87, 797-807.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 54
Kanfer, R., Crosby, J. R., & Brandt, D. M. (1988). Investigating behavioral antecedenty of turnover
at three job tenure levels. Journal of Applied Psychology, 73, 331-335.
Kehoe, J. (2002). General mental ability and selection in private sector organizations: A commentary.
Human Performance, 15, 97-106.
Klimoski, R. J., & Childs, A. (1986). Successfully predicting career success: An application of the
biographical inventory. Journal of Applied Psychology, 71, 3-8.
Markoff, J. (2013, April 4). Essay-Grading Software Offers Professors a Break. NY Times. Retrieved
February 25, 2015, from http://www.nytimes.com/2013/04/05/science/new-test-for-
computers-grading-essays-at-college-level.html?_r=0
McCrae, R. R., & Costa, P. T. (1997). Personality trait structure as a human universal. American
Psychologist, 52, 509-516.
McCrae, R. R., & John, O. P. (1992). An introduction to the five-factor model and its applications.
Journal of Personality, 60, 175-215.
McCrae, R. R., Jang, K. L., Livesley, W. J., Riemann, R., & Angleitner, A. (2001). Sources of
structure: Genetic, environmental, and artifactual influences on the covariation of personality
traits. Journal of Personality, 69, 511-535.
McGrew, K. (2008). CHC theory and the human cognitive abilities project: Standing on the shoulers
of the giants of psychometric intelligence research. Intelligence, 37, 1-10.
Miller, J. D., Gentile, B., Wilson, L., & Campbell, W. K. (2013). Grandiose and vulnerable narcissism
and the DSM-5 Pathelogical Personality Trait Model. Journal of Personality Assessment, 95, 284-
290.
Minton, E., Gurel-Atay, E., Kahle, L., & Ring, K. (2013). Comparing data collection alternatives:
Amazon mTurk, college students, and secondary analysis. AMA Winter Educators' Conference
Proceedings, 24, (pp. 36-37).
Mount, M. K., & Barrick, M. R. (1995). The Big Five personality dimensions: Implications for
research and practice in human resources management. In K. R. (Eds.), Research in Personnel
and Human Resources Management (Vol. 13) (pp. 153-200). Greenwich, CT: JAI Press.
Navarro, G. (2001). A guided tour to approximate string matching. ACM Computing Surveys, 33, 31-
88.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 55
Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a
biodata measure and situational jusgment inventory as predictors of college student
performance. Journal of Applied Psychology, 89, 187-207.
Paruchuri, V. (2015, February 25). Retrieved from YouTube:
https://www.youtube.com/watch?v=zFeP678054U
Paunonen, S. V., Rothstein, M. G., & Jackson, D. (1999). Narrow reasoning about the use of broad
personaloty measures for personnel selection. Journal of Organizational Behavior, 20, 389-405.
Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., Fleishman, E. A., Levin, K.
Y., . . . Dye, D. M. (2001). Understanding work using the occupational information network
(O*NET): Implications for practice and research. Personnel Psychology, 54, 451-492.
Rothstein, H. R., Schmidt, F., Erwin, F. W., Owens, W. A., & Sparks, C. P. (1990). Biographical data
in employment selection: Can validities be made generalizable? Journal of Applied Psychology, 75,
175-184.
Saad, S., Carter, G. W., Rothenberg, M., & Israelson, E. (2014, March 9). Testing and Assessment: An
Employer's Guide to Good Practices. Retrieved from O*NET:
http://www.onetcenter.org/dl_files/empTestAsse.pdf
Salgado, J. E. (1997). The Five Factor Model of personality and job performance in the European
community. Journal of Applied Psychology, 82, 30-43.
Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & De Fruyt, F. (2003). International validity
generalization of GMA and cognitive abilities: A European community meta-analysis.
Personnel Psychology, 56, 573-605.
Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., de Fruyt, F., & Rolland, J. P. (2003). A Meta-
Analytic Study of General Mental Ability Validity for Different Occupations in the
Eurpopean Community. Journal of Applied Psychology, 88, 1068-1081.
Salgado, J., & Anderson, N. (2003). Validity generalization of GMA tests across countries in the
European community. European Journal of Work & Organizational Psychology, 12, 1-17.
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel
psychology: Practical and theoretical implications of 85 years of research findings.
Psychological Bulletin, 124, 262-274.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 56
Schneider, R. J., Hough, L. M., & Dunnette, M. D. (1996). Broadsided by broad traits: How to sink
science in five dimensions or less. Journal of Organizational Behavior, 17, 639-655.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass Corrlations: Uses in Assessing Rater Reliability.
Psychological Bulletin, 86, 420-428.
Society for Industrial Organizational Psychology. (2003). Principles for the Validation and Use of Personnel
Selection Procedures (4th ed.). Bowling Green, OH: Author.
Tett, R. P., & Burnett, D. D. (2003). A personality trait-based interactionist model of job
performance. Journal of Applied Psychology, 88, 500-517.
Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job
performance. Personnel Psychology, 44, 703-742.
U.S. Census Bureau. (2015, February 17). American Fact Finder. Retrieved from factfinder.census.gov:
factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=ACS_12_1YR
_CP05&prodType=table
Wee, S., Newman, D. A., & Joseph, D. L. (2014). More than g: Selection quality and adverse impact
implications of considering second-stratum cognitive abilities. Journal of Applied Psychology, 99,
547-563.
Winkler, W. E. (1990). String comparator metrics and enhanced decision rules in the Fellegi-Sunter
Model of record linkage. Proceedsings of the Section on Survey Research Methods (pp. 354-359).
American Statistical Association.
Woo, S. E., Chernyshenko, O. S., Stark, S. E., & Conz, G. (2014). Validity of six openness facets in
predicting wok behaviors: A meta-analysis. Journal of Personality Assessment, 96, 76-86.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 57
Appendix A: Summary of the HR Avatar Solutions
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Account Manager 43-4051.00 Professional Professional General Office
Workplace Customer Service
Fundamentals √
Accountant / Auditor 13-2011.00 Professional Professional Business/Finance Accounting Concepts
√
Administrator - Elementary and
Secondary School 11-9032.00 Professional Professional
General Office Workplace
√
Agent - Purchasing 13-1023.00 Professional Professional Business/Finance
Aide - Home Health 31-1011.00 Professional Professional Face-to-Face
Customer Service
Aide - Personal Care 39-9021.00 Entry-Level Entry-Level Face-to-Face
Customer Service
Analyst - Financial 13-2051.00 Professional Professional Business/Finance Accounting Concepts
√
Analyst - Market Research
13-1161.00 Professional Professional Business/Finance Marketing Concepts √
Assembler - Electrical and Electronic
Equipment 51-2022.00 Entry-Level Entry-Level Basic Entry-Level
Electrician Fundamentals
Assembler / Fabricator - Other
51-2099.00 Entry-Level Entry-Level Basic Entry-Level
Attendant - Amusement /
Recreation 39-3091.00 Entry-Level Entry-Level Basic Entry-Level
Attendant - Food Services
35-3022.00 Entry-Level Entry-Level Face-to-Face
Customer Service
Attorney 23-1011.00 Professional Professional General Office
Workplace √
Baker 51-3011.00 Entry-Level Entry-Level Basic Entry-Level
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 58
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Bank Teller 43-3071.00 Professional Professional Face-to-Face
Customer Service Banking
Fundamentals
Bartender 35-3011.00 Entry-Level Entry-Level Face-to-Face
Customer Service
Carpenter 47-2031.00 Entry-Level Entry-Level Basic Entry-Level Carpentry
Fundamentals
Cashier 41-2011.00 Professional Entry-Level Face-to-Face
Customer Service
Chief Executive 11-1011.00 Professional Professional Manager √
Childcare Worker 39-9011.00 Professional Professional Entry-Level Office Childcare
Fundamentals
Claims Adjuster, Examiner,
Investigator 13-1031.00 Professional Professional
Entry-Level Business & Finance
√
Cleaner - Vehicles and Equipment
53-7061.00 Entry-Level Entry-Level Basic Entry-Level
Clerk - Billing and Posting
43-3021.00 Entry-Level Entry-Level Entry-Level
Administration
Clerk - Bookkeeping, Accounting, and
Auditing 43-3031.00 Entry-Level Entry-Level
Entry-Level Administration
Accounting Concepts
Clerk - Counter / Rental
41-2021.00 Entry-Level Entry-Level Face-to-Face
Customer Service
Clerk - File 43-4071.00 Entry-Level Entry-Level Entry-Level
Administration √
Clerk - General Office 43-9061.00 Entry-Level Entry-Level Entry-Level
Administration
Clerk - Hotel, Motel, Resort
43-4081.00 Professional Professional Face-to-Face
Customer Service Hospitality Concepts
Clerk - Information and Record Clerks
43-4199.00 Entry-Level Entry-Level Entry-Level
Administration √
Clerk - Insurance 43-9041.00 Professional Professional Entry-Level Business Insurance
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 59
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Claims / Policy Processing
& Finance Fundamentals
Clerk - Order Processing
43-4151.00 Entry-Level Entry-Level Entry-Level
Administration √
Clerk - Payroll and Timekeeping
43-3051.00 Entry-Level Entry-Level Entry-Level
Administration √
Clerk - Production, Planning, and
Expediting 43-5061.00 Entry-Level Entry-Level
Entry-Level Administration
Clerk - Shipping / Receiving
43-5071.00 Entry-Level Entry-Level Entry-Level
Administration
Clerk - Stockroom 43-5081.00 Entry-Level Entry-Level Entry-Level
Administration
Coach / Scout 27-2022.00 Professional Professional General Office
Workplace √
Collections Specialist 43-3011.00 Professional Professional Remote Customer
Service
Compliance Officer 13-1041.00 Professional Professional General Office
Workplace √
Computer Applications Software
Developer 15-1132.00 Professional Professional
Information Technology
√
Computer Programmer
15-1131.00 Professional Professional Information Technology
√
Computer Programmer - Actionscript
15-1131.00 Professional Professional Information Technology
Core ActionScript 3 √
Computer Programmer - C
15-1131.00 Professional Professional Information Technology
Core C √
Computer Programmer - C++
15-1131.00 Professional Professional Information Technology
Core C++ √
Computer 15-1131.00 Professional Professional Information Core Java √
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 60
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Programmer - Java Technology
Computer Programmer - Java
EE 15-1131.00 Professional Professional
Information Technology
Java EE √
Computer Programmer -
Javascript 15-1131.00 Professional Professional
Information Technology
Core Java Script √
Computer Programmer - PHP
15-1131.00 Professional Professional Information Technology
Core PHP √
Computer Programmer - Python
15-1131.00 Professional Professional Information Technology
Python √
Computer Programmer - Ruby
15-1131.00 Professional Professional Information Technology
Core Ruby √
Computer Programmer - Web
Developer 15-1131.00 Professional Professional
Information Technology
Html5/CSS3 √
Computer Programmer - Web
Developer with jQuery 15-1131.00 Professional Professional
Information Technology
Html5/CSS3 √
Computer Systems Analyst
15-1121.00 Professional Professional Information Technology
√
Computer Systems Software Developer
15-1133.00 Professional Professional Information Technology
√
Cook - Fast Food 35-2011.00 Entry-Level Entry-Level Basic Entry-Level Food Safety
Cook - Institution and Cafeteria
35-2012.00 Entry-Level Entry-Level Basic Entry-Level Food Safety
Cook - Restaurant 35-2014.00 Entry-Level Entry-Level Basic Entry-Level Food Safety
Cook - Short Order 35-2015.00 Entry-Level Entry-Level Basic Entry-Level Food Safety
Correctional Officer 33-3012.00 Professional Professional General Office
Workplace
Cost Estimator 13-1051.00 Professional Professional Entry-Level Business
& Finance
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 61
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Counselor - Educational,
Guidance, School, Vocational
13-1051.00 Professional Professional General Office
Workplace √
Customer Service Face-to-Face
21-1012.00 Professional Entry-Level Face-to-Face
Customer Service
Customer Service Representative
43-4051.00 Professional Entry-Level Remote Customer
Service Customer Service
Fundamentals √
Data Entry Keyers 43-9021.00 Entry-Level Entry-Level Entry-Level
Administration √
Database Administrator
15-1141.00 Professional Professional Information Technology
Relational Database Concepts
√
Dental Assistant 31-9091.00 Entry-Level Entry-Level Basic Entry-Level
Dental Hygienist 29-2021.00 Professional Professional Basic Entry-Level
Dishwasher 35-9021.00 Entry-Level Entry-Level Basic Entry-Level
Dispatcher - General 43-5032.00 Professional Professional General Office
Workplace
Driver - Heavy and Tractor-Trailer
53-3032.00 Professional Professional Entry-Level Office
Driver - Light Truck / Delivery
53-3033.00 Professional Professional General Office
Workplace
Driver - Sales and Delivery
53-3031.00 Professional Professional Retail Sales
Driver - School Bus 53-3022.00 Entry-Level Entry-Level Basic Entry-Level
Driver Transit and Intercity Bus
53-3021.00 Professional Professional Basic Entry-Level
Electrical Engineer 17-2071.00 Professional Professional General Office
Workplace √
Electrician 47-2111.00 Entry-Level Entry-Level Technician Electrician
Fundamentals
Enforcement Officer 33-3051.00 Professional Professional General Office
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 62
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Workplace
Engineer - Civil 17-2051.00 Professional Professional General Office
Workplace Construction Fundamentals
√
Engineer - Industrial 17-2112.00 Professional Professional General Office
Workplace √
Engineer - Mechanical
17-2141.00 Professional Professional General Office
Workplace √
Executive Secretary / Administrative
Assistant 43-6011.00 Professional Professional Administration √ √
Fast Food Worker 35-3021.00 Entry-Level Entry-Level Face-to-Face
Customer Service
Firefighter 33-2011.00 Professional Professional Entry-Level Office
First Line Supervisor - Office and
Administrative Support
43-1011.00 Professional Professional First-Line Supervisor First Line Supervisor
Fundamentals √
First-Line Supervisor - Construction /
Extraction 47-1011.00 Professional Professional First-Line Supervisor
First Line Supervisor
Fundamentals & Construction Fundamentals
First-Line Supervisor - Food Preparation /
Serving 35-1012.00 Professional Professional First-Line Supervisor Food Safety
First-Line Supervisor - Helpers, Laborers,
and Material Movers 53-1021.00 Professional Professional First-Line Supervisor
First-Line Supervisor - Housekeeping and
Janitorial 37-1011.00 Professional Professional First-Line Supervisor
First-Line Supervisor - 49-1011.00 Professional Professional First-Line Supervisor First Line
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 63
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Mechanics, Installers, Repairers
Supervisor Fundamentals
First-Line Supervisor - Non-Retail Sales
41-1012.00 Professional Professional First-Line Supervisor First Line Supervisor
Fundamentals √
First-Line Supervisor - Production / Operations
51-1011.00 Professional Professional First-Line Supervisor First Line Supervisor
Fundamentals √
First-Line Supervisor - Retail Sales
41-1011.00 Professional Professional First-Line Supervisor First Line Supervisor
Fundamentals
First-Line Supervisor - Transportation and
Material-Moving 53-1031.00 Professional Professional First-Line Supervisor
Food Preparation Worker
35-2021.00 Entry-Level Entry-Level Basic Entry-Level Food Safety
Food Server - Nonrestaurant
35-3041.00 Entry-Level Entry-Level Basic Entry-Level
General Project Manager
11-1021.00 Professional Professional Manager Project Management
Concepts √
Graphic Designer - Web Development
27-1024.00 Professional Professional General Office
Workplace Html5/CSS3 &
HTML/CSS
Hairdresser, Hairstylist,
Cosmetologist 39-5012.00 Entry-Level Entry-Level Basic Entry-Level
Helper - Dining Room and Cafeteria
35-9011.00 Entry-Level Entry-Level Basic Entry-Level
Helper - Production 51-9198.00 Entry-Level Entry-Level Basic Entry-Level
Host / Hostess - Restaurant
35-9031.00 Entry-Level Entry-Level Face-to-Face
Customer Service
Fundamental Hospitality Concepts
Information 15-1199.09 Professional Professional Information Project Management √
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 64
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Technology Project Manager
Technology Concepts
Inspector, Tester, Sorter, Sampler,
Weigher 51-9061.00 Professional Professional Technician
Installer / Repairer - Telecommunications
Equipment 49-2022.00 Entry-Level Entry-Level Basic Entry-Level
Janitor 37-2011.00 Entry-Level Entry-Level Basic Entry-Level
Laborer - Agricultural 45-2092.00 Entry-Level Entry-Level Basic Entry-Level
Laborer - Construction
47-2061.00 Entry-Level Entry-Level Basic Entry-Level Construction Fundamentals
Laborer - Freight and Warehouse
53-7062.00 Entry-Level Entry-Level Basic Entry-Level
Laborer - Landscaping and Groundskeeping
37-3011.00 Entry-Level Entry-Level Basic Entry-Level
Laborer - Packing / Packaging
53-7064.00 Entry-Level Entry-Level Basic Entry-Level
Laundry and Dry-Cleaning Worker
51-6011.00 Entry-Level Entry-Level Basic Entry-Level
Legal Secretary 43-6012.00 Professional Professional Administration √ √
Loan Interviewer 43-4131.00 Professional Professional Business/Finance Banking
Fundamentals √
Loan Officer 13-2072.00 Professional Professional Business/Finance Banking
Fundamentals √
Machinist 51-4041.00 Entry-Level Entry-Level Basic Entry-Level
Maid / Housekeeping Cleaner
37-2012.00 Entry-Level Entry-Level Basic Entry-Level
Mail Carrier 43-5052.00 Entry-Level Entry-Level Face-to-Face
Customer Service
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 65
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Management Analyst 13-1111.00 Professional Professional Business/Finance Accounting Concepts
√
Manager - Administrative
Services 11-3011.00 Professional Professional Manager √
Manager - Architectural and
Engineering 11-9041.00 Professional Professional Manager √
Manager - Computer and Information
Systems 11-3021.00 Professional Professional Manager √
Manager - Construction
11-9021.00 Professional Professional Manager Construction Fundamentals
Manager - Financial 11-3031.00 Professional Professional Manager Accounting Concepts
√
Manager - Food Service
11-9051.00 Professional Professional Manager Food Safety √
Manager - Human Resources
11-3121.00 Professional Professional Manager Human Resources
Fundamentals √
Manager - Industrial Production
11-3051.00 Professional Professional Manager √
Manager - Marketing 11-2021.00 Professional Professional Manager Marketing Concepts √
Manager - Medical and Health Services
(General) 11-9111.00 Professional Professional Manager
Health Care Administration (US)
√
Manager - Medical and Health Services
(US) 11-9111.00 Professional Professional Manager
Health Care Administration (US)
√
Manager - Other 11-9199.00 Professional Professional Manager √
Manager - Real Estate and Community
Association 11-9141.00 Professional Professional Manager √
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 66
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Manager - Sales 11-2022.00 Professional Professional Manager Sales Concepts √
Mechanic - Bus,Truck, Diesel
Engine 49-3031.00 Entry-Level Entry-Level Technician
Mechanic - Heating, Air Conditioning,
Refrigeration 49-9021.00 Professional Professional Technician
HVAC Fundamentals
Mechanic - Industrial Machinery
49-9041.00 Entry-Level Entry-Level Technician
Medical / Clinical Laboratory
Technologist 29-2011.00 Professional Professional Technician
Medical Assistant 31-9092.00 Professional Professional General Office
Workplace
Network and Computer Systems
Administrator 15-1142.00 Professional Professional
Information Technology
Computer Network Concepts
√
Nurse - Licensed Practical / Vocational
29-2061.00 Professional Professional General Office
Workplace
Nurse - Registered 29-1141.00 Professional Professional General Office
Workplace √
Nursing Assistant 31-1014.00 Professional Professional Face-to-Face
Customer Service
Operating Engineer 47-2073.00 Professional Professional General Office
Workplace
Operator - Packaging / Filling Machines
53-7051.00 Entry-Level Entry-Level Technician
Operator - Industrial Trucks / Tractors
51-4031.00 Entry-Level Entry-Level Basic Entry-Level
Operator - Machine - Metal and Plastic
51-9111.00 Entry-Level Entry-Level Technician
Painter - Construction 47-2141.00 Entry-Level Entry-Level Basic Entry-Level
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 67
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
and Maintenance
Paralegal /Legal Assistant
23-2011.00 Professional Professional General Office
Workplace √ √
Personal Financial Advisor
13-2052.00 Professional Professional Business/Finance Accounting Concepts
√
Pharmacist (General) 29-1051.00 Professional Professional General Office
Workplace Health Care
Administration (US) √
Pharmacist (US) 29-1051.00 Professional Professional General Office
Workplace Health Care
Administration (US) √
Physical Therapist 29-1123.00 Professional Professional Entry-Level Office
Plumber, Pipefitter, Steamfitter
47-2152.00 Entry-Level Entry-Level Basic Entry-Level Plumbing
Fundamentals
Printing Press Operators
51-5112.00 Entry-Level Entry-Level Technician
Production Worker 51-9199.00 Entry-Level Entry-Level Entry-Level Office
Radiologic Technologist
29-2034.00 Professional Professional General Office
Workplace
Receptionist 43-4171.00 Entry-Level Entry-Level Entry-Level
Administration
Recreation Worker 39-9032.00 Entry-Level Entry-Level Basic Entry-Level
Recruiter 43-4111.00 Professional Professional General Office
Workplace Recruiting Concepts √
Retail Salesperson 41-2031.00 Professional Entry-Level Retail Sales Sales Concepts
Sales Agent - Insurance
41-3021.00 Professional Professional Business Sales
Sales Situation Analysis & Insurance
Fundamentals
Sales Agent - Real Estate
41-9022.00 Professional Professional Business Sales Sales Situation
Analysis & Sales Concepts
√
Sales Agent - 41-3031.00 Professional Professional Business Sales Sales Situation √
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 68
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Securities, Financial Services
Analysis & Sales Concepts
Sales Representative - Services
41-3099.00 Professional Professional Business Sales Sales Situation
Analysis & Sales Concepts
√
Sales Representative - Technical and
Scientific 41-4011.00 Professional Professional Business Sales
Sales Situation Analysis &
Sales Concepts √
Sales Representative - Wholesale &
Manufacturing 41-4012.00 Professional Professional Business Sales
Sales Situation Analysis &
Sales Concepts √
Salesperson - Parts and Accessories
41-2022.00 Professional Professional Retail Sales
Secretary - Medical (General)
43-6013.00 Professional Professional Administration √
Secretary - Medical (US)
43-6013.00 Professional Professional Administration Health Care
Administration (US) √
Secretary / Administrative
Assistant 43-6014.00 Professional Entry-Level Administration √ √
Security Guard 33-9032.00 Professional Professional Entry-Level Office
Social / Human Service Assistant
21-1093.00 Professional Professional Entry-Level Office Social Services Fundamentals
Social Worker - Child, Family, School
21-1021.00 Entry-Level Entry-Level Face-to-Face
Customer Service Social Services Fundamentals
Specialist - Computer Network Support
15-1152.00 Professional Professional Information Technology
Computer Network Concepts
Specialist - Computer User Support
15-1151.00 Professional Professional Information Technology
Customer Service Fundamentals
Specialist - Human Resources
13-1071.00 Professional Professional Manager Human Resources
Fundamentals √
Specialist - Office and 43-9199.00 Entry-Level Entry-Level Entry-Level √
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 69
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Administrative Support
Administration
Specialist - Public Relations
27-3031.00 Professional Professional General Office
Workplace Marketing Concepts √
Specialist - Training and Development
13-1151.00 Professional Professional General Office
Workplace √
Taxi Driver / Chauffeur
53-3041.00 Entry-Level Entry-Level Basic Entry-Level
Teacher - Elementary School
25-2021.00 Professional Professional General Office
Workplace Education Delivery
Fundamentals √
Teacher - Health Specialties -
Postsecondary 25-1071.00 Professional Professional
General Office Workplace
Education Delivery Fundamentals
√
Teacher - Kindergarten
25-2012.00 Professional Professional General Office
Workplace Education Delivery
Fundamentals √
Teacher - Middle School
25-2022.00 Professional Professional General Office
Workplace Education Delivery
Fundamentals √
Teacher - Other 25-3099.00 Professional Professional General Office
Workplace Education Delivery
Fundamentals
Teacher - Postsecondary
25-1199.00 Professional Professional General Office
Workplace Education Delivery
Fundamentals √
Teacher - Preschool 25-2011.00 Entry-Level Entry-Level Face-to-Face
Customer Service Education Delivery
Fundamentals
Teacher - Secondary School
25-2031.00 Professional Professional General Office
Workplace Education Delivery
Fundamentals √
Teacher - Self-Enrichment Education
25-3021.00 Professional Professional General Office
Workplace Education Delivery
Fundamentals √
Teacher - Special Education
25-2052.00 Professional Professional General Office
Workplace Education Delivery
Fundamentals √
Teacher - Substitute 25-3098.00 Professional Professional General Office
Workplace Education Delivery
Fundamentals √
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 70
Solution O*NET SOC AIMS Form Behavioral
History Form Cognitive Work
Simulation Job Specific
Knowledge Test Essay Test
Typing Test
Teacher Assistant 25-9041.00 Professional Professional Entry-Level Office Education Delivery
Fundamentals
Team Assembler 51-2092.00 Entry-Level Entry-Level Basic Entry-Level
Technician - Automotive Service
49-3023.00 Entry-Level Entry-Level Technician
Technician - Emergency Medical /
Paramedic 29-2041.00 Professional Professional Technician
Technician - General Maintenance and
Repair 49-9071.00 Professional Professional Technician
Technician - Medical and Clinical Laboratory
29-2012.00 Professional Professional Technician
Technician - Medical Records and Health
Information 29-2071.00 Professional Professional Entry-Level Office
Health Care Administration (US)
Technician - Pharmacy
29-2052.00 Professional Professional Technician
Telemarketer 41-9041.00 Professional Professional Business Sales Sales Situation
Analysis & Sales Concepts
Trainer - Athletic 39-9031.00 Professional Professional Entry-Level Office
Trimmer - Meat, Poultry, and Fish
51-3022.00 Entry-Level Entry-Level Basic Entry-Level Food Safety
Waiter / Waitress 35-3031.00 Entry-Level Entry-Level Face-to-Face
Customer Service
Welder, Cutter, Solderer, Brazer
51-4121.00 Entry-Level Entry-Level Technician
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 72
Appendix B: Historical Validity Evidence for the
Cognitive Scales
Table 31. Compiled Validity Evidence for the Original Content of the Cognitive Work Simulation Scales
Performance Factor (various organizations)
Attention to Detail
Analytical Thinking
Listening Score (Organization A) .37
Performance Rating .50
Listening Score (Organization B) -.43
Listening Score (Organization C) -.35 -.34
Performance Rating (Organization A) .42
Performance Rating (Organization B) .39
Sales/Hour -.26 -.34
Cross Selling .37 .39
Response Quality .33
Average 2nd Contact .21
Schedule Conformance -.09
Performance Rating (Organization C) .39 .16
*Study results provided by original content developer. Sample sizes are all larger than 200 and p<.05
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 73
Appendix C: Historical Validity Evidence for AIMS
Tables 32-38 contain the results of criterion-related validity studies conducted by the developer of the original AIMS content. Please note that
these results are based on the longer, 100-item version of the assessment. Table 36 summarizes the statistically significant relationships (p<.05)
between the original AIMs scales and various measures of performance. The table includes results for over 5000 applicants and over 12 different
organizations from multiple industries including Financial Services, Insurance, Hospitality, Market Research, and Pharmaceuticals. Note that all
significant correlations are reported and although some of these correlations are negative, we would not expect all of the scales to be positively
related to all job performance dimensions (see above section on previous personality research).
Table 32. Study 1 Results: Concurrent Validation Study for Insurance Consultants N=122-136
Competency Performance
Appraisal Average
Policy Count Average
QRF Level Average
Idle Time Average CPH
Service Average Second
Contact
Needs Structure .18 -.15 .18
Innovative & Creative .16 .19
Enjoys Problem-Solving .24
Seeks Perfection
Exhibits a Positive Work Attitude -.19
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 74
Table 33. Study 2 Results: Concurrent Validation Study for Inside Sales N=105
Competency Perfectionism Quality
Innovative & Creative .26
Enjoys Problem-Solving .26
Competitive
Seeks Perfection .19
Develops Relationships .28 .16
Expressive & Outgoing .27 .18
Corporate Citizenship -.17
Exhibits a Positive Work Attitude
-.18
Adaptable -.17
Table 34. Study 3 Results: Concurrent Validation Study for an Internet Services Order Processing N=84
Competency Quality
Enjoys Problem-Solving .19
Develops Relationships -.27
Competitive -.22
Table 35. Study 4 Results: Concurrent Validation Study for an Internet and Cable Sales and Service Position N=72-93
Competency Performance Quality Attendance
Needs Structure .30 .26
Enjoys Problem-Solving .20
Competitive
Seeks Perfection .21
Develops Relationships .24
Exhibits a Positive Work Attitude .25
Adaptable -.20
Table 36. Study 5 Results: Concurrent Validation Study Auto Rental Sales Role N=80-92
Competency Quality Yield Productivity
Needs Structure .30 .25 .29
Innovative & Creative .32
Seeks Perfection .34 .23
Adaptable .19
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 75
Table 37. Study 6: Concurrent Validation Study for Paramedics N=85
Competency Reading Learning
Office Procedures
Assessing Following Procedures
Dealing with
Difficulty
Treating Others
with Respect
On time
Overtime Flexibility Sick
Needs Structure
-.21 -.25
Innovative & Creative
-.26 -.15
Enjoys Problem-Solving
-.27
.18 -.26 -.20
Competitive -.22 .20 .24
Seeks Perfection
-.21 .18
Develops Relationships
.19 -.21
Expressive & Outgoing
-.24 -.29
Corporate Citizenship
.24 .23
Exhibits a Positive Work Attitude
-.24 -.22
Adaptable .36 .26
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 76
Table 38. Summary of Relationships between Performance Measures and Original AIMS Scales
Performance Measure
Enjoys Proble
m Solving
Innovative and
Creative
Needs Structur
e
Adaptable
Develops Relationship
s
Expressive and
Outgoing
Competitive
Seeks Perfectio
n
Positive Work Attitud
e
Corporate Citizenshi
p
Total Score -.15 .20
Conversion Rate
-.15 -.16 .21 -.12
Calls Per Hour -.28 .22 -.17 -.17
Unavailability .18 .34 .14
Positive Attitude
Team Attitude -.14 .13 .14
Service Attitude
Caring Attitude .12
Ownership -.12
Performance -.13 .13
Range .13
Ranking -.12 .23 -.17
Sales -.13 -.15 -.10
Talk Time -.20
Account Weight -.14 -.18 -.14 -.14
Total Score -.49 .58
Conversion Rate
.47
Ranking (A) .20 .19
Ranking (B) .20 .28 .21 .21
Call Management
.26 .28 .27 .26 .19 .15 -.13
Average Hold Time
.15
Weighted Rating
.27
Supervisor Rating of Overall Performance
.23 .15
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 77
Performance Measure
Enjoys Proble
m Solving
Innovative and
Creative
Needs Structur
e
Adaptable
Develops Relationship
s
Expressive and
Outgoing
Competitive
Seeks Perfectio
n
Positive Work Attitud
e
Corporate Citizenshi
p
Supervisor Rating of Skill Acquisition
.34 .29 .25
Supervisor Rating of Summary Performance
-.41 .39
Supervisor Rating of Overall Performance
-.36 .28 .30
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 78
Appendix D: Scoring Rubric for Essays
Grammar Content Structure
Syntax, vocabulary, spelling, sentence structure, natural-sounding, command of English
Conciseness, appropriateness, addressed prompt
Logic, flow, organization, format
0 - 3 Latent with mechanical errors. May include errors in spelling, punctuation use, contractions, prepositions, missing words, verb conjugation and tense, etc. Contains a generally poor command of English. Errors are to the point that the writing is barely understandable, if at all.
Does not address more than half of requirements set up by prompt. Response may be incomplete or extremely off-topic. Response may have been written in a completely inappropriate tone for the intended audience. May be extremely unclear.
Ideas are so jumbled by illogical organization that they may be hard or impossible to follow. Format is not apparent or completely inappropriate for intended audience. There may be little to no transition between different ideas.
4 - 6 There are a few errors in grammar, but they do not severely hinder comprehension of the writing. English/wording may sound somewhat unnatural. Main idea of writing is still understood.
Addresses many or all requirements set up by prompt, but they may not have been thoroughly developed, may be missing information, or may be unclear. Response may be slightly off-topic.
Organization of ideas is slightly off-putting and confusing, but reader should be able to follow them and the main idea is still communicated. Ideas may lack transition when needed.
7 - 10 There may be one or two small errors, but they are very minor and do not affect comprehension of the writing. English sounds natural and flows well.
All requirements set up by the prompt are addressed clearly and well developed. There are no confusing spots. Response is not off-topic or missing information.
Organization of ideas is logical and easy to follow. Transitions are usually or always apparent where appropriate.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 79
Appendix E: Directions for Rating Essay Tests
Scoring with a Rubric:
1. Read the rubric.
2. Read the prompt
3. Outline all requirements listed in the prompt
4. Read the response
5. Choose an appropriate score for the writing sample in each of the areas listed on the
rubric. Be sure to rate each area separately, and not to allow a good/bad score in one area to
affect the way you score another area in the same writing sample; a writing sample with
many spelling errors may still reflect all the required content.
Example of a writing sample with a perfect score:
Prompt:
Pretend you are an administrative assistant. Your boss wants to have an offsite team meeting to set
goals for you and the rest of the team next year. Write an email asking team members to attend an
all-day meeting on the first Monday of next month. Tell team members to write down their goals for
the year and come prepared to present them to the group. Additionally, ask if anyone wants to
volunteer to plan a team activity.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 80
Response:
This person scored a 30/30 on their writing sample, on a basis of three factors: grammar, content,
and structure.
There are no errors in grammar or spelling. The English sounds natural and the writing flows well.
The writer scored a 10/10 for grammar.
The writer answered the question thoughtfully and thoroughly. Despite the fact that the author was
not presented with a date, time, or place in the prompt, he/she recognized that, in an actual work
setting, these figures would be required in a successful email, and included all necessary information.
The response addresses all requirements outlined in the prompt and leaves nothing unclear. 10/10
content.
The sample is structured logically so that it flows without abrupt jumps or changes in idea. It is easy
to read and follow. The writer also included a subject line and list of recipients, an appropriate
introduction (Team Members:) and conclusion (Thank you,). The sample received a 10/10 in
structure, earning it a 30/30 overall.
Example of a poor response to the same prompt:
Response:
We are going have a meeting on the first Monday next month. I want everybody to think of
some goals and write them to present to everyone. Does anyone want to plan a team activity?
Team Members:
Arthur is hosting a mandatory team meeting to set goals for next year. The meeting will take
place January 30, 2014 from 1:00 P.M. to 4:00 P.M. at the Ritz Carlton in Vienna, VA in
Conference Room B.
Arthur is expecting each of you to attend. Also, he is expecting you to prepare for the meeting
by documenting your goals and being prepared to present them to the team.
Arthur is looking for a volunteers to lead a team activity during the meeting. If you would like to
volunteer, contact Arthur with your idea by January 15th.
Thank you,
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 81
This person scored a 14/30 on their writing sample based on the same three factors: grammar,
content, and structure.
The grammar is understandable and there are no glaring errors, however, there is a missing word
(We are going have) and the wording is not always clear. The writer scored a 7 for grammar.
Only a few of the requirements set up by the prompt are addressed here, and when they are, the
ideas are hardly developed or elaborated on. Were this a real email, much would be left to confusion.
This person received a 3/10 for content.
There is little transition between ideas or logical flow in this response. Also, this response is not
structured in an email format. The sample received a 4/10 for structure, giving it a 14/30 overall.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 82
Appendix F: Validity Evidence for HR Avatar Tests
The HR Avatar Automated Essay Scoring System
January, 2016
Introduction
Written communication is a key skill in many positions. Communicating via email, writing
reports, and creating presentations all require the ability to communicate effectively.
Traditionally, essays written for assessment purposes are scored using human raters and a pre-
defined scoring rubric. However, the cost of human scorers is relatively high and humans can
become fatigued and erratic in high volume situations.
Luckily, machine learning has advanced to the point where computers can substitute for human
raters reliably. The HR Avatar Essay Test is an implementation of this technique, which results
in lower cost and faster scoring turn-around.
The HR Avatar Essay Test consists of several writing prompts. The writing prompts were
designed to be general enough to provide an opportunity for anyone to be able to write a short
essay. It is easy to add additional prompts for specific situations or for general use.
Applicants are asked to write a short essay with a minimum of 100 words and are given an
unlimited time to do so. The essays are scored using Discern, an open source, machine learning
program. Discern was designed by edX, a nonprofit organization founded by Harvard and the
Massachusetts Institute of Technology (MIT) (edX, 2015; Markoff, 2013). The system produces
a score that ranges from 0 to 100. A confidence estimate for the score is also computed, which
ranges from 0 to 1. Scores with confidence estimates less than .10 are not considered valid.
How it works
HR Avatar uses open source essay scoring software originally published by EDX Corporation, a
spin-off of The Massachusetts Institute of Technology (MIT).
Software addresses and performs regression to produce a score for each submitted essay along a
continuous scale. This is different from classification, in which the software would simply
attempt to categorize each essay into one or more 'groups' or to rank the essays relative to one
another.
Each essay is written according to a predetermined set of instructions typically referred to as the
"Prompt." A typical prompt might be: "In a short essay of 100-400 words, explain whether it's
better to be a planner or to be a dreamer."
All essays are scored by the machine learning algorithm based on a "Training Set" upon which a
regression model has been built. The algorithm essentially analyzes all of the training essays and
produces a best guess at how the human scorers who created the training set would have judged
the new essay.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 83
The application is written in Python and utilizes several open source machine-learning tools and
is centered around a machine-learning library called scikit-learn (http://scikit-learn.org), which in
turn uses a number of other open source mathematical and data manipulation packages
In order perform its task, the application converts each essay into a number of different
"features." Features are measurable aspects of the essay, such as spelling errors per character, or
grammar errors per character. In concept, the essay is reduced to an N-dimensional vector
containing all of the essay's feature scores. However, some features are complex vectors in and
of themselves.
Each feature is measured using specialized software for text analysis. For example, the grammar
errors are determined by looking for good and bad 'ngrams' which are essentially models of
either good or bad grammar.
Another important feature known as a "bag of words" is also generated. The bag-of-words model
is a simplifying representation used in natural language processing and information retrieval
(IR). In this model, a text (such as a sentence or a document) is represented as the bag (multiset)
of its words, disregarding grammar and even word order but keeping multiplicity. This results in
a vector with a length equal to the number of unique words in the largest essay evaluated.
It's helpful to understand the Bag of Words approach in terms of how it's used to filter out junk
email.
In Bayesian spam filtering, used by many spam filters, an e-mail message is modeled as an
unordered collection of words selected from one of two probability distributions: one
representing spam and one representing legitimate e-mail ("ham"). Imagine that there are two
literal bags full of words. One bag is filled with words found in spam messages, and the other
bag is filled with words found in legitimate e-mail. While any given word is likely to be found
somewhere in both bags, the "spam" bag will contain spam-related words such as "stock",
"Viagra", and "buy" much more frequently, while the "ham" bag will contain more words related
to the user's friends or workplace.
To classify an e-mail message, the Bayesian spam filter assumes that the message is a pile of
words that has been poured out randomly from one of the two bags, and uses Bayesian
probability to determine which bag it is more likely to be.
Along with the bag of words feature, another feature is generated that represents how 'topical' the
essay is, by using the bag of words vector that was generated.
Once the features are generated, the application formulates a model, using all training essays,
and their accompanying human-generated scores. The model is essentially a catalog of all feature
measurements for all of the training essays, along with their scores. Once created, this model can
then be used to determine where in the score space a new, unscored essay lies, based on its
feature measurements. In addition to score values, error values, which indicate how consistent
the training essay set was, can be calculated. This can provide a confidence value for the final
score.
The software uses a technique called Gradient Boosting Regression to pinpoint the score within
the model for a given essay by comparing the features for the new essay against the feature sets
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 84
of the pre-scored or 'training' essays. This is a well-established machine learning technique. Data
theory shows that this technique yields excellent results for regression problems like essay
scoring.
How does it perform?
The best indication of how well the machine learning algorithm works is to measure how well it
predicts the score a human rater would have come up with for any given essay. To do this we can
evaluate the machine-human rater reliability.
Reliability is a critical aspect of any assessment. It describes whether the score is consistent, and
puts an upper limit on the validity of the assessment. The data were analyzed to ascertain the
reliability of the machine scores of the essays to represent the scoring of human essay raters.
One thousand, two hundred and fourteen (N=1,214) essays were scored using both human
scoring and machine scoring. The correlation between the ratings was .73, representing an inter-
rater reliability of .73, which indicates that the machine scoring reliably rates the essays similarly
to human raters. In the world of testing, a reliability value of 0.73 is generally considered more
than acceptable.
Therefore, the machine scoring of the HR Avatar essay test was demonstrated to be a reliable
method for scoring essays that is similar to human ratings, but significantly more efficient,
requiring little or no human time or effort to arrive at an assessment of a large number of
applicants’ writing skills.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 85
Appendix G: Validity Evidence for HR Avatar Tests
Results for the HR Avatar High Potential Solution
February 23, 2016
Validation studies have been conducted to confirm that the HR Avatar High Potential Solution
predicts job performance. Two separate studies are reported below. The first compared performance
on the assessment to a dichotomous outcome measure. In the second study, scores on the
assessment were correlated with a 1-100 performance rating.
Study 1
Three companies were included in the first study of the ability of the HR Avatar assessment to
predict manager job performance. The three companies were Johnson & Johnson, Harte Hanks, and
Manulife. They were identified as leaders in the Philippines in terms of using best practices in
personnel management. Individually, their sample sizes were too small to conduct separate
validation studies, but together the number of participants was high enough to enable confidence in
the results.
Results are reported below. The managers received a job performance rating of either “high” or
“marginal.” The sample included 130 managers. It is likely that the performance measure attenuated
the correlation because it is only two levels, which limits the amount of variance, accuracy, and
consistency it can provide. However, it does provide a measure of performance. Based on the
literature review of the components in the HR Avatar Assessment, we expect validity to be very
strong, approximately .35 - .40 uncorrected, and in the .50 - .60 range when corrected for criterion
unreliability.
Table 1 presents the results of the correlation analysis. The overall score predicted job performance
significantly (r=.25; p<.01). Although we are pleased to see a statistically significant correlation that
approaches .30, we believe this is an underestimate of the actual validity, due to the two-level
performance measure. Among the subcomponents, Attention to Detail was particularly robust in
predicting performance (r=.22; p<.05), as was Adaptable (r=.21; p<.05).
The high/marginal performance rating probably has low reliability. Thus, if we use a low reliability
of .40 for the correction for criterion unreliability, the validity increases to .63. We also used a higher
estimate for criterion reliability of .60, which is typically used as an estimate of reliability for well-
developed multi-level performance ratings, as an estimate of the reliability of the high/marginal
performance rating. The result was a more conservative correction for attenuation that yielded a
corrected correlation is .42.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 86
Further support of the assessment’s prediction of job performance was provided by T-Test analyses,
which indicated that the high-performing group had a significantly higher average overall score
(X=63.16) on the assessment (p<.01) than those who were in the low-to-average job performance
group (X= 54.31).
In addition to predicting test performance, various scores on the assessment were related to
Leadership Engagement and Leadership Aspiration, suggesting that the underlying constructs are
related to important attitudes for leadership potential. For example, Needs Structure, Develops
Relationships, and Corporate Citizenship were related to Leadership Engagement. Further, several
test component scores were related to Leadership Aspiration, including Innovative and Creative,
Enjoys Problem Solving, Develops Relationships, and Adaptable.
Table 1: Correlations between Test Components and Job Performance and Job Attitude
Measures
Test Score Performance (High/
Marginal)
Leadership Engagement
Leadership Aspiration
Overall Score .25** .00 .16
Writing -.06 -.15 .09
Analytical Thinking .17 -.07 -.07
Attention to Detail .22* .01 .09
Adaptable .21* .10 .39**
Develops Relationships .02 .35** .50**
Enjoys Problem Solving .16 .26** .51**
Expressive and Outgoing -.01 -.13 .02
Innovative and Creative .18* .30** .67**
Needs Structure .02 .35** .36**
Seeks Perfection .13 .24** .36**
Frontline Management Fundamentals
.03 .03 .16
Corporate Citizenship .04 .28** .28**
Exhibits A Positive Work Attitude -.04 .11 .06
Competitive -.01 -.07 .00
*=p<.05; **=p<.01. N=130.
Study 2
The second study was conducted for an organization called UCPB LEAP. The sample was 64
managers who completed the HR Avatar assessment, and their scores were compared to their
previous year’s performance appraisal. The sample was small, which necessitated using a
nonparametric form of correlation called Spearman’s Rho, which indicates the extent to which the
rank order on the test was similar to the rank order on the performance measure. The correlation
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 87
was r=.25, which was significant (p<.05). This supports the assertion that managers who scored
higher on the assessment achieved higher performance ratings.
Conclusion
Based on the two studies presented above, we can say with confidence that managers who score
higher on the assessment perform better on the job. The uncorrected correlation of .25 between
overall score and the high/marginal performance measure is significant. When corrected for
criterion unreliability using a conservative approach, the correlation becomes .42. T-Tests also
support the same conclusion. More research is needed to build on the initial, yet promising results
described above. We expect that the results will demonstrate larger effect sizes and more robust
prediction of job performance when we are able to obtain measures of job performance that have
more variance and subjects are not simply placed into “high” and “marginal” categories. When we
have the time to adjust the scoring and weighting of the overall scores and have better criterion
measures, we believe it will be closer to .35 or more uncorrected, which would then yield a corrected
validity of approximately .60.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 88
Appendix H: Emotional Intelligence Test
HR Avatar Emotional Intelligence Test
August 31, 2016
Emotional intelligence (EI; Sometimes called EQ) is the capacity to be aware of and understand
one’s own emotions, as well as other people’s emotions, and to use that information to manage
one’s own reactions and respond effectively in social situations. The concept gained considerable
attention in the 1990s, with the publication of a book called Emotional Intelligence (Goleman, 1995).
Since then, it continues to be researched and used to help people interact with others more
effectively. Emotional Intelligence has been measured in different ways by different test developers.
Self awareness, empathy, and self control are at the core of EI.
Previous validity evidence
The ability of EI to predict performance has been established. A growing body of research
demonstrates that EI predicts job success, as well as other important outcomes on the job (Carmeli,
2003; Farh et al., 2012; O’Boyle at al., 2011; Semadar et al., 2006; Van Rooy & Viswesvaran, 2004;
Wong & Law, 2002). One study, published by Cote and Miners in 2006, showed that employees
with relatively low cognitive intelligence can achieve strong job performance by compensating for it
with high emotional intelligence under the right conditions.
Most jobs include some degree of frustration, and require employees to control their emotions.
Those with higher EI tend to have better social relations with people, including management,
coworkers, and customers. This leads to a wide range of positive outcomes. Having highly-tuned
skills in sensing how other people are feeling and being aware of how one’s own emotions are
impacting one’s thinking can be very valuable in managing conflict, dealing with complex social
situations, and solving problems in team settings. This can help avert escalations of conflict, and
enable the person to solve problems proactively.
For example, a sales associate may be interacting with a customer. The customer is not
communicating very clearly about what they want, but they are giving subtle cues about their
communication style and how they are feeling. Someone who has low emotional intelligence may
misread what the person wants and respond in a way that is frustrating to that customer.
Alternatively, someone high in emotional intelligence is more likely to accurately interpret the
customer’s needs and respond in a way that effectively influences the customer, resulting in stronger
sales performance (Huggins et al, 2016).
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 89
High Emotional Intelligence can also improve work success by helping to:
Use mental capacity on work tasks instead of getting carried away with emotional reactions
Diffuse situations where there is potential for conflict and other nonproductive behaviors
Anticipate others’ reactions and adjust approach to enable effective communications
Not offend others, not alienate customers or partners, not cause problems
Demonstrate better impulse control, avoiding distractions and staying focused
One recent meta-analysis found a correlation of p=.29 between EI and supervisor-rated job
performance (Joseph et al., 2015). The authors characterize EI as a measure that is a shorthand
version that contains components of multiple established measures. The table below presents the
relationships they found between some of those related measures and job performance. Cognitive
ability demonstrated the highest power of prediction of success on the job. EI provides a snapshot
view of a certain combination of skills that are important for success at work.
Example Correlations between Test Performance and Supervisor Ratings of Job
Performance*
Test Type Validity
Cognitive Ability .44
Emotional Intelligence .29
Conscientiousness .21
Emotional Stability .11
Extraversion .09
* Source is Joseph et al. (2015) meta-analysis
Emotional Intelligence can also impact other important outcomes, such as organizational citizenship.
Research has shown that employees who have higher EI tend to focus more on the welfare of their
organization, and put more effort into actions that help the organization to function effectively, even
though the activities are not directly required of them (Shrestha & Baniya, 2016). Similarly, higher EI
is related to adaptability for frontline service employees, which in turn is related to job outcomes,
such as job satisfaction and job performance (Sony & Nandakumar, 2016).
There is debate about Emotional Intelligence. Some have suggested that EI is not really a new
concept, and that some of the published research may overstate its prediction of job performance
(e.g., Joseph et al., 2015). Rather, it is simply a clever combination and repackaging of other things
we already knew predicted job performance, and still does not relate to job performance as well as
cognitive ability. These are fair criticisms. Nonetheless, emotional intelligence has become as
popular as it has because it makes sense, and, as noted above, a growing body of research does show
that it predicts success on the job, as well as other important outcomes.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 90
Measuring Emotional Intelligence
In roles where employees must interact with other people, for example, Customer Service, Sales, and
Management, the higher their capacity to be in touch with their own and others’ emotions,
understand those emotions, and behave in ways that are socially appropriate and demonstrate
impulse control, the better they will be received by other people. This pattern of skills and abilities
aligns with the observation that EI is related to Conscientiousness, Agreeableness, and Emotional
Stability, as well as cognitive ability. Emotional Intelligence can be used as one component of a good
selection process for hiring employees, as long as the potential overlap among different tools being
used is considered, and the combination of tools covers important skills for success on the job.
There are a number of available EI measures. Some focus on EI as an ability, and some are self-
report measures that represent EI in what is called a “mixed model,” which generally includes self
awareness, self regulation, social skill, empathy, and motivation. Three competencies in particular are
core to the concept of EI, because they involve the process of dealing with emotions directly. Being
aware of one’s own emotions and how one should behave in social situations, being aware of other
people’s emotions and caring about others, and being adept at impulse control and maintaining calm
can be thought of as “minimum requirements” for EI. Though its effectiveness probably is
overstated in certain publications, EI is a tool that can provide useful information about a candidate
in a relatively efficient way. A streamlined EI measure is described below.
Three Core Competencies of EI
EI Competency Definition Relation to Job Performance
Emotional Self Awareness
Monitors and understands how and why one reacts in particular ways to different situations, and knows how to conduct oneself appropriately and effectively in social situations
Interacts with customers and coworkers in an appropriate and measured way that reflects calm competence and inspires confidence
Empathy Senses and understands other people’s feelings, feels sympathy for other people, and sees things from other people’s point of view
Improves customer loyalty and wallet share through improved relationships, reducing levels of conflict in the workplace
Emotional Self Control Manages the desire to satisfy urges or impulses, shows restraint and manages behaviors to ensure appropriate and effective work habits and interactions with others, maintains composure in stressful situations
Prioritizes work tasks effectively, meeting long-term goals, and ability to form and leverage cooperative work relationships for better outcomes
The HR Avatar EI Test measures the three core components described above. They are three of the
most common aspects of the somewhat diverse set of competencies that make up EI. HR Avatar’s
EI Test is a streamlined mixed-model job-related questionnaire, and is offered as a new addition to
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 91
HR Avatar’s personality assessment to provide an alternate window into candidates’ interpersonal
skills, without having to use a separate test. It measures the core competencies of EI in a self-report
assessment.
An example EI item: I can usually tell how other people are feeling.
It’s important to note that while EI can help predict success on the job, it provides just one piece of
the puzzle for predicting success on the job. The best way to evaluate a candidate is to measure
multiple traits, including cognitive ability, past behavior and related job knowledge.
Reliability
The descriptive statistics and reliability estimates of the EI scales were estimated using data collected
via MTurk. As can be seen from the table below, the EI scales have acceptable levels of internal
consistency.
Descriptive Statistics and Reliability Evidence for the EI Scales
Competency N M SD Alpha
Emotional Self Awareness 320 25.84 4.25 .67
Empathy 320 25.38 4.34 .70
Emotional Self Control 320 25.18 5.49 .78
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 92
References
Carmeli, A. (2003). The relationship between emotional intelligence and work attitudes, behavior
and outcomes: An examination among senior managers. Journal of Managerial Psychology, 18(8),
788-813.
Cote, S., Miners, C.T.H. (2006). Emotional intelligence, cognitive intelligence, and job performance.
Administrative Science Quarterly, 51, 1-28.
Farh, C.I.C.C., Seo, M., Teslluk, P.E. (2012). Emotional intelligence, teamwork effectiveness, and
job performance: The moderating role of job context. Journal of Applied Psychology, 97(4), 890-900.
Goleman, D. (1995). Emotional Intelligence, New York, NY, England: Bantam Books, Inc.
Huggins, K.A., White, D.W., & Stahl, J. (2016). Antecedents to sales force job motivation and
performance: The critical role of emotional intelligence and affect-based trust in retailing managers.
International Journal of Sales, Retailing & Marketing, 5(1), 27-37.
Joseph, D.L., Jin, J., Newman, D.A., & O’Boyle, E.H. (2015). Why does self-reported emotional
intelligence predict job performance? A meta-analytic investigation of mixed EI. Journal of Applied
Psychology, 100(2), 298-342.
O’Boyle, E.H., Humphrey, R.H., Pollack, J.M., Hawver, T.H., & Story, P.A. (2011). The relation
between emotional intelligence and job performance: A meta-analysis. Journal of Organizational
Behavior, 32, 788-818.
Semadar, A., Robbins, G., & Ferris, G.R. (2006). Comparing the validity of multiple social
effectiveness constructs in the prediction of managerial job performance. Journal of Organizational
Behavior, 27(4), 443-461.
Shresna, A.K., Baniha, R. (2016). Emotional intelligence and employee outcomes: Moderating role
of organizational politics. Business Perspectives & Research, 4(1), 15-26.
Sony, M., & Nandakumar, M. (2016). The relationship between emotional intelligence, frontline
employee adaptability, job satisfaction and job performance. Journal of Retailing & Consumer
Services, 30, 20-32.
Van Rooy, D.L., Viswesvaran, C. (2004). Emotional intelligence: A meta-analytic investigation of
predictive validity and nomological net. Journal of Vocational Behavior, 65(1), 71-95.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 93
Wong, C.S., Law, K.S. (2002). The effects of leader and follower emotional intelligence on
performance and attitude: An exploratory study. The Leadership Quarterly, 13(3), 243-274.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 94
Appendix I: Work Competency Tests
HR Avatar Work Competency Tests
August 31, 2016
The HR Avatar Work Competency Tests were developed to address the need for assessments of
workplace competencies that focus on interpersonal skills. The Work Competency Tests use a
situational judgment test (SJT) format, in which candidates make decisions in situations that
employees typically encounter at work. The SJT format has proven particularly effective at
measuring critical workplace competencies, which can impact the overall performance and bottom
line of an organization.
SJT’s have been used for many years to predict job performance (Christian, Edwards & Bradley,
2010; McDaniel et al, 2007). Christian et al (2010) conducted a meta-analysis that considered the
ability of SJT’s to predict job performance in a number of domains. SJT’s that measure applied
social skills predicted job performance ranging from r=.25 to r=.38 (See below Table). Additionally,
Christian et al (2010) found that video-based SJT’s tend to predict performance better than paper-
and-pencil tests.
Results of SJTs that Measure Applied Social Skills in Predicting Job Performance*
Domain Number of studies Number of people Validity**
Interpersonal skills 17 8,625 .25
Teamwork skills 6 573 .38
Leadership 51 7,589 .28 *From Christian et al (2010) meta-analysis.
**Validity corrected for criterion unreliability.
Content Development
Four assessments were created, one for each of the following job families: 1) Team Member, 2)
Service, 3) Sales, and 4) Leader. Content was developed by conducting research on job descriptions
and typical activities conducted within each job family. Competency models were created for each
job family, and a panel of SMEs was engaged to develop scenarios for each test. Responses
represented different course of action that could be taken, given the scenario. Test-takers are asked
to rate the effectiveness of each. SMEs were further engaged to generate alternative actions for each
scenario, and to estimate the effectiveness of each response.
MTurk was used to gain additional SME evidence for building the assessment. SMEs from each job
family completed the assessments, and provided their ratings of the effectiveness of each alternative
(Team Member N=49, Service N=49, Sales N=52, Leader N=49). Their mean responses were used
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 95
as the true score for each of the assessments. The test-taker score is the difference between
responses and the true scores, combined for each competency. SMEs also rated the extent to which
the scenarios appeared to be similar to things that happen in real work situations relevant to the job
family, and whether each competency was important for performing the job scenario. Ratings for all
competencies for all four job families supported the assertion that the scenarios were similar to real-
life job situations and the competencies were important for performing the situation represented in
the scenario.
In the HR Avatar Work Competency Tests, the work competency scenarios are presented using HR
Avatar’s simulation technology. First, a general description of the context is presented. Then, the
test-taker views a scene in which characters are interacting, and a situation occurs. They then
respond to a list of 5-8 alternative responses, indicating how effective each response is on a scale of
1-5.
Sample Work Competency Test Item
Context: You work on a team that has an important deadline coming up.
Coworker: Female: You know how we are all supposed to get our work done by tomorrow close of business? Well I just overheard Jim telling Jessica that he is going to a concert tonight and will probably call in sick tomorrow. He also said that he’s not even close to getting his assignment finished, so it doesn’t matter anyway. What should I do?
Rate the effectiveness of the following responses:
1. Tell Jim you are going to tell the Team Leader what you heard if he calls in sick tomorrow. 2. Talk to the team leader now to share what you heard. 3. Just ignore it and let Jim fail on his own. 4. Put an anonymous note on the Team Leader’s desk explaining what you overheard.
Team Member Work Competency Test Description
The Team Member Work Competency Test was designed to assess critical competencies within jobs
where the employee must interact with others, both inside and outside an organization, in order to
be successful. Virtually all jobs require interaction, so this assessment is a good choice for just about
any job.
Team Member Work Competencies
Building Relationships with Customers and Coworkers: Effective work relationships
help produce better results, because most roles in organizations require interacting with
others and relying on them to accomplish objectives. This includes developing constructive
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 96
and cooperative working relationships with others, and maintaining them over time. It also
involves effectively managing 1:1 interactions with others, maintaining positive relationships
while accomplishing objectives, and building credibility through mutual trust and respect.
Demonstrating Customer Focus: Customers are the reason most organizations exist.
Keeping the customer in mind and providing a positive customer experience is critical to
success. Demonstrating Customer Focus includes working with customers to understand
needs and ensure that products and services meet their needs, doing what can be done to
resolve issues, getting the customer to someone who can help if needed, and following up to
ensure there is resolution. It also involves handling the situation in a conscientious and
customer-focused way.
Resolving Conflicts and Meeting Customer Needs: Being able to effectively resolve
conflicts and meet customer needs is a critical aspect of being an effective team member. It
involves creatively solving problems, maintaining a calm demeanor, and managing through
differences of opinion. It includes handling complaints, looking for ways to solve problems
collectively and agree on next steps, settling disputes and resolving grievances and conflicts,
or otherwise negotiating with others. Additionally, effective conflict management requires
the team member to work to understand the views of both sides of a disagreement, ensure
relevant information is shared and considered, and help the parties in a conflict to find
common objectives.
Working Well with Teams: When team members are cooperative, helpful, and respectful
of others, it results in a more positive and cooperative workplace. Demonstrating good
teamwork results in other people feeling good about working with you, and increases the
chances that they will be helpful in return. This involves working effectively with other
people and teams, supporting and showing respect for others, showing interest in other
people’s work, and saying positive things about the work and organization. It also involves
helping others get their work done, making sure the team’s work gets done, and being ready
to put team goals ahead of your own individual goals.
Maintaining Flexibility and Adaptability: Organizations face an increasing amount of
change, whether due to technology, mergers or acquisitions, or changing strategy to keep up
with the competition. Flexibility and adaptability can help you be effective in organizations
during times of change. This includes responding well to change, modifying your approach
in light of new demands, and adapting to, supporting, and accepting change. It also involves
demonstrating resilience, hardiness, and effective coping skills during difficult times.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 97
Customer Service Work Competency Test Description
The Customer Service Work Competency Test was designed to assess critical competencies within
jobs where the employee is required to provide customer service in order to be successful.
Customer Service Work Competencies
Building Relationships with Customers and Coworkers: Effective work relationships
help produce better results, because most roles in organizations require interacting with
others and relying on them to accomplish objectives. This includes developing constructive
and cooperative working relationships with others, and maintaining them over time. It also
involves effectively managing 1:1 interactions with others, maintaining positive relationships
while accomplishing objectives, and building credibility through mutual trust and respect.
Demonstrating Customer Focus: Customers are the reason most organizations exist.
Keeping the customer in mind and providing a positive customer experience is critical to
success. Demonstrating Customer Focus includes working with customers to understand
needs and ensure that products and services meet their needs, doing what can be done to
resolve issues, getting the customer to someone who can help if needed, and following up to
ensure there is resolution. It also involves handling the situation in a conscientious and
customer-focused way.
Resolving Conflicts and Meeting Customer Needs: Being able to effectively resolve
conflicts and meet customer needs is a critical aspect of being an effective team member. It
involves creatively solving problems, maintaining a calm demeanor, and managing through
differences of opinion. It includes handling complaints, looking for ways to solve problems
collectively and agree on next steps, settling disputes and resolving grievances and conflicts,
or otherwise negotiating with others. Additionally, effective conflict management requires
the team member to work to understand the views of both sides of a disagreement, ensure
relevant information is shared and considered, and help the parties in a conflict to find
common objectives.
Working Well with Teams: When team members are cooperative, helpful, and respectful
of others, it results in a more positive and cooperative workplace. Demonstrating good
teamwork results in other people feeling good about working with you, and increases the
chances that they will be helpful in return. This involves working effectively with other
people and teams, supporting and showing respect for others, showing interest in other
people’s work, and saying positive things about the work and organization. It also involves
helping others get their work done, making sure the team’s work gets done, and being ready
to put team goals ahead of your own individual goals.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 98
Helping Customers Understand Information: Communicating effectively with customers
is at the core of delivering customer service. It depends largely on how effectively you help
customers understand information. Customers tend to have a wide range of knowledge
levels, so it is important to meet them where they are and provide the appropriate level of
detail to ensure their understanding, in a way that makes them feel good about the
experience. This involves asking questions to understand their needs, issues, and current
level of understanding, translating or explaining what information means and how it can be
used, and providing a sufficient amount of detail. It also includes demonstrating empathy
with the other person, helping the customer to review information, and looking for details
that will help the customer understand and/or address the problem.
Sales Work Competency Test Description
The Sales Work Competency Test was designed to assess critical competencies within jobs where
the employee is required to make sales in order to be successful.
Sales Work Competencies
Building Relationships with Customers and Coworkers: Effective work relationships
help produce better results, because most roles in organizations require interacting with
others and relying on them to accomplish objectives. This includes developing constructive
and cooperative working relationships with others, and maintaining them over time. It also
involves effectively managing 1:1 interactions with others, maintaining positive relationships
while accomplishing objectives, and building credibility through mutual trust and respect.
Demonstrating Customer Focus: Customers are the reason most organizations exist.
Keeping the customer in mind and providing a positive customer experience is critical to
success. Demonstrating Customer Focus includes working with customers to understand
needs and ensure that products and services meet their needs, doing what can be done to
resolve issues, getting the customer to someone who can help if needed, and following up to
ensure there is resolution. It also involves handling the situation in a conscientious and
customer-focused way.
Resolving Conflicts and Meeting Customer Needs: Being able to effectively resolve
conflicts and meet customer needs is a critical aspect of being an effective team member. It
involves creatively solving problems, maintaining a calm demeanor, and managing through
differences of opinion. It includes handling complaints, looking for ways to solve problems
collectively and agree on next steps, settling disputes and resolving grievances and conflicts,
or otherwise negotiating with others. Additionally, effective conflict management requires
the team member to work to understand the views of both sides of a disagreement, ensure
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 99
relevant information is shared and considered, and help the parties in a conflict to find
common objectives.
Driving for Results: Being successful at sales generally requires a relentless persistence and
focus on goals. The constant pursuit of closing deals will translate into increased sales. This
requires you to push yourself to achieve objectives, stay focused on making sales,
aggressively close deals, and to work toward achieving both personal and team goals. It also
involves asking questions to determine what obstacles there are and to find the best way to
drive to making a sale, and influencing customers to buy.
Conveying Value: Conveying value to customers is critical to sales success. It requires
flexibility in messaging, thinking about what you have to offer, and understanding the
context of the customer and the sale. It requires some degree of creativity. Conveying value
involves recognizing and describing value to customers, determining customer needs and
connecting value to those needs, and influencing others. It also involves describing how
value is related to important customer outcomes (e.g., saving time, saving money, increasing
speed to market).
Leader Work Competency Test Description
The Leader Work Competency Test was designed to assess critical competencies within jobs where
the employee is required to manage a team in order to be successful.
Leader Work Competencies
Guiding, Directing, and Motivating Others: Guiding, Directing, and Motivating Others
is a core aspect of managing people. It is how a manager gets the team to act, and how a
manager ensures that action is effective. It involves providing direction and guidance to
subordinates, including setting performance standards and monitoring performance. It also
includes coordinating the work and activities of others, encouraging goal accomplishment,
making detailed plans that consider what is most important, and communicating priorities to
team members. To be effective at guiding others, a leader must effectively hold the team
accountable for their work, and provide advice that is reasonable and socially aware.
Coaching and Developing Others: Coaching and Developing Others has a significant
impact on a leader's long-term effectiveness. It involves understanding employees' needs and
helping them to grow and improve their job performance. It includes identifying the
development needs of others and coaching, mentoring, or otherwise helping others to
improve their knowledge or skills. Effective coaching and developing starts with building a
relationship of mutual trust, working together to decide what to accomplish, goal(s) to set,
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 100
and developing a roadmap for reaching the goal, and giving feedback along the way. The
leader should provide specific behavioral examples when giving feedback on performance
issues, clarify expectations, and get a commitment from the employee to act.
Resolving Conflicts and Meeting Customer Needs: Being able to effectively resolve
conflicts and meet customer needs is a critical aspect of effective leadership. It involves
creatively solving problems, maintaining a calm demeanor, and managing through
differences of opinion. It includes handling complaints, looking for ways to solve problems
collectively and agree on next steps, settling disputes and resolving grievances and conflicts,
or otherwise negotiating with others. Additionally, effective conflict management requires
the leader to work to understand the views of both sides of a disagreement, ensure relevant
information is shared and considered, and help the parties in a conflict to find common
objectives.
Exercising Political Savvy: Building relationships and managing impressions with
decision-makers and others who have influence in organizations can help you be effective
and successful. When they are deciding whom to include in strategic discussions and where
to put resources, people are more likely to think to include those they know and trust.
Political savvy can help you to be more likely to be included in decisions made by leaders in
your organization. Political savvy includes understanding how to position yourself and
communicate objectives in the context of organizational issues and other personnel to
maximize outcomes both for your group and the organization, getting people to cooperate
with you, socializing your ideas, and building bridges to meet others halfway.
Team Building: Teams that are characterized by trust, support, and positive interactions
tend to be more effective. Leaders can help facilitate positive interactions among their teams.
Team building involves engaging and participating in activities that support improved team
social relations, building mutual trust, respect, communication, understanding and
cooperation among team members. It also includes focusing on providing a team
environment that is conducive to collaboration, fostering innovation and creativity,
promoting increased comfort level and celebration among team members.
HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31, 2016 101
References
Christian, M.S., Edwards, B.D., & Bradley, J.C. (2010). Situational judgment tests: Constructs
assessed and a meta-analysis of their criterion-related validities. Personnel Psychology, 63, 83-117.
McDaniel, M.A., Hartman, N.S., Whetzel, D.L, & Grubb III, W.L. (2007). Situational judgment tests,
response instructions, and validity: A meta-analysis. Personnel Psychology, 60, 63-91.