Computers & Education 99 (2016) 1e13

Contents lists available at ScienceDirect

Computers & Education

journal homepage: www.elsevier .com/locate/compedu

Closing the gender gap in STEM with friendly maleinstructors? On the effects of rapport behavior and gender ofa virtual agent in an instructional interaction

Nicole C. Kr€amer a, *, Bilge Karacora a, Gale Lucas b, Morteza Dehghani c,Gina Rüther a, Jonathan Gratch b

a University of Duisburg-Essen, Forsthausweg 2, 47057, Duisburg, Germanyb USC Institute for Creative Technologies, 12015 Waterfront Drive, Playa Vista, CA 90094-2536, USAc University of Southern California, 3620 South McClintock Ave., Los Angeles, CA 90089-1061, USA

a r t i c l e i n f o

Article history:Received 5 January 2016Received in revised form 5 April 2016Accepted 6 April 2016Available online 7 April 2016

Keywords:STEMRapportVirtual agentsGenderPerformance in mathematicsAdult learningHuman-computer interfaceIntelligent tutoring systemsTeaching/learning strategies

* Corresponding author.E-mail addresses: nicole.kraemer@uni-due.de (N.

edu (M. Dehghani), Gina.ruether@stud.uni-due.de (

http://dx.doi.org/10.1016/j.compedu.2016.04.0020360-1315/© 2016 Published by Elsevier Ltd.

a b s t r a c t

While numerous research endeavors address the effects of pedagogical agents, the role ofthe agent's gender and its rapport behavior has been neglected. We hypothesize that aminimal amount of behavioral realism induced by display of rapport is necessary for anysocial effects to occur in human-computer interaction. Further, in line with results fromSTEM research on female role models, we assume that especially for female learners asame sex agent will be beneficial. In a 2 (student gender) � 2 (agent gender) � 2 (rapportbehavior yes/no) between subjects design, we investigate whether virtual agents can helpenhance participants' performance, effort and motivation in mathematics. Female andmale participants (N ¼ 128) interacted with a male or female virtual agent that eitherdisplayed rapport or no rapport. Our results confirm the expected main effect of rapport.However, against expectations, our results do not support the assumption that a same sexagent is beneficial for female learners. Participants’ performance and effort were signifi-cantly enhanced when interacting with an agent of opposite gender that displayed rapport.Our results have implications on designing agents for education and training purposes.

© 2016 Published by Elsevier Ltd.

1. Introduction

A common goal in educational endeavors is to increase interest and performance in science and math related subjects.Compared to the need for future experts in the realm, there is a lack of students in STEM fields (Science, Technology, Engi-neering and Mathematics) and especially women are underrepresented (Hill, Corbett, & Rose, 2010). It is therefore of highsocietal relevance to encourage women to enter STEM related fields. However, previous findings suggest that due to socio-cultural factors like implicit and explicit gender stereotypes, a gender difference exists in STEM fields especially with re-gard to motivation and willingness to enter the corresponding disciplines (Correll, 2004; Nosek, Banaji, & Greenwald, 2002a;Nosek, Banaji, & Greenwald, 2002b; Steele & Aronson, 1995). These stereotypes are mainly perceived and learned through

C. Kr€amer), Bilge.karacora@stud.uni-due.de (B. Karacora), lucas@ict.usc.edu (G. Lucas), mdehghan@usc.G. Rüther), gratch@ict.usc.edu (J. Gratch).

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e132

interaction with other social beings (Eccles, Wigfield, & Schiefele, 1998; Jacobs, 1991). Women seem to underestimate theircompetence inmathematics (and other STEM fields) because of the socially common belief that women have lower abilities inthose fields than men (Beyer & Bowden, 1997; Bong, 1999). Apparently, educating women about the inappropriateness ofthose gender stereotypes is not sufficient since those beliefs can continue to stay implicitly present in their minds (Noseket al., 2002a, 2002b). According to the expectation-value-model (Eccles et al., 1983), women underestimate their owncompetence and have lower expectation for success and therefore tend to have a lower preference for a career in STEM fields.

Recently, there has been an upsurge of interest in educational technology which exploits social and motivational factorsthat enhance math performance in general, and reduces gender inequality in particular (Baylor & Ryu, 2003; Kim, 2004).Specifically, pedagogical agents have been suggested as a means to individually motivate and instruct students. Theseautonomous agents have been widely expected to transfer the benefits of humanehuman tutoring and instructionalcommunication to the area of computer-supported learning. Future systems are anticipated to enhance e-learning programsfor individual learning with a human-like motivator or to support teachers in the classroom by attending to small groups ofstudents (Lester, Towns, Callaway, Voerman, & Fitzgerald, 2000; Moreno, 2004). Advocates of embodied automatic tutorsclaim that one major advantage of these pedagogical agents is that they can, or will be able in the future, communicate viaverbal and nonverbal means, thus facilitating and personalizing the interactionwith an e-learning program (e.g., Lester et al.,2000). Furthermore, increased motivation is expected: Baylor and Ryu (2003) suggest that the key advantage is that human-likeness creates more positive learning experiences and provides a strong motivating effect (Kr€amer & Bente, 2010). Thisassumption is also in line with the phenomenon that people often treat computers as social actors (Reeves & Nass, 1996).Here, it has been demonstrated that people automatically display social behavior towards computers and virtual agents evenif they are convinced that the artificial interlocutor does not warrant human-like treatment (Kr€amer, 2005; Nass & Moon,2000). Therefore, it can be assumed that psychological factors which improve people's performance in traditional face-to-face instructional settings (Andersen, 1979) can be successfully simulated by technologies in form of virtual learning com-panions or virtual instructors e at least in the sense that people will tend to react to them in a similar way as they wouldtowards human instructors.

One factor that has been shown to be beneficial in terms of facilitating interaction, relationship building and instructionalcommunication is rapport. In social psychology, rapport is described as the establishment of a positive relationship amonginteraction partners by rapidly detecting and responding to each other's nonverbal behavior (Gratch, Wang, Gerten, Fast, &Duffy, 2007a). This includes displaying behaviors that indicate positive emotions (such as head nods and smiles), showingmutual attentiveness (such as mutual gaze) and certain coordination behaviors (such as postural mimicry and synchronizedmovement, Tickle-Degnen & Rosenthal, 1990). Not only is rapport beneficial in human-human-interaction but also withinhuman-agent-interaction: Niewiadomski, Mancini, Hyniewska, and Pelachaud (2010) have shown that when an agent dis-plays appropriate and socially adapted emotional expressions it is perceived as more human-like than an agent that showshuman expressions which are inappropriate or not socially adapted. This indicates that rapport is an important feature inorder for the agent to be perceived as human-like and for any social effects to occur.

In this study, our goal is to focus on the agent's social role and social effects within instructional settings. Therefore, we donot employ a virtual agent as a tutor as in most other research on pedagogical agents (see Graesser, Wiemer-Hastings,Wiemer-Hastings, Kreuz, & Tutoring Research Group, 1999; Lester et al., 2000). Instead, the agent's role is to motivate byobserving the learners' success and e by asking about the experiences e giving the learner the opportunity to self-reflect onone's own performance. We therefore do not primarily focus on the role of the agent in terms of guiding and instructingcognitive processes but rather address the agent's social role and the corresponding processes (Kr€amer & Bente, 2010).Although traditionally, research on pedagogical agents has been dominated by a cognitive perspective, recent approachesexplicitly state the importance of complementing this by a social psychological framework. For instance, Kim and Baylor(2006) argue that learning environments should provide situated social interaction since it is well documented that thecognitive functioning of learners is framed by social contexts. Kim and Baylor (2006) sum up that teaching and learning arehighly social activities. They conclude that the reason for the weak impact of virtual agents (compared to human tutors)concerning learning outcomes might lie in the lack of empathetic social encouragement and caring aspects.

In this paper, we seek to address two goals. First, we aim to show that employing socio-emotional cues can enhance mathperformance in a human-computer setting. Specifically, we aim to show that the presence of an agent which provides rapportand fosters self-reflection can improve performance on standardized math tests. By this, we seek to provide further evidencethat people do treat computers as social actors and help elucidate the design principles that foster this effect. Second, westrive to contribute to the questionwhether the gender of the agent matters and influences the effects in interactionwith thestudent's gender. Here, we do not strictly follow up on the research on competent role models and their potential to motivateyoung women to engage in STEM fields but e in order to provide a new foundation to the discussion e rather focus on theeffects of the mere presence of a pedagogical agent which is either of same or different sex as the learner.

2. Theoretical background

The importance of motivating factors in instructional processes has long been demonstrated. Especially the expectation toexcel in terms of self-efficacy plays a crucial role: according to the expectancy-value-model of academic motivation, an in-dividual's engagement and performance with regard to an academic task is best predicted by the two factors expectation andvalue of success (Eccles, Wigfield, Harold, & Blumenfeld, 1998; Pintrich & Schrauben, 1992; Pintrich & Schunk, 1996). Marks

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e13 3

(2000) defines engagement as a psychological process which is mainly defined by the factors attention, interest, investmentand effort put to a certain task or subject. Engagement has been validated as a reliable predictor for academic performancewith high engagement being associated with good grades and test results (Arhar& Kromrey, 1993; Connell&Wellborn,1991;Mounts& Steinberg,1995; Voelkl, 1995). The expectation of success with regard to a certain task is strongly determined by anindividual's belief of having the required capabilities to conclude that task successfully. This expectation is a critical factorwhen it comes to an individual's motivation to engage and thrive to performwell in a task; the more an individual believes tohave the necessary capabilities to dowell, the more they are motivated to engage and actually be successfulewhich seems tobe true for STEM related subjects in particular (Greene, DeBacker, Ravindran, & Krows, 2002; Singh, Granville, & Dika, 2002).This means, that, according to the expectation-value-model, the subjective perception of personal task-specific competenceand self-efficacy strongly influence an individual's behavior with regard to a certain task (Eccles et al., 1998). Previous studiesshowed that children's self-evaluation of their own competence was a stronger predictor for their performance in differentsubjects than their actual previous performance (Eccles, 1994; Eccles et al., 1983; Parsons, Adler, & Kaczala, 1984; Wigfield &Eccles, 1992). These results have implications for the motivation and success in STEM related subjects since it has been shownthat girls and women tend to have a lower esteem of their own STEM related competences and a lower expectation of success(Meece, Wigfield, & Eccles, 1990). One of the most important goals for teaching in STEM related fields therefore is tostrengthen students' and especially female students' self-evaluation of their own competence. Two aspects which might beinfluential in this context are teacher gender and teacher (rapport) behavior which will be discussed in the following sections.

2.1. Influence of social aspects in instructional settings

With regard to the expectancy-value-model, Eccles et al. (1983) stress the impact of the social environment on an in-dividual's academic motivation. Previous research repeatedly showed that social actors play a significant role when it comesto self-evaluation of personal competence and academic value beliefs. Different authors (Eccles & Jacobs, 1986; Grolnick,Gurland, Jacob, & Decourcey, 2002; Grolnick & Ryan, 1989; Wigfield & Eccles, 1992) concluded in their research that in-dividual's academic motivation was strongly influenced by their perceived support and encouragement from parents andteachers. These perceptions seemed to have an even stronger impact on performance-related decisions and academicengagement than their actual performances. Additionally, other social processes play a role: a number of studies showed thatcompetent female role models in STEM, especially mathematics, contributed successfully to minimizing stereotype threats(Aronson, Fried, & Good, 2002; Marx & Roman, 2002; Mallam, 1993; McIntyre, Paulson, & Lord, 2003; McIntyre et al., 2005).Until now, there have been very few studies specifically focusing on the effect of role models. However, numerous studiesshow that social comparison with a competent person who is considered similar to oneself has a positive influence on one'sacademic performance by inspiring (Lockwood & Kunda, 1997; Tesser, 1986), improving self-evaluation, increasing motiva-tion (Blanton, Crocker, & Miller, 2000; Collins, 1996; Major, Sciacchitano, & Crooker, 1993; Taylor & Lobel, 1989) and evenimpacting student's academic goals (Hackett, 1985; Lockwood & Kunda, 1997). This is in line with previous classical socialpsychology research which shows that similarity of personal characteristics (like gender, ethnicity, age) is a key factor withregard to a role models' influence on motivation and perceived self-efficacy in a learning environment (Bandura, 1997). Marxand Roman (2002) tried to use those positive effects of role models to counteract the influence of stereotypical beliefs onfemale's academic performance in mathematics. Their research showed that girls' performance in a mathematical testshowed no significant difference compared to the boys' performancewhen theywere instructed by a competent female tutor.The authors argued that female role models can help overcome mathematically talented girls' (explicit and implicit) ste-reotypical beliefs of being less mathematically talented as boys. They further explained that the female role model isperceived as a kind of counter evidence that girls have lower competence and therefore a lower chance of success inmathematics than boys do. Also, girls' fear of confirming that stereotype in front of others is minimized. Therefore, it has beenconcluded that especially for girls, a same sex tutor whichmight serve as a role model is important and beneficial. There areealbeit a few e studies which directly test this assumption: a series of experiments revealed that exposure to female STEMexperts promoted positive implicit attitudes and stronger implicit identificationwith STEM, greater self-efficacy in STEM, andmore effort on STEM tests (Stout, Dasgupta, Hunsinger, & McManus, 2011). The results were driven by greater subjectiveidentification which predicted enhanced self-efficacy.

However, other studies show that academic motivation and engagement does not significantly vary as a function of theteacher's gender (Martin & Marsh 2005). Moreover, Betz and Sekaquaptewa (2012) demonstrated that the counter-stereotypic-yet-feminine success of female role models may actually be demotivating, particularly to young girls. Resultsshowed that feminine STEM role models reduced middle school girls’ current math interest, self-rated ability, and successexpectations relative to gender-neutral STEM role models.

The studies so far did not address the question of which mechanisms exactly contribute to the effects of role models: it isnot sufficiently clear whether the driving factor is primarily the role models' content-related competence, the fact that thereis less fear of confirming the stereotype in front of others or merely the presence of a same-sex person which could lead to amore comfortable situation. In order to take a first step towards disentangling these explanations, the present studywill focuson the mere presence of a same-sex person.

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e134

2.2. Influence of social signals within instructional communication

It has long been shown that the quality of a relationship between students and their teachers strongly influences student'smotivation to engage in those academic subjects and is very important to students' self-concept (Eccles, Wigfield, Harold, &Blumenfeld,1993; Learner& Kruger,1997).With regard tomathematics, Chouinard and Karsenti (2005) even showed that thefeeling of being supported by teachers is more influential for students than feeling supported by their own parents. In socialpsychology, rapport has been repeatedly shown as a correlate of a positive relationship between an individual and theirinstructor, which leads to an increase of motivation and engagement with regard to a certain task (Granitz, Koernig,& Harich,2009; Thomas, Ribich, & Freie, 1982). Rapport is said to have positive influences in “negotiations, management, psycho-therapy, teaching and caregiving” (Gratch, Wang, Gerten, Fast, & Duffy, 2007a, p.1). Rapport has been defined as verbal andnonverbal codependence between interaction partners (Stumm & Pritz, 2000). Tickle-Degnen and Rosenthal (1990)described rapport as consisting of three essential components: mutual attention, positivity and coordination (e.g., in termsof turn taking). Also, rapport occurs on three levels. On the behavioral level both interaction partners align their bodymovements (e.g. posture shifts, nods), on the emotional level both conversation partners feel comfortable and perceive theinteraction as rewarding. Finally, on the cognitive level there is a shared understanding (Gratch et al., 2007a; Tickle-Degnen&Rosenthal, 1990). Rapport is a sign of quality within an interaction and does not arise in every interpersonal conversation.When rapport occurs people, for instance, are more responsive to what the counterpart is saying, broaden their variety oftopics, keep more eye-contact, smile more frequently and increase proximity (Cassell, Gill, & Tepper, 2007; Grahe & Bernieri,1999). Rapport happens both on a verbal and on a nonverbal level. However, several researchers (Grahe & Bernieri, 1999;Tickle-Degnen & Rosenthal, 1990) argued that the nonverbal cues are more essential indicators (and drivers) for rapport.In line with these results, Andersen (1979) demonstrated that in instructional communication immediate social cues such assmiling and nodding displayed by the instructor had a positive effect on the participant's interest and motivation resulting ina better performance.

2.3. Virtual agents in instructional settings

Eccles and Midgley (1989) summarize that students are highly motivated when the learning environment is adapted totheir individual need, interests and capabilities. Using virtual figures as tutors or instructors can help to achieve this kind ofindividual learning environment. Their variable features can be adapted to the individual needs and goals of certain users anduser groups. In addition, virtual figures can be used independently from time and locationwithout any human oscillations liketiredness, time pressure, negative mood etc. This is one reasonwhy the development and evaluation of so called pedagogicalagents has been flourishing in the last 20 years. Johnson, Rickel, and Lester (2000) found evidence that the use of virtualagents influences students' performance in a positive way. A number of studies showed that the mere presence of anautonomous virtual figure can increase interest and attention with regard to a specific learning situation thereby enhancingthe learning performance (Baylor& Ryu, 2003; Lester et al., 2000). However, numerous studies demonstrate that it is not firstand foremost the agent's presence which is influential but that the agent's behavior is decisive. Garau, Slater, Pertaub, andRazzaque (2005) conducted a study showing that only participants who interacted with an agent with behavior that wasresponsive to their movements experienced a sense of personal contact with the agent which influenced them to behavemore socially considerate as opposed to interacting with a static or moving but unresponsive agent. A certain amount ofbehavioral realism is indeed necessary for a virtual agent to be perceived as human-like and therefore trigger social reactionsin their human interaction partner; this behavioral realism is created by a virtual agent's appropriate emotional andbehavioral reactions towards their human interaction partner (Blascovich, 2002; Von der Pütten, Kr€amer, Gratch, & Kang,2010). A virtual agent needs to be capable of showing appropriate emotional reactions towards a human interaction part-ner in order to cause a feeling of support and encouragement which can have a motivational effect (Elliott, Rickel, & Lester,1997; Kim, 2004). In line with these assumptions, Andersen's (1979) findings that an instructor's immediate social cues (e.g.smiling and nodding) increase motivation and learning, were replicated for human-computer-interaction (Kim, 2004). Shedemonstrated that positive emotional facial expressions displayed by virtual agents had a positive effect on participants'motivation and engagement to work on a given task. When it comes to human-computer-interaction the display of appro-priate emotional reactions and immediate social cues by a virtual agent creates a feeling of rapport in the user and contributesto a positive relationship (Gratch et al., 2006, 2007a; Kr€amer & Bente, 2010; Wang & Gratch, 2009).

2.4. The influence of virtual agents dependent on their gender

For the area of STEM, research has shown that virtual agents can enhance females' preference for STEM fields in general(Eisenhart, 2008; Turner& Lapan, 2005). Of interest is whether differential characteristics of female ormale virtual agents canbe expected to affect this preference. In line with the results from face-to-face instructional communication, several studiesdemonstrated that individuals were influencedmore strongly by a virtual agent with the same ethnicity (Baylor& Kim, 2004;Pratt, Hauser, Urgay, & Patterson, 2007) or the same gender (Baylor, 2005; Baylor & Kim, 2004; Guadagno, Blascovich,Bailenson, & McCall, 2007; Kim, Baylor, & Shen, 2007) compared to a virtual agent with different characteristics. Plant,Baylor, Doerr, and Rosenberg-Kima (2009) showed that (male and female) undergraduate students reported more positiveattitudes and higher interest in engineering after the interaction with a female virtual character (compared to prior to the

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e13 5

interaction, a control group and those interacting with a male agent). Further studies demonstrate that female agents have astronger influence on students' self-efficacy with regard to engineering and were more efficient in counteracting negativegender stereotypes (Baylor & Plant, 2005). Especially agent models that were similar to the young women (i.e., young, femaleand “cool”) tended to be the most effective for positively influencing the women's self-efficacy (Rosenberg-Kima, Baylor,Plant, & Doerr, 2008).

However, the findings are not consistent. In a study with female participants and female agents, Zanbaka, Ulinski,Goolkasian, and Hodges (2007) demonstrate that the presence of a virtual agent inhibits the performance of participantson a mathematical task. Hayes, Ulinski, and Hodges (2010) found a similar decrease in performance with regard to maleparticipants interacting with an agent of the same gender. However, the study demonstrated that when male participantsinteracted with an agent of the opposite gender their performance improved. The authors argued that participants experi-enced a stronger feeling of “being in the room with the agent” when interacting with an agent of opposite gender.

Although in sum there is evidence that female virtual agents have advantages especially for female students, moreresearch is needed to fully understand themechanismse especially with regard to rapport and immediacy behaviorse and toclarify inconsistent findings.

2.5. The current study

In our study, we examine whether the presence of a virtual agent can enhance motivation and performance in STEMsubjects and whether this influence depends on the virtual agent's gender and its rapport behavior. As mentioned above, thevirtual agent is not a tutor in the sense that s/he instructs the student how to learn but is supposed to motivate students bygiving the chance to reflect on one's own performance and by giving feedback. Based on the literature summarized above, wehypothesize that women will especially benefit from a female agent and that rapport will lead to positive effects:

H1. When the agent displays rapport behavior, participants perceive a stronger feeling of rapport compared to the agentdisplaying no rapport behavior.

H2. When the agent displays rapport behavior, motivation, effort and performance is higher than when the agent displaysno rapport behavior.

H3. For women, the interaction with a female virtual agent causes higher increase in motivation, engagement and per-formance than when interacting with a male virtual agent.

3. Method

The goal of the study was to investigate whether a virtual agent can motivate participants and help to improve theirperformance in a mathematical task. For this purpose, we examined whether beneficial effects occur when participantsinteract with virtual agents and how gender of the agent as well as of the learner and the rapport displayed by the agentinfluence these effects. In order to test the assumptions that a female agent will be especially beneficial for women and thatrapport will yield increased motivation and performance, we designed an experiment in which we manipulated virtualagents' gender and rapport behavior. We recruited men and women and had them perform two mathematical tasks, onebefore interacting with an agent and one during the interaction. Each participant was either pairedwith an agent of matchinggender or of opposing gender.We also investigated participants’motivation and increase of effort to solve themath problems.

3.1. Participants

We recruited 128 participants (53.1% females) from the greater Los Angeles area. Their age ranged from 18 to 34 years withan average age of 23.85 (SD¼ 3.07).16.0% of participants had a high school education, 5.3% completed some college, and 78.7%had a college education. Data was not collected on SAT completion or score, and college education included either two or fouryear degrees. Participants were recruited by responding to recruitment posters posted on craigslist.com and were paid $30.Data were collected between November 2011 and June 2012.

3.2. Design

We used a 2 � 2 � 2 full factorial between subjects design, with the first variable being the gender (male/female) of theagent and the second variable being whether or not the agent displayed rapport (rapport/no rapport). Participants wererandomly assigned to one of the four conditions (it was, however, considered that participant gender was distributed equallyacross conditions). In all four conditions the agent showed a minimum of human-like nonverbal behavior by blinking andperforming slight occasional posture shifts. In the conditions with no rapport, there was no additional nonverbal behavioradded. In the rapport-condition, the agent additionally smiled and nodded corresponding to the participants' behavior(verbal and nonverbal input). Additionally, gender of the participants was considered as a variable (see Table 1).

Table 1Study design and number of participants per condition.

Gender of the agent Rapport

Rapport (blinking, posture shift, backchanneling nod, comprehension nod, smiling) No Rapport (blinking, posture shift)

Female agent 19 women and 14 men 20 women and 14 menMale agent 16 women and 16 men 13 women and 16 men

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e136

3.3. Procedure

After an explanation of the study and obtaining consent, participants were led to a private roomwhere they completed theexperiment individually. Participants were seated at a desk with two monitors that were positioned next to each other. Theywere then instructed to work on a mathematical task for 10 min, which was presented as a computer-based survey on one ofthe monitors. The secondmonitor was turned off at this point. To minimize self-presentation concerns, the anonymity of taskperformance results and the non-competitiveness of the task were emphasized. The experimenter left the room for theduration of the working period. Participants' answers were not reviewed by the experimenter and they were not given anyperformance feedback. Next, the experimenter provided detailed (verbal and written) instructions on the following inter-action with the virtual agent. It was emphasized that the agent is a computer program and that participants will be aloneduring their interaction period. The agent was then launched and displayed on the second monitor. The experimenter wouldleave the room before the interaction started and thenwould control the virtual agent's speech over a separate computer in adifferent location, without the participants' knowledge. The agent's nonverbal behaviors were automated by the systemaccording to the condition (rapport/no rapport) as described above. First, the agent asked how the participants estimatedtheir previous performance on a 5-point Likert-scale (very poor to very good). Next, theywere asked to rate how difficult theythought the previous math problems were on another 5-point Likert-scale (easy to hard). This was followed by the secondtask period that was part of the interaction. The agent would instruct the participants to load the task on the other monitorand work on it for a time period of 10 min. During that time the agent reminded the participants twice of the time remaining(5 min and 1 min left) and also let them know when time was up. Afterwards, the agent asked the participants to estimatetheir performance with regard to the second task on the same scale. During the last part of the interaction, the agentinterviewed the participants concerning their experiences while working on the tasks. Then, the agent would announce theend of the interaction and would disappear from the monitor. Finally, situational motivation and demographic variables weremeasured in a post survey without the virtual agent visible or the experimenter present. Subsequently, participants weredebriefed. During debriefing it was made sure that participants had not been aware during the experiment that the exper-imenter or any other human was involved and/or had any part in the interaction. The experiment lasted about 60 min.

3.4. Mathematical tasks

Two mathematical tasks were presented to the participants. Each task consisted of a set of 24 math problems comparableto original GRE and SAT math items (examples: The child care center charges $11 an hour plus a daily $3 drop-off fee. Howmany hours of childcare did Robert pay for if he dropped his son off 3 days last week and paid $130 at the end of the week? B)What is the value of the expression 3 þ 4 � 6e2(5 � 8)?). The math problems were selected out of a larger set of questionspretested with regard to their difficulty. By pretesting the problems we made sure that both tasks have approximately thesame levels of difficulty and require the same sets of skills. Also, tasks had enough number of questions to prevent ceilingeffects due to participants finishing all themath problems in less than 10min. To avoid an improvement in performance in thesecond task due to simple learning/practice, the math problems were modified with regard to their wording and surfacefeatures. This way the first and the second task appeared distinct from each other while they still each required the same setof skills.

3.5. Rapport agent

Participants interacted with a female or male virtual agent with a human-like appearance. Four different characters wereused: 2male and 2 female (see Fig.1) to control for possible effects of particular agents.We used the Rapport Agent developedby Gratch et al. (2006). To create rapport with the participant, the agent displayed positive listening behaviors (such asnodding and smiling) that correspond to the verbal and nonverbal behavior of a human speaker. Previous studies of therapport agent have shown that it is highly capable of creating the experience of rapport comparable with a face-to-facecondition (Gratch et al., 2006, 2007a, 2007b). To produce listening behaviors, the Rapport Agent first collects and analyzesaudiovisual features from the speaker's voice (silence, speech) and upper-bodymovements (head nod, smile, eye gaze) in realtime. This happens via amicrophone and a Videre Design Small Vision System stereo camera, which was placed in front of theparticipants to capture their movements.Watson, an image-based tracking library, uses images captured by the stereo camerato track the participants' head position and orientation. Acoustic features are derived from properties of the pitch and in-tensity of the speech signal using a signal processing package, LAUN (Gratch et al., 2006). The Rapport Agent displays

Fig. 1. Female and male agents used in the study.

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e13 7

behaviors that show that the animated character is “alive” (eye blinking, breathing), and listening behaviors such as postureshifts and head nods automatically triggered by the system corresponding to participants' verbal and nonverbal behavior. Thisallows the agent to provide contingent feedback while the speaker is speaking by following a response model (Huang,Morency, & Gratch, 2011) to decide which behavioral response would be most appropriate (such as head nod or smile).The different animations are converted into Behavior Markup Language (BML) (Kopp et al., 2006), sent to an action scheduler(to determine the duration of each animation) and passed on to Smartbody, an animation system that blends the differentanimations naturally into each other (Thiebaux & Marsella, 2007). The commercial game engine Gamebryo then renders theanimations and displays them to the user.

3.6. Measures

Improvement in performance, increase in effort, motivation and rapport weremeasured as dependent variables by using acomputer-based survey and audio recordings of participants' verbal behavior during the interaction.

3.6.1. Improvement of performanceWe calculated participants’ performance improvement by the difference in their performance before and during the

interactionwith the agent. For this purpose, we subtracted the number of math problems they solved correctly in the first taskfrom the number they solved correctly in the second task. We used this relative performance improvement, assuming thatparticipants have indifferent pre-knowledge, education level, practice and talent affecting their absolute performance but nottheir relative score during the manipulation.

3.6.2. Increase in effortBy subtracting the number of solved math problems in the first task from the number solved in the second task, we

calculated the increase in effort. Here, we did not take into account whether the solutions were correct or not. This variablewas interpreted as an additional indicator for their situational motivation.

3.6.3. MotivationTo measure participants' motivational state with regard to the mathematical tasks we used the Situational Motivation

Scale (SIMS) by Guay, Vallerand, and Blanchard (2000). The scale includes 16 items asking for the motivation to work on thetasks and comprises four subscales of four items each: Intrinsic Motivation (“Because I think the activity is interesting“),Identified Regulation (“Because I am doing it for my own good”), External Regulation (“Because I am supposed to do it”) undAmotivation (“There may be good reasons to do this activity, but personally I don't see any”). All items are rated on a seven-point Likert-scale (1 ¼ corresponds not at all to 7 ¼ corresponds exactly) and were summed for further analysis (M ¼ 60.20,SD ¼ 11.94).

3.6.4. RapportThe Rapport Scale (Kang & Gratch, 2012) with 39 items was used in order to assess how participants perceived the

interaction and the Rapport Agent per se (e.g., “I felt I had a connection with the listener”; “The interaction was frustrating”;“The listener's body language encouraged me to continue talking”). People were asked to evaluate the interaction on an 8-point Likert scale (strongly disagree e strongly agree). In order to conduct further analyses the 39 items were all com-bined into one variable (M ¼ 3.72, SD ¼ 0.66).

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e138

4. Results

We conducted all analyses using Statistical Package for the Social Sciences (SPSS, Version 18). Before starting the analysesof hypotheses, we verified that agent appearance did not affect the results. As anticipated, there were no significant differ-ences between agents with the same gender but different appearances. Therefore, the data was collapsed for further analysis.Also, variances were not significantly different for any test (ps > 0.51), therefore the equal variances assumption holds for alloutcome variables. Visual inspection of residuals through a kernel density plot demonstrated normality of residuals. As re-siduals are normally distributed, and thus the data do not violate this assumption, it is appropriate to use ANOVA.

First, we analyzed the effect of rapport on improvement in performance using a between-subjects t-test. Participants whointeracted with an agent that displayed rapport behaviors improved their performancemore (M¼ 1.65, SD¼ 1.88) than thosewho interactedwith an agent that did not engage in rapport behaviors (M¼ 0.57, SD¼ 2.17, t(126)¼ 3.00, p¼ 0.003, d¼ 0.59).Participants who interacted with the rapport-agent also increased their effort (in the sense that they attempted to solve moreproblems in the second working period compared to the first; M ¼ 1.75, SD ¼ 2.42) more than those who interacted with anagent that did not engage in rapport behaviors (M ¼ 0.79, SD ¼ 2.54, t(126) ¼ 2.19, p ¼ 0.03, d ¼ 0.42). Although resultsconfirmed this expected effect of rapport on improvement in performance and effort, there were no effects of rapport on self-reported rapport (t(125) ¼ �0.40, p ¼ 0.69, d ¼ 0.07) or motivation (t(126) ¼ 0.06, p ¼ 0.95, d ¼ 0.01). Therefore, there is nosupport for H1 suggesting an improvement on self-reported rapport while H2 is supported for performance and effort (butnot for motivation).

To test H3 stating that the effect of female role models among women in the STEM domain could be moderated bypresence of a same-sex agent who displayed rapport behaviors, we conducted a 2 (participant gender: male vs. female) � 2(agent gender: male vs. female) � 2 (rapport: rapport vs no rapport) ANOVA on performance improvement. This analysisyielded a significant main effect of rapport (F(1,120) ¼ 10.23, p ¼ 0.002, d ¼ 0.59), as described above. The three-wayinteraction was also significant on the 10% level (F(1,120) ¼ 3.35, p ¼ 0.07, d ¼ 0.33). As depicted in Fig. 2a, among men,there is only an effect of rapport among those exposed to the female agent, however, no effect of rapport among thoseexposed to the male agent. Likewise, for women, rapport only has an effect among those exposed to the male agent (Fig. 2b).

Although unexpected, the three-way interaction suggests an effect of rapport for participants exposed to agents of theopposite gender. To test this, we ran an exploratory 2 (rapport: rapport vs no rapport)� 2 (gender matched agent: opposite vssame gender) ANOVA on improvement in performance. This analysis revealed an interaction on the 10% level of significance(F(1,124) ¼ 3.64, p ¼ 0.059, d ¼ 0.35), reflecting a significant effect of rapport on performance improvement among

-1.000

-.500

.000

.500

1.000

1.500

2.000

2.500

3.000

Male agent Female agent

Rapport

No rapport

-1.000

-.500

.000

.500

1.000

1.500

2.000

2.500

3.000

Male agent Female agent

Rapport

No rapport

Fig. 2. a. Effects of rapport and agent gender on performance for men. b. Effects of rapport and agent gender on performance for women.

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e13 9

participants who interacted with an opposite gender agent (M ¼ 1.97, SD ¼ 1.75 vs. 0.12, SD ¼ 2.34; t(124) ¼ 3.43, p ¼ 0.001,d ¼ 0.60), but no significant effect of rapport among those who interacted with an agent of the same gender (M ¼ 1.37,SD ¼ 1.97 vs. 0.89, SD ¼ 2.01; t(124) ¼ 1.10, p ¼ 0.32, d ¼ 0.18; see Fig. 3). The confidence intervals around a) the differencebetween rapport and no rapport of male agent for men (Fig. 2a), b) the difference between rapport and no rapport of femaleagent for women (Fig. 2b), and c) the difference between rapport and no rapport of same gender agent (Fig. 3) all contain zero(�0.91 to 1.91, �0.76 to 1.74, and �0.44 to 1.40, respectively), confirming that differences are not significant.

We confirm that this rapport X gender-matching interaction remains for both men and women with a 2 (participantgender: male vs. female) � 2 (gender matched agent: opposite vs same gender) � 2 (rapport: rapport vs no rapport) ANOVA.Indeed, this effect holds for both males and females as no significant three-way interaction emerges from this analysis(F(1,120) ¼ 0.40, p ¼ 0.53, d ¼ 0.11), while the two-way rapport X gender-matching interaction remains (F(1,120) ¼ 3.66,p ¼ 0.058, d ¼ 0.35), as does the main effect of rapport (F(1,120) ¼ 10.46, p ¼ 0.002, d ¼ 0.59).

We conducted 2 (participant gender: male vs. female) � 2 (agent gender: male vs. female) � 2 (rapport: rapport vs norapport) ANOVAs on effort improvement, as well as on self-reported rapport andmotivation. While the main effect of rapporton effort improvement (F(1,120) ¼ 5.23, p ¼ 0.02, d ¼ 0.42) was again significant, there were no other significant effects foreffort improvement (Fs < 1.56, p > 0.21, d < . 23), and no significant effects at all for self-reported rapport (Fs < 1.84, p > 0.17,d < 0.25) or motivation (Fs < 2.80, p > 0.10, d < 0.31).

Therefore, there is no support for H3, but interestingly, performance yielded significant results, while there were nosignificant effects of agent gender for self-reported rapport, motivation and effort improvement. The significant results,however, are opposite of what was assumed in the corresponding hypothesis.

5. Discussion

Our results yield several interesting insights. The aim of the study was to demonstrate the effectiveness of rapport duringinstructional communication in fostering performance and effort in STEM learning as well as to find further evidence for theadvantage of same sex instructors for women. While the former was clearly supported, the latter was not only unsupportedbut the opposite of our assumptions was substantiated. In the following, we will discuss these results in greater depth.

As hypothesized, rapport was able to foster performance and effort. Those participants who interacted with an agent whodisplayed rapport behavior showed a better performance and a larger increase in effort compared to those participants whointeractedwith an agent who did not showany rapport. This yields further support to the notion that by providing reciprocityand positivity (Gratch et al., 2007a; Tickle-Degnen & Rosenthal, 1990), rapport is able to motivate students in a similar way aspositive nonverbal behavior in teaching contexts does (Andersen, 1979). Likewise, research in social psychology has shownthat establishing rapport between people and their instructors in face-to face interactions increases desirable outcomes suchas motivation and improvement in task success (Granitz et al., 2009; Thomas et al., 1982). Our results show that rapport has asimilar positive effect on performance in human-computer-interaction.

However, although the availability of rapport behavior influenced the participants' behavior, there was no influence onsubjective experience, neither on self-reported motivation nor on perceived rapport. Here, it has to be considered that otherstudies also tended to show more pronounced effects on participants' behavior instead of participants' perception andsubjective experiences (Gratch et al., 2006; Huang et al., 2011; Von der Pütten et al., 2010). Therefore, rapport seems to havethe potential to be influential even without people consciously noticing it. The same might be true here: while rapportbehavior of the agent impacts participants' behavior, they do not notice rapport to be present. It is possible that more pro-nounced differences in agent rapport behavior between conditions would allow the rapport to be perceived, but this remainsto be tested in future research.

Additionally, it is important to consider the effect sizes observed. We find a medium (d ¼ 0.59) effect of rapport behavioron participants' performance and a small (d ¼ 0.42) effect of rapport behavior on participants' effort. These effect sizes are

-1.000

-.500

.000

.500

1.000

1.500

2.000

2.500

3.000

Opposite gender agent Same gender agent

Rapport

No rapport

Fig. 3. Interaction of rapport and gender match on performance.

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e1310

consistent with the range from medium to small effects found in other work on computer human-interaction (e.g., Gratchet al., 2007a; Lucas, Gratch, King, & Morency, 2014). While the effect on performance is larger than the effect on effort,given that effect sizes are merely estimates of effect size based on samples, it is possible that -in the population- the effectsizes are even more similar than they appear to be here. So, readers should hesitate to draw the conclusion that the effect onperformance is larger than the effect on effort and further research should be considered first to replicate this difference.While these are not large effects by standard conventions (Cohen, 1988), the manipulation used is also fairly subtle. Partic-ipants in the rapport and non-rapport conditions experienced exactly the same experimental setup; the only difference wasthe non-verbal and verbal behavior of the rapport agent included rapport-building. The small to medium effects are moreimpressive in light of the subtlety of the manipulation.

While the main effect of rapport was in line with the hypotheses, the results with regard to the gender of the agent wasopposite to our hypothesis. However, our hypothesis regarding the potential influence of the virtual instructor's gender wasbased on inconsistent prior research. Although several studies demonstrate the positive effect of female role models in STEMfields (Cheryan, Siy, Vichayapai, Drury, & Kim, 2011; Stout et al., 2011), others indicate that academic motivation andengagement does not significantly vary as a function of the teacher's gender or even show that female teachers may hindergirls' STEM success (Betz & Sekaquaptewa, 2012; Martin &Marsh 2005). The latter is now supported by our data in the sensethat the dyads which are NOT gender-matched lead to most performance and effort. It has to be acknowledged, though, thatthe role of the virtual agent here, was not the one of a competent role model as mentioned in the literature but the role of amotivating interviewer. Therefore, we can at least conclude that the mere presence of a woman in a learning situation is notsufficient to lead to increased motivation and learning. This does not preclude, however, that results might be different whenpresenting a competent role model.

More specifically, the results indicate that when rapport was displayed, participants' improvement was higher when theyinteracted with an agent of opposite gender than with an agent of matching gender. This indicates that a facilitation ofperformance only occurs under a certain gender condition, i.e. opposing gender, and when rapport is displayed by the agent.Participants' performance improvedmost when they interactedwith a virtual agent of opposite gender that displayed rapportand it improved least with an agent of the opposite gender who did not display rapport. This indicates that rapport has aneffect on participants' improvement in performance only when the agents’ gender does not match their own. Rapport istherefore especially beneficial when it is shown by an agent of opposite sex (e.g. a male agent talking to a woman), while anagent of opposite sexwho does not display rapport, is detrimental. While this seems to be plausible for women (whomight bepositively surprised and especially encouraged by a rapport-displaying male agent asking her about her experiences andsuccess, while they are appalled by a male character who does not show rapport), the pattern is more difficult to explain formen. Here, men might not have been pleasantly surprised because reciprocity and positivity is attributed to women quastereotype. Instead, it is more likely that they would simply react especially positive towards a rapport displaying womancompared to a rapport displaying man e and additionally do not want to disgrace themselves in the presence of a responsivewoman.

However, the specific mechanisms have to be targeted in greater detail in future studies. As yet, it is not sufficiently clearwhether rapport fosters self-reflection and by this self-efficacy, or whether the rapport directly influences motivation whichincreased effort to perform well by attempting to solve a higher number of math problems.

5.1. Limitations

The study certainly has some limitations: the most important limitation probably is the laboratory situation whichquestions the generalizability to every day instructional communication. Especially as solving the math tasks did not haveactual consequences for the participants' future, their behavior and experiences might have been very different from real lifesituations.

Also, our results are confined to the specific learning situation that was tested: the virtual agent merely acted as anobserver who triggered reflections on self-efficacy. The results therefore cannot be transferred to a classic tutor system. Withregard to clarifying themechanisms of female competent rolemodels it can, however, be concluded that themere presence ofa woman is not sufficient for women to feel and learn better.

Additionally, the sample was mixed and heterogenous in age and education. Although this usually is a strength, it couldhave been detrimental here because some participants might have been too different from the group of people this research ismeant to be most relevant for, which is high school students or first-year college students.

With regard to sample size, power analyses reveal power of 85% for performance, but only 59% for effort. Therefore,although we had a sufficient sample size to detect a medium effect, we were underpowered (by conventional standards of80% power) to detect a small effect such as the one we found for effort. Thus, if future research efforts were undertaken toreplicate the current results (especially for effort), we suggest an increase in sample size. Specifically, traditional estimates ofpower would lead one to recommend a sample size of 212 (106 per condition); however, other perspectives on power suggestthat an even greater number of participants may be needed for replication (e.g., Maxwell, Lau, & Howard, 2015).

Also, we have to mention and consider some violations to normal distribution in the data: Shapiro-Wilk tests reveal that,while rapport and motivation are normally distributed (Shapiro-Wilks > 0.98, ps > 0.19), improvement in performance andeffort were not (Shapiro-Wilk ¼ 0.96, p ¼ 0.001, and Shapiro-Wilk ¼ 0.97, p ¼ 0.02). Despite this violation of normalityassumption for ANOVA for performance and effort, wewere able to find significant effects for these variables. On the variables

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e13 11

where we found no significant effects (rapport and motivation), normality held; thus, finding non-significance with rapportand motivation cannot be attributed to violating this assumption.

With regard to the appearance of the agent, several aspects have to be reflected critically. First, although we tried toalleviate the effects of the specific appearance by using two different male and two different female characters, the charactersstill represent a specific type of virtual agent that may not resemble all participants. This may be important since it has beendemonstrated that virtual characters that resemble the user are assessed more positive (Iacobelli & Cassell, 2007) andtherefore are probably more influential. Specifically an “in-group” member agent is perceived as more intelligent andcompetente evenwhen it has the same narratives as a virtual character that does not resemble the user. In our study, severalparticipants, for example with different ethnicity from the agents, might not have felt similar to the agent.

Finally, our results are derived from an interactionwhich lasted less than an hour. Especially in the learning and instructionrealm, usually instructor-learner-relation exist for longer periods of time. Therefore, long-term studies are needed.

6. Conclusion

In summary, contributions of this work are three-fold. First, the study adds to literature on human-computer-interactionby showing that virtual agent rapport is an important factor for achieving desirable outcomes such as effort and performancewith regard to mathematical tasks. Virtual Agents with rapport can contribute to improve people's performance, specificallywith standardized math tests. This observation may support the development of useful and effective applications in math-ematical education and training, such as virtual instructors, tutors or learning companions. Second, the study shows that anagent of opposing gender, not a same sex agent, is most successful in boosting participants' performance. In this context, theresults of this study may motivate future research on gender differences with regard to effects of specific gender conditions.Finally, this work makes a methodological contribution to the fields of experimental psychology and human-computer-interaction.

Acknowledgements

This research was supported by the National Scientific Foundation under grant # IIS-0916858. Two of the authors receivedsupport by the PROMOS Program of the German Academic Exchange Service (DAAD).

References

Andersen, J. F. (1979). Teacher immediacy as a predictor of teaching effectiveness. Communication Yearbook, 3, 543e559.Arhar, J. M., & Kromrey, J. D. (1993). Interdisciplinary teaming in the middle level school: creating a sense of belonging for a-risk middle level students. In

Paper presented at the annual meeting of the american educational research association, atlanta, GA.Aronson, J., Fried, C. B., & Good, C. (2002). Reducing the effects of stereotype threat on African American college students by shaping theories of intelligence.

Journal of Experimental Social Psychology, 38(2), 113e125. http://dx.doi.org/10.1006/jesp.2001.1491.Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: W.H. Freeman.Baylor, A. (2005). The impact of pedagogical agents' image on affective outcomes. In Paper presented at the international conference on intelligent user in-

terfaces, san Diego.Baylor, A. L., & Kim, Y. (2004). Pedagogical agent design: the impact of agent realism, gender, ethnicity, and instructional role. In J. C. Lester, R. M. Vicari, & F.

Paraguacu (Eds.), Lecture notes in computer science: Vol. 3320. Intelligent tutoring systems (pp. 592e603). Berlin/Heidelberg: Springer. http://dx.doi.org/10.1007/978-3-540-30139-4_56.

Baylor, A. L., & Plant, E. A. (2005). Pedagogical agents as social models for engineering: the influence of appearance on female choice. In C.-K. Looi, G.McCalla, B. Bredeweg, & J. Breuker (Eds.), Artificial intelligence in education: Vol. 125. Supporting learning through intelligent and socially informedtechnology (pp. 65e72). Bristol, UK: IOS Press.

Baylor, A. L., & Ryu, J. (2003). The effects of image and animation in enhancing pedagogical agent persona. Journal of Educational Computing Research, 28(4),373e394. http://dx.doi.org/10.2190/v0wq-nwgn-jb54-fat4.

Betz, D. E., & Sekaquaptewa, D. (2012). My fair physicist? feminine math and science role models demotivate young girls. Social Psychological and PersonalityScience, 3(6), 738e746. http://dx.doi.org/10.1177/1948550612440735.

Beyer, S., & Bowden, E. M. (1997). Gender differences in self-perceptions: convergent evidence from three measures of accuracy and bias. Society of Per-sonality and Social Psychology Bulletin, 23, 157e162. http://dx.doi.org/10.1177/0146167297232005.

Blanton, H., Crocker, J., & Miller, D. T. (2000). The effects of in-group versus out-group social comparison on self-esteem in the context of a negativestereotype. Journal of Experimental Social Psychology, 36, 519e530. http://dx.doi.org/10.1006/jesp.2000.1425.

Blascovich, J. (2002). A theoretical model of social influence for increasing the utility of collaborative virtual environments. In Proceedings of the 4th in-ternational conference on collaborative virtual environments, bonn, Germany (pp. 25e30). http://dx.doi.org/10.1145/571878.571883.

Bong, M. (1999). Personal factors affecting the generality of academic self-efficacy judgments: gender, ethnicity, and relative expertise? The Journal ofExperimental Education, 67(4), 315e331. http://dx.doi.org/10.1037/e527772014-429.

Cassell, J., Gill, A., & Tepper, P. (2007). Coordination in conversation and rapport. In J. Cassell, & D. Heylen (Eds.), Embodied NLP '07. Proceedings of theworkshop on embodied language processing (pp. 41e50). Stroudsberg: ACL.

Cheryan, S., Siy, J. O., Vichayapai, M., Drury, B. J., & Kim, S. (2011). Do female and male role models who embody STEM stereotypes hinder women'santicipated success in STEM? Social Psychological and Personality Science, 2(6), 656e664. http://dx.doi.org/10.1177/1948550611405218.

Chouinard, R., & Karsenti, T. (2005). The impact of expectancy and value of mathematics on students' commitment. In Paper presented at the Europeanassociation for research on learning and instruction, nicosie, chypre.

Collins, R. L. (1996). For better or worse: the impact of upward social comparisons on self-evaluation. Psychological Bulletin, 119, 51e69. http://dx.doi.org/10.1037//0033-2909.119.1.51.

Connell, J. P., & Wellborn, J. G. (1991). Competence, autonomy, and relatedness: a motivational analysis of self-system processes. In M. R. Gunnar, & L. A.Sroufe (Eds.), Self-processes in Development: Vol. 23. Minnesota symposium on child psychology (pp. 43e77). Chicago: University of Chicago Press.

Correll, S. J. (2004). Constraints into preferences: gender, status, and emerging career aspirations. American Sociological Review, 69(1), 93e113. http://dx.doi.org/10.1177/000312240406900106.

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e1312

Eccles, J. S. (1994). Understanding women's educational and occupational choices: applying the Eccles et al. model of achievement-related choices. Psy-chology of Women Quarterly, 18(4), 585e609. http://dx.doi.org/10.1111/j.1471-6402.1994.tb01049.x.

Eccles, J. S., Adler, T. F., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J. L., et al. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.),Perspective on achievement and achievement motivation (pp. 75e146). San Francisco: W. H. Freeman.

Eccles, J. S., & Jacobs, J. E. (1986). Social forces shape math attitudes and performance? Signs: Journal of Women in Culture and Society, 11(2), 367e380. http://dx.doi.org/10.1086/494229.

Eccles, J. S., & Midgley, C. M. (1989). Stage-environment fit: developmentally appropriate classrooms for young adolescents. In C. Ames, & R. Ames (Eds.),Research on motivation in education (Vol. 3, pp. 139e186). San Diego: Academic Press.

Eccles, J. S., Wigfield, A., Harold, R. D., & Blumenfeld, P. (1993). Age and gender differences in children's self- and task perceptions during elementary school.Child Development, 64, 830e847. http://dx.doi.org/10.1111/j.1467-8624.1993.tb02946.x.

Eccles, J. S., Wigfield, A., & Schiefele, U. (1998). Motivation to succeed. In W. Damon, & N. Eisenberg (Eds.), Handbook of child psychology (5th ed., Vol. 3, pp.1017e1095). New York: Wiley.

Eisenhart, M. (2008). We can't get there from here: high school girls consider engineering. In Presentation for a women in engineering. ProActive network(WEPAN) national webcast.

Elliott, C., Rickel, J., & Lester, J. C. (1997). Integrating affective computing into animated tutoring agents. In Proceedings of the IJCAI workshop on animatedinterface Agents: Making them intelligent (pp. 113e121).

Garau, M., Slater, M., Pertaub, D. P., & Razzaque, S. (2005). The responses of people to virtual humans in an immersive virtual environment. Presence:Teleoperators and Virtual Environments, 14, 104e116. http://dx.doi.org/10.1162/1054746053890242.

Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., & Tutoring Research Group. (1999). AutoTutor: a simulation of a human tutor. CognitiveSystems Research, 1(1), 35e51.

Grahe, J. E., & Bernieri, F. J. (1999). The importance of nonverbal cues in judging rapport. Journal of Nonverbal Behavior, 23(4), 253e269. http://dx.doi.org/10.1023/A:1021698725361.

Granitz, N. A., Koernig, S. K., & Harich, K. R. (2009). Now it's personal: antecedents and outcomes of rapport between business faculty and their students.Journal of Marketing Education, 31(1), 52e65.

Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S., Morales, M., van der Werf, et al. (2006). Virtual rapport. In 6th international conference on intelligentvirtual agents. Berlin, Germany: Springer-Verlag.

Gratch, J., Wang, N., Gerten, J., Fast, E., & Duffy, R. (2007a). Creating rapport with virtual agents. Proceedings of the 7th International Conference on IntelligentVirtual Agents (IVA), 4722, 125e138. http://dx.doi.org/10.1007/978-3-540-74997-4_12.

Gratch, J., Wang, N., Okhmatovskaia, A., Lamothe, F., Morales, M., & Morency, L.-P. (2007b). Can virtual humans be more engaging than real ones? In J. Jacko(Ed.), Human-computer interaction. HCI intelligent multimodal interaction environments (pp. 286e297). Berlin Heidelberg, Germany: Springer-Verlag.http://dx.doi.org/10.1007/978-3-540-73110-8_30.

Greene, B. A., DeBacker, T. K., Ravindran, B., & Krows, A. J. (2002). Goals, values, and beliefs as predictors of achievement and effort in high schoolmathematics classes. Sex Roles: A Journal of Research, 40, 421e458.

Grolnick, W. S., Gurland, S. T., Jacob, K. F., & Decourcey, W. (2002). The development of self-determination in middle childhood and adolescence. Devel-opment of Achievement Motivation, 147e171.

Grolnick, W. S., & Ryan, R. M. (1989). Parent styles associated with children's self-regulation and competence in school. Journal of Educational Psychology,81(2), 143.

Guadagno, R. E., Blascovich, J., Bailenson, J. N., & McCall, C. (2007). Virtual humans and persuasion: the effects of agency and behavioral realism. MediaPsychology, 10, 1e22.

Guay, F., Vallerand, R. J., & Blanchard, C. (2000). On the assessment of situational intrinsic and extrinsic motivation: the situational motivation scale (SIMS).Motivation and Emotion, 24, 175e213.

Hackett, G. (1985). Role of mathematics self-efficacy in the choice of math-related majors of college women and men: a path analysis. Journal of CounselingPsychology, 32, 47e56. http://dx.doi.org/10.1037/0022-0167.32.1.47.

Hayes, A. L., Ulinski, A. C., & Hodges, L. F. (2010). That avatar is looking at me! social inhibition in virtual worlds. In J. Allbeck, N. Badler, T. Bickmore, C.Pelachaud, & A. Safonova (Eds.), Lecture notes in computer Science: Vol. 6356. Intelligent virtual agents (pp. 454e467). Berlin, Germany: Springer-Verlag.http://dx.doi.org/10.1007/978-3-642-15892-6_49.

Hill, C., Corbett, C., & St Rose, A. (2010). Why so few? women in science, technology, engineering, and mathematics. Washington, D.C: AAUW.Huang, L., Morency, L.-P., & Gratch, J. (2011). Virtual rapport 2.0. In J. Allbeck, N. Badler, T. Bickmore, C. Pelachaud, & A. Safonova (Eds.), Lecture notes in

computer Science: Vol. 6895. Intelligent virtual agents (pp. 68e79). Berlin, Germany: Springer. http://dx.doi.org/10.1007/978-3-642-23974-8_8.Iacobelli, F., & Cassell, J. (2007). Ethnic identity and engagement in embodied conversational agents. In C. Pelachaud, et al. (Eds.), Intelligent virtual agents

(IVA) (pp. 57e63). Berlin: Springer.Jacobs, J. E. (1991). Influence of gender stereotypes on parent and child mathematics attitudes. Journal of Educational Psychology, 83(4), 518. http://dx.doi.

org/10.1037//0022-0663.83.4.518.Johnson, W. L., Rickel, J. W., & Lester, J. C. (2000). Animated pedagogical agents: face-to-face interaction in interactive learning environments. The Inter-

national Journal of Artificial Intelligence in Education, 11, 47e78.Kang, S.-H., & Gratch, J. (2012). Socially anxious people reveal more personal information with virtual counselors that talk about themselves using intimate

human back stories. Annual Review of Cybertherapy and Telemedicine, 181, 202e207.Kim, Y. (2004). Pedagogical agents as learning companions: The effects of agent affect and gender on learning, interest, self-efficacy, and agent persona. Talla-

hassee, FL: Florida State University.Kim, Y., & Baylor, A. L. (2006). A social-cognitive framework for pedagogical agents as learning companions. Educational Technology Research & Development,

54(6), 569e590. http://dx.doi.org/10.1007/s11423-006-0637-3.Kim, Y., Baylor, A. L., & Shen, E. (2007). Pedagogical agents as learning companions: the impact of agent emotion and gender. Journal of Computer Assisted

Learning, 23, 220e234. http://dx.doi.org/10.1111/j.1365-2729.2006.00210.x.Kopp, S., Krenn, B., Marsella, S., Marshall, A. N., Pelachaud, C., Pirker, H., et al. (2006). Towards a common framework for multimodal generation: the

behavior markup language. In Gratch, et al. (Eds.), Lecture notes in computer Science: Vol. 4133. Intelligent virtual agents intelligent virtual agents (pp.205e217). Berlin, Germany: Springer.

Kr€amer, N. C. (2005). Social communicative effects of a virtual program guide. In T. Panayiotopoulos, et al. (Eds.), Lecture notes in computer Science: Vol. 3661.Intelligent virtual agents intelligent virtual agents (pp. 442e543). Hamburg, Germany: Springer. http://dx.doi.org/10.1007/11550617_37.

Kr€amer, N. C., & Bente, G. (2010). Personalizing e-Learning. The social effects of pedagogical agents. Educational Psychology Review, 22, 71e87. http://dx.doi.org/10.1007/s10648-010-9123-x.

Learner, D. G., & Kruger, L. J. (1997). Attachment, self-concept, and academic motivation in high school students. American Journal of Orthopsychiatry, 67,485e492. http://dx.doi.org/10.1037/h0080249.

Lester, J. C., Towns, S. G., Callaway, C. B., Voerman, J. L., & Fitzgerald, P. J. (2000). Deictic and emotive communication in animated pedagogical agents. In J.Cassell, J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied conversational agents (pp. 123e154). Boston: MIT Press.

Lockwood, P., & Kunda, Z. (1997). Superstars and me: predicting the impact of role models on the self. Journal of Personality and Social Psychology, 73,91e103. http://dx.doi.org/10.1037//0022-3514.73.1.91.

Lucas, G. M., Gratch, J., King, A., & Morency, L.-P. (2014). It's only a computer: virtual humans increase willingness to disclose. Computers in Human Behavior,37, 94e100. http://dx.doi.org/10.1016/j.chb.2014.04.043.

N.C. Kr€amer et al. / Computers & Education 99 (2016) 1e13 13

Major, B., Sciacchitano, A. M., & Crocker, J. (1993). In-group vs. out-group comparisons and self-esteem. Personality and Social Psychology Bulletin, 19,711e721. http://dx.doi.org/10.1177/0146167293196006.

Mallam, W. (1993). Impact of school-type and sex of the teacher on female students? Attitudes toward mathematics in Nigerian secondary schools.Educational Studies in Mathematics, 24(2), 223e229. http://dx.doi.org/10.1007/bf01273693.

Marks, H. M. (2000). Student engagement in instructional activity: patterns in the elementary, middle, and high school years. American Educational ResearchJournal, 37, 153e184.

Martin, A., & Marsh, H. (2005). Motivating boys and motivating girls: does teacher gender really make a difference? Australian Journal of Education, 49(3),320e334.

Marx, D. M., & Roman, J. S. (2002). Female role models: protecting women's math test performance. Personality and Social Psychology Bulletin, 28, 1183e1193.http://dx.doi.org/10.1177/01461672022812004.

Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? what does “failure to replicate” really mean? AmericanPsychologist, 70(6), 487e498. http://dx.doi.org/10.1037/a0039400.

McIntyre, R. B., Lord, C. G., Gresky, D. M., Eyck, T., Frye, L. L., J., G. D., et al. (2005). A social impact trend in the effects of role models on alleviating women'smathematics stereotype threat. Current Research in Social Psychology, 10(9), 116e136.

McIntyre, R. B., Paulson, R. M., & Lord, C. G. (2003). Alleviating women's mathematics stereotype threat through salience of group achievements. Journal ofExperimental Social Psychology, 39(1), 83e90. http://dx.doi.org/10.1016/s0022-1031(02)00513-9.

Meece, J. L., Wigfield, A., & Eccles, J. S. (1990). Predictors of math anxiety and its consequences for young adolescents' course enrollment intentions andperformances in mathematics. Journal of Educational Psychology, 82, 60e70. http://dx.doi.org/10.1037//0022-0663.82.1.60.

Moreno, R. (2004). Decreasing cognitive load for novice students: effects of explanatory versus corrective feedback in discovery-based multimedia.Instructional Science, 32(1), 99e113.

Mounts, N. S., & Steinberg, L. (1995). An ecological analysis of peer influence on adolescent grade point average and drug use. Developmental Psychology, 31,915e922. http://dx.doi.org/10.1037//0012-1649.31.6.915.

Nass, C., & Moon, Y. (2000). Machines and mindlessness: social responses to computers. Journal of Social Issues, 56(1), 81e103. http://dx.doi.org/10.1111/0022-4537.00153.

Niewiadomski, R., Mancini, M., Hyniewska, S., & Pelachaud, C. (2010). Communicating emotional states with the Greta agent. A Blueprint for AffectiveComputing: A Sourcebook and Manual, 256.

Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002a). Harvesting implicit group attitudes and beliefs from a demonstration web site. Group Dynamics:Theory, Research, and Practice, 6(1), 101e115. http://dx.doi.org/10.1037//1089-2699.6.1.101.

Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002b). Math ¼ male, me ¼ female, therefore math s me. Journal of Personality and Social Psychology, 83(1),44e59. http://dx.doi.org/10.1037//0022-3514.83.1.44.

Parsons, J., Adler, T. F., & Kaczala, C. M. (1984). Socialization of achievement attitudes and beliefs: parental influences. Child Development, 53, 322e339.http://dx.doi.org/10.2307/1128973.

Pintrich, P. R., & Schrauben, B. (1992). Students' motivational beliefs and their cognitive engagement in classroom academic tasks. In D. H. Schunk, & J.Meece (Eds.), Student perceptions in the classroom (pp. 149e179). Hillsdale, NJ: Lawrence Erlbaum.

Pintrich, P., & Schunk, D. (1996). Motivation in education: Theory, research & applications. Englewood Cliffs, NJ: Prentice-Hall.Plant, E. A., Baylor, A. L., Doerr, C. E., & Rosenberg-Kima, R. B. (2009). Changing middleschool students' attitudes and performance regarding engineering

with computer-based social models. Computers and Education, 53(2), 209e215. http://dx.doi.org/10.1016/j.compedu.2009.01.013.Pratt, J. A., Hauser, K., Ugray, Z., & Patterson, O. (2007). Looking at human-computer interface design: effects of ethnicity in computer agents. Interacting

With Computers, 19, 512e523. http://dx.doi.org/10.1016/j.intcom.2007.02.003.Reeves, B., & Nass, C. I. (1996). The media equation: How people treat computers, television, and new media like real people and places. New York: Cambridge

University Press.Rosenberg-Kima, B. R., Baylor, A., Plant, A. E., & Doer, E. C. (2008). Interface agents as social models for female students: the effects of agent visual presence

and appearance on female students' attitudes and beliefs. Computers in Human Behavior, 24(6), 2741e2756. http://dx.doi.org/10.1016/j.chb.2008.03.017.Singh, K., Granville, M., & Dika, S. (2002). Mathematics and science achievement: effects of motivation, interest, and academic engagement. The Journal of

Educational Research, 95(6), 323e332. http://dx.doi.org/10.1080/00220670209596607.Steele, C. M., & Aronson, J. (1995). Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology,

69(5), 797.Stout, J. G., Dasgupta, N., Hunsinger, M., & McManus, M. A. (2011). STEMing the tide: using ingroup experts to inoculate women's self-concept in science,

technology, engineering, and mathematics (STEM). Journal of Personality and Social Psychology, 100(2), 255. http://dx.doi.org/10.1037/a0021385.Stumm, G., & Pritz, A. (2000). W€orterbuch der Psychotherapie. Vienna, Austria: Springer.Taylor, S. E., & Lobel, M. (1989). Social comparison activity under threat: downward evaluation and upward contacts. Psychological Review, 96, 569e575.

http://dx.doi.org/10.1037//0033-295x.96.4.569.Tesser, A. (1986). Some effects of self-evaluation maintenance on cognition and action. In R. M. Sorrentino, & E. T. Higgins (Eds.), Handbook of motivation and

cognition: Foundations of social behavior (pp. 435e464). New York: Guilford.Thiebaux, M., & Marsella, S. (2007). Smartbody: behavior realization for embodied conversational agents. In C. Pelachaud, et al. (Eds.), 7th International

conference on intelligent virtual agents (IVA). Berlin: Springer.Thomas, D., Ribich, F., & Freie, J. (1982). The relationship between psychological identification with instructors and student ratings of college courses.

Instructional Science, 11(2), 139e154.Tickle-Degnen, L., & Rosenthal, R. (1990). The nature of rapport and its nonverbal correlates. Psychological Inquiry, 1(4), 285e293. http://dx.doi.org/10.1207/

s15327965pli0104_1.Turner, S. L., & Lapan, R. T. (2005). Evaluation of an intervention to increase non-traditional career interests and career-related self-efficacy among middle-

school adolescents. Journal of Vocational Behavior, 66(3), 516e531. http://dx.doi.org/10.1016/j.jvb.2004.02.005.Voelkl, K. E. (1995). School warmth, student participation, and achievement. Journal of Experimental Education, 63, 127e138. http://dx.doi.org/10.1080/

00220973.1995.9943817.Von der Pütten, A. M., Kr€amer, N. C., Gratch, J., & Kang, S.-H. (2010). “It doesn't matter what you are!” explaining social effects of agents and avatars.

Computers in Human Behavior, 26(6), 1641e1650.Wang, N., & Gratch, J. (2009). Can a virtual human build rapport and promote learning?. In Proceedings of the 2009 Conference on Artificial Intelligence in

Education (pp. 737e739).Wigfield, A., & Eccles, J. S. (1992). The development of achievement values: a theoretical analysis. Developmental Review, 12, 265e310. http://dx.doi.org/10.

1016/0273-2297(92)90011-p.Zanbaka, C. A., Ulinski, A. C., Goolkasian, P., & Hodges, L. F. (2007). Social responses to virtual humans: implications for future interface design. In Proceedings

of the SIGCHI conference on human factors in computing systems (pp. 1561e1570). New York: ACM.