My Professor Cares: Experimental Evidence on the Role of ... · My Professor Cares: Experimental Evidence on the Role of Faculty Engagement Scott Carrell and Michal Kurlaender University

My Professor Cares: Experimental Evidence on the

Role of Faculty Engagement

Scott Carrell and Michal Kurlaender University of California-Davis

1

Presentation Outline

•  Research and Policy Context •  Exploratory Work—Focus Groups •  Theore<cal Mo<va<on •  Pilot at UCD •  Scale up at California State University Campus

– Results – Mechanisms

•  Next Steps

2

Research & Policy Context

•  Low persistence and degree comple<on rates at non-‐selec<ve colleges and universi<es.

•  High rates of remedial/developmental course taking at non-‐selec<ve colleges and universi<es.

•  Large dispari<es in college persistence, degree aMainment, and developmental course taking by race/ethnicity and socioeconomic status.

•  Increased policy aMen<on at California’s broad-‐access public postsecondary ins<tu<ons (California State Universi<es and Community Colleges)

3

California Public Higher Education

•  California’s structured system of public higher educa<on—The Master Plan (1960) – UC: reserved for the top 1/8th (9 campuses) – CSU: reserved for the top 1/3rd (23 campuses) – CCC: “any student capable of benefi<ng from instruc<on” (113 campuses)

4

Racial/Ethnic Composition of Students at the California State University System

0%

20%

40%

60%

80%

100%

2004 2006 2008 2010 2012 2014

First-Time Freshmen by Race/Ethnicity

Hispanic White Black Asian Other

5

Persistence and Completion Rates at the California State University System

6

Six-Year Completion Rates by Race/Ethnicity at the California State University System

58.8% 64.1%

38.3% 41.8%

53.2% 61.0%

45.4% 51.5%

0%

100%

2004 2005 2006 2007 2008 2009

White Black Asian Hispanic

7

College Completion Puzzle

•  Why do so many students who start college don’t finish? – Rising complexity of higher educa<on enrollment – Loss of Interest – Academic prepara<on – Lack of informa<on –  Ins<tu<onal policies/prac<ces

8

High remediation and low graduation rates at CSUs by race/ethnicity

Data from CSU Analytic Studies: http://www.asd.calstate.edu/performance/proficiency.shtml

9

0

0.1

0.2

0.3

0.4

0.5

0.6

African American Asian American Latino White Other

CSU Systemwide Remediation Need: Fall 2016 First-Time Freshmen

Math

English

Prior Literature—College Access

A substan<al body of empirical work on improving college access and success, par<cularly for low income students and other underrepresented groups •  Academic rigor in high school and informa<on are key determinants of college access and success. – Curricular intensity in high school – Recent RCTs of interven<ons focused on improving student informa<on on college access

10

Prior Literature—Persistence/Completion

•  Much less evidence on persistence and comple<on – A light touch individualized student coaching interven<on led to higher reten<on and comple<on rates

– Quasi-‐experimental work suggests that professor characteris<cs maMer, informa<on about college readiness, academic rigor in high school

11

Exploratory Work

•  Focus groups at CSU with men of color (Winter 2014) – Lack of interac<on with faculty – Disconnected from courses and faculty – Lack of understanding about course success

12

Professor Engagement

•  Can we improve professor engagement in students’ course success?

•  Can we improve students informa<on and sense of self-‐efficacy through student-‐professor engagement?

•  Does this impact student success; how/why; and for whom?

13

Pilot Light Touch Intervention at UC Davis (Spring 2014)

•  Two personalized emails from the professor during the term, providing informa<on and feedback: (1)  how to be successful in the class (2)  how students are progressing in the class (3)  the availability of the professor and other

supports

14

Theoretical Motivation

Promo%ng students’ sense of self-‐efficacy and strengthening help-‐seeking behavior •  Interven<on: Feedback and Informa<on

–  Individualized feedback about course performance and success

–  Focused on the process underlying the tasks expected of students in the course

– Offer strategies students can incorporate to improve performance

Ø Focusing on posi<ve steps to be taken rather than shorgalls in performance can enhance self-‐efficacy, aspira<ons, and performance

15

Pilot: Study Design

•  Sehng: UC Davis large introductory Economics course (420 students)

•  Randomiza<on at 1st homework submission –  Treatment (n=35), Control (n=34)

•  Outcomes –  Course performance: Exam scores (two midterms and a final), and Final course grade

–  Effort: Homework scores (unlimited aMempts) and <me spent •  Pre-‐treatment background characteris<cs

–  High School GPA, 1st genera<on, Race, Gender, Residency status •  Mechanisms

–  Office hour aMendance, mo<va<on, percep<ons of professor

16

Pilot: Random Assignment

•  Students randomly assigned to the treatment and control group irrespec<ve of background characteris<cs – Demographic characteris<cs unavailable at the <me of randomiza<on

•  Data matched to student demographic characteris<cs, ex post. – Treatment uncorrelated with student pre-‐treatment background characteris<cs

17

Variable 1 2-0.155 -0.091(0.173) (0.173)0.058 0.05

(0.166) (0.169)0.153 0.081

(0.210) (0.214)-0.227 -0.296(0.196) (0.196)

0.2 0.139(0.244) (0.243)-0.091 -0.162(0.252) (0.249)0.083 -0.042

(0.239) (0.240)Observations 53 53P-value: Joint Significance of all individual covariates 0.8905 0.8449Includes TA Fixed Effects No Yes

Entering Cohort==2012

Male

First Generation College Goer

HS GPA

Under-represented Minority

CA Resident


Pilot Randomness Checks

18

Pilot: Estimating Equation

€

Yit = φ0 + φ1Treatment + βXi + λt +ε it

•  Where, X are pre-‐treatment characteris<cs •  is a TA fixed effect

19

€

λt

Pilot: Results (Homework) 20

Mean .93 .55 7.05 .85

Pilot: Results (Exams & Course Grade) 21

Mean .74 .68 .64 .68 .72 2.40

Pilot: Results (Mechanisms) 22

Mean .45 2.39 3.02 2.32 .23

Pilot: Mechanisms (Student Feedback)

•  I'd…like to thank you for offering your help in such a kind manner, I've rarely seen teachers at this school respond to missed assignments the way you have. I'll be sure to complete future assignments in a %mely manner, the first prac%ce homework was indeed preFy helpful.

•  Thanks for talking to me about my homework and test scores. Even though you have a couple hundred students, I really appreciate the effort you put into making it personal for your students. I would have gone to office hours the first %me you emailed me, but I simply forgot by the end of the week.

23

Pilot: Mechanisms (Student Feedback)

•  This class is fulfilling a GE for me and my other classes do rank higher on my list. While I do enjoy microeconomics and usually aim to only take GE's that interest me, I'm just too busy to spend too much %me on the homework. I definitely plan to study and prac%ce extensively for the tests, but not make it a priority, which i know is not ideal but a reality for this quarter. But once again, thanks for the email, I really appreciate what you're doing and I do wish more professors do what you are doing.

24

Pilot: Summary of Results

•  Aker the first “nudge” regarding homework: – Students in the treatment group increased the <me spent and performance on homework

•  Aker the second “nudge” regarding performance on the first exam: – Students in the treatment group scored significantly higher on the second exam

•  Significant posi<ve effect on overall course performance – No significant effects on dropping the course

•  Noisy evidence on poten<al mechanisms

25

Scale-Up of Intervention at CSU

•  Funded by College Futures Founda<on •  Large undergraduate, broad-‐access, four-‐year ins<tu<on in Northern California – Among the largest CSU campuses, enrolling approximately 24,000 undergraduate students.

–  The campus enrollment is approximately 50% students from underrepresented backgrounds.

– Over two-‐thirds of all first-‐<me freshmen require some remedial coursework in English or math

•  In partnership with Campus Center for College and Career Readiness

26

Persistence Rates at CSU Campus

First-‐<me freshmen entering in 2014

27

74

76

78

80

82

84

86

88

URM (N=1131)

Asian (N=681)

Foreign (N=86)

White (N=939)

One-Year Retention Rate

Persistence Rates at CSU Campus

First-‐<me freshmen entering in 2014

28

58

60

62

64

66

68

70

72

74

76

78

URM (N=931)

Asian (N=590)

Foreign (N=51)

White (N=744)

Three-Year Retention Rate

Degree Completion 29

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

African American Latino Asian/Pacific Islander White

6-Year CSU Graduation Rate: Fall 2009 Cohort

CSU Implementation

•  Iden<fy Large Courses: –  Fall 2015 iden<fied classes with 80+ enrollment for Spring 2016 –  Randomly selected 42 large undergrad for recruitment of faculty

•  Faculty Recruitment: –  Faculty invited to par<cipate (by CSU Provost and Dean of Undergraduate Studies)

–  Three informa<on sessions for faculty to learn about the interven<on and determine if they wanted to par<cipate.

–  21 Faculty (24 classes) included in the study (Biology, History, Economics, Psychology, Math, Chemistry, and Philosophy, among others).

•  Faculty compensa<on: –  $500.00 to par<cipate + administra<ve assistance to implement.

–  $100.00 gik card for survey comple<on at the end of the term.

30

CSU Intervention Courses

•  Art 3B •  Astronomy 4 •  Biology 22 •  Chemistry 1A (2 courses) •  Economics 1B •  Engineering 45 (2 courses) •  English 50B •  Ethnic Studies 11 •  Family and Consumer Sciences 10 •  Geology 7 (2 courses) •  History 7 •  History 17B •  History 51 •  Math 24 (2 courses) •  Music 18 •  Philosophy 4 •  Psychology 2 (3 courses)

31

CSU Study Implementation

•  January 2016: students in the 24 courses randomly assigned to treatment and control groups within classes. –  Treatment=1,425; Control =1,475

•  Interven<on: three emails sent by the professor to students in the treatment group. (Researchers provided faculty with email templates.) –  Email 1: Introductory (sent 2-‐3 weeks into the semester); –  Email 2: Feedback based on performance (sent approximately half way through the semester);

–  Email 3: Feedback based on performance (sent about 1 to 1.5 weeks before the final exam).

32

Email 2 Template

STUDENT PROFILE 2 (B/ C performance) Dear XXX, We are approaching the mid point in the semester. You’ve done well so far on [Fill-‐in: Midterm, quiz, homework/?]. To strengthen your grade in the course and do well in the upcoming [Fill-‐in Final, next assignment, HW, ??], I want to encourage you to [fill-‐in: come to class regularly, and review lecture notes, go to office hours]. This will help you be7er understand the class material. Sincerely, Professor X STUDENT PROFILE 1 (< C acceptable performance) Dear XXX, We are approaching the mid point in the semester. I am concerned that based on your performance on the [Fill-‐in: Midterm, quiz, homework/?] you may be struggling in this course. However, don’t be discouraged, there is s<ll plenty of <me to recover. To do well in the upcoming [Fill-‐in Final, next assignment, HW, ??] I encourage you to [fill-‐in: come to class regularly review lecture notes, go to office hours]. This will help you be7er understand the class material. Sincerely, Professor X STUDENT PROFILE 3 (B+ or higher performance) Dear XXX, We are approaching the mid point in the semester. You’ve done very well so far on [Fill-‐in: Midterm, quiz, homework/?]. To keep up your grade and do well in the upcoming [Fill-‐in Final, next assignment, HW, ??], I encourage you to [fill-‐in: come to class regularly, and review lecture notes, go to office hours]. Sincerely, Professor X

33

Email 2 Template

STUDENT PROFILE 3 (B+ or higher performance) Dear XXX, We are approaching the mid point in the semester. You’ve done very well so far on [Fill-‐in: Midterm, quiz, homework/?]. To keep up your grade and do well in the upcoming [Fill-‐in Final, next assignment, HW, ??], I encourage you to [fill-‐in: come to class regularly, and review lecture notes, go to office hours]. Sincerely, Professor X

34

1 2 3 4 5

All Students Latino Black Asian White

2.49 2.41 2.21 2.60 2.73(1.22) (1.25) (1.33) (1.15) (1.11)0.56 0.54 0.47 0.60 0.65

(0.50) (0.50) (0.50) (0.49) (0.48)0.81 0.78 0.75 0.84 0.87

(0.40) (0.41) (0.44) (0.36) (0.33)0.15 0.15 0.19 0.19 0.13

(0.36) (0.36) (0.39) (0.40) (0.34)0.56 0.60 0.60 0.54 0.52

(0.50) (0.49) (0.49) (0.50) (0.50)0.32 0.57 0.28 0.35 0.12

(0.47) (0.50) (0.45) (0.48) (0.32)0.12 0.06 0.10 0.07 0.20

(0.33) (0.24) (0.30) (0.26) (0.40)948.78 914.10 893.13 934.70 1026.09

(142.08) (123.12) (125.53) (141.70) (141.28)0.72 0.82 0.85 0.69 0.57

(0.45) (0.39) (0.35) (0.46) (0.49)0.78 0.85 0.83 0.85 0.62

(0.41) (0.35) (0.38) (0.36) (0.49)0.50 0.52 0.54 0.53 0.41

(0.50) (0.50) (0.50) (0.50) (0.49)Observations 2,782 734 199 652 652

SAT Score

Remediation Math

Remediation English

Freshman

Transfer

Female

First Generation College

Course Grade

Earned Course A/B

Passed Course

Dropped Course

CSU Summary Statistics 35

CSU Randomness Checks 36

Specification 1 2 3 4 5


0.04* 0.03 0.04 -0.02 0.08***(0.02) (0.04) (0.07) (0.05) (0.03)-0.02 0.04 -0.09 -0.08 -0.03(0.03) (0.03) (0.14) (0.05) (0.06)-0.10 -0.03 -0.37 0.14 -0.23(0.06) (0.19) (0.27) (0.13) (0.19)-0.01 -0.04 0.12 -0.03 -0.05(0.02) (0.04) (0.08) (0.04) (0.05)-0.06 -0.02 -0.24 0.03 -0.26(0.11) (0.22) (0.50) (0.20) (0.27)0.01 0.06 0.07 -0.02 -0.04

(0.04) (0.08) (0.20) (0.07) (0.07)Remediation English 0.01 -0.01 -0.01 0.06 -0.01

(0.05) (0.06) (0.13) (0.07) (0.07)Latino 0.01

(0.04) Black 0.06

(0.07) Asian 0.00

(0.05) -0.03 (0.03)

Observations 2,782 734 199 652 606

SAT Score

Remediation Math

White

Female

First Generation College

Transfer

Freshman

CSU Estimating Equation

€

Yit = φ0 + φ1Treatment + βXi + λt +ε it

•  Where, X are pre-‐treatment characteris<cs •  is a class fixed effect

37

€

λt



Panel A. Class Fixed Effects0.03 0.24** -0.19 -0.09 -0.02

(0.06) (0.10) (0.25) (0.10) (0.10)

Panel B. Demographic Controls

0.03 0.24** -0.20 -0.14 0.01(0.06) (0.10) (0.25) (0.10) (0.10)

Panel C. Academic Preparation Controls0.04 0.26** -0.23 -0.14 0.00(0.06) (0.11) (0.24) (0.10) (0.11)

Observations 2,230 599 150 495 499

Treatment Effect

Treatment Effect

Treatment Effect

CSU Results: Course Grade 38

Notes: Each column reports results from a separate regression. All specifications include class fixed effects. Demographic characteristics include indicator variables for gender, transfer and first generation college. Academic preparation variables include SAT score (verbal + math), indicator variables for needing remediation in math and English. Standard errors in parentheses are clustered at the class level.

CSU Results: Pass Course 39




Panel A. Class Fixed Effects0.00 0.07* -0.09 -0.03 0.02

(0.02) (0.04) (0.09) (0.03) (0.04)


0.00 0.07* -0.10 -0.05* 0.03(0.02) (0.04) (0.10) (0.03) (0.04)

Panel C. Academic Preparation Controls0.01 0.07* -0.11 (0.04) 0.03(0.02) (0.04) (0.09) (0.03) (0.04)

Observations 2,230 599 150 495 499

Treatment Effect

Treatment Effect

Treatment Effect

CSU Results: Earned an A or B 40




Panel A. Class Fixed Effects0.01 0.08** -0.09 0.01 -0.02

(0.03) (0.03) (0.09) (0.04) (0.05)


0.01 0.08** -0.08 -0.01 -0.01(0.03) (0.03) (0.09) (0.03) (0.05)

Panel C. Academic Preparation Controls0.01 0.08** -0.08 -0.01 -0.01(0.03) (0.03) (0.08) (0.04) (0.06)

Observations 2,230 599 150 495 499

Treatment Effect

Treatment Effect

Treatment Effect



Panel A. Class Fixed Effects0.02 -0.04 0.11** 0.00 0.05*

(0.02) (0.03) (0.05) (0.02) (0.03)


0.02 -0.04 0.14*** 0.01 0.06* (0.01) (0.03) (0.05) (0.02) (0.03)

Panel C. Academic Preparation Controls0.02 -0.04 0.13** 0.01 0.06** (0.01) (0.03) (0.05) (0.02) (0.03)

Observations 2,782 734 199 652 606

Treatment Effect

Treatment Effect

Treatment Effect

CSU Results: Dropped Course 41


CSU Results: Grades in Other Courses 42




Panel A. Class Fixed Effects0.01 0.13 -0.20 -0.09 0.00

(0.05) (0.08) (0.17) (0.08) (0.08)


0.01 0.12 -0.16 -0.12 -0.01(0.05) (0.07) (0.18) (0.08) (0.08)

Panel C. Academic Preparation Controls0.02 0.14* -0.20 -0.12 -0.01(0.05) (0.08) (0.16) (0.08) (0.08)

Observations 2,204 591 147 491 492

Treatment Effect

Treatment Effect

Treatment Effect



Panel A. Class Fixed Effects0.09 0.14 0.82* 0.10 0.35*

(0.09) (0.17) (0.46) (0.18) (0.19)


0.06 0.12 1.09** 0.15 0.26(0.08) (0.15) (0.45) (0.17) (0.20)

Panel C. Academic Preparation Controls0.09 0.13 0.83 0.24 0.43* (0.10) (0.16) (0.63) (0.18) (0.23)

Observations 472 111 41 142 87

Treatment Effect

Treatment Effect

Treatment Effect

CSU Results: Grades in Other Courses (students who dropped)

43


CSU Results: Heterogeneity 44


Specification 1 2 3 4 5 6 7 8 9 10

Male Female First GenNot

First GenNeed

RemediationNo

Remediation Low SAT High SAT FreshmanUpper

Classman

-0.11 0.09 -0.02 -0.03 0.02 0.00 -0.01 -0.02 0.11 -0.10(0.07) (0.09) (0.10) (0.10) (0.08) (0.13) (0.09) (0.07) (0.10) (0.08)

Observations 841 1,092 366 792 1,451 305 815 806 937 961

Treatment Effect

Student Survey

•  A week aker the term ended, all students in the experimental courses were recruited via email to par<cipate in a survey (n=2,400)

•  Response rate: 23% •  Survey Instrument:

– Student percep<ons of faculty and classroom engagement in courses, and overall campus sense of belonging.

45

CSU Results: Mechanisms (Intervention Instructor)

46




Panel A. Instructor cares about my success0.10 0.45** 0.04 0.26 0.29

(0.13) (0.19) (0.70) (0.19) (0.21)

Panel B. Instructor kept me informed

0.00 0.14 0.18 0.62** 0.14(0.14) (0.24) (1.30) (0.28) (0.19)

Panel C. Instructor motivated me to do well

0.07 0.22 0.10 0.06 0.34(0.10) (0.14) (0.67) (0.14) (0.22)

Observations 561 162 32 168 139

Treatment Effect

Treatment Effect

Treatment Effect

CSU Results: Mechanisms (Other Instructor)

47




Panel A. Instructor cares about my success-0.28 0.14 -0.89 -0.55** -0.25(0.18) (0.30) (0.65) (0.25) (0.34)

Panel B. Instructor kept me informed

0.03 0.12 -1.29 -0.18 -0.09(0.13) (0.23) (1.09) (0.20) (0.39)

Panel C. Instructor motivated me to do well

0.21* 0.42** 0.17 0.02 0.24(0.12) (0.20) (0.42) (0.13) (0.34)

Observations 528 159 31 157 127

Treatment Effect

Treatment Effect

Treatment Effect

Email Responses (1)

•  “I apologize for missing your class wednesday akernoon, i was stuck in lassen hall trying to pay my monthly installment for tui<on. I will definitely be at mondays lecture. ” [History 7B]

•  “Hello! Thank you for emailing. I actually do have a ques<on regarding the week three course work. I cannot find it on Blackboard and I was wondering if I am looking in the wrong folder. Any help would be greatly appreciated. Thank you!” [Family and Consumer Sciences 10]

•  “Thank you professor. I have not missed a single day in your class because I take this class very serious and you are doing a lot beMer at helping us as students succeed then my previous professor. I was wondering if maybe the system is not showing that I am present but like I said I have not missed a day. And I will defiantly come to your office hours when I need your help ” [Psychology 2]

•  “Hi Professor, Thank you for all of this informaFon. It's very useful and I'm looking forward to learning a lot from your class. I was struggling in the beginning because I've never taken a one part lecture and one part discussion based class, but I think I'm starFng to get the hang of it. If I have any quesFons, I'll be sure to stop by your office hours. Thanks once again!“ [English 50B]

48

Email Responses (2)

•  “Hello Professor, I was wondering why I was sent this message. I believe that I have been coming to class regularly as well as taking notes acFvely. Thank you for your Fme.” [Art 3B]

•  “Hello professor, Yes I am definitely not doing well. I have been distracted this semester. I suppose I can take the day off of work tomorrow to come during office hours. I'm also comfortable with the amount of extra credit I have accumulated.” [Art 3B]

•  “Thank you for email! I hope to do well on the next 2 exams. I also apologize for my poor performance on the first exam, there was a personal problem I had to deal with the day before and it affected my studying & performance on the exam. Thank you for reaching out, I really appreciate it. ” [Psychology 2]

•  “Hello professor. I aMend every class, go to the review sessions, and have turned in the extra credit so I am defiantly trying to do well but I am s<ll struggling. I will come to office hours and try to meet up with our TA as well. Let me know if there is anything else I can do. Thank you” [History 51]

49

Email Responses (3)

•  “Hi Professor, thank you for your concern about my grade, history is a subject I struggle on. This course has been excep<onally hard because its history from another country which has made it very hard for me. Is there a way I can s<ll pass the course? I am going to dedicate a lot of <me studying for this final exam. I really cannot afford an F's in my transcript as it will impact my overall Gpa greatly.” [History 7]

•  “I just saw your message, and would like to thank you for your support. I have taken a few of your classes over my Fme here at Sac State and would like to say that I have really enjoyed them all. I apologize for missing more classes then id have liked this semester and would like you to know that it was in no way meant as disrespecXul to you. I will be graduaFng in December and I do not think I will have the privilege of taking any more of your classes. Thank you for everything. You're enthusiasm for History and life in general has been inspiring. Have an amazing summer, hopefully you can find some Fme to surf! Thanks Again!” [History 51-‐ (top bin)]

50

Faculty Survey

•  Survey of all faculty par<cipants aker Spring 2015 and interven<on concluded

•  $100 Amazon Gik Card for survey comple<on

51

What was the nature of the student responses to your emails?

•  “All were posi<ve -‐ most thanking me or apologizing for their performance or sta<ng that they would try harder.” [Family Consumer Sciences 10]

•  “I don't think I saw any responses that I would characterize as nega<ve. There were definitely some posi<ve ones, and many were simply responding with a ques<on regarding some aspect of the course.” [Astronomy 4]

•  “A couple were concerned about why they received the first email -‐ this is because it went out aker the first midterm but it was really a "welcome to the class" email so the disconnect worried them.” [Geology 7]

•  “Curious, mostly; a few worried” [English 50B]

52

Did any responses surprise you?

•  “With the first email, I was pleasantly surprised to see a number of student responses that didn't actually have a ques<on or issue, but were just responses to say hello and thank you for the email.” [Astronomy 4]

•  “Yes, students thanked me for touching base with them. It was surprising how thankful they were for such a simple email.” [History 7]

•  “Yes, three students thanked me for caring about whether they succeeded or not.” [Psychology 2]

•  “Yes and no. The responses generally came from what I would consider already conscien<ous students. They weren't defined by grade, but by ac<ve involvement. If they were really engaged in the classroom, they were more interested in the emails. Students that didn't really care probably ignored them.” [Engineering 45]

53

How important do you think it is for you to interact with your students outside of class?

•  “I think it is a good idea to reach out to students who are struggling to see if I can help.” [Family and Consumer Sciences 10]

•  “I think it's very important-‐-‐it helps to know who is willing to do the addi<onal work required to be successful in the class.” [Econ 1B]

•  “I think it depends on the level of interac<on that is able to be achieved during class. I find that interac<on is extremely beneficial, and generally try to promote it during lecture itself. With a class of this size, I think these emails really did serve a useful purpose of establishing some level of one-‐on-‐one interac<on between myself and the students.” [Astronomy 4]

•  “I think it's important, but some of them really don't care. The ones that come see me, or interact in class get a lot more out of their educa<on. I can't force them to come to my office.” [Engineering 45]

54

Conclusions and Next Steps

•  Sugges<ve evidence that feedback from the professor can affect student effort and course performance

•  But important differences by racial/ethnic groups, why? – Differences in response to email language – Spillovers across T/C? – Explore students who dropped

55

Next Steps

•  Currently in the field: 14 new classes – Spillovers: add mul<-‐sec<on classes – Change <ming of messages –  In-‐class evalua<ons for understanding mechanisms

•  Spring 2018: Replicate pilot in large introductory UCD class

56

Thank You!

•  College Futures Founda<on •  Research Support

–  CSU Campus –  Joy Salveh, Director of the College & Career Readiness Center

– Monica BhaM, Erika Jackson, Elizabeth Zeiger Friedman, Angelica Cortes, Andrew Perez, Claudia Escobar, Ryan Bandy, Brian Fernandez-‐Oquendo

•  Ques<ons/Comments: – [email protected] –  [email protected]

57

•  Extra Slides

58

Pilot Summary Statistics 59

Pilot Summary Statistics 60

Variable 1 2-0.155 -0.091(0.173) (0.173)0.058 0.05

(0.166) (0.169)0.153 0.081

(0.210) (0.214)-0.227 -0.296(0.196) (0.196)

0.2 0.139(0.244) (0.243)-0.091 -0.162(0.252) (0.249)0.083 -0.042

(0.239) (0.240)Observations 53 53P-value: Joint Significance of all individual covariates 0.8905 0.8449Includes TA Fixed Effects No Yes


Male

First Generation College Goer

HS GPA

Under-represented Minority

CA Resident


Pilot Randomness Checks

61

CSU Intervention Courses

•  Art 3B •  Astronomy 4 •  Biology 22 •  Chemistry 1A (2 courses) •  Economics 1B •  Engineering 45 (2 courses) •  English 50B •  Ethnic Studies 11 •  Family and Consumer Sciences 10 •  Geology 7 (2 courses) •  History 7 •  History 17B •  History 51 •  Math 24 (2 courses) •  Music 18 •  Philosophy 4 •  Psychology 2 (3 courses)

62

CSU Results: Grades in Other Courses 63


Specification 1 2 3 4 5All

Students Latino Black Asian White0.06 0.17** 0.05 -0.07 0.01

(0.05) (0.07) (0.18) (0.09) (0.07)

-0.08 -0.12 0.70 0.04 0.03(0.09) (0.14) (0.50) (0.16) (0.26)

Observations 2,676 702 188 633 579

Table X: Treatment Effects on Other Course Grades

Treatment Effect

Treatment * Droppped Intervention Course

Email 1 Template

Dear [insert student name], Welcome to [insert course name]. I would like to provide you with some useful informa<on that will help you succeed in my course. First, it is really important to aMend class on a regular basis so that you don’t fall behind. Second, I highly encourage you to come visit me with your ques<ons regarding the course material during my regularly scheduled office hours [INSERT TIMES]. Taking these steps will help you [INSERT: e.g. “prepare for the midterm and final exam, which are a substan<al propor<on of your grade.” Or “beMer understand the reading needed to complete the assignments”]. I look forward to a great semester with you [OR your own sign-‐off] Sincerely, Professor XXX

64

Email 3 Template

STUDENT PROFILE 1 (< C acceptable performance) Dear XXX, We are nearing the end of the semester. Although your performance to date has been lower than you may have hoped, there are s<ll opportuni<es to improve your grade in the course. Remember, we s<ll have [Fill-‐in: Final exam, paper, etc., ??] To do well in the upcoming [Fill-‐in exam, paper, etc., ??] I encourage you to [fill-‐in: come to remaining classes, review lecture notes, go to office hours, etc]. Sincerely, Professor X STUDENT PROFILE 2 (B/ C performance) Dear XXX, We are nearing the end of the semester. You’ve con<nued to do well on [Fill-‐in: Midterm, quiz, homework/?]. To strengthen your grade in the course and do well in the upcoming [Fill-‐in: Final exam, paper, etc., ??], I want to encourage you to [fill-‐in: come to remaining classes, review lecture notes, go to office hours, etc]. Sincerely, Professor X STUDENT PROFILE 3 (B+ or higher performance) Dear XXX, We are nearing the end of the semester. You’ve con<nued to do very well on [Fill-‐in: Midterm, quiz, homework/?]. To keep up your grade and finish strong on the upcoming [Fill-‐in: Final exam, paper, etc., ??], I want to encourage you to [fill-‐in: come to remaining classes, review lecture notes, go to office hours, etc]. Sincerely, Professor X

65

Email 1 Responses

•  “I apologize for missing your class wednesday akernoon, i was stuck in lassen hall trying to pay my monthly installment for tui<on. I will definitely be at mondays lecture. ” [History 7B]

•  “Hello! Thank you for emailing. I actually do have a ques<on regarding the week three course work. I cannot find it on Blackboard and I was wondering if I am looking in the wrong folder. Any help would be greatly appreciated. Thank you!” [Family and Consumer Sciences 10]

•  “Thank you professor. I have not missed a single day in your class because I take this class very serious and you are doing a lot beMer at helping us as students succeed then my previous professor. I was wondering if maybe the system is not showing that I am present but like I said I have not missed a day. And I will defiantly come to your office hours when I need your help ” [Psychology 2]

•  “Hi Professor, Thank you for all of this informa<on. It's very useful and I'm looking forward to learning a lot from your class. I was struggling in the beginning because I've never taken a one part lecture and one part discussion based class, but I think I'm star<ng to get the hang of it. If I have any ques<ons, I'll be sure to stop by your office hours. Thanks once again!“ [English 50B]

66

Email 2 Responses

•  “Hello Professor, I was wondering why I was sent this message. I believe that I have been coming to class regularly as well as taking notes ac<vely. Thank you for your <me.” [Art 3B]

•  “Hello professor, Yes I am definitely not doing well. I have been distracted this semester. I suppose I can take the day off of work tomorrow to come during office hours. I'm also comfortable with the amount of extra credit I have accumulated.” [Art 3B]

•  “Thank you for email! I hope to do well on the next 2 exams. I also apologize for my poor performance on the first exam, there was a personal problem I had to deal with the the day before and it affected my studying & performance on the exam. Thank you for reaching out, I really appreciate it. ” [Psychology 2]

•  “Hello Professor, I've just received your email on grades. I know I've been struggling but I do not know why I've done all my reading and I take the quizzes mul<ple <mes. This is my second <me taking PSYC 2 because the first <me I got a D. I would really appreciate any recommenda<ons you have for me I've gone over many quizzlets as well. Any recommenda<ons would help.” [Psychology 2]

•  “Hello professor. I aMend every class, go to the review sessions, and have turned in the extra credit so I am defiantly trying to do well but I am s<ll struggling. I will come to office hours and try to meet up with our TA as well. Let me know if there is anything else I can do. Thank you” [History 51]

67

Email 3 Responses

•  “Hi Professor, thank you for your concern about my grade, history is a subject I struggle on. This course has been excep<onally hard because its history from another country which has made it very hard for me. Is there a way I can s<ll pass the course? I am going to dedicate a lot of <me studying for this final exam. I really cannot afford an F's in my transcript as it will impact my overall Gpa greatly.” [History 7]

•  “I just saw your message, and would like to thank you for your support. I have taken a few of your classes over my <me here at Sac State and would like to say that I have really enjoyed them all. I apologize for missing more classes then id have liked this semester and would like you to know that it was in no way meant as disrespecgul to you. I will be gradua<ng in December and I do not think I will have the privilege of taking any more of your classes. Thank you for everything. You're enthusiasm for History and life in general has been inspiring. Have an amazing summer, hopefully you can find some <me to surf! Thanks Again!” [History 51-‐ (top bin)]

•  “I am trying my best to do good in this last exam. I haven't done so great on my other exams since I have too many classes this semester, and I am not so great in remembering terms. History was never my strong subject, always interested in history but could not find a right approach to study for the exams. I've never missed a class and always take notes but when it gets to the exam I just can't remember much of the names. For this exam I am doing another approach hopefully it works (note cards). It's been a while since I had an exam other than calcula<ons (engineering) and maybe I forgot how to study for these kinds of exams! I also went to see the documentaries this Friday and will be wri<ng the essay to help with my grade. Hope everything works out. Thanks for your concern and it’s been a great semester! I've learned a lot even though my grades may not show it. Thanks!” [History 51-‐ (top bin)]

•  “Hello! I wanted to talk to you about what my grade was right now up to date and what I could get it up to by the end of the semester. My goal is a C. This is my first online class and it has been hard to adapt to the different learning style, I know that's not an excuse but I really do care for my learning and unfortunately the quizzes do not reflect that. Thank you!“ [Family and Consumer Sciences 10, bin 2]

68

Documents

My Professor Cares: Experimental Evidence on the Role of ... · My Professor Cares: Experimental Evidence on the Role of Faculty Engagement Scott Carrell and Michal Kurlaender University