2009-2010 Annual Program Assessment for the MS Experimental … · 2018-05-13 · 2009-2010 . Annual Program Assessment for the . MS Experimental Psychology Graduate Program . Final

2009-2010 Annual Program Assessment for the

MS Experimental Psychology Graduate Program Final Report

Submitted to Dr. Stephanie Stein, Chair Department of Psychology

by

Dr. Wendy A. Williams Program Director

June 14, 2009

1

Table of Contents

I. Introduction and Overview……………………………………………………….….3 II. Program Enrollment …………………………………………………………..……4 Admissions, Enrollment Changes, Enrollment Trends, Recruitment, Degrees conferred, Attrition and Retention III. Assessment of Student Learning Overview …………………………………….….9 Review of Record Keeping Procedures, Number of Enrolled Students IV. Major Milestones ……………………………………………………………….…12 General Academic Assessment Data, Course of Study Forms, Option Approval Forms, IACUC/HSRC Approvals, Theses Accepted, Degrees Conferred, Conference presentations V. Time to Completion Data …………………………….……………………………15

Thesis Defense Meetings/Time to Completion VI. Programmatic Course Offerings …………………………………….…………….17 VII. Quarterly Survey of Student Progress …………………………………………....18

Academic Performance Summary, Cumulative GPA, Itemized Assessments: • Content knowledge • Skills mastery • Quantitative analytical skills • Logic-based analytical skills • Comprehension • Written and verbal expressive skills •Appreciation of empirical evidence • Ability to succeed in the experimental psychology program • Disposition consistent with the field of experimental psychology

VII. Assessing the Quarterly Survey ………………………………………………….27 Previous Limitations, Results of solutions Implemented, Limitations to the

new 2009-2010 Assessment IX. MS Experimental Psychology Program Assessment (Student Version) ………….29

Overview, Breakdown by Program, Breakdown by Year, Open-ended Comments

X. Concerns raised in Previous Assessments (2007-2008; 2008-2009) ………………33 XI. Conclusions ………………………………………………………………………37

2

2009-2010 Annual Program Assessment MS Experimental Psychology

3

2009-2010 Annual Program Assessment for the MS Experimental Psychology Graduate Program

Final Report

Introduction and Overview The 2009-2010 academic year was spent continuing to improve the MS Experimental Psychology program assessment procedures, making important changes to the ABA specialization program, developing a new colloquium course for all MS Experimental Psychology students, moving the assessments to an on-line format, and recruiting for the 2009-2010 school year. A significant change in assessment procedures was issued on May 20, 2009. Program and departmental assessment data must be received by Dr. Tracy Pellet on or before June 15th. Every effort has been made to gather all data available for Spring 2010, however, some data was not available and was therefore not included. Moreover, it makes more sense to talk about our program in terms of the academic year because it keeps cohorts of students together. Program changes and university changes make it difficult to lump students together across academic years. Therefore, the qualitative nature of this report will refer specifically to the 2009-2010 academic year. All evaluations are based on grades or formalized assessments submitted before June 14, 2010. Averages may not reflect responses from all faculty, due to the early deadline.


4

Program Enrollment Admissions Thirteen new students were admitted to the MS Experimental Psychology program this year. Three females and ten males. Eight were accepted into the General Experimental program; five were admitted to the ABA specialization. Their stats are outlined in Table 1.

Table 1. Average GPA and GRE scores (verbal and quantitative) for 2009-2010 incoming graduate class in the MS Experimental psychology program.

N GPA GRE-Verbal GRE-Quant

2008-2009 6 3.40 487 625 2009-2010 13 3.53 472 585

The range for undergraduate GPA (last 90 credits) for the thirteen applicants was 2.88-3.98; the range for GRE-verbal scores was 350-620; the range for GRE quantitative scores was 450-720. Applicant Targets. The MS Experimenal psychology program’s minimum entrance requirements are:

1) BA/BS in psychology or a related field 2) GPA of 3.0 3) combined GRE score (verbal+quantitative) of 900.

This year, one student fell below the 3.0 GPA cutoff. This student was a CWU graduate; well known in the department; had earned a 3.89 in winter 2009; and was working to complete an honors thesis. Based on his reputation, his most recent work, and on his interest to continuing his work with his honors thesis mentor, he was admitted. Thus far he has proved to be highly successful in the program. The range for the combined GRE score was 860 – 1340. Two students’ GRE total scores fell below the 900 cutoff. Both student earned GPA scores well above the minimum. The overall average GRE combined score was 1058 – significantly above the minimum requirement. All applicants also need to have a research interest that aligns with at least one of our faculty members. This was true for all thirteen students admitted to the program. Enrollment changes During the course of the 2009-2010 school year, the status of several students’ enrollment changed. Three students announced they were dropping from the program. The program director met several times with each student to discuss their individual issues. None of the students left the program for reasons related to the curriculum, funding, staffing, or performance. All three students left for personal reasons. Two left after deciding that graduate school was not a good fit for them. The third has decided to remain active but to


5

not enroll. She will be searching for a MS program in Organizational Development, a program that is no longer available at CWU. Also during the 2009-2010 school year, two students changed their program affiliations. One student changed from the General Experimental program to the ABA program. The other student changed from the ABA program to the General Experimental program. Both students changed their specializations under the guidance of (and approval from) their academic advisors, the program coordinators and the program director. Enrollment Trends for 2009-2010 Enrollment in the MS Experimental Psychology programs increased from 5 students in 2007-2008 to 6 students in 2008-2009 to 13 students in 2009-2010. This year (2009-2010) the enrollment in the MS Experimental Psychology program more than doubled from last year, and was greater that the two previous years combined. This year (2009-2010) was the second year since the creation of the Primate Behavior and Ecology Program, which was previously expected to significantly reduce enrollments in the Psychology MS Experimental Psychology program. Only one student enrolled with an interest in primate behavior in fall 2009, but he changed his interest to social psychology after the first quarter. Hence, our reliance on primate behavior (CHCI) students has decline from more than half (3/5) in 2007-2008 to one third (2/6) in 2008-2009 to zero (0/13) in 2009-2010. This change is noteworthy in that primate behavior has historically been the primary draw to the MS Experimental Program for over 15 years. The addition of new faculty and the new ABA program have more than compensated for the elimination of CHCI as part of the psychology department. Similarly, 2009-2010 was also the second year for the Applied Behavior Analysis specialization within the MS Experimental Program. Enrollment in the ABA specialization more than double from 2 students in 2008-2009 to 5 students in 2009-2010. Furthermore, the Ms Experimental Psychology program appears to be growing as it reaches out to potential students with a more diverse faculty interest base (with research interests that include physiological psychology and neuroscience, human cognition and memory, applied behavior analysis, social psychology, and animal learning and behavior). This broadening of research interests and the development of curricula that meet national standards for licensure (in ABA and Animal Behavior) appears to be having a dramatic influence on enrollment. While we don’t expect every year to result in such a dramatic increase in enrollment, it does suggest that we are on the right track in terms of offering the kinds of programs that are desired by recent college graduates. Enrollment Targets. There was no explicit program targets for enrollment in 2009 when the admissions process was underway. Since that time, it has become our goal to admit a minimum of 1-2 students per faculty member. In 2008-2009, we had 9 regular tenure-track faculty members on staff so that would place our new targets between 9-18 students. In 2008-2009 academic year, we admitted 6 students. In 2009-2010, our enrollment jumped to 13 – well within our target goal. It seems that our recent attempts to


6

strengthen the MS Experimental Psychology program through new hires and the addition of the ABA specialization has been successful. Next year we will see the loss of one tenure-track faculty member (Dr. Ryan Zayac, ABA program coordinator), although we hope to search to replace that position in Fall 2010. Also, Dr. Elizabeth Street will rejoin the MS Experimental Psychology Program in winter 2011 which may help to offset the loss of Dr. Zayac. Her background in applied behavior analysis will help us through the transition to finding a new program coordinator who is BCBA certified. Because the ABA program is in flux, we do not anticipate growing the program immediately. Rather, we anticipate a temporary stabilization in recruitment, enrollment and retention resulting in a total enrollment in the ABA specialization somewhere between 10-15 students over the next two years. However, changes to the ABA faculty and programming should result in another period of growth sometime around 2012-2013. Recruitment for 2010-2011 This year, the MS Experimental Psychology program set target goals for recruitment and enrollment. Our goal was approximately 1-2 students per faculty member. With a reduction in staffing by 1.5 faculty members (Drs. Anthony Stahelski and Marte Fallshore), we were left with seven full-time tenure-track faculty members. Thus, our goal for admissions for 2010-2011 was for between 7-14 students. In 2009-2010, the responsibility of recruitment and admissions was given to Dr. Kara Gabriel. She worked collaboratively with the MS Experimental psychology faculty to match students with individual faculty. We implemented an acceptance policy that requires at least one full-time faculty member with a shared research interested to accept responsibility as an applicant’s academic advisor. Also, at least one other faculty member must agree to take over in the event a change of advisor is needed. Dr. Ryan Zayac, the Advisor for the Applied Behavior Analysis Specialization attended the Association of Behavior Analysis International Conference in San Antonio, Texas to promote the ABA specialization in MS experimental psychology at CWU. As a result f the combined efforts of Srs. Gabriel and Zayac, 8 new graduate students accepted positions in the MS Experimental Psychology program (3 in General Experimental and 5 in the ABA specialization). This is the first year that students were turned away in the ABA program. Enrollment in the ABA specialization is limited by the supervision requirements. Last year Dr. Zayac was our only fulltime, tenure-track Board Certified Behavior Analyst (BCAB) faculty member. Some assistance was provided by Season Almasen, M.A, and a BCBA from Children’s village (one our practicum sites). With only Dr. Zayac able to conduct supervision, a cap of 5 students was placed on enrollment. Now that Dr. Zayac is leaving, Ms. Almasen will be teaching courses in the ABA specialization as an ABD doctoral candidate and adjunct, and Dr. Elizabeth Street is beginning the process of submitting her paperwork for her BCBA. She has all the educational and supervised hours qualifications but she must sit for the Board exam. Dr. Wendy Williams has agreed to complete her ABA supervised hours under Dr. Street in


7

2011-2012. Additionally, it is anticipated that a search for Dr. Zayac’s replacement will be approved for Fall 2010. It is anticipated that the cap on enrollment in the ABA specialization will be raised for 2012-2013, after the staffing concerns have been resolved. Incoming students. Of the students slated to arrive in September 2010, three are coming from other universities (2 from out-of-state, plus one international student). and the remaining 5 are CWU graduates. Five students are joining the ABA specialization and will be working with Drs. Street and Williams; one student is coming from Ghana to do social/health psychology research with Dr. Williams. One student will be working with Dr. Gabriel; another will be working with Dr. Polage. Drs. Greenwald, Lonborg, Fallshore, and Matheson will not be advising new students in the fall but the admission’s process for the MS Experimental Psychology (General Experimental specialization) remains open throughout the year. It is not unusual for us to accept students on an ongoing basis. A comparison of enrollment trends between 2007-2008, 2008-2009 and 2009-2010 is noted in Table 2.

Table 2. Enrollments for 2007-2009 and offers made and accepted for the 2009-2010 academic year.

Academic Advisor 2007-2008 2008-2009 2009-2010 2010-

2011 K. Gabriel 1 0 2 1 R. Greenwald n/a 0 2 (-1) 0 S. Lonborg 0 0 2 0 M. Matheson 0 1 1 (-1) 0 D. Polage n/a 0 1 1 E. Street† n/a n/a n/a (5) W. Williams† 0 1 2 6 R. Zayac n/a 2 3 n/a M. Jensvold (CHCI) 3 2 0 0 S. Schepman* n/a n/a n/a n/a M. Fallshore (.5) 0 0 0 0 A. Stahelski (.5) 1 0 0 retired

*Dr. Schepman works primarily with students in the School Counseling. School Psychology and Mental Health counseling programs. † Dr. Williams will share thesis advisor duties with Dr. Street for all of the ABA specialization students from 2008-2009 to 2010-2011. Her additional students work in general experimental psychology. Note: All MS Experimental faculty may also supervise thesis projects for student in the other Psychology graduate program. Hence, not all faculty members’ thesis advisees are noted in Table 2.


8

Degrees Conferred in 2009-2010 Last year the program director, Dr. Wendy Williams, made serious overtures to Nth year students (3rd year and beyond) who had lost momentum in their programs, or taken leaves of absence. Her goal was to clarify each student’s standing and intention, as well as to attempt to find ways to help them return and complete their programs in a timely fashion. Ten students were contacted. Of those, five students have completed their degree programs. Two students completed their coursework and successfully defended their theses within the 2-year timeframe set as the goal to completion in both specializations. An additional three students are poised to defend their theses this summer. Attrition and Retention Three students (1st-Nth year) resigned from the program this year. Of the students who did not complete the program, three are poised to defend their theses this summer. The remaining students four appear to have stagnated and will approached again in the fall by the program director to see what we can do to facilitate completion of their programs. All remaining students continue to be actively working toward degree completion.


9

Assessment of Student Learning Overview

In 2007-2008, the MS Experimental (MS EXP) Psychology Graduate Program Committee met to revise the Student Learning Objectives (SLOs) and SLO Assessments. At the time, the Psychology Department had charged the MS Experimental Psychology Program Committee with reviewing and updating the SLOs and assessments in order to bring them more in line with the assessment procedures of the other graduate programs within the department. The work on the SLOs and Program goals was met with strong approval by the CWU Assessment Team. In 2007-2008, a new tracking system was developed that included individual files, survey materials, and summary data. At the time, only a preliminary report based on a narrow application of those tools was available. Last year, the tools were implemented more comprehensively and thoroughly. This year, an electronic survey system was developed in an attempt to get greater cooperation from the faculty in terms of completing the forms. The assessment of student learning for the 2009-2010 academic year is based on these new, electronic assessment tools. direct comparisons between 2008-2009 and 2009-2010 will be made whenever possible. However, it is clear that the electronic form requires minor changes to ensure that the data are gathered in ways that allow such comparisons. Review of Record Keeping Procedures Supplemental Individual Files for ALL current students in the MS Experimental Psychology Program (going back to 2004-2005) are now created for every graduate student and maintained in the Director’s office. Much of the data in these files was gleaned from the departmental records and/or graduate school records, and was then summarized within each file. (See attached Sample Tracking Form). As each student progresses through the program, copies of the course of study forms, option approval forms, IACUC and/or HSRC approvals, updated annual transcripts and other relevant documents are placed in this file. Major milestones like proposal meetings, thesis defense meetings, theses submissions and approvals are also noted at the end of the year. Individual assessment reviews by course faculty, academic advisors and thesis chairs are maintained here as well. Initial fall assessments are averages across instructors; winter and spring are then added into that average across the year. By late June we have an overall assessment for each student that encompasses one entire academic year (quarterly averages are not currently being computed but could be if an individual case warranted such fine grain analyses) This year, electronic data collection was implemented in an attempt to streamline this system, increase user friendliness, and to provide printed summaries for the students’ files. Strengths. The student files have centralized important information like names, ID numbers, course of study, major advisor, contact information. The use of the summary forms has allowed us to focus on the progress and needs of individual students. We are able to check off completed courses and note progress in a timely manner by cross-


10

referencing quarterly transcripts with their official course of study forms. This has been easily implemented because the information is available on-line to the director. Furthermore, the new assessment data have allowed us to directly address many of the issues raise by the CWU Assessment committee last year, namely assessments based on measures other than grades and academic progress. These summary forms also make it easy to contact students and to maintain a record of conversational notes as a way to make sure that students are getting what they need in terms of resources, support and guidance. Limitations. Tracking student milestones continues to be difficult. In the past, duplicate copies of the Course of Study forms were sent to Dr. Williams before being filed in the

department office. This has become less than reliable due to budget cuts and reductions in staffing. Furthermore, proposal meetings, option approval forms, defense meetings, and other types of information do not have a regular track that funnels through the Director’s

office. As such, tracking relies on self-report by advisors, or via manual review of departmental files. This system is too slow and requires repeated checks (almost on a weekly basis by the Director). We failed to develop an acceptable alternative tracking

system with the graduate school that would pass through the Program Director’s office. Hence this year, the data were gathered by hand and only at the end of the year. Some faculty did report milestones electronically, others did not. The goal for next year is to

make minor changes to the electronic assessment surveys that will make reporting easier (e.g. check boxes rather than narrative boxes). We anticipate even less departmental help

as we have lost one office manager, and the secretary’s hours have been reduced.

Number of enrolled students An overview of students who have yet to complete the MS Experimental Psychology program is shown in Table 3. It includes data from last year for comparison purposes. As you can see 9 students completed the program in 2009-2010. At the end of 2009-2010, a total of 24 students are enrolled in the program (with an addition of 13 students this year). Eight additional students will join the program in the fall bringing the total to 32.

Table 3. Comparison of the number of graduate students currently enrolled in the MS Experimental Psychology program (2009-2010) and the data from last year (2008-2009) by the year accepted into the program (current/last year)

2009-2010

2008-2009

2007-2008

2006-2007

2005-2006

2004-2005

2003-2004

13 4/6 3/5 0/2 3/4 1/3 0/1 Sixty percent of 2007-2008 first-year students had prerequisites that they need to fulfill before completing the program. 33% of 2008-2009 first-year students had prerequisites that they need to fulfill before completing the program. Only 23% of the 2009-2010 students had prerequisites that they need to fulfill before completing the program.


11

Below, in Table 4, are the numbers of students needing each specific prerequisite course. Beginning in 2010-2011, prerequisite courses changed to Psychology 300 (Research Methods) and the two statistics courses only. Psychology 301 (Learning) will only be recommended for ABA students. Psychology 461 (History and Systems in Psychology) will no longer be required.

Table 4. Number of students entering the program and prerequisite courses needed. PSY 300 PSY 301 PSY 362 PSY 363 PSY 461

2007-2008 N/A 3 0 1 2 2008-2009 N/A 2 0 1 1 2009-2010 0 0 0 2 N/A

Nth year Students. We began this year with 10 Nth year students. Five have completed all course work and thesis data collection and have defended their theses. Degrees were conferred on all of them since the last assessment. Of the remaining 5 students, one is slated to defend this summer. Three of the four remaining Nth year students have failed to make adequate progress (coursework and/or thesis work) this year; one has completed his data collection. The Program Director will be contacting these four students in the fall to see what we can do to help them complete their programs/theses. Three students decided to leave the program. One left to join the military (Emmett FL09). Another left for personal reasons (Fulton, FL09). And one has merely made her intentions to leave the program known but continues to remain officially active (Zukowski SP08) while she considers MSOD programs elsewhere. Program Target: Our program target is to graduate 100% of students who are admitted. Whenever we loose students to jobs, alternative careers or other programs, the student, his/her advisor and the director meet to discuss whether the loss is due to a selection/admission problem, a programmatic problem, or whether it is an isolated incident. In the case of the three students lost this year, the students, the advisors and director agreed that the students’ decisions to leave were restricted to the individual interests of these particular students, and were unlikely to be related to the nature of the program.


12

Major Milestones General Academic Assessment Data

Table 5. Number of students meeting departmental and graduate school benchmarks and percent of program targets met

2007-2008

2009-2010 2008-2009

Nth year students Total/Target

# of students enrolled

13

(100%) 6

(100%) 16

(100%)

35/35

(100%) # of Course of Study Approvals on file √

13/13 (100%)

5/6 (83%)

16/16 (100%)

34/35 (97%)

# of COS forms approved in 1st qtr

13/13 (100%)

5/6 (83%) unknown

18/19 (95%)

# of Option Approvals on file†

0/13 (0%)

2/6 (33%)

15/16 (94%)

17/22 (77%)

# of Thesis Defense meetings*

n/a

1/6 (33%)

6/16 (37.5%)

8/22 (36%)

# of Theses Accepted by Grad School £

n/a

0/6 (0%)

6/16 (37.5%)

6/22 (27%)

# of students who presented @ Conferences

2/13

(15%) 0/6

(0%) 0/16 (0%)

2/22 (27%)

† Two 2nd year students will complete their proposal meetings and option approval forms this summer. IACUC forms for these two students are blanket approvals at CHCI (using archival videotaped data). £ One 2nd year student has defend her theses and is waiting for the degree to be conferred. * Another 2nd year student will defend her theses on July 13th. √Note that academic advisors responsible for those students who have yet to submit their course of study forms to the graduate school have been notified and asked to help their advisees complete and submit this paperwork immediately.


13

Course of Study Forms. By the end of the 2009-2010 academic year, 97% of all students had completed, submitted and gained approval from the graduate school for their Course of Study forms. one student has not yet submitted and had her course of study accepted by the graduate school. This will be corrected before fall quarter. Program Target: Our program goal is to have course of study forms completed preferably during the fall quarter of each student’s first year, and no later than the end of the first year. For the 1st and 2nd year students, all but one student had an approved course of study by the end of their first quarter in the program. Ninety-seven percent of all students have approved course of study forms on file. Option Approval Forms. Of the 22 students who entered the program in 2008-2009 or earlier, 17 have successfully proposed a thesis and submitted their option approval forms. Of the five remaining students, four should hold proposal meetings early next fall and submit the Option Approval forms at that time. One Nth year student had continued to delay making progress since last year. The Program Director met with him last year and helped him to revise his course of study. Unfortunately, his thesis advisor has reduced her affiliation with the department (dropping to 50%). I suspect he is not getting the support he needs. I will contact him over the summer to see what I can do to help him move forward. None of the first year students managed to develop a thesis proposal in time to appoint a committee and complete the Option Approval form. Ideally, we would like to see all first year students reach this point in the thesis process by the end of their first year, but it is not unusual for them to propose in the fall of their second year. This year all of the 1st year students are expected to achieve that milestone by the end of Fall 2010. Program Target: Last fall, the MS Experimental psychology faculty agreed to set a realistic program target for proposing. Based on the current patterns of behavior, setting a program target for Option Approval Form completion by the end of fall quarter of each student’s second year seemed realistic and achievable. However, we also agreed that under ideal circumstances, proposal meetings (and option approval form submissions) would occur at the end of the 1st year. We agreed to revisit the discussion next fall after the data for this variable are in. It may be that the faculty will decide to move this target goal forward to the end of the 1st academic year. If so, we will need to discuss programmatic changes that will make such a change a realistic and achievable target. IACUC/HSRC Approvals. The goal of obtaining IACUC/HSRC approvals for thesis research were not assessed. It is difficult to reliably obtain these data when the approval letters are issued to the student directly. Although the goal remains, alternative methods of obtaining reliable, timely and accurate data on this goal will be explored directly with the IACUC and HSRC directors next year.


14

Program Target: The program goal is for students to get HSRC/IACUC approvals within one quarter of their proposal meeting. Theses accepted/ Degrees conferred. It is the goal that 100% of thesis submitted to the CWU Graduate School be accepted, and 100% of degree filings be conferred. This goal has been met. Revisions are common but relatively minor (formatting, spelling, etc.). We would also like to see the delay between thesis submission and degree confirmation reduced significantly. At present, the delay ranges between 3-9 months. This is something that must be addressed by the CWU Graduate School, as the delay is occurring there. This delay is hurting the school’s relationship with graduate alumni. Changes have been implemented at the level of the graduate school but it will take time to determine how effective these change are. Program Target: The program goal is to have 100% of thesis submissions accepted by the Graduate School with minor revisions, and for 100% of degree filings to be conferred within one quarter. Conference Presentations. At least two of our graduate students presented original research at regional conferences. Program Target. The program target for conference presentations is 100% for second year+ students. Again, this goal may seem high, but efforts to see this number grow are a program priority. Moreover, this goal is directly tied to thesis data collection. Delays in data collection can functionally delay a conference presentation by an entire year. This too will be part of the 2010-2011 MS Experimental Psychology program committee’s agenda.


15

Time to Completion Data Currently, the program director has data going back to Fall 2003 and time-to completion data for ten students. See Table 6. An additional five students are expected finish within the next 3-6 months (depending on the approve time for the graduate office). The average overall time to completion is 4.325 years.

Table 6. Time-to-Completion data for students in the MS Experimental psychology program.

Student/Advisor(s) Program Year

enrolled Year

Completed Time-to-

Completion Manjarrez (Eubanks/Street)

60-credit Gen Exp

Fall 2003 Summer 2008 5.25 years

Todd (Fallshore) 60-credit Gen Exp

Fall 2003 Spring 2008 4.75 years

Buckner (Jensvold) 60-credit Gen Exp

Fall 2003 Fall 2009 6.25 years

Tate (Stahelski) 60-credit Gen Exp


Redfield (Williams) 60-credit Gen Exp

Fall 2005 Winter 2008 2.50 years

Sherrill (Matheson) 60-credit Gen Exp


Stadner (Jensvold) 60-credit Gen Exp


Jones (Matheson) 60-credit Gen Exp


Rutledge (Gabriel) 60-credit Gen Exp

Fall 2007 Summer 2009 2.25 years

Crnich (Matheson) pending

45-credit Gen Exp

Winter 2009

Spring 2010 1.75 years


16

Thesis Defense Meetings/Time to Completion. This year seven students defended their theses. This represents only 36% of students those that should be nearing the completion of their programs- a marked improvement over last year (13%). Six of these students were Nth year students; the other completed the program in four quarters plus a summer (a noteworthy accomplishment). We expect another second-year student to defend on July 13, 2010. Program Target. Our program target is for all students to defend their thesis at the end of their second year. As you can see from the table, this is a goal that needs work on our part. Many students extend their enrollment into a third year and some even well beyond that. Because the credit load for the MS Experimental Psychology (General Experimental specialization) degree was reduced from 60 to 48 credits in 2007-2008, we believe that a 2-year target for degree completion should be possible for most students. While there may be exceptions due to thesis data collection restraints, the two-year target goal is worthwhile for these students. The program committee will address this issue in the Fall 2010 in an attempt to identify way to assist students achieve this goal. Students enrolled in the ABA program have a much higher course load requirement (68+ credits). Further, the combined goals of additional hours, extensive internships hours and thesis completion creates a difficult situation for these students in terms of time to completion. None of the ABA specialization students have finished their program so no data on time-to completion are available. Both of our 3rd year students in the program have yet to propose. The MS Experimental Psychology faculty will meet in fall 2010 to discuss this goal. It is likely that two separate goals are needed for these two programs.


17

Programmatic Course Offerings MS Experimental Psychology students are expected to maintain an overall GPA of 3.0, and receive no grade lower than a C in required core content courses listed on their Course of Study. In 2009-2010, eleven graduate courses in the MS Experimental Psychology program were offered. Nearly all students (95%) passed their classes and the overall average GPA in each of the course ranged from 3.67 to 4.0. Table 7 outlines the number of students in each class, the number and percent that passed the class, the average GPA for the class, and a breakdown by year in the program. Note that a course offering may reflect more than one section offered across multiple quarters by various instructors. Also, what appears to be courses with low enrollments are courses that are offered to students in other programs as well. Table 7. Content course offerings, percent of students passing and average GPA

Core Course

Course Offered?

# of MS Exp

Students

# of Exp

students who

Passed Percent Passed

Average GPA

1st year students passing

2nd year students passing

Nth year students passing

Psy 504 Y 11 11 100% N/A 11 0 0 Psy 525 N Psy 541 Y 2 2 100% 3.85 2 0 0 Psy 542 Y 2 1 50% 2.0 1 0 1 Psy 544 Y 1 1 100% 4.0 1 0 0

Psy 550* Y 5 4 80% 2.54 4 1 0 Psy 551 Y 12 12 100% 3.73 12 0 0 Psy 553 Y 7 7 100% 4.00 7 0 0 Psy 554 N Psy 555 Y 8 8 100% 4 8 0 0

Psy 558* Y Psy 562 N Psy 565 N Psy 576 N Psy 578 Y 13 12 92% 3.42 12 1 0 Psy 580 Y 12 11 92% 3.56 11 1 0 Psy 684 Y 3 3 100% N/A 0 3 0

*Spring 2010 sections of these courses wee taught but the grade rosters were unavailable at the time the report was due. Program Targets: All but two graduate students enrolled in MS Experimental Psychology courses in 2009-2010 have maintained GPAs higher than 3.0, nor did they have any grades lower than a C. The two students with failing grades are the ones who chose to leave the program. They both maintained a 3.0 and passed their courses until the quarter they left. Despite advising to the contrary, neither student officially withdrew from courses, so they failed.


18

Quarterly Survey of Student Progress (Fall 2009, Winter 2010, Spring 2010)

As part of the our annual program assessment, the MS Experimental Program gathers quantitative and qualitative data on 1st and 2nd year students related to the general skills, knowledge, behaviors and dispositions that characterize professionals in experimental psychology. In addition to the GPA results and the benchmark achievements noted in the Student Learning Objectives (SLOs – see attached), faculty are surveyed and asked to assess student qualitatively on a quarterly basis. Most of the qualitative assessments provide insight into the requisite skills that contribute to grades. However, several qualitative assessments address issues that contribute only indirectly to grades (if at all) such as expressive skills, appreciation for empirical evidence, potential for success within the field, and attitudes and dispositions. It is important to track each student’s progress in these areas to ensure that our program provides appropriate and timely feedback as part of their training, and makes data-driven improvements designed to address particular needs. This year, the survey was administered on-line to help facilitate responding, as well as data analyses. Faculty members in the MS Experimental Psychology program who have had experimental psychology graduate students in their graduate classes during the previous quarter are asked to complete a survey that rates students on a variety of measures. Faculty were asked to complete the survey for each student in each class they taught this year. Eighty-one surveys were returned. Academic Performance Summary. Faculty were asked to assess overall academic performance (which includes assignment completion, class participation, work effort, organization, problem solving, critical thinking skills, data-based conceptualization and analyses, and overall performance). Table 8 outlines the results of the ACADEMIC PERFORMANCE summary item. Table 8. Academic Performance Summary Table RANKING Responses Percent 5. Superior performance 33 41.8% 4. Good performance 26 32.9% 3. Adequate performance 15 19.0% 2. Less that adequate performance 3 3.8% 1. Unsatisfactory performance 2 2.5% 0. Insufficient knowledge to comment or question is irrelevant to the course

2 0%

Nearly ninety-eight percent (93.7%) of the rankings in Table 8 suggest adequate or better progress in academic performance. Nearly 75% of all rankings met the program’s criterion goal of a 4 (good performance) or better. Only 6.3% of rankings were less than adequate or worse. These low rankings were given to 5 of the first-year students. Two of them have left the program. The remaining three had apparent difficulty in only one class. All three individuals were assessed by other instructors and ranked as making adequate progress academically.


19

Cumulative GPA. Table 9 shows the Cumulative GPA rankings for both 1st and 2nd year students. Students are required to maintain a 3.0 GPA throughout their graduate careers. Note that 85% of the first year students were able to maintain the 3.0 minimum-GPA by the end of the first year; 100% of the second year students had GPAs over the 3.0-minimum at the end of two years. Note that the two first-year students (15%) with GPA scores lower that 3.0 (italicized) are the two students who chose to leave the program. However, it is important to note that both students maintained an adequate GPA until the quarter they decided to leave. Neither of them completed their courses during their final quarter, which negatively affected their overall GPA. The average GPA for the remaining first year students is 3.86. Table 9. Cumulative GPA Scores for First and Second Year Students

Cumulative GPA First Year Students

Cumulative GPA Second Year Students

2.46 3.94 1.90 3.86 3.41 3.98 3.96 3.80 3.56 4.00 3.91 3.65 4.00 3.93 3.76 3.95 4.00 4.00 4.00

Mean = 3.60 Mean = 3.87


20

Itemized Assessments Overview MS Experimental Psychology program faculty members were asked to rate students in all of their classes for the year on 10 individual performance measures related to the SLO and suggestions from the University Assessment Team, including:

A. Content knowledge mastery B. Skill mastery C. Quantitative analytical skills D. Logic-based analytical skills E. Comprehension skills F. Written expressive skills G. Verbal expressive skills H. Appreciation for empirical evidence I. The instructor’s view of the student’s ability to succeed in the program J. The instructor’s opinion of whether the student display’s an attitude or

disposition that is consistent with the field of experimental psychology Each skill is rated on a scale from 1 to 5 (1 being the lowest and 5 being the highest). Once the surveys are returned, they are compiled, and individual forms are placed in the student’s file. These ten measures are tied to our outcomes goals, and we would like our students to maintain a score of 4.0 as an overall average score for each item. Table 10 shows a summary of those data from Fall 2009-Spring 2010 for the 10 qualitative items listed above (A-J). Twenty-three students were included in the analysis and included first, second, and Nth-year students. The number of students at each ranking is listed, along with a calculated percentage of the total responses. Scores reflect assessments by class instructors in Psychology courses relevant to the students’ courses of study. The data in Table 10 represent all the responses submitted for all students. There were a total of 81 responses submitted across 6 faculty members. Rankings were provided by MS Experimental Psychology program faculty only. Courses offered outside the Psychology Department were not surveyed. As such, individual students are assessed (and represented) more than once as a result of being in multiple classes across 3 quarters.


21

Table 10. Number of students (and percent) ranked at each level on the 10 itemized assessments noted above. “Unable to Observe or Irrelevant to the Class (0)” is the alternative option. Faculty could also skip questions. Assessments scoring 0 or SKIP were deducted from the total assessments for calculation purposes. Rank Assessment Items A B C D E F G H I J 5 32

(40%) 33

(41%) 7

(31%) 36

(52%) 39

(50%) 28

(42%) 25

(33%) 33

(54%) 35

(44%) 7

(8%) 4 26

(33%) 25

(31%) 14

(64%) 22

(32%) 23

(29%) 25

(37%) 35

(46%) 25

(40%) 31

(38%) 52

(64%) 3 18

(23%) 18

(23%) 1

(4%) 8

(12%) 13

(16%) 11

(16%) 11

(14%) 3

(5%) 9

(11%) 17

(22%) 2 2

(2%) 2

(2%) 0

(0%) 1

(1%) 1

(1%) 1

(1%) 4

(5%) 1

(1%) 3

(5%) 4

(5%) 1 2

(2%) 2

(2%) 0

(0%) 2

(3%) 2

(3%) 2

(3%) 1

(1%) 0

(0%) 2

(2%) 1

(1%) 0 1

(0%) 1

(0%) 59

(0%) 12

(0%) 3

(0%) 13

(0%) 5

(0%) 19

(0%) 1

(0%) -

Skip 0 (0%)

0 (0%)

0 (0%)

0 (0%)

0 (0%)

1 (0%)

0 (0%)

0 (0%)

0 (0%)

0 (0%)

% @ Target

73% 73% 95% 84% 79% 79% 79% 93% 83% 72%

Program Target. Our program goal is for all (100%) students to maintain ratings at or above 4.0 (good performance) on each measure. Our obtained measures range between 73% - 96%. Four of the ten assessments show that 80% or more of the assessments ranked our students as having good or superior performance. Three of the remaining six items all had 79% of all assessment rankings as good performance or better. The remaining three items had over 70% of assessments falling in the target range (4.0 or higher).Only a very few individual assessments (5% or less in any cell) were ranked at the less than adequate or unsatisfactory levels. It is possible that our target goal is overly ambitious since it requires that all students perform more than adequately on all measures. (Note that adding in rank level 3 (adequate performance) would substantially raise the percentages to over 80% for all assessments. Students enroll in graduate school to learn things they do not already know. And not all students have completed the relevant course by the end of their first year. The assessments presented above are not broken down by year(s) in the program. (Also note that the two students who left the program were included in the data set. Their assessments represent several of the low ranking in each case.) Although our target is set very high, it is currently the view of the graduate faculty within the program that such high goals provide motivation to improve program delivery. Failure to meet that goal is not seen as failure. Rather it provides a rubric for assessing how we are doing at delivery the best possible program. Last year’s indicators suggested that 2nd year students outperformed the 1st year students. As such, it is unlikely that this year’s grouped data will even allow us to reach our goal of


22

100% at or above 4.0 (good performance) on each measure. Currently the data analysis system does not gather information that would allow such an analysis. However, an item (for office use only) can be added next year so that the data can be sorted by year in the program. Unfortunately, it cannot be added to completed surveys. But the correction can be made over the summer and future assessments will include those analyses. In the meantime, our primary targets for improving scores will continue to be 1st year students. Furthermore, I will be identifying the individual students who were assessed below level 3 (adequate performance) to explore ways to improve those skills. Comparison of Student Averages. Last year, the data were analyzed at the individual level and average scores per student were presented. This year the amount of data was tremendous (81 assessments x 10 items x 22 students) with a total of 193, 820 individual data points for the itemized assessment alone. Due to the short timeline, averaged data are not presented here. But it is the goal of the program director to analyze those data over the summer so that comparisons can be made with last year’s data.

Itemized Assessment Review Content knowledge (Item A) This item is linked directly to Student Learning Objective 5: Describe and contrast major theoretical and practical concepts in core content areas (see attached SLOs). Seventy-three percent of assessments of content mastery were rated at or above the target level of 4.0. This suggests that instructors of courses relevant to our students’ courses of study felt that the students are making good to superior progress on the skills addressed in those classes. Furthermore, 40% of the assessments received perfect scores (5.0) while another 33% earned good performance scores (4.0). Twenty-three percent of the assessments earned scores of 3.0. These data suggest that our most of our students demonstrate reasonable mastery of the content knowledge in their classes; still there is room for improvement. Faculty may wish to consider alternative or supplemental teaching methods that encourage content mastery. Skills mastery (Item B) This item is linked directly to Student Learning Objectives 2: Describe and perform data analyses, 3: Describe, assess and utilize research designs in psychology, 4: Design, conduct and report psychological research findings, and 7: Perform professional skills in a supervised practice setting (see attached SLOs). Seventy-three percent of assessments of skills were rated at or above the target level of 4.0. This suggests that instructors of courses relevant to our students’ courses of study felt that the students are making good to superior progress on the skills required in those classes. Furthermore, 41% of the assessments received perfect scores (5.0) while another 31% earned good performance scores (4.0). Twenty-three percent of the assessments earned scores of 3.0. These data suggest that our most of our students demonstrate


23

reasonable mastery of the skills required in their classes; still there is room for improvement. Faculty may wish to consider additional practice to encourage skill mastery. Quantitative analytical skills (Item C). This item is linked directly to Student Learning Objectives 2: Describe and perform data analyses, 3: Describe, assess and utilize research designs in psychology, and 4: Design, conduct and report psychological research findings (see attached SLOs). This item had more faculty members opting not to assess student performance than any other item. Of the 81 assessments return, 59 responded that they were unable to assess or the question was not relevant to the course they were reviewing. This implies that the faculty members are not simply marking a single score for all items for every student. It shows that they are attending to the question in a meaningful way and are only reporting assessments when they are relevant. Quantitative analyses are not conducted in every class in the program. However, many classes do deal indirectly with data and as such a conversation with the faculty is warranted regarding whether this item could be assessed in ways other than through direct data analyses. As for the assessments that were reported, 95% of assessments met the level 4.0 ranking target. Therefore, when the students are actively engaged with data analyses, they are showing excellent proficiency. (Note that by adding in the single level 3 ranking, the percentage jumps to 100%). Logic-based analytical skills (Item D). This item is linked directly to Student Learning Objectives 2: Describe and perform data analyses, 3: Describe, assess and utilize research designs in psychology, 4: Design, conduct and report psychological research findings, and 5: Describe and contrast major theoretical and practical concepts in core content areas Perform professional skills in a supervised practice setting (see attached SLOs). Eighty-four percent (84%) of the assessments achieved the level 4.0 minimum target goal. An additional 12% reached the 3.0 level. Again, we see nearly 100% of assessments (minus 3 single rankings) suggesting that our students are capable of doing adequate or better work. For those students scoring below the 3.0 rank, brief conversations with either the instructional faculty or with the students’ advisors may shed light on the difficulty and provide ideas for remediation. Comprehension (Item E) This item is linked directly to Student Learning Objectives 3: Describe, assess and utilize research designs in psychology, 4: Design, conduct and report psychological research findings, and 5: Describe and contrast major theoretical and practical concepts in core content areas of research (see attached SLOs).


24

Most graduate courses require a great deal of reading of professional literature. Hence, there is a need for advanced comprehension skills. Here too, our students did well. Seventy-nine percent of assessments earned ratings at or above our target, with 50% earning perfect ratings of 5.0. Another 29% scored in the 4.0 range. A total of 92% of all assessments fell in the adequate or better range (3.0 or higher). Again, here is an item that should show a marked difference between first year students and more experienced students. Next year’s, breakdown by year of admission to the program will shed more light on this prediction. Written Expressive Skills (Item F) This item is linked directly to Student Learning Objectives 1: Write in the language of the discipline, using the elements of style described in the APA Publication Manual. 3: Describe, assess and utilize research designs in psychology, 4: Design, conduct and report psychological research findings, and 6: Engage in scholarly and professional activities including presenting research at formal and/or informal professional events (see attached SLOs). Experimental psychologists, whether in the field of basic research or applied science, need to be articulate – particularly in writing. Written language using the APA writing format is the primary mechanism of dissemination of research findings in experimenal psychology. Seventy-nine percent of the student assessment items met the program goal of 4.0 or higher. These data suggest that written expression is being mastered by most of the graduate students across many settings. If we consider rankings of 3.0 or higher (adequate performance) the number of assessments jumps to 95%. These data suggest that most of our students could benefit greatly from writing practice and feedback across many courses. Such practice and feedback can come in various forms. This assessment item warrants discussion among the faculty to identify creative instructional and thesis-based opportunities for students to write and receive meaningful feedback. Some courses in our program might benefit from additional learner objectives that target writing. Furthermore, students scoring below the 3.0-level will be approached to discuss additional resources for assistance with writing (e.g. CWU writing center, Psych 300). Verbal Expressive Skills (Item G) This item is linked directly to Student Learning Objectives 2: Describe and perform data analyses for particular data sets; 3: Describe, assess and utilize research designs in psychology, 4: Design, conduct and report psychological research findings, 5: Describe and contrast major theoretical and practical concepts in core content areas of research and 6: Engage in scholarly and professional activities including presenting research at formal and/or informal professional events and 7: Perform professional skills in a supervised practice setting (see attached SLOs). Whether teaching a college class, presenting research findings as a conference, or interacting with the parents of a child with autism, verbal skills are key to professional


25

success. Seventy-nine percent of the assessments met the target goal of 4.0 or higher (with 33% scoring in the superior range). That number jumps to 90% if adequate performance is considered as well. It is the goal of the MS Experimental psychology faculty to give our students ample opportunities to speak publically in both formal and informal settings. Some courses in our program might benefit from additional learner objectives that target and public speaking. the lowest scoring individual may benefit from taking a communications/speech class. Such suggestions could come from instructional faculty or from academic/thesis advisors. Appreciation of empirical evidence (Item H) This item is linked directly to Student Learning Objectives 2: Describe and perform data analyses for particular data sets; 4: Design, conduct and report psychological research findings, and 5: Describe and contrast major theoretical and practical concepts in core content areas of research (see attached SLOs). It is very important to note that the program faculty instructors view our students as placing high value on empirical evidence and data. This is particularly noteworthy given the diversity of interests within the program. Not all of our students will pursue careers in fields dominated by statistical inference. Student in applied behavior analysis often rely on single-subject designs, case studies or pure description. Ninety-three percent of the assessments related to the appreciation of empirical evidence met the program criterion of 4.0 (good appreciation) or higher. These students have clearly developed their analytical skills and an appreciation for data-driven approaches that are appropriate to their own or to other sub-disciplines. The inclusion of adequate appreciation (3.0) raises the total to 98%. Ability to succeed in the experimental psychology program (Item I) This item is indirectly linked to all of the Student Learning Objectives (1-7) 1: Write in the language of the discipline, using the elements of style described in the APA Publication Manual; 2: Describe and perform data analyses for particular data sets; 3: Describe, assess and utilize research designs in psychology ; 4: Design, conduct and report psychological research findings, 5: Describe and contrast major theoretical and practical concepts in core content areas of research; 6: Engage in scholarly and professional activities including presenting research at formal and/or informal professional events; and 7: Perform professional skills in a supervised practice setting (see attached SLOs). Predicting academic success is a bit like predicting the weather. We take great pride and great credit in our successes, and we blame our failures on powers beyond our control. Yet it is something that all of us are asked to do regularly as professionals when we make guesses about undergraduate and graduate school admissions. Although it is difficult to quantify, it is something that experts within the field recognize in others. As such, we decided to ask the MS Experimental Psychology program faculty directly about whether


26

they thought the graduate students were likely to complete the program. Eight-four percent of the assessments were consistent with the program goal of 4.0 (good likelihood or better). Recall that one assessment predicted one student was unlikely to succeed in the program and that very student dropped out. The other student who dropped out was repeatedly assessed as having between adequate to superior chances of completing the program. Oh well, looks like rain or shine – you choose. In terms of testing the accuracy and validity of such assessments, we will simply have to wait to see who completes the program and who does not. Disposition consistent with the field of experimental psychology (Item J) This item is directly linked to all of the Student Learning Objectives (1-7) 1: Write in the language of the discipline, using the elements of style described in the APA Publication Manual; 2: Describe and perform data analyses for particular data sets; 3: Describe, assess and utilize research designs in psychology ; 4: Design, conduct and report psychological research findings, 5: Describe and contrast major theoretical and practical concepts in core content areas of research; 6: Engage in scholarly and professional activities including presenting research at formal and/or informal professional events; and 7: Perform professional skills in a supervised practice setting (see attached SLOs). One of the most difficult attributes to measure is a person’s disposition toward their chosen field of study. To do so, we must ask ourselves whether the person behaves and interacts in a manner consistent with the values, opinions and approaches of the field. We asked the faculty directly about whether they thought the students seemed to be developing a sense of identity with the field of experimental psychology. The responses were very consistent with our other measures (probably because items A-H are necessary for such dispositions to develop). Ninety-two percent of the assessments suggested that the program faculty believe that most of our students were adequately (3.0 or higher) demonstrated the dispositions consistent with the field. Seventy-two percent of the assessments met the program criterion of 4.0 or higher. Again it is important to note that 2 students left the program because they, themselves, felt that experimental psychology was a poor fit for them. Of the assessments that met the criterion, only 8% received the 5.0 ranking (similar to a professional in the field). The remaining 64% of were described as holding a attitude/disposition that is very consistent (4.0) with the field. It would appear that there is substantial room for improvement. Again, this item warrants discussion among the program faculty to identify opportunities to model/encourage activities and behaviors that typify professionals in the field of experimental psychology.


27

Assessing the Quarterly Survey

The quarterly survey of student progress has addressed several of the concerns raised in last year’s final report. Specifically, it allows us to more directly assess the link between student behavior, learning objectives and program goals. Until 2007-2008, we had relied exclusively on grades and program milestones (such as time to thesis completion, HSRC and IACUUC approvals, and grades). This quarterly assessment allows us to capture a more dynamic view of the development of these important skill sets. Moreover, progress can be tracked on an individual basis, and intervention by advisors or the director can target individual student needs. Last year’s survey did, however, have some limitations. Previous Limitations.

1) Return rates are not 100%. 2) Perspective is based solely on instructor point of view; the student’s point of

view of how the program is working for them is missing. 3) The current system is unwieldly and relies too much on xeroxed handouts that

can be misplaced or lost. 4) Integration with academic data (grades, progress) is not yet possible due to the

paper-based nature of all the data. There is no single database where all the information can be entered.

Results of solutions implemented.

1) An electronic version of the assessment was created and implemented. As a result, return rates were much higher. More faculty members participated, even though the number of students being evaluated increased dramatically. Hundreds of thousands of data points were gathered. There is now a rich source of information from which to draw when considering improvements to the program.

2) An electronic assessment of the MS Experimental Psychology program was developed for students to complete. Fourteen students completed the survey. The survey was greeted with enthusiasm.

3) The problem with reliance on Xerox copies has not been solved. A formal attempt was made to modify the course of study form and to then redirect the paper path from the graduate school through the program director’s office, which was rejected by the graduate school. An attempt was made to make arrangements for the secretarial staff to provide copies but it was inconsistently completed. 4) Integration of assessment results with academic data (grades, progress) continues to be a problem. However, the difficulty is now an electronic one. There is still no single database where all the information can be entered. No formal efforts were made to resolve this problem this year because the focus was on developing electronic assessments.


28

Limitations to the new 2009-2010 Assessment

1) The newly adopted timeframe for assessment activites creates barriers to a thorough analyses of the data and completion of the assessment. Accommodations must be found that will allow the process to begin earlier. If three major assessments are to be completed (Quarterly, Annual and Student), time must be available to compile and interpret the results. Consultation with Dr. Pellett is recommended.

2) Data analyses need to be refined. This will surely result in changes to the survey instrument. Some changes may preclude comparisons across academic years. This problem should begin to diminish as the survey instruments are perfected.

3) A solution to the paper trail issue is paramount. At present, we have SLOs that require the tracking of IACUC/HSRC forms, proposal and defense meetings, etc. for which there is no documentation. If these non-documented activities continue to be part of the SLOs, a method for documenting them is essential.

4) Clarification is needed to really understand the CWU Assessment Teams request for a clearer picture regarding how outcomes, methods and results fit together. Example reports will be requested.

5) Requests for assessment data to be posted on the website still needs to be discussed among the faculty. Which data? Directed at whom? The current assessment data may not reflect changes that are put in place for the subsequent year and hence, may be misleading. Solutions need to be found that will allow us to make available data that are relevant, valid and useful.


29

MS Experimental Psychology Program Assessment (Student Version)

This year, the MS Experimental Psychology students were invited to take part in an on-line survey/assessment of their graduate program (See attached copy of the blank survey). Fourteen students completed the survey. The following are the results from the five questions designed to address demographics, academic advising, thesis advising, and satisfaction with the program.

Table 11: Demographics: In which program are you enrolled?

Program Response Count Response Percent MS Exp Psych (General) 7 53.8% MS Exp Psych (ABA) 6 46.2%

Table 12: Demographics: In what year did you start your graduate program?

Year Response Count Response Percent 2004-2005 0 0.0% 2005-2006 1 7.1% 2006-2007 0 0.0% 2007-2008 0 0.0% 2008-2009 5 35.7% 2009-2010 8 57.1%

Table 13: How would you rate the academic advising (re: course, adjustment to the program, pre-thesis advising) that you have received?

Rating Response Count Response Percent Excellent 5 35.7% Good 4 28.6% Fair 4 28.6% Poor 1 7.1%

Table 14: How would you rate the thesis advising you have received?

Rating Response Count Response Percent Excellent 5 35.7% Good 5 35.7% Fair 1 7.1% Poor 2 14.3% No Thesis Advisor yet 1 7.1%


30

Table 15: How satisfied are you with the training in the MS Experimental psychology Program thus far?

Rating Response Count Response Percent Very Satisfied 2 14.3% Satisfied 4 28.6% Unsatisfied 6 42.9% Very Unsatisfied 2 14.3%

Breakdown by Program

Students in the two programs (General Experimental Psychology and Applied Behavior Analysis) responded differently to the 3 substantive questions. Generally speaking, the view of academic advising is very mixed for both groups. This issue warrants review by the Program faculty in terms of seeking methods to standardize academic advising somewhat. Table 16. Responses x Program: Academic Advising

Ranking MS Exp Psy (Gen) MS Exp Psy (ABA) Excellent 42.9% (3) 33.3% (2) Good 28.6% (2) 33.3% (2) Fair 14.3% (1) 33.3% (2) Poor 14.3% (1) 0.00% (0)

Thesis advising is less consistent across programs. The MS Exp Psy (Gen) program appears to be more satisfied with the thesis advising in general (although one student was very unhappy). It would appear that who the thesis advisor may be having a direct on this item. The MS Exp Psy (ABA) students are more satisfied in general but the range is still broad. More ABA students gave thesis advising a rating of “good” while the largest number of GEN students rated their thesis advising as “excellent”. Table 17. Responses x Program: Thesis Advising


Training Satisfaction is also spread broad across both programs. More MS Exp Psy (ABA) students have negative things to say about he program n general when compared with the MS Exp Psy (GEN) students.


31

Table 18. Responses x Program: Training Satisfaction


Breakdown by Year Responses also varied as a function of time spent in the program. Second year and Nth year students appear to be fairly satisfied with the academic advising. First year students overwhelmingly rate the academic advising as fair to good. Table 19. Responses x Year: Academic Advising

Ranking 2005-2006 2008-2009 2009-2010 Excellent 100.0% (1) 60.0% (3) 12.5% (1) Good 0.00% (0) 20.0% (1) 37.5% (3) Fair 0.00% (0) 0.00% (0) 50.0% (4) Poor 0.00% (0) 20.0% (1) 0.00% (0)

Second year and Nth year students appear to be fairly satisfied with the thesis advising. However, one of them still does not have a thesis advisor. This is concerning. Six of the eight first year students rate the academic advising as good to excellent. Table 20. Responses x Year: Thesis Advising

Ranking 2005-2006 2008-2009 2009-2010 Excellent 100.0% (1) 40.0% (2) 25.0% (2) Good 0.00% (0) 20.0% (1) 50.0% (4) Fair 0.00% (0) 0.00% (0) 12.5% (1) Poor 0.00% (0) 20.0% (1) 12.5% (1) No thesis Advisor Yet 0.00% (0) 20.0% (1) 0.00% (0)

Students from every year appear to have mixed feelings about the program. But first and second year students seem less satisfied. This may reflect the new and somewhat rocky history with the ABA program. The program will loose its director next year. This has creates angst among the first-year students and the data reflect this. The data from the second year students is less clear. Two of these students are in the ABA program so that could account for some of the low ratings. But overall, these data suggest that we need to discuss and reflect on how we can help our students to be more satisfied. We may also want to gather more detailed data next year to see what aspects of the program are causing students to feel less than satisfied.


32

Table 21. Responses x Year: Training Satisfaction

Ranking 2005-2006 2008-2009 2009-2010 Excellent 0.00% (0) 40.0% (2) 0.00% (0) Good 100.0% (1) 0.00% (0) 37.5% (3) Fair 0.00% (0) 60.0% (3) 37.5% (3) Poor 0.00% (0) 0.00% (0) 25.0% (2)

It is important to note that the number of respondents is quite small because there are few graduate students in the MS Experimental Psychology program overall. However, the data do seem to suggest that discussions among the program faculty regarding academic advising, thesis advising and training satisfaction are warranted.

Open-ended comments Students were also asked to respond to three open-ended questions. The primary purpose of these questions was to identify information that could be added to the Graduate MS Experimental Psychology Handbook. The three questions were:

1. What do new students need to know about the MS Experimental Psychology program? What advice would you give them?

2. What are the best aspects of the MS Experimental Psychology program? 3. What recommendations do you have that might help us make the MS

Experimental Psychology program better? The students provided candid, honest and thoughtful responses. (See attached Response Summary). Recommendations. 1. Address these three issues with the program faculty in the fall with the goal of

standardizing our advising through the development of uniform advising materials. 2. Consider adding clarifying questions to the survey in an attempt to identify relevant

components of advising (e.g. frequency of meetings with advisor, level of feedback provided, etc.)

3. Have the program director meet the incoming graduate student at their first meeting to clarify the role of advisors, and the flexibility available to students to change advisors.

4. Also advise students that they can come to the Program Director for advice regarding advising, or with concerns about the program.


33

Concerns raised in Previous Assessments (2007-2008; 2008-2009) The 2008-2009 CWU Assessment Team provided several points for consideration. What student outcomes are being assessed this year and why? How were they assessed?

What methods were used? Who assessed? When was it assessed?

Much of this question has been answered above. Changes have been put in place that involve quarterly and annual assessments. The assessment documents were translated into an on-line electronic data gathering system. Additional modifications have become apparent and will be implemented next year (e.g. identifiers for year in the on-line program). In reality this question more accurately addresses what isn’t included in the assessment. For example, some course grades were unavailable. Changes to the deadlines for assessment data have made the development of a complete picture difficult. Rather than exclusively relying on grades, individual assessments that are tied to the SLOs are now being made by academic advisors, thesis advisors, and course instructors. But it takes time to complete these assessments. The assessments for academic/thesis advisors was not completed in time to include in this year’s report. Next year, the Annual thesis advisor survey will be administered mid-quarter to avoid overloading the faculty at the end of Spring quarter. Finally, a new program assessment survey for graduate students was developed and implemented this year in the interest of gaining the students’ perspective. It was greeted enthusiastically by the students in the program.

Is there a way to assess students at the beginning and ending of their program to determine what was learned?

Due to the nature of Experimental Psychology, students arrive with a variety of different degrees and skills. The distinction between basic and applied research is large and will directly impact students’ training. Some students even arrive without an undergraduate degree in psychology. One student may plan to be trained to work for the US Census Bureau and need strong statistical skills, while a future applied behavior analyst needs minimal statistical and stronger single-subject design skills. As such, attempting to derive a standard set of skills for students in this broad area of graduate training would not necessarily reflect the diversity of the program. Nor would all students meet a given standard even at the end of the program. There are not national standards or norms for experimental psychology. There is no licensing entity. The Director plans to discuss this idea with the chair of the undergraduate assessment committee within psychology.


34

What general improvements in the program were implemented this year? 1. ABA specialization committee met to discuss the possibility of having the students

complete their coursework in the first year, leaving the second year for internships. This is the model used in the mental health counseling program, the school counseling program and the school psychology program. It has been successfully employed in those programs and seemed attractive or the ABA progam. with coursework out of the way, students would be free to complete internships are a wide set of agency sites – some that have BCBA professionals on staff that could supervise some of their hours. After careful review of the course requirements, credit-loads, course programming and faculty workloads, we were able to redesign the program such that the incoming class (2010-2011) will be able to complete their coursework by the end of their first academic year.

2. All MS Experimental Psychology students now agree to a background check as a way of protecting vulnerable populations with regard to internships and thesis work.

3. The Assessment process has begun to move to electronic formats that should facilitate

data collection and analyses. Also, faculty assessment surveys are being returned in higher numbers as a result of the new format.

4. A year-long colloquium class was developed to assist students with aspects of

experimental psychology that are not traditionally part of any currently taught courses, such as grant writing, conference presentation preparation, internship searches, and professional discourse about one’s own research.

5. All faculty are more involved in recruitment and admissions as a result of the new

policies put into place (2 faculty members committed seeing every student through the process).

6. A new program assessment survey for graduate students was developed and

implemented this year in the interest of gaining the students’ perspective. Are there discipline specific rubrics or standards that could be used to evaluate student progress? Experimental psychology does not have a governing agency that sets standards or

establishes certification criteria. As such, students in the General Experimental Psychology specialization do not have a set of standardized goals or outcomes set by the APA or any other national board. Applied behavior analysts are required to take a Board Examination before being granted their certification. At this time, board certification is not offered through CWU. Currently, we provide only the course work and the practicum hours required for certification. Students must arrange to take the certification exam elsewhere. However, it is our long-term goal to have the CWU psychology department’s ABA specialization board certified. At that point, we will


35

be able to offer the exam as part of our program, which will allow us to keep track of those data. (Currently, none of our students have progressed far enough to take the exam.) The timeframe for program certification is between 5-8 years. Until then, we do not have a nationally standardize measure of student progress beyond successful course and program completion.

Have you considered admission interviews, standardized advising, portfolios?

As yet we have not begun to formally interview candidates for the MS Experimental Psychology Program. Dr. Gabriel did encourage faculty to contact applicants with overlapping interests. But a formal interview process has not been established. This year, we have had some issues with professionalism and maturity among some of the new students. As a result, a section on professionalism is being added to the MS Experimental Psychology handbook. It also appears that a discussion regarding formal interviews would be useful. It will be part of the agenda for Fall 2010. Finally, interviews may be particularly necessary for the ABA specialization since interpersonal skills are mandatory for the career. It seems clear form the student versions of the assessment survey that a serious discussion about advising among the program faculty is warranted. It will be high on our list of agenda items for fall 2010. The department of psychology has only recently implemented the use of portfolios for the undergraduate students. Formal portfolios are rare in graduate school but not unheard of. This too is worth a discussion by the faculty.

Have we considered developing a rubric for thesis defense?

This is on our list for discussion next year. If approved, a subcommittee will develop a draft rubric that will be reviewed/modified by the entire program committee. The program committee will bring the finalized draft version to the department chair for feedback and final approval.

Have your considered making your assessment data available on-line? Given the radically changing nature of the program at this time, making such data

available would not be meaningful to program applicants. We would prefer to wait until 2012-2013 before placing our assessment data on-line so that we have time to gather sufficient longitudinal data as to be meaningful, and not confusing.

Do you have a MS Experimental Psychology Handbook? In 2008-2009, Dr. Williams develop a 60+ page handbook (with inserts) for new

incoming graduate students in the MS Experimental Psychology program. It includes important information regarding the faculty, program objectives, prerequisites,


36

internships, malpractice and liability insurance, the handling of complaints, and thesis and non-thesis regulations and guidelines. As part of this project, she also had the program committee discuss important issues like mandatory criminal background checks and fingerprinting for any graduate students that plan to conduct research or work directly with children or adults with developmental disabilities. A copy of the handbook was supplied to the assessment team last year.

Next year, handbook inserts will be developed for the ABA specialization, and an advising handbook will be need to be developed for Internship Supervision (to be used by faculty and practicum/internship supervisors). But this might be best held until Dr. Zayac’s replacement has been hired. This year, we have had some issues with professionalism and maturity among some of the new students. As a result, a short section on professionalism in the workplace is being added to the MS Experimental Psychology handbook.

Have we made comparisons across programs?

In 2009-2010, the four graduate program directors and program coordinators met to ensure that our programming would accommodate courses that overlap across programs in ways that maximized efficiency. There are several classes that are required in multiple programs. We wanted to make sure that course offerings would be offered in ways that meet several criteria:

1. Graduate classes have sufficient enrollment (6 students minimum) 2. Graduate classes are not over-enrolled in ways that are detrimental to the

content. 3. Graduate classes are offered at programmatically appropriate times for each

cohort. Mutually beneficial solutions were agreed upon and course programming was based

on our recommendations. Beyond course offerings, we have not attempted to compare the MS Experimental Psychology assessment data with that of other programs within the CWU College of the Sciences, within the Psychology Department, nor with similar Experimental Psychology programs at other universities. The CWU Psychology Department does conduct a self-study every three years with an outside reviewer. Currently, it is unclear what it would cost to establish a similar program for our graduate program alone. Next year, Dr. Williams hopes to meet with the Directors of Mental Health Counseling, School Counseling and School Psychology to see how they conduct their assessments.


37

Conclusions Accomplishments. Overall, the assessment data suggest that as a program we are doing well, and our new assessments provide a deeper and more thorough view of how our students are progressing through the MS Experimental Psychology Program. Changes in recruiting styles have led (in part) to a substantive increase in enrollments over the last 2 years. Many current first and second year students are progressing nicely toward timely completion of their degrees. The reduction in credits for the MS General Experimental degree program may help to facilitate this. Several Nth year students returned to the program and completed their degrees. Between 5 and 10 students should have their degrees conferred by this time next year. Communication and tracking of student progress by the Director continues to improve but still requires changes to the way in which paperwork is tracked. Of particular interest is the need to better track IACUC/HSRC documentation. Another important change has been the improved connection between assessment documents and the student learning objectives for the program. The development of a MS Experimental Psychology Graduate Handbook assisted new incoming students in adjusting to the MS Experimental Psychology program and to CWU in general. New regulations for working with human and animal subjects were established including requiring all incoming student participate in a background check (and mandatory fingerprinting for those planning to work with sensitive populations). The creation of Psychology 505 Colloquium should help facilitate improved progression through the thesis process; introduce new topics like grant writing, and encourage conference participation. The modification of the ABA specialization to allow all course work to be completed in the 1st year, followed by supervised internships in distant locations should help to increase the number of BCBA-certified supervisors without having to hire new faculty. It will also allow students to focus intensely on their practicum skills without having to divide their attention between internship and coursework. This change will also allow students to gather thesis data at their internship sites that should improve time-to-completion somewhat. Finally, the ABA specialization faculty are planning to begin a conversation about migrating the ABA specialization into an independent master’s degree program in Applied Behavior Analysis. The evidence is clear that the program has great potential for growth, but that potential is limited by the time-to-completion. Competing programs Concerns. Concerns to be addressed in the future include faculty compliance with assessment measures, thesis review delays at the graduate office, and the mediation of student problems and/or concerns in a timely fashion. Plans for addressing and/or ameliorating these concerns are in place for next year. Additionally, the Director would like to meet with the CWU Assessment Team to discuss inter-program exchanges where successful and creative assessment ideas could be shared.


38

Closing Comments As noted last year, it continues to be necessary to allow the new and updated programs to become well-established, and to give new tracking and assessment systems sufficient time to glean enough information to make relevant comparisons across academic years. To see the effects of current and future modifications to the programs, it will be necessary to allow students to progress far enough into their programs for meaningful data to be gathered. Similarly, the program faculty also face with changes to old routines; it takes time to make new advising and assessment instruments salient, meaningful and important. We are beginning to see the fruits of our efforts on these fronts. We have also begun to see data-driven, programmatic changes within the MS Experimental Psychology degree program that result directly from these ongoing assessment activities. Of course, our best data will come when we can look across multiple years to see how well the changes that we implement produce meaningful change for the program and for our graduate students, for our faculty and for the MS Experimental Psychology Program. Respectfully submitted, Wendy A. Williams, PhD. MS Experimental Psychology Program Director Department of Psychology Central Washington University


39

Appendix A. Summary of issues to be addressed by the MS Experimental Psychology program committee and others in 2010-2011.

1. Thesis Rubric Development 2. How to get students to attend/present at conferences 3. Course of Study form completion in Fall / standardized advising 4. Thesis proposal and Option Approval forms completed by end of first year? 5. Dialog with IACUC and HSRC to improve student progress 6. A handbook for mentoring new graduate faculty (Academic and Thesis Advising) 7. Assessment completion compliance by faculty 8. Benchmark reporting by faculty (discuss with grad school) 9. Formalized internships for both specializations. 10. Revise Program Webpage 11. Time-to-completion: How can we get it closer to 2 years? 12. Inter-program assessment conversations within the department 13. Inter-program assessment exchanges across the College of the Sciences 14. Development of an ABA Handbook Insert for the new MS Exp Psy Graduate

handbook.


40

Appendix B. Student Learning Outcomes for the M.S. Experimental Psychology Program established in 2007-2008.

1. Write in the language of the discipline, using the elements of style described in the Publication Manual of the American Psychological Association.

2. Describe and perform data analyses for particular data sets such as traditional descriptive statistical analyses, inferential statistical analyses, sequential analyses, single-subject designs and/or multivariate analyses.

3. Describe, assess and utilize common research designs in psychology.

4. Design, conduct, and report psychological research.

5. Describe and contrast major theoretical and practical concepts in core content areas of research including behavior analysis, cognitive psychology, comparative psychology, and/or physiological psychology.

6. Engage in scholarly and professional activities, including presenting research at formal and/or informal professional events.

7. Performing professional skills in a supervised practice setting such as research teams, internships, and graduate research courses.

Documents

2009-2010 Annual Program Assessment for the MS Experimental … · 2018-05-13 · 2009-2010 . Annual Program Assessment for the . MS Experimental Psychology Graduate Program . Final