Embed Size (px)
Citation preview
Tiger Woods Learning Center: Using Data to Inform Program Planning and Improvement
University of California Irvine
December 2010
Femi Vance Deborah Lowe Vandell
2
TABLE OF CONTENTS
Introduction................................................................................................................................................................3 Study Design and Evaluation Research Goals..............................................................................................4 Study Findings...........................................................................................................................................................6
Attendance Findings.................................................................................................................................6 Parent Survey Findings.........................................................................................................................10 Youth Survey Findings..........................................................................................................................17
Conclusions and Recommendations..............................................................................................................24 References.................................................................................................................................................................25 Appendix A................................................................................................................................................................26 Appendix B................................................................................................................................................................33
3
INTRODUCTION
Researchers at the Department of Education at the University of California Irvine have provided ongoing research and evaluation of the Tiger Woods Learning since 2007. In an initial two-‐year study of program implementation and student outcomes, Vandell, Warschauer, O’Cadiz, & Hall, (2008) determined that youth who attended the Center for 30 or more days during the academic year benefited relative to youth who attended the program for fewer days in several ways, including developing supportive relationships with adults and peers, practicing leadership skills in service learning projects, and developing academic skills. Following a recommendation based on the two-‐year implementation and outcome evaluation, TWLC upgraded its attendance data system. The new system, TraxSolutions, has the capacity to measure daily program attendance for individual students at the Center and in particular classes, but implementation of the attendance data system presented challenges. TWLC turned to UCI for technical assistance in developing its internal capacity to collect high quality attendance data on an on-‐going basis. In addition, TWLC asked UCI to provide background research and evaluation to support the Center’s development of strategies to boost overall attendance in the Career Exploration Program, the STEM focused after-‐school program for adolescents. From the outset, it was determined that efforts to increase students’ daily attendance to 30 or more days would be facilitated by assessments of the program from key stakeholders, namely youth and parents. This report describes the attendance and program assessment findings with the goal of providing practical implications for the use of these data for program planning and improvement.
4
STUDY DESIGN AND EVALUATION RESEARCH GOALS Research Goals The purpose of the current study is to strengthen ongoing program planning and improvement at the Tiger Woods Learning Center (TWLC) through the use of evaluation data. The specific project goals were developed in concert with the TWLC leadership to reflect the priority on increasing youths’ daily attendance in the Career Exploration Program (CEP) to 30 or more days. The specific research goals for this project were to:
• Assist the TWLC in the development of a robust attendance monitoring system • Analyze attendance data to identify specific patterns and characteristics of high
attending youth in the Career Exploration Program • Assess parent and student perceptions of the program
Goal One
The first goal for the current project was to assist TWLC to implement a reliable attendance monitoring system. For TWLC, the value of developing an attendance monitoring system lies in the ability to understand current attendance levels in relation to its goal of increasing the number of students that attend 30 days or more and in tracking the effectiveness of future strategies to increase attendance. The investment in TraxSolutions was a necessary first step to develop an attendance monitoring system. The UCI Research team with the assistance of TWLC staff customized the use of TraxSolutions to the TWLC program structure. The attendance monitoring procedures are detailed in an accompanying document, TWLC Attendance Data Collection Procedures Manual. This manual outlines data management guidelines, the ongoing attendance monitoring process, and procedures for collecting youth feedback with TraxSolutions. The manual, submitted to TWLC in October, 2010, should increase the quality of the attendance data. In addition, the UCI research team trained TWLC staff to implement the attendance-‐tracking procedures outlined in the manual.
Goals Two and Three The data sources for Goals Two and Three consisted of attendance records, parent surveys, and student surveys. Data were predominantly gathered using TraxSolutions. Relying on TraxSolutions as the primary data source demonstrates the potential of the management information system (MIS) for ongoing program planning and improvement. Attendance Records Using TraxSolutions, TWLC collects two types of attendance data: 1) Center Attendance which keeps track of youth when they enter the Center and; 2) Course attendance which documents youth attendance in their enrolled classes. These data are gathered using
5
electronic scanners located at the front door of the facility and in each classroom or program area. Each student is given a membership card with a unique identifier that can be scanned upon entry and exit to the Center and each class within the program. The current study reports center attendance during the program years 2008 – 09 and 2009 – 10, coupled with the attendance data findings from the initial two-‐year evaluation and outcomes study to identify long-‐term attendance trends. Surveys Youth Surveys Youth surveys were collected using TraxSolutions. At the end of each term, participating youth were asked to complete an online survey about their experience in the course. The surveys were developed in partnership with TWLC instructors to insure that the survey data would be relevant to their practice. Survey questions were designed to assess youth’s interest in the topic, reasons for attending the course, student learning, and features of the course that youth enjoyed or thought needed to be improved. This report uses youth survey data collected in the Fall and Spring terms of the 2009 – 2010 program year. Youth surveys from the Winter term were not available1. A sample youth survey can be seen in Appendix A. Parent Exit Surveys At the end of each term in program year 2009 -‐10, parents were sent an exit survey. The surveys were distributed with study information sheets informing parents that their participation was voluntary. Completed exit surveys were either mailed to the UCI research team or returned to TWLC in a sealed envelope. The exit survey assessed parent perceptions of TWLC program features, their child’s experiences in the program, the perceived impact of program participation, and overall satisfaction with the program. Survey items were measured on a five-‐point scale that ranged from “Strongly Disagree” to “Strongly Agree”. The exit survey also included space for parents to write in additional comments about what they particularly enjoyed about the program or to identify areas for improvement. The parent exist survey is included in Appendix A. Parent Demographic Surveys Parent demographic surveys were sent at the same time as the exit survey. The one page demographic survey asked parents to report their household size, education, work status, income, and access to computers and the Internet. A copy of the parent demographic survey is located in Appendix A. 1 In the 2009 – 2010 program year the Winter term was a series of one-‐week sessions compared to the seven week courses delivered in the Fall and Spring terms. Due to the short course length youth surveys were not conducted in the Winter.
6
STUDY FINDINGS
The findings are reported in three sections. The first section summarizes the attendance findings. The second section presents the findings from the parent surveys. The third section presents the findings from the youth surveys.
Section 1. Attendance Findings The attendance monitoring system is documented in the TWLC Attendance Data Collection Procedures Manual. This report focuses on understanding the current attendance patterns of TWLC youth. Attendance Overview In the current study, TWLC youth could attend the CEP program at five different levels:
• None • Low: below 17 days of attendance • Average: 17 days – 29 days of attendance • High: 30 days – 59 days of attendance • Very High: 60 or more days of attendance
Center Attendance 2008 – 2010 The center attendance data demonstrate that the attendance at TWLC has been very stable over the last two program years. In 2008, 65 percent of the youth attended at a low level and 14 percent achieved a high or very high level of attendance. These percentages were similar in the 2009 program year, which was 67 and 12 percent, respectively (See Figure 1).
5%
67%
16%
9% 3%
Attendance Level of TWLC Members Program Year 2009 -‐ 2010
None Low Average High Very High
7
Figure 1. Attendance Levels in 2008-‐09 and 2009-‐10 Comparing the 2008 – 2010 students’ daily attendance data to similar data gathered as part of the evaluation and outcome study provides a look at long-‐term attendance trends at TWLC. For students who attended the Center at least one day during the program year the average attendance was approximately 17 days for the last four years. This would be considered an average level of attendance. Table 1. Average Organizational Attendance from 2006 – 2010
Number of Youth
Minimum Maximum Mean Standard Deviation
Year One (2006 -‐ 2007) 1,037 1 121 17.03 18.50
Year Two (2007 – 2008) 573 1 94 17.57 17.05
Year Three (2008 – 2009) 830 1 175 17.63 19.53
Year Four (2009 – 2010) 953 1 156 16.02 17.34
The number of terms that a youth attends the Center has direct implications for his/her attendance level. An in-‐depth analysis of term enrollment in 2009 – 2010 illustrates this relationship (See Table 2). More than half of TWLC students are attending only one term and they are attending at a low level. The youth who are attending at a high level enroll in three or more terms.
9%
65%
12%
10%
4% Attendance Level of TWLC Members
Program Year 2008 -‐ 2009
None
Low
Average
High
Very High
8
Table 2. Average Organizational Attendance by Number of Terms Attended 2009 -‐ 2010
Total Terms Average Days Number of Youth Percent of Youth
One Term 12.50 532 53%
Two Terms 28.53 137 14%
Three Terms 40.34 29 3%
Four Terms 71.83 24 2%
Organizational Usersa 6.40 284 28%
a Organizational users are members that are eligible to come to the Center but do not have to sign up for a class because they participated in a previous term. Differences in Youth Characteristics and Attendance Levels Statistical tests were conducted to identify differences in the center attendance by youth demographic characteristics. The demographic data available to conduct comparisons were gender, race, and school level (middle or high school). Although, it is common to also explore differences in attendance by family income TWLC does not routinely collect this information, making these comparisons impossible. As shown in Figure 2, high school youth were more likely than middle school youth to attend at high or very high levels of attendance. In 2009, the percent of high school youth who attended at high or very high levels was almost twice that of middle school youth. This is also true in 2008 in which 19 percent of high school students had high or very high attendance compared to 9 percent of middle school youth. This finding is particularly striking because attendance research (Grossman et al., 2002; Mahoney, Vandell, Simpkins, & Zarret, 2009) indicates that as youth get older their attendance declines in youth programs.
9
Figure 2. High Attending Youth by School Level in 2009 -‐ 2010 No differences in attendance were observed by gender or race. Additional analysis of class enrollment data pointed to the three classes that enrolled the largest proportion of high attending youth. These include Universal Science (30 percent), Short Game (25 percent) and Engineering (25 percent). Organizational Changes that Might Yield High and Very High Attendance Levels In accordance with the findings from the initial two-‐year UCI evaluation, the target attendance level for TWLC youth is high, i.e. 30 or more days. The TWLC program structure influences the ways in which youth can attain a high level of attendance. In the CEP program there are 14 class days that a student can attend during each 7-‐week term. In addition, the student can also attend the Center on days that their course is not meeting. To attend at a high level a teen can adopt one of three attendance patterns: 1) enroll in one course for a minimum of three terms; 2) enroll in multiple courses per term for a minimum of two terms and; 3) enroll in one or two courses and attend TWLC on days that the class does not meet. Considering that adolescents have other constraints on their time such as family responsibilities and school-‐based clubs and sports, one of the most promising strategies to increase attendance at TWLC may be to enroll youth in more than one term and encouraging youth to attend the Center on days that courses do not meet. If a student enrolled in one course and attended just one non-‐course day each week of the term, they would reach a high level of attendance in two terms.
9%
16%
0%
5%
10%
15%
20%
25%
Middle School High School
Perc
ent
School Level
Percent of High Attending Youth by School Level
Program Year 2009 - 2010
10
Section 2. Parent Survey Findings Characteristics of TWLC Youth and Families A total of 126 parents returned the exit survey and/or demographic survey. The demographic and attendance data for youth whose parents returned a survey was compared to that of youth whose parents did not return the survey. It was determined that no significant differences existed between these youth on gender, race, or school level. There was, however, a difference based on attendance. Youth whose parents returned a survey attended an average of 28 days while youth whose parent did not return a survey attended an average of 13 days during the program year. Additional data gathered in the demographic survey tells us more about youth’s families who returned the survey. Seventy percent of these youth live in two-‐parent household and 89 percent include households in which at least one parent is employed. On average parents reported having 11 years of education or the equivalent of a high school junior. More than half of TWLC families earn less than $45,000 a year. Compared to the two-‐year evaluation and implementation findings more parents with lower incomes returned the survey in the 2009 -‐ 10 program year. TWLC youth seem to have access to computers and the Internet at home with 93 percent of parents reporting having at least one computer in the home and some form of internet connection.
Figure 3. Income Distribution of TWLC Families in 2009 – 2010 Parent Perceptions of TWLC The findings for parent perceptions are organized into five areas. These include parent perceptions of:
• TWLC features • Staff quality • Youth experiences • Perceived youth outcomes • Overall parent satisfaction.
16%
23%
23%
17%
8%
13%
0% 5% 10% 15% 20% 25% 30% 35%
<15 Thousand
15 -‐ 29 Thousand
30 -‐ 44 Thousand
45 -‐ 59 Thousand
60 -‐ 74 Thousand
75+ Thousand
Income Level of TWLC Families 2009 -‐ 2010
11
TWLC Program Features Parents were asked to report their perceptions of TWLC program features, including the location of the Learning Center, the safety of their children while at the Center, the variety of activities, and if the program environment promoted friendships among students. Nearly all parents agreed that they were satisfied with these features of the TWLC.
Figure 4. Parent Satisfaction with TWLC Program Features 2009 -‐ 2010 Interestingly, while 89 percent of parents agreed that the TWLC was in a convenient location the additional comments communicated praise and dissatisfaction with transportation to the Learning Center. The comments below illustrate this tension. In response to the prompt, “The best things about the TWLC are....”
“the many options for classes, the trained staff, it is free, and best of all they pick up and drop off my child free of charge. Thanks so much.”
In response to the prompt, “I recommend the following to improve the TWLC....”
“Very difficult to get kids to the center. Bus comes 1 hr after school ends so they do not have time to get homework done, not conducive to working parents”
When parents expressed concerns about transportation they centered around the lengthy wait time for students, how the extended wait time cut into the time designated for students to complete their school work, and expanding transportation to additional schools and/or neighborhoods.
89% 97% 99% 94% 87%
0% 20% 40% 60% 80% 100%
Location Safety Variety Quality Environ.
Percent Agree
TWLC Program Features
Parent Perceptions of TWLC Features 2009-‐2010
12
Staff Quality Parents’ perceptions of staff quality included assessment of three staff characteristics: 1) Positive relationships with students 2) If staff were fair and respectful when enforcing rules and; 3) Staff communication with parents about their child.
Figure 5. Parent Satisfaction with Staff Quality 2009 – 2010 Parents reported high levels of satisfaction with staff and youth relationships and with staff’s ability to enforce rules while remaining respectful and fair. The parent surveys point to an area for improvement for staff. Only 44 percent of parents surveyed agreed that staff communicated with them about how their child was doing. In additional comments parents mentioned the type of communication that they would like to receive from staff. In response to the prompt, “I recommend the following to improve the TWLC....”
“I would like to receive at the end of each session some info about the kids development and behavior in class.” –Translated from Spanish “Perhaps at the end of the 7 week class we could get a sort of report, informal would be fine. Perhaps a grade of satisfactory and a teacher comment would be great.” “It would be nice for a note or progress report to be sent home from the instructors so we can see what they think of our children behavior and their learning progress. Email would be good too.”
The parent reports of youth experiences captured parents’ perceptions of their child’s enjoyment while at the TWLC and their child’s access to and enjoyment of computers and new technology. Ninety-‐four percent of the Parents agreed that their child enjoyed their time at the Center and 84 percent reported that their children enjoyed the use of computers and technology. In addition, parents felt that their child had access to technology. One area for improvement is parents’ perception of their child’s access to new software program.
85% 82%
44%
0% 20% 40% 60% 80% 100%
Positive Relations Enforce Rules Update Parents
Percent Agree
Staff Quality
Parent Perceptions of Staff Quality 2009-‐2010
13
Unfortunately, additional comments from parents do not shed light on their child’s experiences with new software or programs that they would like to see their children exposed to.
Figure 6. Parent Satisfaction with Youth Experiences 2009 -‐ 2010 Perceived Youth Outcomes Parents were asked how they thought program participation had impacted their child on a range of developmental outcomes. Overwhelmingly, parents felt that their children had gained new knowledge from participating at TWLC with 95 percent of parent’s reporting that their child had learned a lot at the Center. Additional comments from parents suggest that the new knowledge gained by their children was related to the content of the TWLC classes and 21st Century skills such as social skills, public speaking, technological savvy, and leadership style. In response to the prompt, “The best things about the TWLC are....”
“The excellent learning environment that gives students the opportunity to learn and explore science and technology beyond public schools curriculum.” “Golf programs free time before classes....has peaked an interest in my son. He has been given an opportunity to explore the sport which we would not otherwise been able to give him.” “The amount of opportunities that they offer the students they learn real leadership skills.” “In school they say that my son is not sociable and in the Center he is learning to be sociable.” –Translated from Spanish
94% 84%
62% 80%
0%
20%
40%
60%
80%
100%
Child Enjoys Likes Tech. Software Access Tech.
Percent Agree
Youth Experiences
Parent Perceptions of Youth Experiences 2009-‐2010
14
“Students benefit from the technology at the TWLC by using what they learn in the sessions when they are required to give presentation in their school.”
Furthermore, 86 percent of parents reported that TWLC had made their children more aware of future career options. This is a clear indication that parents value the college and career focus at TWLC.
*S. is an abbreviation for school. Figure 7. Parent Perceptions of the Impact of Program Participation 2009 -‐ 2010 Nearly two thirds of the parents agreed that coming to TWLC had improved their child’s performance. In additional comments parents complemented the academic support that TWLC offers. In response to the prompt, “The best things about the TWLC are....”
“The help my son receives in the center on any school subject where he is behind. That the program is free and that they learn different things.” –Translated from Spanish “That when my daughter was needing help with her homework they were always available to help her.” –Translated from Spanish “That they encourage the kids to keep studying.”-‐ Translated from Spanish
Overall Satisfaction In general Parents were very satisfied with the TWLC. Nearly all parents agreed that they hoped that their child continued to attend the Learning Center and that they would recommend the program to other parents.
95%
63% 73% 71% 65%
81% 86%
0%
20%
40%
60%
80%
100%
Learn Behavior S.Conoidence S. Attitude S. Pform. College Career
Percent Agree
Youth Outcomes
Parent Perceptions of Youth Outcomes 2009 -‐ 2010
15
Figure 8. Overall Parent Satisfaction 2009 – 2010 At the same time, parents did offer some suggestions for improving the program. In some instances, even these comments were evidence of their satisfaction with the Center. For example, some parents recommended improving access to TWLC programming by opening the CEP program up to elementary school students, expanding existing program offerings to Saturdays or evenings during the Summer, and opening more Learning Centers. In addition, parents expressed that they wanted their child to have more time in class and a few parents were particularly dissatisfied with the one-‐week courses because they felt it was too short of a time period for their child to learn the course content. It is apparent that at the center of these suggestions is parents’ desire to have greater access to what TWLC is offering. Some parents also suggested that TWLC add to the existing course offerings. Not all parents were clear about the types of additional courses they would like to see added to the curriculum. Those that were suggested including more courses on the Arts (drama, music etc.), sports other than golf, computer software, and classes specifically geared toward high school students. These suggestions should be tempered against the praise that other parents gave TWLC for the variety in the course listing.
98% 97% 98%
0% 20% 40% 60% 80% 100%
Overall Continue Prog. Recomm. Prog.
Percent Agree
Overall Parent Satisfaction
Overall Parent Satisfaction 2009-‐2010
16
Parent Voices: The Best Things about TWLC
1. No cost! I could not afford a High-‐quality program if they were charging going rate 2.Transportation to TWLC is provided (great for working parents) 3. Career oriented programs that are highly engaging and stimulating 4. Academic support 5. Emphasis on good character and healthy choices 6.scholarships opportunities
The staff, the curriculum, and the environment. I am very humbled that we are a part of the TWLC family. We are very grateful for all that the TWLC has given to our child & family. We appreciate it very much. The best thing about the TWLC is that "it is" and it is available for my child to benefit from greatly at the cost of only $5…it is one of the blessings and privileges I have ever been able to take advantage of...... When my daughter looks forward to the day of [the] week that she gets to go, she gets really excited about [the] new things she learns. I would not change anything because we love the way it is, the building is clean, bathrooms are clean :) people that work there are so nice, helpful, the snacks are great plus healthy the activities are fun and (WE) actually learn from them. The variety and quality of the activities offered and above all the improvement in my son’s self esteem before he had an EIP. -‐Translated from Spanish My child is in a safe place makes new friends and learns important skills that are going to help him in his education. My husband and I are able to work while my son is in TWLC. Special projects for the students such as youth's leadership, special outings such as ducks playoff game, hallowing party, golf incentives, such as balls, hats, clubs, hand on learn by doing projects. Special projects such as TWLC/Warner Bros. movie and field trips. The best things about the TWLC are the friendly and helpful staff, interesting classes, free transportation, and beneficial learning experiences to all people who attend. Love the college-‐like environment. Like all the information given to students about college scholarships, etc.
17
Section 3. Youth Survey Findings
A total of 568 youth completed the surveys at the end of each term. Youth respondents were primarily middle school students and more boys completed the survey than girls.
Figure 9. School Level and Gender Distribution of Youth Survey Respondents 2009 – 2010 TWLC participants completed their surveys in their respective classes. The percent of surveys that were contributed by each class that was offered by TWLC in 2009 -‐2010 can be seen below. Engineering students completed the most surveys, however, this was also the class that was offered the most. Where appropriate, findings are presented by class to facilitate comparisons across courses.
Figure 10. Survey Distribution by TWLC Course 2009 – 2010
74%
47%
0% 20% 40% 60% 80% 100%
Middle School High School
Percent
School Level
School Level of Surveyed Youth 2009 -‐ 2010
58% 42%
0%
20%
40%
60%
80%
100%
Male Female
Percent
Gender
Gender of Surveyed Youth 2009 -‐ 2010
1% 6%
2% 0%
3% 1%
15% 3%
8% 8%
7% 3%
6% 7%
2% 6% 6%
8% 10%
0% 5% 10% 15% 20% Adv. Engineering
Aerospace Aerospace Rocketry
Business Video Production
Digital Manuf. Engineering
Forensic Science Full Swing
Golf Academy Communications
Hip Hop Nutrition & Fitness Performing Arts NXT Robotics Short Game
TAP Universal Science
Video Game Design
Percent of Surveys Completed by Course 2009-‐2010
18
The youth survey findings are organized into three topics: • Youths’ reasons for enrolling in the course • Youths’ perceptions of what they learned in their courses • TWLC features youth would like to take advantage of in the future
Youths’ Enrollment Decisions Youth were asked to report the reasons that they chose to enroll in their class. The most frequently cited reason for enrolling in a class was interest in the topic. Class content was reported far more than peers, which one would suspect would be very attractive to adolescents. Youth also reported a continued interest in the content of their courses. More than 75 percent of TWLC participants indicated that they were interested in taking another class in the same content area. It seems that a portion of TWLC did just that. Fourteen percent of youth were returning students who wanted to learn more about the topic. TWLC instructors were the second most cited reason for enrolling in a course.
Figure 11. Factors Considered in TWLC Youths’ Enrollment Decisions TWLC Student Learning TWLC instructors developed two or three learning goals that were specific to each class to describe what they would like students to gain from their course. In addition, four additional learning goals, coined the Universal Goals, were developed to reflect the mission of the Learning Center. The Universal Goals are:
• Youth will learn to work in groups. • Youth will learn about careers. • Youth will understand the assets that they have • Youth will learn new technology
52%
17%
11%
3% 10% 7%
Youth's Reasons for Enrollment 2009-‐2010
Interested in Class Like the Teacher Taken Before and Want to Learn More Taken Before and Want to Finish Project Friends in Class Class Wanted Was Full
19
The Universal Goals and the class specific learning goals were used to assess student learning. Youth were presented with a learning goal statement and asked how true this statement was for them on a four-‐point scale from “Not at all true” to “Really True”. Survey results reveal that students clearly felt that they learned how to work in groups, were exposed to career options and became aware of the assets that they possess. Comparatively, students were less certain that they had learned about new technology. One factor that may help explain student reports about new technology is that 93 percent of TWLC youth have access to a computer and Internet in their homes. It may be that youth have previous knowledge about technology from using it at home. Also, little is known about the technology accessible to TWLC youth during the school day.
Figure 12. Youth Perceptions of TWLC Universal Learning Goals 2009 – 2010 The Universal Goals were also examined by TWLC course to understand how each class contributed to the overall goals of the Learning Center. A snapshot of the percent of youth who agree that they learned about new technology by course reveals that the classes that youth may not be expected to use extensive technology are likely driving the score on this learning goal. Examples of such courses include golf courses, Hip Hop, Performing Arts and TAP2.
2 TAP is a community service course that has since been renamed TWLC Help Your Community but is the acronym is still used.
81% 71% 81% 57%
0% 20% 40% 60% 80% 100%
Learned to Work in Groups
Learned About Careers
Understand Assets
Learned about Technology
Percent True
Universal Goals
Youth Perceptions of TWLC Universal Goals 2009 -‐ 2010
20
Figure 13. Youth Perceptions of the Use of Technology by TWLC Course 2009 – 2010 Additional graphs that present the Universal Goals by TWLC course can be seen in Appendix B. The interpretation of these graphs will depend on the standards set by TWLC for achieving the Universal Goals. The class specific learning goals were also explored. An average was created for class specific learning goals using the four-‐point scale from “Not at all true” to “Really true”. To do this for a TWLC course with two learning goals, the score for each learning goal was added together and then the total was divided by two. If the course had three learning goals the total was divide by three. The averages for the class specific learning goals are presented in Table 3. The average for the class specific learning goals was approximately 3 for all courses. This suggests that, on average, youth feel that they mastered the course content.
57% 59%
80%
100% 69% 88%
81% 73%
50% 63%
41% 44%
76% 44%
37% 33%
61% 69%
76%
0% 20% 40% 60% 80% 100%
Adv. Engineering Aerospace
Aerospace Rocketry Business
Communications Digital Manufacturing
Engineering Forensic Science
Full Swing Golf Academy
Hip Hop Nutrition & Fitness
NXT Robotics Performing Arts
Short Game TAP
Universal Science Video Game Design Video Production
Percent True
Universal Goal: Learn New Technology by Course 2009 -‐ 2010
21
Table 3. Average of Class Specific Learning Goals by 2009 -‐ 2010 courses Course Average Learning Goal Score Range
Adv. Engineering 3.14 1.5 – 4
Aerospace 2.89 1 – 4
Aerospace Rocketry 2.94 1.3 – 4
Business 4 4 – 4
Communications 2.96 1 – 4
Digital Manufacturing 3.19 2 – 4
Engineering 3.01 1 – 4
Forensic Science 2.97 2 – 4
Full Swing 3.30 1 – 4
Golf Academy 3.44 1.5 – 4
Hip Hop 2.96 1.5 – 4
Nutrition & Fitness 3.50 1 – 4
NXT Robotics 3.46 2 – 4
Performing Arts 3.39 1 – 4
Short Game 3.25 1.5 – 4
TAP 3.60 2 – 4
Universal Science 3.12 1 – 4
Video Game Design 3.23 1 -‐ 4
Video Production 3.26 2 – 4
TWLC Course Features TWLC youth were asked to report the features of the course that they enjoyed and the aspects of the course that they felt could be improved. The students could choose more than one course strength and improvement. The survey results indicate that youth think that making the lessons more engaging and strengthening behavior management in classrooms would improve the TWLC courses. It should be noted that the number of students that reported course improvements was far fewer than those that reported course strengths. A weakness of this survey item is that it did not allow students to report that they did not feel that the course could be improved. Perhaps if students did not feel that the course could be improved they chose to skip this question. This may help explain the drop in the number of youth who answered this survey
22
item. This question was revised in the 2010 – 2011 youth surveys to be a short answer item where youth are not confined to predetermined survey options.
Figure 14. Course Improvements for TWLC Courses 2009 – 2010 In comparison, the survey results for the TWLC course strengths do not point to any particular course features, instead, the course strengths seem almost equally appealing to students. For example, almost two-‐thirds of the respondents cited hands-‐on projects and meeting new people as strengths of TWLC courses.
Figure 15. TWLC Course Strengths 2009 – 2010
34%
15%
16%
10%
10%
18%
34%
22%
0% 10% 20% 30% 40%
Lessons Were Boring
Too Much Writing/Note Taking
Not Enough Projects
Class Too Long
Too Much Work
Didn't Like Questions after Lessons
Peers Disrupted Class
Need More Time to Learn about Topic
Percent of Youth
Youth Perceptions of TWLC Course Improvements 2009 -‐ 2010
67% 59% 68% 62% 60%
0% 20% 40% 60% 80% 100%
Hands on Projects
Working with Others
Meeting New People
Learning New Skills
The Teacher
Percent of Youth
Class Strengths
Youth Perceptions of TWLC Course Strengths 2009-‐2010
23
Future TWLC Experiences Students were also asked which TWLC programs they intended to take advantage of in the future. According to youth, the top three reasons to return to the Center are to take another class, (29 percent), go to the Computer Clubhouse (18 percent) and finally to practice in Open Golf (15 percent).
Figure 16. TWLC Programs that Encourage Continued Participation 2009 -‐ 2010
29%
18%
13%
12%
13%
15%
Youths' Reasons for Returning to TWLC 2009-‐2010
Another Class Computer Clubhouse
Academic Support Golf Clinic
College Info/Access Workshops Open Golf
24
CONCLUSIONS AND RECOMMENDATIONS
The findings in this report represent the types of valuable program insights that can be garnered from continuous program evaluation. TWLC’s investment in a robust attendance monitoring system has resulted in a better understanding of the attendance patterns of TWLC students and youth perceptions of the program. Both of which can be used to inform program delivery. The attendance data indicate that the level of individual students’ participation in the Learning Center has been fairly stable since it opened its doors four years ago. Increasing the daily attendance level of students is a process that began with the development of the monitoring system. The next step is to identify the factors that act as barriers to participation. The feedback from low attending youth and their families will be essential to understanding what those barriers are and will also help TWLC develop strategies to break down these barriers. The survey data indicate that parents and youth are very satisfied with TWLC. The parent surveys describe the views of families whose children attend the center more than other TWLC students. The perceptions of other parents whose children attend the center less frequently are still unclear. To provide a more complete picture of parent perceptions of the program the Learning Center should think creatively about how to reach these parents. Some options to consider are getting youth involved in soliciting parent perceptions or offering incentives for parents to complete surveys. The parent surveys included in this report were collected via mail. An alternate delivery method such as email may also increase parent participation especially given that most TWLC families have access to the internet in their homes. Youth surveys showed that students have a strong interest in the program content offered at the Learning Center. Many youth indicated that they would be interested in taking another course in the same content area. TWLC may consider offering course sequences that guide novice students to become more advanced in a specific content area. Offering courses in this manner aligns with student’s intent to take another TWLC course and may also help increase attendance since youth will have to enroll in more terms to continue learning about their desired topic. The youth survey results also revealed that students are mastering the course content. For program improvement purposes, it would be helpful for the Learning Center to establish standards for the class specific and universal learning goals. With a standard in place the Center would know when to and where to direct program improvement efforts.
25
REFERECENCES Feister, L. M., Simpkins, S. D., & Bouffard, S. M. (2005). Present and accounted for:
Measuring attendance in out-‐of-‐school-‐time programs. New Directions for Youth Development, 105, 91-‐105.
Grossman, J., Price, M., Fellerath, V., Jucovy, L., Kotloff, L., Raley, R. & Walker, K. (2002).
Multiple choices after school: Findings from the extended-‐service schools initiative (No. 48). Philadelphia: MDRC.
Mahoney, J., Vandell, D., Simpkins, S., & Zarrett, N. (2009). Adolescent Out-‐Of-‐School
Activities. In Handbook of Adolescent Psychology: Contextual Influences on Adolescent Development (3rd ed., Vol. 2, pp. 228 – 269). Hoboken, New Jersey: John Wiley and Sons, Inc.
26
APPENDIX A: SURVEYS
Youth Survey
27
28
29
30
Parent Exit Survey
31
32
Parent Demographic Survey
33
APPENDIX B: UNIVERSAL GOALS BY TWLC COURSE
86% 86%
40% 100%
72% 100%
81% 100%
89% 98%
86% 94%
81% 100%
72% 97%
85% 66%
94%
0% 20% 40% 60% 80% 100%
Adv. Engineering Aerospace
Aerospace Rocketry Business
Communications Digital Manuf. Engineering
Forensic Science Full Swing
Golf Academy Hip Hop
Nutrition and Fitness NXT Robotics
Performing Arts Short Game
TAP Universal Science
Video Game Design Video Production
Percent True
Universal Goal: Work in Groups by TWLC Course 2009 -‐ 2010
34
71% 79%
64%
100% 69%
75% 70% 73%
77% 83%
67% 59%
50% 78%
69% 80%
73% 78%
82%
0% 20% 40% 60% 80% 100%
Advanced Engineering Aerospace
Aerospace Rocketry Business
Communications Digital Manuf. Engineering
Forensic Science Full Swing
Golf Academy Hip Hop
Nutrition & Fitness NXT Robotics
Performing Arts Short Game
TAP Universal Science
Video Game Design Video Production
Percent True
Universal Goal: Learn about Careers by Course 2009 -‐ 2010
35
71% 72%
91% 100%
81% 88%
82% 100%
89% 85%
80% 79% 82%
100% 72%
90% 78%
72% 77%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Advanced Engineering Aerospace
Aerospace Rocketry Business
Communications Digital Manufacturing
Engineering Forensic Science
Full Swing Golf Academy
Hip Hop Nutrition and Fitness
NXT Robotics Performing Arts
Short Game TAP
Universal Science Video Game Design Video Production
Percent True
Universal Goal: Understanding Assets by Course 2009 -‐ 2010
