View
218
Download
0
Category
Tags:
Preview:
Citation preview
The STARS AllianceStudents & Technology in Academics, Research, and Service
STARS Alliance Evaluation
Tiffany Barnes
Evaluation Team
Kim Buch, Audrey Rorrer
Web: Anthony Chow
Qualitative & PP: Sally Berenson
The STARS AllianceStudents & Technology in Academics, Research, and Service 2Barnes, Eval Team
NSF Site Visit 2007
Evaluation• Successes:
– Refined approach to evaluate Organizational, program, individual level outcomes
– Instrument development– Combined qualitative and quantitative measures for:
• SLC• Alliance effectiveness
– Articulated new goals and measures (STARS/Computing Identity)
• Challenges:– Changing team– New research questions arise all the time– Comprehensive program has many goals - too many surveys for same
people– Data collection from different people/institutions– Small team
The STARS AllianceStudents & Technology in Academics, Research, and Service 3Barnes, Eval Team
NSF Site Visit 2007
Impact on pipeline
• Q1: The STARS Alliance features a broad scope (middle school to graduate school) across three categories of population (gender, ethnic, disability), with a variety of outreach, research, mentoring, and career counseling components. Characterize, quantitatively if possible, what has been attempted within this scope of activity and what has been achieved to date. To what extent have each of the target groups been reached? Each of the pipeline stages?
• Approximately 110 college SLC students each year– About 40% returning in Year 2
• Over 500 K-12 students at outreach events• Over 500 parents• Over 100 teachers and counselors• > 5 major community events (AU, FSU, UNCC)
The STARS AllianceStudents & Technology in Academics, Research, and Service 4Barnes, Eval Team
NSF Site Visit 2007
Measures
• Q3: What new measures of impact/outcome have been developed to capture what the Alliance is seeking to achieve, i.e., beyond traditional pre-post test measures appropriately disaggregated (by gender, ethnicity, disability, and cohort) such as GPA, year-to-year retention, and degree attainment? For example, how have the observations in students' electronic journals been used by the Alliance institutions and Advisory Boards?
• Selected interviews each semester• Journaling currently not being analyzed• Climate survey to measure impact of STARS on
computing culture at schools• New outreach evaluation instruments
The STARS AllianceStudents & Technology in Academics, Research, and Service 5Barnes, Eval Team
NSF Site Visit 2007
Comparison
• Q7: What data is the Alliance using for comparison purposes? Are they national sources, local institutional information on students, some combination, or other? Explain.
• Comparison of applicants, enrollments across years
• Comparison with NSF and Taulbee survey data
The STARS AllianceStudents & Technology in Academics, Research, and Service 6Barnes, Eval Team
NSF Site Visit 2007
Celebration impacts
• Q4: Describe the benefits of the summer workshop to different categories of participant.
• Faculty- networking, project collaborations, information exchange, e.g.
– o Junior Faculty Round Table– o Implementing Pair Programming
2007 I tem 100% My expectations of the conference were met. 100% I w as satisfied with the conference overall.
100% I have developed helpful professional collaborations through attending the conference.
100% I believe that the celebration had a positive impact on individual student who attended.
90% I think that the conference prepared students to conduct successful SLC projects.
90% The STARS Celebration inspired me to become more involved in BPC efforts.
The STARS AllianceStudents & Technology in Academics, Research, and Service 7Barnes, Eval Team
NSF Site Visit 2007
STARS Celebrations
The STARS Celebration emphasizes STARS core values of excellence, leadership, civic engagement, service, and community – values intended to foster student success.
•2007 at UNC Charlotte– 113 students– Over 40 faculty– Over 50 partners and community
•2006 at Georgia Tech– 105 students– 23 faculty/staff
The STARS AllianceStudents & Technology in Academics, Research, and Service 8Barnes, Eval Team
NSF Site Visit 2007
STARS Celebration Results
• Students felt the Celebration provided opportunities for community building. (2006: 97% , 2007: 92% )
Students learned how computing professionals can improve quality of life. (2006: 83%, 2007: 89%)
Students learned about disparity of representation in IT: by gender and race (2006: 98%, 2007: 88%) by persons with disabilities (2006: 86%, 2007: 69%)
Students had ample time to select leadership assignments to fit their professional needs (2006: 80%, 2007: 88%)
The STARS AllianceStudents & Technology in Academics, Research, and Service 9Barnes, Eval Team
NSF Site Visit 2007
Demonstration Projects
The STARS Alliance serves as an
incubator for new demonstration projects and the
scaling and replicating of best practices among
the diverse alliance
institutions.
STARS Leadership Corps
Pair Programming
Tiered mentoring model
Teaching Math to the Visually Impaired
African American Researchers in Computer Science
Culturally Situated Design Tools
The STARS AllianceStudents & Technology in Academics, Research, and Service 10Barnes, Eval Team
NSF Site Visit 2007
STARS Leadership Corps Model
The STARS AllianceStudents & Technology in Academics, Research, and Service 11Barnes, Eval Team
NSF Site Visit 2007
STARS Tiered Mentoring and Role Models
High school student
Middle school student
Professional or Grad student
Undergrad student
Regional Partnerships -A computing
community from Kindergarten to the Workforce
The STARS AllianceStudents & Technology in Academics, Research, and Service 12Barnes, Eval Team
NSF Site Visit 2007
Evaluation
• Three-Tiered Outcomes Model:– Organizational Outcomes (e.g., Alliance)– Program Level Outcomes (e.g., Demonstration
Projects)– Individual Level Outcomes (e.g., Students)
• Evaluation model: CIPP– Context (setting of project)– Input (project resources)– Process (activities & strategies)– Product (ultimate results)
The STARS AllianceStudents & Technology in Academics, Research, and Service 13Barnes, Eval Team
NSF Site Visit 2007
Alliance Outcomes
• Increased student enrollment in computing and IT programs
• Increased student awareness about computing and IT• Increased student readiness to enter computing and IT
graduate school and workforce• Increased participation of undergraduates entering
computing and/or IT graduate school or workforce• Increased graduate rates of computing and IT students• Increased persistence and declaration of students
majoring in computing or IT• Increased college adjustment and GPA for students in
computing or IT
The STARS AllianceStudents & Technology in Academics, Research, and Service 14Barnes, Eval Team
NSF Site Visit 2007
Alliance Outcomes
• Sustained Alliance Efficacy• Institutionalize Alliance Partnerships• Increased national awareness of effective practices for
Alliance development• Serve as a model and repository for BPC
The STARS AllianceStudents & Technology in Academics, Research, and Service 15Barnes, Eval Team
NSF Site Visit 2007
Year 1 Accomplishments
• Steering Committee:– Jan. 2006-2007, & Aug. 2006-2007– Regular teleconferences for planning & update
• STARS Celebration– Successful Annual Conference held in August 06-07– 2006: 105 students, 23 faculty, 2007: 113 students, >40 faculty– 2007: New evening for partners, >50 additional attendees
• Student Leadership Corps– 2 universities implemented in Summer 2006– All implemented in Fall 2006, Spring 2007, Fall 2007
• Workshops– Pair Programming, Mentoring, IBPC: Jan & Aug 2007– CSDT at Celebration Aug 2007– Qualitative methods workshops Aug 06-07, April 2007
The STARS AllianceStudents & Technology in Academics, Research, and Service 16Barnes, Eval Team
NSF Site Visit 2007
Accomplishments
• Web portal: StarsAlliance.org• Evaluation
– Instrument development– IRB exploration: applying for waivers for K-12
outreach evaluation– Interviews & Training
• Coordination with UNC Charlotte REU Site• Faculty recruiting at UNC Charlotte!
The STARS AllianceStudents & Technology in Academics, Research, and Service
Results
The STARS AllianceStudents & Technology in Academics, Research, and Service 18Barnes, Eval Team
NSF Site Visit 2007
Institutional Enrollment
The STARS AllianceStudents & Technology in Academics, Research, and Service 19Barnes, Eval Team
NSF Site Visit 2007
SLC ’06-’07 Participants
SLC Participants
Females Males Total
African American 20% 23% 43%
Asian 7% 7%
Caucasian 29% 19% 48%
Total 58% 42% 100%
The STARS AllianceStudents & Technology in Academics, Research, and Service 20Barnes, Eval Team
NSF Site Visit 2007
2006-2007 SLC
107 students participated in 2006-2007 STARS Leadership Corps
K-12 Outreach18%
Peer Ambassadors
13%
Campus Organization
17%Community
Service7%
Research25%
Internships3%
Peer Mentors17%
The STARS AllianceStudents & Technology in Academics, Research, and Service 21Barnes, Eval Team
NSF Site Visit 2007
SLC Survey Results
• Retention of SLC students: Over 95% are forming meaningful peer & faculty relationships 88% more committed to computing majors 94% interested in graduate education 83% felt the SLC helped them develop computing skills 91% would recommend the SLC to others
Recruiting: 90% are helping others understand the value of computing Hundreds of K-12 students have participated in outreach events
Building a “like” community:88% felt the SLC gave them opportunities to work with people like themselves
The STARS AllianceStudents & Technology in Academics, Research, and Service 22Barnes, Eval Team
NSF Site Visit 2007
Who is SLC?
• NOT the “average” UG student in Computing– Older, average age= 24 years– Higher GPA in Computing, M=3.3– Higher SES
• 85% “middle class”• 64% had parents with at least some college; 53% were college graduates
– Much more diverse• Women 51%• Under-represented minorities 62%• Physical disability 14%
– Extremely confident that they will obtain UG degree (4.7 on 5-pt scale)– Very likely to attend graduate school
• 66% believe they will obtain Masters• 50% believe they will obtain PhD
The STARS AllianceStudents & Technology in Academics, Research, and Service 23Barnes, Eval Team
NSF Site Visit 2007
SLC & Computing
• 93% are confident that Computing is the right major• 87% plan to stay in the field of Computing long-term• Have a very strong “Computing Identity” (M=4.4/5)
– “Feel I fit in field of Computing” M=4.2– “Expect to be treated fairly in Computing career” M=4.1– “Can relate to the people around me in my classes” M=4.3– “Committed to applying Computing to benefit society” M=4.5– “Committed to promoting Computing to others” M=4.3
• Have a very strong “Computing Efficacy” (M=4.3/5)– “Feel someone like me can succeed in a Computing career” M=4.6– “I can complete the programming requirements for Computing majors” M=4.5– “I can complete the math requirements for Computing majors” M=4.6– “I can complete the science requirements for Computing majors” M=4.6– “I can complete my current Computing degree” M=4.6
The STARS AllianceStudents & Technology in Academics, Research, and Service 24Barnes, Eval Team
NSF Site Visit 2007
SLC & Computing departments?
• 64% at least weekly contact with Computing faculty• 77% satisfied+ with the degree of interaction with faculty• 67% satisfied+ with the availability of extra-curricular
activities, eg, ACM/IEEE• 73% satisfied+ with “welcoming” atmosphere in the
department• 47% satisfied+ with availability of
“cutting-edge”/innovative courses• 70% feel satisfied+ that “I am part of the department”• 73% overall satisfied+ with the department
The STARS AllianceStudents & Technology in Academics, Research, and Service 25Barnes, Eval Team
NSF Site Visit 2007
SLC & Diversity in Computing
• 66% satisfied+ w/ the diversity of dept. students and faculty• 64% agree that dept. students are sensitive to minority
issues• 77% agree that dept. faculty are sensitive to minority issues• 71% agree that dept. students are sensitive to women’s
issues• 78% agree that dept. faculty are sensitive to minority issues• 69% agree that dept. students are sensitive to
disability/accessibility issues• 77% agree that dept. faculty are sensitive to
disability/accessibility issues
The STARS AllianceStudents & Technology in Academics, Research, and Service 26Barnes, Eval Team
NSF Site Visit 2007
SLC Concerns• Computing faculty contact per semester
– 6% reported NO contact– 5% reported they only had ONE contact– 12% reported they had one contact per month
• Students rated their high school technology/ computing program as:– 36% Not good– 23% Somewhat good– 14% Very good, 14% Excellent
• Students felt that their depts were NOT sensitive to:– 36% minority issues– 30% women’s issues– 31% disabilities/access issues
The STARS AllianceStudents & Technology in Academics, Research, and Service 27Barnes, Eval Team
NSF Site Visit 2007
SLC quote
• “The STARS Alliance has engaged me to use computing to make a positive impact by exposure to other areas of technology and to be of service to other students and community members. It has provided them (middle school students) access to computing resources. It has allowed them to see women and minorities as role models.”
The STARS AllianceStudents & Technology in Academics, Research, and Service 28Barnes, Eval Team
NSF Site Visit 2007
SLC Interviews
• Emergent themes toward careers are altruism, creativity, and mentorship.
• Initial early interest in computing: game playing that requires problem solving, along with a motivation to be self-sufficient.
• Overall, students stated that a family member or mentor had been influential in their decision to pursue a career in computing.
• The overwhelming majority of students indicated that their reason for persistence in computing is due to a commitment to giving back to their communities and an excitement about the creativity inherent in computing careers.
The STARS AllianceStudents & Technology in Academics, Research, and Service 29Barnes, Eval Team
NSF Site Visit 2007
Celebration Results
• 97% of students felt the workshop provided adequate opportunities for community building.
• 83% agreed with the statement that the SLC Workshop emphasized how computing and IT professionals can use their skills to improve quality of life
The STARS AllianceStudents & Technology in Academics, Research, and Service 30Barnes, Eval Team
NSF Site Visit 2007
Alliance structure
• Collaborations are beneficial to both students and faculty
• Participation facilitates faculty and student career development
• Alliance goals are being achieved and represent meaningful impact to the computing field.
• Training, dissemination, and sustainability goals are being met
The STARS AllianceStudents & Technology in Academics, Research, and Service 31Barnes, Eval Team
NSF Site Visit 2007
Presentations
• Grace Hopper Celebration, Oct 2006• SIGCSE presentation to computing K-12
teachers, funded by Microsoft• Presentation at NIU, August 2007• Panel at GHC 2007• Panel & Birds of a Feather at Tapia 2007• Paper at Frontiers in Education 2007
The STARS AllianceStudents & Technology in Academics, Research, and Service
Successes & Challenges
The STARS AllianceStudents & Technology in Academics, Research, and Service 33Barnes, Eval Team
NSF Site Visit 2007
Alliance infrastructure
• Time is major issue for both faculty & students• Need tools to facilitate:
– Collaboration– Communication– Management– Timelines & deadlines– Tracking– Reporting– Sharing resources
The STARS AllianceStudents & Technology in Academics, Research, and Service 34Barnes, Eval Team
NSF Site Visit 2007
Cross-institutional collaboration
• Faculty see this as very successful
• Students excited to meet students from other places
• Recruiting across Alliance!
• Looking forward to building more
The STARS AllianceStudents & Technology in Academics, Research, and Service 35Barnes, Eval Team
NSF Site Visit 2007
Synergy with existing programs
• Essential to success
• Examples:– Charlotte REU & CRA-DMP Summer 2006– Mentoring on NCSU campus– Intel Opportunities at Georgia Tech
• Share resources and minimize work
The STARS AllianceStudents & Technology in Academics, Research, and Service 36Barnes, Eval Team
NSF Site Visit 2007
Recruiting students
• A challenge at first: What is STARS?
• Now: More students want to join!
• Creating ways for non-funded students to join– STARS Alliance club– Course credit, other avenues
• Creating ways to recognize participation
The STARS AllianceStudents & Technology in Academics, Research, and Service 37Barnes, Eval Team
NSF Site Visit 2007
Replication of best practices
• A Challenge!• How do you make professors do something
new?• You must show how it benefits them• Need a cookbook approach
The STARS AllianceStudents & Technology in Academics, Research, and Service 38Barnes, Eval Team
NSF Site Visit 2007
Sustaining changes
• Student STARS Alliance club• STARS course added at UNC Charlotte• Faculty recruiting at UNC Charlotte:
– 2 new black faculty, 2 new women
• Existence of Alliance is raising awareness:– Deans and departments are using STARS Leaders
for recruiting and involving them in dept activities and policy-making, marketing
The STARS AllianceStudents & Technology in Academics, Research, and Service 39Barnes, Eval Team
NSF Site Visit 2007
Effects of Alliance on faculty careers
• Writing circles– Planned for Year2
• Mentoring within the Alliance– happening informally
• Collaborations– Landmark College, REU Site proposal, Gaming
programs at UNCC, Auburn
STARSAlliance.orgStudents & Technology in Academia, Research, and Service
PI: Teresa Dahlberg tdahlber@uncc.eduCo-PI: Tiffany Barnes tbarnes2@uncc.edu
Evaluation TeamKim BuchAudrey RorrerSarah BerensonAnthony ChowJennifer Thomas
Recommended