Rising to the Occasion: Systematic STEM Academy Assessment Practices

Virginia Department of EducationOffice of Career and Technical Education Services

October 25, 2012

“Using Data for Program Planning and Continuous Improvement”

Rising to the Occasion: Systematic STEM Academy Assessment Practices

Nathan Carter, Ph.D.Research Consultant

Background Information

• In 2008, the Virginia Board of Education approved the criteria for establishing Governor’s STEM Academies.

• As part of the criteria, the Virginia Department of Education (VDOE) was required to conduct annual evaluations of all established STEM Academies.

• In order to evaluate STEM Academies effectively VDOE needed to:• Acquire current descriptions of each career pathway offered at a

STEM Academy

• Develop an informative template that STEM Academies could use to complete an Annual STEM Academy Program Assessment Report

Typical Problems with Program Assessments

Pressure for doing a comprehensive program assessment can cause a “rush to completion syndrome” and produce results that are not very useful.

When there is not a team-oriented approach to completing a comprehensive program assessment important voices can be left out of the discussion.

In some situations comprehensive program assessment is not based on a clear understanding of core program values or agreement on what the short and long-term goals of the program are.

Another Common Problem: DATA OVERLOAD!!!!!

If I’d known they wanted me to use all of this quantitative data – I would NEVER have asked for it!!!

Four General Purposes of Quality Comprehensive Program Assessment

To improve – The assessment process should provide feedback to stakeholders on how a school program can be improved.

To inform – The assessment process should inform decision makers and program stakeholders of the continual contributions and impacts of a program on meeting short and long-term goals.

To prove – The assessment process should demonstrate what the program is accomplishing and how these accomplishments are changing overtime.

To support- The assessment process should support faculty, administrators, and others as they think about what actions need to be taken to continue to support students.

Requirements for a Complete Assessment Process It must be comprehensive – must cover the full

range of learning outcomes that are connected to program values and goals

It must include multiple judgments – multiple sources of evidence must be used

It must include multiple dimensions – different facets of student performance must be included

It must collect direct evidence – direct measures of student attainment must be used

It must include qualitative interpretations of the data and evaluations of patterns overtime.

Iterative Model of the STEM Academy Annual Assessment Process

Plan

CollectData

Impl

emen

tA

nalyze

Data

Reflec

t

& Rev

ise

STEM Academy Career Pathway Description Sheet

Grade Level

CoreCourse Name

StateCourse Code

Content AreaCareer Pathway Course Options

(indicate all that apply for each course)

English Math ScienceHistory &

SocialScience

CTE Other

CourseRequirement

For CTE Program

Completion

CourseRequirement

ForAdvanced

StudiesDiploma

CourseRequirement

ForAdvancedTechnicalDiploma

Industry Certificatio

nAvailable

Course Available To Be Taken As

A Dual Enrollment

Course

Non-Traditional

CareerPreparation

Course(M or F)

Career Pathway Name___________________________

Workplace Experiences Offered With Career Pathway

Job Shadowing Internship

Mentorship Cooperative Education

Student Apprenticeship Service Learning

Other

Sections of the STEM Academy Assessment Report

SECTION DATA ELEMENTS INCLUDED

Background InformationName of STEM Academy, year STEM Academy opened, STEM Academy director’s name & contact information, STEM Academy’s purpose, STEM Academy’s current short & long-term goals.

STEM Academy Acceptance Rates

Number of applications received, number of applications accepted, number of students placed on a waitlist

STEM Academy Experiences

Number of elementary or middle school students in STEM Academy summer activities, number of elementary or middle school students in STEM Academy classes during the school year

STEM AcademyHigh School Enrollment

Total number of high school students enrolled, newly enrolled, and number who were expected to return but did not (by grade level and by career pathway)

Each section has a quantitative and qualitative data collection requirement

Sections of the STEM Academy Assessment Report (cont.)

SECTION DATA ELEMENTS INCLUDED

STEM Academy AP Course & AP Test Participation

Number of STEM Academy students enrolled in AP courses, number enrolled in AP courses and taking AP tests, number taking AP tests without being enrolled in AP courses.

STEM Academy Dual Enrollment Participation

Number of STEM Academy students taking 1, 2, or 3+ dual enrollment courses, number of students passing 1, 2, or 3+ dual enrollment courses

STEM Academy SOL Achievements

Number of STEM Academy students scoring Advanced Proficient, Proficient, or Not Proficient on the different SOL exams

STEM AcademyCredential Production

Number of STEM Academy graduates/completers taking and passing Workplace Readiness Skills Assessments, or Occupational Assessments, State Licensure Exams, or National Industry Certification Exams

Sections of the STEM Academy Assessment Report (cont.)

SECTION DATA ELEMENT INCLUDED

STEM AcademyGraduation Data

Number of STEM Academy graduates/completers earning Board approved diplomas with a special emphasis on whether STEM Academy graduates earn advanced studies diplomas and/or CTE credentials upon graduation

STEM AcademyPost-graduation Data

Number of STEM Academy graduates/completers who enrolled in credit-bearing courses, or earned college degrees, or transitioned into high-skill, high-wage, high-demand careers, etc.

Summary of STEM Academy Progress

1. STEM Academies compile their qualitative and quantitative data and assess to what extent they are meeting their short and long-term goals.

2. STEM Academies identify their strengths as well as what areas need attention.

3. STEM Academies adjust their short and long-term goals where needed.

VDOE provided each STEM Academy with a PDF document that showed which tables and available data in the template could be populated with state-collected data. rt.

STEM Academies will organize their own data

management team and set deadlines for populating

tables in the STEM Academy Assessment Report using

locally stored data sources.

STEM Academies will compare the

data tables provided by the

VDOE with their own data tables on the different areas

team.

STEM Academies will create a list of

all the relevant data inaccuracies between the two

sources and document possible

reasons for the differences.

In the case of discrepancies ,

STEM Academies will use their own

locally validated data to populate the tables

in the Assessment Report and complete

their annual evaluation.

START

END

STEM Academies will submit their

Annual Assessment Report and a list of any discrepancies

uncovered during the data validation

process to VDOE along with

descriptions of career pathways

Data Validation Flow Chart

Important Questions To Think About As You Move Forward

How are you currently embracing a data-driven decision-making process that leads to actionable

results?

Consider including people from various stakeholder groups to work colloboratively on the comprehensive program assessment plan.

Who is going to be a part of the program assessment team?

When will we administer the assessment?When will we collect the data?When will we analyze the data?When will we reflect upon the data?When will we make recommendations?When will we make the decisions about the recommendations?When will we provide written documentation about the decisions?When will we share the data with other stakeholders?

What is a realistic schedule for completing the program assessment?

The next CTE Directors

Video Streaming Session will be held

November 8, 2012

Selecting Appropriate Industry Certification Examinations

Presenters:

Gordon Creasy

Joseph Wharff

Continuing Professional Development via Video Streaming

George R. Willcox, CTE Coordinator

Office of Career and Technical Education

Virginia Department of Education

P. O. Box 2120

Richmond, VA 23218-2120

804 225-2052

804 530-4560

cte.@doe.virginia.gov

For Additional Information Contact