Embed Size (px)
Citation preview
Rising to the Occasion: Systematic STEM Academy Assessment Practices
Virginia Department of EducationOffice of Career and Technical Education Services
October 25, 2012
“Using Data for Program Planning and Continuous Improvement”
Rising to the Occasion: Systematic STEM Academy Assessment Practices
Nathan Carter, Ph.D.Research Consultant
Background Information
• In 2008, the Virginia Board of Education approved the criteria for establishing Governor’s STEM Academies.
• As part of the criteria, the Virginia Department of Education (VDOE) was required to conduct annual evaluations of all established STEM Academies.
• In order to evaluate STEM Academies effectively VDOE needed to:• Acquire current descriptions of each career pathway offered at a
STEM Academy
• Develop an informative template that STEM Academies could use to complete an Annual STEM Academy Program Assessment Report
Typical Problems with Program Assessments
Pressure for doing a comprehensive program assessment can cause a “rush to completion syndrome” and produce results that are not very useful.
When there is not a team-oriented approach to completing a comprehensive program assessment important voices can be left out of the discussion.
In some situations comprehensive program assessment is not based on a clear understanding of core program values or agreement on what the short and long-term goals of the program are.
Another Common Problem: DATA OVERLOAD!!!!!
If I’d known they wanted me to use all of this quantitative data – I would NEVER have asked for it!!!
Four General Purposes of Quality Comprehensive Program Assessment
To improve – The assessment process should provide feedback to stakeholders on how a school program can be improved.
To inform – The assessment process should inform decision makers and program stakeholders of the continual contributions and impacts of a program on meeting short and long-term goals.
To prove – The assessment process should demonstrate what the program is accomplishing and how these accomplishments are changing overtime.
To support- The assessment process should support faculty, administrators, and others as they think about what actions need to be taken to continue to support students.
Requirements for a Complete Assessment Process It must be comprehensive – must cover the full
range of learning outcomes that are connected to program values and goals
It must include multiple judgments – multiple sources of evidence must be used
It must include multiple dimensions – different facets of student performance must be included
It must collect direct evidence – direct measures of student attainment must be used
It must include qualitative interpretations of the data and evaluations of patterns overtime.
Iterative Model of the STEM Academy Annual Assessment Process
Plan
CollectData
Impl
emen
tA
nalyze
Data
Reflec
t
& Rev
ise
STEM Academy Career Pathway Description Sheet
Grade Level
CoreCourse Name
StateCourse Code
Content AreaCareer Pathway Course Options
(indicate all that apply for each course)
English Math ScienceHistory &
SocialScience
CTE Other
CourseRequirement
For CTE Program
Completion
CourseRequirement
ForAdvanced
StudiesDiploma
CourseRequirement
ForAdvancedTechnicalDiploma
Industry Certificatio
nAvailable
Course Available To Be Taken As
A Dual Enrollment
Course
Non-Traditional
CareerPreparation
Course(M or F)
Career Pathway Name___________________________
Workplace Experiences Offered With Career Pathway
Job Shadowing Internship
Mentorship Cooperative Education
Student Apprenticeship Service Learning
Other
Sections of the STEM Academy Assessment Report
SECTION DATA ELEMENTS INCLUDED
Background InformationName of STEM Academy, year STEM Academy opened, STEM Academy director’s name & contact information, STEM Academy’s purpose, STEM Academy’s current short & long-term goals.
STEM Academy Acceptance Rates
Number of applications received, number of applications accepted, number of students placed on a waitlist
STEM Academy Experiences
Number of elementary or middle school students in STEM Academy summer activities, number of elementary or middle school students in STEM Academy classes during the school year
STEM AcademyHigh School Enrollment
Total number of high school students enrolled, newly enrolled, and number who were expected to return but did not (by grade level and by career pathway)
Each section has a quantitative and qualitative data collection requirement
Sections of the STEM Academy Assessment Report (cont.)
SECTION DATA ELEMENTS INCLUDED
STEM Academy AP Course & AP Test Participation
Number of STEM Academy students enrolled in AP courses, number enrolled in AP courses and taking AP tests, number taking AP tests without being enrolled in AP courses.
STEM Academy Dual Enrollment Participation
Number of STEM Academy students taking 1, 2, or 3+ dual enrollment courses, number of students passing 1, 2, or 3+ dual enrollment courses
STEM Academy SOL Achievements
Number of STEM Academy students scoring Advanced Proficient, Proficient, or Not Proficient on the different SOL exams
STEM AcademyCredential Production
Number of STEM Academy graduates/completers taking and passing Workplace Readiness Skills Assessments, or Occupational Assessments, State Licensure Exams, or National Industry Certification Exams
Sections of the STEM Academy Assessment Report (cont.)
SECTION DATA ELEMENT INCLUDED
STEM AcademyGraduation Data
Number of STEM Academy graduates/completers earning Board approved diplomas with a special emphasis on whether STEM Academy graduates earn advanced studies diplomas and/or CTE credentials upon graduation
STEM AcademyPost-graduation Data
Number of STEM Academy graduates/completers who enrolled in credit-bearing courses, or earned college degrees, or transitioned into high-skill, high-wage, high-demand careers, etc.
Summary of STEM Academy Progress
1. STEM Academies compile their qualitative and quantitative data and assess to what extent they are meeting their short and long-term goals.
2. STEM Academies identify their strengths as well as what areas need attention.
3. STEM Academies adjust their short and long-term goals where needed.
VDOE provided each STEM Academy with a PDF document that showed which tables and available data in the template could be populated with state-collected data. rt.
STEM Academies will organize their own data
management team and set deadlines for populating
tables in the STEM Academy Assessment Report using
locally stored data sources.
STEM Academies will compare the
data tables provided by the
VDOE with their own data tables on the different areas
team.
STEM Academies will create a list of
all the relevant data inaccuracies between the two
sources and document possible
reasons for the differences.
In the case of discrepancies ,
STEM Academies will use their own
locally validated data to populate the tables
in the Assessment Report and complete
their annual evaluation.
START
END
STEM Academies will submit their
Annual Assessment Report and a list of any discrepancies
uncovered during the data validation
process to VDOE along with
descriptions of career pathways
Data Validation Flow Chart
Important Questions To Think About As You Move Forward
How are you currently embracing a data-driven decision-making process that leads to actionable
results?
Consider including people from various stakeholder groups to work colloboratively on the comprehensive program assessment plan.
Who is going to be a part of the program assessment team?
When will we administer the assessment?When will we collect the data?When will we analyze the data?When will we reflect upon the data?When will we make recommendations?When will we make the decisions about the recommendations?When will we provide written documentation about the decisions?When will we share the data with other stakeholders?
What is a realistic schedule for completing the program assessment?
The next CTE Directors
Video Streaming Session will be held
November 8, 2012
Selecting Appropriate Industry Certification Examinations
Presenters:
Gordon Creasy
Joseph Wharff
Continuing Professional Development via Video Streaming
George R. Willcox, CTE Coordinator
Office of Career and Technical Education
Virginia Department of Education
P. O. Box 2120
Richmond, VA 23218-2120
804 225-2052
804 530-4560
For Additional Information Contact
