Upload
virgil-small
View
213
Download
1
Tags:
Embed Size (px)
Citation preview
NOVA Evaluation Report 1996-2002
Presented by:Dr. Dennis Sunal
Higher Education faces significant changes.
Data has been collected as part of a formative evaluation effort in the continuous enhancement of NOVA structure and functions to met its objectives.
Ten questions and data follow.
1. What are the characteristics of the institutions, faculty, and students who matriculate in NOVA courses?
- Principles from the National Science Education Standards
- Principles from the National Council of Teacher of Mathematics
- NOVA Professional Development Workshop Components
National Science Education Standards-Five principles
1. Change throughout the entire science education system.
2. What students learn is influenced by how they are taught.
3. The actions of teachers are influenced by their perceptions of science.
National Science Education Standards-Five principles, cont.
4. Student understanding is constructed through individual and social processes.
5. Teaches actions are influenced by the relationships with their students.
National Council of Teacher of Mathematics-Six principles
1. Math education requires equity. 2. The curriculum must be coherent, and
well articulated across the grades. 3. Effective math requires the teacher to
understand students prior knowledge, and continue to challenge and support the students to learn.
National Council of Teacher of Mathematics-Six principles, cont.
4. Students must learn math by actively building new knowledge from experiences and prior knowledge.
5. Assessment needs to support the learning of math.
6. Technology influences math that is taught, and enhances students’ learning.
NOVA Development Workshop Components-Twelve skills
1. NASA strategic enterprises2. Inquiry-based learning3. Innovative instructional strategies4. Assessment strategies that
facilitate learning5. Teaching science for all Americans6. New curriculum goals
NOVA Development Workshop Components-Twelve skills, cont.
7. Interdisciplinary approaches8. Using technology to facilitate learning9. Overcoming barriers10.Action research for college faculty11. Writing proposals for NOVA12. Connections to NSES, NCTM, ISTE
standards.
2. To what extent is the NOVA Model being disseminated and implemented by other faculty at NOVA institutions or at other institutions?
100 publications in journals, manuscripts, and presentations at national and regional conferences.
Regional and local workshops, consultancies, and other professional activities.
3. How congruent is the NOVA program with recommendations found in major reports on the preparation of teachers?
The NOVA Model for Professional development includes eight major elements aimed at changing how science and math are taught in K-12 classrooms by influencing “how” content is taught at the college level.
Eight major elements
1. Continuous professional development.
2. Sustained collaboration among interdisciplinary teams of faculty and administrators.
3. Courses based on national standards for science, math, and technology.
4. Courses utilize content based on NASA Strategic Enterprises.
Eight major elements, cont.
5. Program improvement at NOVA institutions through ongoing research.
6. Instructional strategies that are inquiry based and center on student interaction.
7. Extensive use of technology. 8. Collaborations among institutions in
the NOVA network.
4. What are the participant reactions to the NOVA professional development model? (Two part question)
Part A: Participant Reactions to the NOVA program
PositiveUseful workshopsNOVA course online modules has
been considered time well spent in professional development.
4. What are the participant reactions to the NOVA professional development model? (Two part question)
Part B: The change process for NOVA faculty
Commitment to the project. Belief in the model. The use of NASA resources. Monetary support for innovations. Introduce innovative teaching and/or
interdisciplinary courses to faculty.
Part B, cont.
Inspiration that came from changes in state education requirements and/or content standards.
Team of teachers attend a Phase I NOVA workshop.
Staff development.
5. How has the implementation of the NOVA Model impacted the collaborative work and organizational climate of science, mathematics, engineering, and education faculties? (Two part question)
Part A: Collaborative work changes When a team member leaves the
university. Teams no longer exist at the
institution. Faculty turnover and retirement. New faculty has not been integrated
into the team.
Part A, cont.
Collaboration continues with institutionalization of courses.
Team approach is vital to course development.
5. How has the implementation of the NOVA Model impacted the collaborative work and organizational climate of science, mathematics, engineering, and education faculties? (Two part question)
Part B: Organizational climate changesThe NOVA course has become the
“favorite lab.” The collegiality and impact on faculty
and students. Collaboration between the College of
Education and the science disciplines in the College of Arts and Sciences.
Part B, cont.
Time barriers. Institutional support barriers. Funding was a barrier. Conflicts
between or within department conflicts and the lack of mutual respect.
Students who had weak backgrounds, lacked motivation, or were fearful of math and science.
6. What new and modified courses are being offered as a result of NOVA?
135 courses were developed and offered at 84 NOVA institutions in the topic areas of :
-Biology-Life science-Life in space-Environmental science
7. To what extent does the NOVA course content integrate data and information that are unique to NASA, its fundamental questions, and its missions?
The NOVA courses integrate at least one of the five NASA Strategic Enterprises as a fundamental course component: ATE, BPRE, ESE, HEDS, and SSE.
8. What aspects of the NOVA professional development model are effective in creating and sustaining intended faculty knowledge and skills for reform action to take place? (Three part question)
Part A: What are the faculty perceptions of effective elements in the NOVA professional development process?
Cognitive apprenticeshipFacilitators of learningHigh self-efficacyKnowledge of pedagogy
8. What aspects of the NOVA professional development model are effective in creating and sustaining intended faculty knowledge and skills for reform action to take place? (Three part question)
Part B: Does action research conducted as a regular part of faculty teaching facilitate development of effective courses in science and mathematics?
Multiple techniques concerning assessment: journals, on-line tasks, rubrics, observation, performance tasks, interviews, and discussion groups, etc.
Part B, cont.
Faculty members may need additional support to engage in action research.
Investigating effective ways to teach science for improved student learning.
The need for instructors to create a “friendly learning environment” for students with poor attitudes.
8. What aspects of the NOVA professional development model are effective in creating and sustaining intended faculty knowledge and skills for reform action to take place? (Three part question)
Part C: What factors relate to sustaining change brought about through the NOVA professional development process?
Continued interaction between departments or disciplines.
Teams should have ‘people with similar ideas.’
9. What aspects of professional development affect course disciplinary content, classroom pedagogy, and specific disciplinary pedagogy related to student learning outcomes? (Three part question)
Part A: Course disciplinary contentNOVA course had significantly impacted
student attitudes to science as inquiry and to the study of climate.
Students taking the NOVA course had significantly higher teaching efficacy.
Students in the NOVA course also displayed more positive attitudes to geology.
Part A, cont.
Students in the NOVA course demonstrated significant gains in content knowledge.
Students valued the hands-on, collaborative approach to learning science.
9. What aspects of professional development affect course disciplinary content, classroom pedagogy, and specific disciplinary pedagogy related to student learning outcomes? (Three part question)
Part B: Classroom pedagogyMore interactive methods to enhance
student learning. Some classes were team-taught by
two or more faculty. Good collaboration helped shift
faculty from teacher-centered teaching to student-centered approaches.
Part B, cont.
The team approach provided the necessary expertise, collaborative preparation, and sharing of the workload.
98% of the new courses were using technology of some sort in course delivery.
9. What aspects of professional development affect course disciplinary content, classroom pedagogy, and specific disciplinary pedagogy related to student learning outcomes? (Three part question)
Part C: Specific disciplinary pedagogyHands-onLess or little lectureLots of group workStudent inputA diversity of teaching stylesThe content was “experienced–not told”
Part C, cont.
NOVA instructor was motivational. Students content knowledge was
improved. Technology enhanced learning. Course was clearly connected to
understanding the content.
10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question)
Part A: What are students’ perceptions of effective learning in science, mathematics or engineering classrooms resulting from the NOVA course?
Hands-on approachThe goal of teaching was student
understanding, not just covering content.
Part A, cont.
Awareness of students’ prior knowledge.
Having the ability to explain things on the students level.
Teaching in diverse ways. The teachers had the ability to
“make science come to life.”
10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question)
Part B: What changes in attitude toward science and mathematics are found in the students who experienced the NOVA course?
Becoming comfortable with various technology aids.
Positive student attitudes. An increase of minority students who
participated in the NOVA classes.
10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question)
Part C: What changes in teaching efficacy and learning outcomes resulted from students taking the NOVA course?
An increase in students’ content knowledge.
An increase in scientific processing skills. A greater understanding of the
mathematical concepts. Increased self-efficacy in science.
Part C, cont.
An increased ability and willingness to teach science and mathematics.
An increase in the students’ ability to use technology.
Enhanced critical thinking skills. A high level of interest in the
activities and were able to relate the content to real life.
10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question)
Part D: What changes in the nature of science and nature of mathematics, related to the literacy goals in the national standards, are found in students taking the innovative NOVA courses?
Undergraduate students in NOVA reform-oriented classes demonstrated a significantly higher end of course growth and level of understanding of the nature of science when compared to students in other science, “non-reformed,” courses.
Part D, cont.
An increased interest of females in science and mathematics literacy.
The learning climate was significantly more positive in the NOVA courses as compared to traditional NOVA courses.
10. Does scientific and mathematics literacy among students increase as a result of the faculty change through NOVA? (Five part question)
Part E: What are the indicators of change in students in NOVA course related to knowledge and performance of standards-driven reforms as a K-12 teacher of science or mathematics?
Improvements in the achievement of science content assessments.
Students had a high level of interest in the activities and were able to relate the content to real life.
Part E, cont.
Students’ self efficacy has increased.
6. The Effect of the NOVA Model on University Courses
A total of 135 courses were developed and offered at 84 NOVA institutions. 24% - biology, life science, life in space, or environmental
science 21% - earth and space science 20% - combined categories of mathematics, mathematics and
technology, mathematics and science, or statistics 16% - chemistry alone or chemistry, physics, or physical
science 10% - nature of science, inquiry science, or integrated
mathematics and science 5% - mathematics or science methods for preservice teachers 4% - engineering based for preservice teachers.
6. The Effect of the NOVA Model on University Courses
10060%
5001%
400
8%30015%
200
16%
NOVA Phase I Course Levels by Percent of Total
6. The Effect of the NOVA Model on University Courses
Course Level for Modified and Created Phase I Courses
0
20
40
60
80
100
L-100 L-200 L-300 L-400 L-500
No. Courses 81 21 20 11 2
7. The Impact of the NASA Mission, Data and Information, and Fundamental
Questions on the NOVA Courses
NASA Strategic Enterprise Representation
0
10
20
30
40
50
60
70
80
ATE BPRE ESE HEDS SSE
Institution 18 3 74 39 51
7. The Impact of the NASA Mission, Data and Information, and Fundamental
Questions on the NOVA Courses
Percent of NOVA Courses Making Connections with each NASA
Enterprise
10%
1%
40%21%
28%
ATE
BPRE
ESE
HEDSE
SSE
7. The Impact of the NASA Mission, Data and Information, and Fundamental
Questions on the NOVA Courses
During workshop sessions, the research program of each NASA Strategic Enterprise is described and participants are provided: NASA Strategic Plan NASA Implementation Plan for Education, 1999-2003 NSES & NCTM standards NRC’s Project 2061 & Science for All Americans NSTA’s College Pathways to the Science Education
Standards ISTE’s standards
7. The Impact of the NASA Mission, Data and Information, and Fundamental
Questions on the NOVA Courses
Many institutions have 3 or more connections to NASA Enterprises
Number of NASA Enterprises implemented remains similar across institutions by Carnegie Classification.
7. The Impact of the NASA Mission, Data and Information, and Fundamental
Questions on the NOVA Courses
An assessment instrument identifying NASA resources indicated: 78% were able to readily identify NASA
Enterprises relevant to their proposed course 70% had course materials that clearly
indicated the NASA Enterprises relevant to the courses being taught
91% stated they had used NASA resources to develop college level activities
7. The Impact of the NASA Mission, Data and Information, and Fundamental
Questions on the NOVA Courses
Examples of use of NASA resources: University of Tennessee at Chattanooga offers
2 courses using NASA materials through Phase III NASA Field Center projects
Brescia University used spreadsheets involving NOVA data as part of course activities
University of the Incarnate Word integrated NASA research data and NASA Internet sources in laboratory investigations
8. Aspects of NOVA Professional Development Model that are Effective in
Creating and Sustaining Change
Sunal, Hodges, Sunal, Whitaker, Freeman, & Edwards (2001) examined faculty conceptions about effective teaching and learning at 30 universities.
Information was gathered on faculty attitude, knowledge, teaching efficacy, course planning, and classroom interactions.
Results showed that faculty who described themselves as facilitators of learning and those with higher self-efficacy were more likely to follow through on implementing change.
Successful implementation of change was also dependent upon knowledge of pedagogy and course design.
a. What are the faculty perceptions of effective elements in the NOVA professional development
process?
8. Aspects of NOVA Professional Development Model that are Effective in
Creating and Sustaining Change
Nine important criteria emerged for successful implementation of change. For example:
interaction between science/mathematics and education faculty
administrative and collegial support a sense of common purpose and similar goals among faculty interpersonal skills and trust between faculty developmental and incremental change importance of action research component network of faculty as a support mechanism
a. What are the faculty perceptions of effective elements in the NOVA professional development
process?
8. Aspects of NOVA Professional Development Model that are Effective in
Creating and Sustaining Change
Fifty-two action research plans were analyzed: Eight were in the novice-apprentice range Thirty-two in the apprentice-proficient range Thirteen in the proficien-distinguished range
Most were weak on the data analysis component Some did not devise actual research questions or
hypotheses Most were a mixture of qualitative and quantitative; most
common was pre-posttest design
b. Does action research conducted as a regular part of faculty teaching facilitate development of
effective courses in science and mathematics?
8. Aspects of NOVA Professional Development Model that are Effective in
Creating and Sustaining Change
Emphasis of action research resulted in the use of a range of techniques for student assessment: • journals • on-line tasks • performance tasks • rubrics • observations • interviews • concept maps • discussion groups • discourse analysis • video and audio tape • self-evaluation • peer review
b. Does action research conducted as a regular part of faculty teaching facilitate development of
effective courses in science and mathematics?
8. Aspects of NOVA Professional Development Model that are Effective in
Creating and Sustaining Change
Most institutions saw students as the only source of data or unit of analysis
Research focused on student content knowledge, student skills, student attitudes, etc.
Cleveland State University used an approach where instructors focused on their own teaching: Two instructors constructed portfolios that included
personal reflections, philosophies, methods, student surveys and suggestions for improvement
b. Does action research conducted as a regular part of faculty teaching facilitate development of
effective courses in science and mathematics?
8. Aspects of NOVA Professional Development Model that are Effective in
Creating and Sustaining Change
Cyclical nature of change is indicated in action research
b. Does action research conducted as a regular part of faculty teaching facilitate development of
effective courses in science and mathematics?
Planning
Acting
Reflecting
Observing
Followed by
Does it answerthe problem?
Leads to
Results in
8. Aspects of NOVA Professional Development Model that are Effective in
Creating and Sustaining Change
Sixteen action research papers presented at the NOVA Leadership Development Conferences, 2001 & 2002: 13 described improvement in science or math content
knowledge, attitudes to the subject, process skills, and teaching efficacy
Others dealt with instructor issues such as classroom environment and teaching strategies
Studies of student outcomes showed improved student outcomes as a result of new courses
b. Does action research conducted as a regular part of faculty teaching facilitate development of
effective courses in science and mathematics?
8. Aspects of NOVA Professional Development Model that are Effective in
Creating and Sustaining Change
2002 LDC surveys showed 91% agreed that NOVA workshop provided opportunities to learn about NASA resources
Site Evaluation Visit interviews showed that continued interaction between disciplines was a valuable form of professional development
There was a relationship between a shift to a more student-centered approach and the effectiveness of NOVA teams
c. What factors relate to sustaining change brought about through the NOVA professional development
process?
9. Impact of the NOVA Model on Classrooms and Students
Most common title was interdisciplinary course in science Second most typical was a course in a single discipline Third - “NOVA Project” Fourth - integration of science, mathematics, and
technology Fifth - mathematics course for preservice teachers Sixth - connection between pedagogy and science or
mathematics Others involved nature of science, inquiry-based science,
etc.
a. Course Disciplinary Content
9. Impact of the NOVA Model on Classrooms and Students
Exemplary action research projects
San Francisco State University Kansas State University Alaska Pacific University Brescia University
a. Course Disciplinary Content
9. Impact of the NOVA Model on Classrooms and Students
San Francisco State UniversityNOVA Geology Course
Higher scores in attitudes toward science as inquiry and nature and study of climate
Better connections between concepts
a. Course Disciplinary Content
9. Impact of the NOVA Model on Classrooms and Students
Kansas State UniversityNOVA Geology Course
Students had significantly higher teaching efficacy More positive attitudes toward geology
a. Course Disciplinary Content
9. Impact of the NOVA Model on Classrooms and Students
Alaska Pacific University
Significant gains in content knowledge Gains did not depend on gender, class standing, or major Students with lower initial scores made more gains
a. Course Disciplinary Content
9. Impact of the NOVA Model on Classrooms and Students
Brescia University
Significant gains in content knowledge No significant difference in students’ science literacy
although NOVA groups scored higher Students valued hands-on, collaborative approach Revised methods improved course in second year
a. Course Disciplinary Content
9. Impact of the NOVA Model on Classrooms and Students
68% were using inquiry-based lessons Majority of faculty have shifted to more student-centered
approaches and others are in transtition Team approach was seen as essential 98% used technology of some sort 80% of instructors had high full-scale scores on the ESTEEM
Science Classroom Observation rubric Content-specific pedagogy subscale was lowest indicating
more NOVA workshop time should be spent on pedagogy
b. Classroom Pedagogy
9. Impact of the NOVA Model on Classrooms and Students
100% indicated national standards were referred to in development of course
Students reported hands-on methods with little or less lecture, lots of group work, motivational instructors, improved content knowledge, and enhancement of learning by technology
Three NOVA institutions experienced poor implementation of the NOVA model Instructors were not original NOVA team members
c. Specific Disciplinary Pedagogy
9. Impact of the NOVA Model on Classrooms and Students
Overall, typical NOVA courses emphasized:construction of knowledge through hands-on, inquiry-
based activitiesfocusing on solving relevant problemssupport from technologyenhancement from cross-college faculty
collaboration
c. Specific Disciplinary Pedagogy
9. Impact of the NOVA Model on Classrooms and Students
Doctoral dissertation studies showed:Modeling, active engagement, project-based
activities, and cooperative learning positively affected science teaching efficacy (Staples, 2002)
Higher achievement in elementary students of teachers experienced in NOVA (Staples, 2002)
Cross-college faculty collaboration is effective but time consuming (Hodges, 1999)
c. Specific Disciplinary Pedagogy
10. Impact of the NOVA Model on Science and Mathematics Literacy of University
Students
Integration of lab & lecture to promote a hands-on approach
Goal should be understanding, not just content Awareness of prior knowledge Ability to explain on students’ level Use diverse ways of teaching Make science “come to life” NOVA model “improved content knowledge, attitude and
confidence”
a. What are students’ perceptions of effective learning in science, mathematics, or engineering
classrooms resulting from the NOVA course?
10. Impact of the NOVA Model on Science and Mathematics Literacy of University
Students
Increased content knowledge Appreciation and enjoyment in becoming comfortable with
technology aids High rates of employment and admission to graduate
school Positive impact on minority students Overall improvement in attitude toward science and
mathematics More confidence in future ability to teach science
b. What changes in attitude toward science and mathematics are found in the students who
experienced the NOVA course?
10. Impact of the NOVA Model on Science and Mathematics Literacy of University
Students
Increased self-efficacy in science Greater understanding of mathematical concepts Increased ability and willingness to teach science and
mathematics Improvement in attitudes toward the language of mathematics Decrease in misconceptions Long-term retention of science content Improvements in achievement on science content assessments A variety of ways to teach mathematical content were learned
c. What changes in teaching efficacy and learning outcomes resulted from students taking the NOVA
course?
10. Impact of the NOVA Model on Science and Mathematics Literacy of University
Students
Overall increase in science and mathematics literacy Increase seen in minorities and females Increased understanding of the nature of science More positive learning climate Positive impact on attitudes toward mathematics Long-term effects such as incorporation of relevant nature
of science topics in secondary content
d. What changes in the nature of science and nature of mathematics, related to the literacy goals in the
national standards, are found in students taking the innovative NOVA courses?
10. Impact of the NOVA Model on Science and Mathematics Literacy of University
Students
Overall achievement improvement in all areas of science and mathematics understanding related to national standards
Improvement in achievement on science content assessments
More comfortable attitudes toward the idea of teaching science
Improved Personal Science Teaching Self-Efficacy Comfort with constructivist learning environments
e. What are the indicators of change in students in NOVA courses related to knowledge and
performance of standards-driven reforms as a K-12 teacher of science and mathematics?