Applying the Rasch Model in KEEP - 1

Applying the Rasch Model in KEEP - 1 -

Running head: Applying the Rasch Model in KEEP

Applying the Rasch Model to Evaluate a High School Science Implementation of the Kentucky

Electronics Education Project (KEEP)

Presenter: Weijia Ren (University of Kentucky)

Co-presenters: Kelly D. Bradley (University of Kentucky)

Janet K. Lumpp (University of Kentucky)

1


Abstract

Kentucky Electronics Education Project (KEEP) uses the multidisciplinary nature of

microelectronics as a theme to connect real world content with K-12 classroom science

education. Science teachers are trained in a series of circuit building activities through summer

workshops and implement them in their classes. Students are given a survey to evaluate their

perspectives after completing the activities. The sample includes 61 students from a high school

in Lexington, Kentucky. Data are fitted into the Rasch rating scale model. Findings indicate a

high reliability and validity of the survey instrument and a high satisfaction of the project among

students. The result shows that most students think the project is easy. However, most

misunderstandings appear to occur in item 3 “Etching the copper”. The least favorite item and

the most difficult item are both item 2 “Ironing the pattern onto the circuit board”. Possible

reasons and solutions for improving the steps in the project and corresponding items are

described at the end.

Keywords: Rasch model, KEEP, circuit building, program evaluation

2


Introduction

With the rising expectations of U.S. students to be the first among students all over the

world in math and science achievement and the widening gap between the United States and

some other countries, such as India and China, in educating and graduating students with the

engineering degree, the Kentucky Electronics Education Project (KEEP) was initiated and

developed to engage K-12 students in math and science education with a hands-on circuit

building activity. This real world example is beyond the traditional “bulbs and batteries” example

and provides students with the basic science knowledge about electricity, which is also aligned

with the national science education standards.

Although nowadays some reform proposals have been posted to improve the K-12

science education, there has never been a specific curriculum or program to connect students’

classroom learning with the real world scientific activities. Therefore, it is the effort and goal of

KEEP to help produce a set of science education curricula to apply real life science activities to

the classroom education. Thus it is necessary and important to get instant feedbacks from

teachers and students and based on them, to improve this “pioneer” project in K-12 science

education.

The Kentucky Electronics Education Project began in 1997. It involves curriculum

development, teacher workshops and classroom activities initiated by the educational outreach

aspects of two National Science Foundation research grants (Lumpp et al., 2003). This project

was initiated by Dr. Janet Lumpp, a professor in the College of Engineering at the University of

Kentucky. The specific objectives of KEEP are to educate teachers regarding electronic assembly

3


technologies and properties of electronic materials; to develop curriculum materials and kits; to

organize hands-on projects and fieldtrips; and to solicit education/industry partnerships (Lumpp

et al.). This circuit building activity incorporates integrated curricula and performance

assessment, and moreover, provides a real world activity that is a foundation for math and

science application to the K-12 science education classroom.

Science Education is an indispensable element of the education system. Students can only

build a comprehensive knowledge background with a competitive science curriculum. In order to

develop such science curriculum, real world scientific inquiries, technologies and examples are

indispensable. Incorporating true scientific inquiry into today’s classrooms, especially K-12

classrooms, is a major focus of current science education reform and is critical to scientific

literacy (Rutherford et al., 1990). In a meta-analysis of 20 studies that examined the effects of

technology use on students’ cognitive, affective, and behavioral learning outcomes, a modest

positive effect was found between technology and other learning outcomes (Waxman et al.,

2002). Also, the report to the nation by the National Commission identifies the need to facilitate

teachers in learning how to integrate technology into science and mathematics education

(USDOE, 2000). Therefore, implementing engineering design is a powerful strategy for the

integration of science, mathematics, technology, and for engaging a broad population of students.

Although engineering design could be used to provide relevant and standards-based

content for integrating engineering into K-12 science classes, nowadays most popular science

textbooks for grades 4-12 incorporate little if any engineering content or activities (Cantrell et al.,

2002). Not long ago, national reports had indicated that science education in America was falling

4


short of its goal for achieving scientific literacy for its youth. Moreover, “experts have also been

warning for a few years now that the United States is at risk of losing its lead position in the

education of science and engineering Ph.D.’s” (Jaschik, 2005). In order to attract more students

to study science or pursue an engineering degree in college, teachers have to begin to cultivate

children’s interest in science when they are in the elementary school. The long-term goal of

KEEP is to enhance the quality of teacher instruction, support gains in student achievement, and

increase students’ interest in STEM curricula and careers (Lumpp et al., 2006).

In KEEP, circuit building is used as an activity to bring the real world microelectronics

models into classroom education. During the past 10 years, the circuit building activity is given

to the students at the end of each spring semester, typically in May. There are altogether seven

steps in the circuit building project, including copper cleaning, ironing the pattern onto the circuit

board, etching the copper, drilling holes, removing toner, inserting components, and soldering.

These steps are related to the following topics in microelectronics: copper etching and plating,

semiconductors, insulators, conductors, energy conversion, heat transfer, materials properties,

dimensions, routing, soldering, circuit symbols, and component types. These topics tie directly

with national standards on math skills and concepts, physical science, science and technology,

environmental issues, and science as inquiry (Lumpp et al., 2003).

This manuscript focuses on the attitudes of students who have participated in the circuit

building activity. Students’ attitudes towards the project are reflected by two sections: students’

satisfaction of each step in the project and the overall project; and students’ perspectives on the

difficulty level of each step and the overall project. Students’ attitudes are very important in order

5


to evaluate teachers’ performance as well as the whole KEEP project. Based on students’

feedbacks, the project could be properly improved, and steps in this activity could be adequately

designed and ordered. Specific steps which most students are confused will be improved and

restructure as well. In addition to the improvement of the project, the application of the survey

instrument will also be improved. The content and order of the survey will be analyzed and

whether it is necessary to add or remove any questions from the current survey will be discussed.

In a word, the purpose of this manuscript is to analyze students’ feedbacks to this project and

improve both the project and the survey instrument.

Evaluation is a tool to help teachers judge whether a curriculum or instructional approach

is being implemented as planned, and to assess the extent to which stated goals and objectives

are being achieved (Fleischman & Williams, 1996). This study combines both process evaluation

and outcome evaluation which determines the effect of the program, and also how the program

produced that effect and how to improve the program. Data collection and record is an important

part of the design in the evaluation, which may include record, keeping forms, questionnaires,

interview guides, tests, or other assessment measures (Fleischman & Williams). In this study,

questionnaire is chosen to be the effective and reliable method to collect data fast and without

causing major inconveniences to the class. To analyze the collected data, Item Response Theory

is often utilized. Item Response Theory is based on assumptions concerning the mathematical

relationship between abilities and item responses (Rudner, 2001). However, traditional IRT

model may not fit in all situations. Rasch model, as a subset of IRT model, works better and

more accurate when analyzing data from assessment to measure things such as abilities, attitudes,

6


and personality traits. In the case of this study, because students’ attitudes are the major factor,

Rasch model will be a better fit.

Method

Participants

The participants to this survey were the 61 students who took part in the KEEP activity in

May 2006 from one high school in Lexington, Kentucky. There were 38 males and 23 females.

The participants include 49 seniors, 11 juniors and one sophomore. Among them, 41 students had

completed similar KEEP project before, while the other 20 students had not taken any similar

courses before. During the analysis, each participant was assigned a person code including

information about their current grade level, gender, and the typical grades they receive in classes,

whether or not they want to do a similar project again and whether or not they have participated

in a similar project before. Currently, students who participate in KEEP are middle school and

high school students in Lexington, KY. Therefore, the population of interest in this study is

middle school and high school students who have participated in the circuit building project. And

the 61 students in the sample are fitted in the population of interest.

Instrumentation

This survey was designed by experts at the University of Kentucky as an onsite paper

questionnaire. Students were asked to complete this pencil-and-paper questionnaire in class. The

project representatives brought the questionnaires to this high school and students were required

to finish the survey within five to ten minutes at the end of the circuit building activity. This

survey utilized a 4-point Likert-type scale to rate students’ satisfaction of each step of the project,

7


including copper cleaning, ironing the pattern onto the board, etching the copper, etc. (1=very

unsatisfied, 2=unsatisfied, 3=satisfied, 4=very satisfied), their perspectives on the difficulty level

of each step (1=very difficult, 2=difficult, 3=easy, 4=very easy), and their perspectives on the

classroom environment (see appendix for details). In addition to the three tables, six close-ended

questions were also designed to gather the demographic information of the students, such as their

grade level, gender, etc. Four open-ended questions were designed to gather information about

what students learned from the project, their favorite and least favorite parts in this project, etc.

(see appendix for details).

Data Analysis

In this manuscript, students’ satisfaction of the project and students’ perspectives on the

difficulty level of the project are highlighted. There were no missing data for these two sections

among the collected responses. Data were analyzed in Winsteps, employing the rating scale

Rasch model. Rasch is mathematically identical to the most basic Item Response Theory (IRT),

however it is a comparatively more viable proposition for practical testing since it can be applied

in the experimental context in which persons interact with assessment questions or items. In this

model, data must fit the model, and no extreme item or person will be analyzed. The rating-scale

Rasch model is an extension of the dichotomous model and a simplified version of the partial

credit model. It specifies that the whole set of items share the same rating scale structure. The

mathematic expression of rating-scale Rasch model is )]([exp)]([expjin

jinnixτδβτδβ+−ΣΣ+−Σ

=Π where

x= 0, 1,…, m. πnix shows the probability of student n responding in category x to item i (Wright

8


et al., 1982).

Throughout the analysis, several tables and figures are used to describe and analyze both

students’ satisfaction and the difficulty level of the project. A statistical summary table is

produced to describe the separation rate and reliability of the students and the items. Separation

is the number of statistically different performance strata that the test can identify in the sample.

The reliability rate indicates the validity and reliability of the instrument, and whether or not the

test discriminates the sample into enough levels for the research purposes. In Rasch model,

separation rate and reliability rate are highly correlated. Then, a construct keymap will be

provided to further test the validity of the categories of the four choices as well as the whole

instrument.

In addition, an item/person misfit table is analyzed. The misfit data is described and

explained. The statistics show how well the data fit the model. Fit implies meeting requirements

or matching intentions. It specifies an indication of the match between a group of persons and a

set of items. If some persons or some items did not perform as expected, they were flagged as

persons or items differing from expectation and therefore misfitting (Shaw, 2006).

Following, a variable map is constructed to illustrate the empirical hierarchy of items in

the survey, as connected to the students’ level of willingness to endorse each item. The variable

map visually reveals the hierarchy and the order of the items as well as the gap between every

two items. It also displays the logit of every student and every item. A logit (log-odds unit) is a

unit of interval measurement which is well-defined within the context of a single homogeneous

test (Winsteps Help, 2007).

9


Lastly, a correlation matrix is used to analyze the correlation coefficient between the

seven steps and the overall project. The matrix is run by Minitab. From the analysis, how these

seven steps correlated with each other as well as with the overall project will be discovered.

Results and Discussion

The results are divided into two sections and discussed separately. Section 1 is students’

satisfaction of the project and section 2 is students’ perspectives on the difficulty level of the

project.

Students’ Satisfaction of the Project

Reliability and separation for students’ satisfaction.

In order to have an overall view of the reliability and validity of the instrument and the

students, the statistical summary tables of both the student (Table 1) and the item (Table 2) are

analyzed at first.

Table 1 shows that the person reliability in the model is 0.77, and the person separation is

1.83. Table 2 shows that the item reliability is 0.89 and the item separation is 2.91. The higher

the reliability is, the more reliable the instrument is. In this case, reliability which is higher than

0.7 is acceptable, thus the reliabilities in both tables (0.77 and 0.89) are acceptable.

In table 1, the person separation of “1.83” indicates that 1.83 levels of performance were

consistently identified by the test for the tested sample. That means students were roughly

separated into 2 groups. These two separated groups include those who were satisfied with the

project and dissatisfied with the project.

10


The person separation is slightly lower than 2, which indicates a narrow range of person

measures or a small number of items (8 items) in the survey. The student reliability in table 1 is

0.77 out of 1, which is quite a good reliability and indicates the model is reliable for the 61

students. If the designers want to increase the reliability, the survey could be lengthened, and

more questions could be added in the survey. Increasing the sample size and testing people with

more extreme attitudes (high or low) may also help.

The reliability and separation rate in table 2 are more acceptable and indicate the survey

is reliable for the eight items and these items are better separated than the students. The item

separation rate “2.91” indicates that the eight items are generally separated into 3 groups: items

that students were satisfied, items that students thought to be “OK”, and items that students were

unsatisfied. The item reliability is 0.89 out of 1, which is higher than the student reliability above,

shows that the eight items as well as the whole survey instrument are reliable.

Construct keymap for students’ satisfaction.

Construct keymap is a figure which shows reliability and validity of the categories in the

instrument visually. In this plot, items are ordered from the least favorite-item 2 “Ironing” to the

most favorite-item 8 “Overall project”. If the category order “1, 2, 3, 4” of one item is not

consistent, that item might cause misunderstanding or unexpected answers. In figure 1, all items

have a consistent category order; therefore, the instrument and scales are given evidence as to

their validity. The survey instrument is thus proved to be both reliable and valid. For items 1, 4, 5,

6, 7and 8, since category 1 does not appear, “very unsatisfied” is not a likely response regardless

of the respondents’ abilities or attitudes.

11


Item/person misfit order table for students’ satisfaction.

In addition to the statistical summary tables, the person and item misfit order tables are

also helpful to test assumptions. The analysis tests whether this instrument was a valid tool for

data collection. Table 3 reflects the misfit statistics of the item.

In table 3, the outfit mean square statistic (MNSQ=1.49) for item 3 is outside the selected

range of item outfit “Mean+S.D.” (0.97 + 0.24). The high outfit value indicates this item

received the most unexpected responses and implies that students’ responses to this item were

out of the project designer’s expectation.

The possible reason for students’ misunderstanding maybe the teachers usually etch the

copper by themselves rather than teaching students to etch because etching will take a long time.

This item is typically carried out by the teacher outside of class time. While the circuit boards

soak in the etching solution, the students would have to wait to perform the next step. Perhaps

the students rated this item inconsistently because they did not personally perform the step. In

order to solve the misunderstanding, teachers may show students a video about how to etch the

copper. Although students may not be able to do it by themselves due to the limited time, they

will at least know how to etch after watching the video.

As for the misfit students, seven misfit students are found to respond with more

unexpected answers than the other 54 respondents. Since the range of student outfit is 0.45-1.49

(Mean+S.D.), person 12 SFBCYY (see Appendix for details) has an outfit figure of MNSQ=2.35,

which is much higher than the upper bound and the highest among all responds. That indicates

she endorsed the most unexpected answers among all respondents. Similarly, student 10

12


SMABYN, student 26 SMABYY, student 18 SMAAYY, student 52 JFAAYN, student 14

JFANYY, and student 41 SMABYY also endorsed more unexpected answers than the rest of

students.

The most unexpected response that the seven persons endorsed is item 3 “Etching the

copper”, which is also shown as the most misfit item in the previous item misfit table. Other top

unexpected items include item 6 “Inserting components” and item 7 “Soldering”. Most of these

misfit persons have a high measure score, which means they are generally satisfied with the

overall project, while they may not as satisfied with items 3, 6, and 7.

Variable map for students’ satisfaction.

The variable map is another important figure to provide visual information about the

relative scales of person and item. In this map, from top to bottom, items are ranked from the

least favorite item to the most favorite item. Respondents are ranked from students who are most

satisfied with the project to those who are least satisfied with the project. The variable map is

shown in figure 2.

In figure 2, students are most satisfied with item 8 “Overall project”, and least satisfied

with item 2 “Ironing the pattern”. It is very interesting that students endorse the overall project as

the most satisfied item. It indicates students are satisfied with the whole project but each single

step in this project may have some weaknesses with which they are not satisfied. Other elements

may also contribute to students’ satisfaction, such as the classroom environment, teachers’

teaching methods, etc.

From the variable map, the item “Overall project” does not reveal any additional useful

13


information than other items. Therefore, this item may be revised or deleted. In addition, since

the mean of students’ satisfaction is almost two standard deviations higher than the mean of item;

therefore, most students are satisfied with most parts of this project.

The significant gap between item 2 “Ironing the pattern” and other items indicates the

need for a better scale coverage. Item 2 is also the most difficult to endorse, indicating students

are more dissatisfied with this item than others. It is possible that students think this step is

difficult, boring and not as important as other steps. The step “Ironing on the pattern” includes

ironing the toner onto the board, and then repairing any missing lines with a permanent match. If

the pattern is beyond repair, the students must repeat both of the cleaning and ironing steps. It

may be the case that the students rate “Ironing” as difficult because of the repair process. From

students’ comments, “ironing” is regarded as difficult because it is hard to control the melting

temperature of the toner and if they make any small mistake, they have to redo the whole process

again. In order to identify the specific difficulty level of ironing, this step and its corresponding

question in the survey could be separated into two— “Ironing” and “Repairing the pattern”.

Teachers may give students more detailed instructions for ironing and help student finish ironing

more quickly. The large gap at the top indicates that most students were satisfied with most of

the project, which again implies students’ satisfaction with this project. The smaller gap between

item “Soldering” and item “Inserting components”, together with other two gaps, divide the eight

items into three groups by students’ satisfaction degree. This is also verified in the previous item

separation table.

Correlation matrix for students’ satisfaction.

14


In this correlation matrix, the correlation coefficients as well as the p-value between each

step and other steps are revealed and analyzed. Since α is set at 0.05, any correlation coefficient

that has a p-value higher than 0.05 is not significant. Therefore, the correlation coefficient

between “Ironing” and “Etching” is not significant with a p-value at 0.176>α=0.05, so it is hard

to predict students’ satisfaction of ironing by their satisfaction of etching, and vise versa.

Similarly, the correlation coefficients between “Copper Cleaning” and “Inserting Components”

(p=0.471), “Copper Cleaning” and “Soldering” (p=0.069) are not significant. And all these three

correlation coefficients are relatively weak. “Inserting Components” has the highest correlation

coefficient (0.528) with “Overall Project” among all the seven steps, and “Drilling” has the

lowest (0.328). It implies if students are satisfied with the step “inserting components”, they will

more likely to be satisfied with the overall project. And students may be satisfied with the overall

project no matter whether they are satisfied with drilling or not.

Students’ Perspectives on the Difficulty Level of the Project

Similar to the analysis above, statistical summary table, construct keymap, item/person

misfit tables, the variable map and item correlation matrix are used to analyze students’ ability

and the difficulty level of the project.

Reliability and separation for the difficulty level.

Table 4 and table 5 show that the item reliability (0.95) and separation (4.23) are higher

than the student reliability (0.77) and separation (1.84) for students’ perspectives on the difficulty

level. The results indicate that both the student sample and the items in the survey are reliable,

15


especially the item reliability. Students are successfully separated into 2 groups: one thinks the

project is difficult and the other thinks it’s easy. The 8 items are separated into 5 difficulty levels,

rating from the easiest to the most difficult. This is also visually displayed in the variable map in

the following figure 4.

Construct keymap for the difficulty level.

In figure 3, for item 3 “Etching the Copper”, there is a disorder between category 1 and 2.

It indicates that for this item, students with less ability are more likely to endorse category 2

“difficult” than category 1 “very difficult”. This also implies that item 3 has the highest misfit

rate. In addition, since category 1 is missing for Item 1, 4, 5 and 8, the category “1=very

difficult” is not the most likely response in this survey, however, if more students with less

ability are included in the sample, this category may be useful for these items. In other words,

these four items are regarded to be easy by all students. In general, most of the categories are in

order, which implies this survey instrument is valid.

Item/person misfit order table for the difficulty level.

Surprisingly, table 6 also indicates that item 3 “Etching the copper” is the most misfit

item since it is outside the upper bound of outfit—“Mean+S.D.” (1.00+0.32). This item is also

the misfit item in students’ satisfaction. Similarly, students’ concepts about this item may be

different from the designed concepts. Since item 3 is the misfit item in both sections, it indicates

a significant mismatch between some students and this item. The wording may be ambiguous or

misleading. As mentioned above, since the step “Etching the copper” is understood at different

levels among students, teachers may want to describe in detail or show students how this step

16


processes via video or other instructional methods.

Student whose mean square is outside the range of Mean + S.D. (1.00+0.59) is

considered to be misfit. Person 52 JFAAYN has a highest mean square of 3.83 among all

students, indicating she’s the most misfit student. Also, student 42 JMABYY, student 58

SMABYY, student 46 SFABNN, student 17 SMABYY, student 33 SMABNY and student 47

SMAAYN all endorsed more unexpected answers than the rest of students.

The most unexpected answers they endorse are “1=very difficult” and “2=difficult” for

item 2 “Ironing” and item 3 “Etching”. Most of the 7 persons have positive measure value, which

means they think this project is easy. Therefore, their perspectives on item 2 and item 3 may be

the reason why the two items become the misfits.

Variable map for the difficulty level.

Figure 4 illustrates the variable map for student perspectives on the difficulty level of the

project, from top to bottom, the 8 items are ranked from the most difficult item to the easiest item,

while the persons are ranked from the most able students to the least able students. The variable

map is shown in figure 4.

In this map, “Ironing the pattern” is regarded as the most difficult item, while the

“Copper Cleaning” is the easiest item. Since “Copper Cleaning” is the first step, students may

generally view it easier than all other following steps. The difficulty level of the overall project is

in the middle of all steps, which implies some steps are easier and others are more difficult. Also,

the students’ mean ability is higher than the items’ mean difficulty level, thus most students

thought the project was easy. The range of the items is almost 4 standard deviations, and there is

17


an obvious gap between “Ironing” and “Soldering”. It implies that students think “Ironing” is

much more difficult than other steps.

Together with the last section, “Ironing” is regarded as the step which is least favorite and

most difficult. It may be concluded that because “Ironing” is the most difficult step, most

students are not satisfied with it. As mentioned above, designers may want to separate this step

into “Ironing” and “Repairing the pattern”. Altogether, there are five visible gaps which divide

the eight items into 5 difficulty levels, which verifies the item separation rate in table 5 as well.

Correlation matrix for the difficulty level.

In the correlation matrix of the difficulty levels of the eight items, the correlation

coefficients between “Copper Cleaning” and other steps are almost all insignificant

(p-value>α=0.05). Similarly, the correlation coefficients between “Ironing” and “Drilling”

(p=0.744), “Ironing” and “Removing the toner” (p=0.569), “Ironing” and “Inserting

Components” (p=0.091) are not significant. Since “Copper Cleaning” and “Ironing” are the first

two steps in this project, students may hardly define their difficulty levels, therefore, students’

answers to these two steps are not related tightly with other steps, that is to say, it is hard to

estimate other steps’ difficulty levels based on students’ answers to these two questions.

Item 2 “Ironing” has a lower correlation coefficient with item 8 “Overall Project” (0.260)

than all other items, which means students who think the overall project is easy may think the

item 2 “Ironing” is either difficulty or easy. Item 4 “Drilling” has a higher correlation coefficient

with the overall project (0.499) than other steps, which indicates that students who think the

overall project is easy tend to think drilling is easy too, and vise versa. In figure 4, the same

18


conclusion was drawn. There is a big gap between “Overall project” and “Ironing”, but not

between “Overall project” and “Drilling”.

Conclusion

Based on the analysis above, most students are satisfied with the circuit building activity,

and think this overall project is easy. Though Rasch model, students’ ability and the

corresponding item are clearly stated and compared along one scale and therefore properly

analyzed. The survey instrument is reliable and can well separate both the sample students and

items. The four categories of the choice are also valid in this instrument. Most students

understand all items and steps well without confusions or misunderstandings.

Item 3 “Etching the copper” is the most misfitting step in both section 1 “students’

satisfactions of the project” and section 2 “students’ perspectives on the difficulty level of the

project”. The possible reason may be that teachers did this step by themselves instead of teaching

students to do it; therefore, teachers may want to give more details about this step and show

students a video about the etching process to make students to be aware of the step.

In addition, item 2 “Ironing the pattern” is regarded as both the least favorite and most

difficult item in this project. Many students put “Ironing” as the most difficult step in their

answers to one of the open-ended questions. Their comments imply that because the toner does

not melt at the expected temperature, it needs several tries to make it work and if any mistake is

made, the whole pattern should be repaired. Therefore this step may need to be separated into

two steps –“Ironing” and “Repairing the pattern” in order to test the specific difficulty level of

19


ironing and teachers may give students more details about the toner’s temperature.

In addition, the results from this survey may be biased since most of the respondents are

students in good standing and they all come from one school. To improve it, more students with

extreme grades may be included. With “good standings”, the students may be interested in all

studies or subjects, consequently it may not be concluded that this project is successful and liked

by all students if all students in the sample like science and engineering. Thus a close-ended

question may be raised to ask students to choose their favorite subject.

This preliminary study evaluates both the project and the survey instrument. Different

from many other studies, this study provides both outcome evaluation and improvement

suggestions to this ongoing project. But when testing students’ satisfaction to this project, some

other aspects should also be taken into consideration, such as the classroom environment, the

tools, and teachers’ instructions. In addition to students’ attitude evaluation, teachers’ attitude

evaluation is also indispensable. Therefore, in future studies, the analysis of students’

achievement, teachers’ instructions, and teachers’ attitudes may also be helpful to develop this

project. Because KEEP is more like a short-term workshop than a long-term curriculum, some

other alternative assessments such students’ portfolio may not fit in this case. With effort of

engaging KEEP in school curriculum, more assessments and evaluations will be conducted to

improve the development of the project.

20


References

Cantrell, P., & Robinson, M. (2002). How do 4th through 12th grade science textbooks address applications in engineering and technology? Bulletin of Science, Technology & Society, 22, 31-41. Fleischman, H.L., & Williams, L. (1996). An Introduction to Program Evaluation for Classroom Teachers, Development Associates, Inc, Arlington, VA. Retrieved on August 22, 2007 from http://teacherpathfinder.org/School/Assess/assess.html Frechtling, J., & Sharp, L., & Carey, N., & Vaden-Kiernan, N. (1995). Teacher enhancement programs: A perspective on the last four decades, National Science Foundation, Washington, DC. Jaschik, S. (2005). Lost dominance in Ph.D. production. Inside Higher Ed. Retrieved October 29th, 2006 from http://www.insidehighered.com/news/2005/07/15/science Lumpp, J.K., & Bradley, K.D. (2003). “Development and Dissemination of KEEP- Kentucky Electronics Education Project”. Proceedings of the Elec. Comp. & Tech. Conference. New Orleans, LA. Lumpp, J.K., & Bradley, K.D. (2006). “Math and Science across the Board: Connecting Professional Development to Classroom Practices via an Embedded Research Initiative”. American Society for Engineering Education Annual Conference Program & Proceedings. Chicago, IL. Rudner, L. M. (2001). Item Response Theory, Retrieved on August 22, 2007 from http://edres.org/irt Rutherford, F. J., & Ahlgren, A. (1990). Science for all Americans, Oxford University Press, New York. Shaw, F. (2006). Fits about “Misfit”, Retrieve November 14th, 2006 from http://rasch.org/rmt/rmt51g.htm USDOE. (2000). “Before It’s Too Late: A Report To The Nation From The National Commission on Mathematics and Science Teaching for the 21st Century”. Washington, D. C: United States Department of Education. Waxman, H.C., & Connell, M.L., & Gray, J. (2002). A quantitative synthesis of recent research on the effects of teaching and learning with technology on student outcomes. Naperville, IL:

21

http://teacherpathfinder.org/School/Assess/assess.html

http://edres.org/irt


North Central Regional Education Laboratory. Winsteps Help (2007). Logit and Probit: What are they? Retrieved March 25th, 2007, from http://www.winsteps.com/winman/whatisalogit.htm Wright, B. D., & Masters, G. N. (1982). Rating scale analysis: Rasch measurement. Chicago: Institute for Objective Measurement, Inc.

22


Appendix Survey

I. Students’ Satisfaction of the Project.

1. Copper Cleaning-Preparing the surface

2. Ironing the Pattern onto the Circuit Board

3. Etching the Copper-Creating Black Patterned Lines

4. Drilling Holes

5. Removing Toner

6. Inserting Components

7. Soldering

8. Overall Project

(A: Very unsatisfied, B: Unsatisfied, C: Satisfied, D: Very satisfied)

II. Difficulty Level of the Project.

1. Copper Cleaning-Preparing the surface

2. Ironing the Pattern onto the Circuit Board

3. Etching the Copper-Creating Black Patterned Lines

4. Drilling Holes

5. Removing Toner

6. Inserting Components

7. Soldering

8. Overall Project

(A: Very difficult, B: Difficult, C: Easy, D: Very easy)

23


III. Person Code:

Example:

Person 12 SFBCYY:

S-senior; F-female; BC-the grades she typically receives in classes is B’s and C’s; Y-she would

like to do a project like this again; Y-she has done a project like this before.

24


Tables and Figures

Table 1. Summary of 61 measured students for students’ satisfaction

____________________________________________ _______

Real RMSE 1.01 Separation 1.70 Student reliability 0.74

Model RMSE 0.95 Separation 1.83 Student reliability 0.77

__________________________________________________________________________

Table 2. Summary of 8 measured items for students’ satisfaction

__________________________________________ _________

Real RMSE 0.29 Separation 2.79 Item reliability 0.89

Model RMSE 0.28 Separation 2.91 Item reliability 0.89

____________________________________________________________________________

Figure 1. Construct keymap for students’ satisfaction.

25


Table 3. Item statistics in order of misfit for students’ satisfaction

______________________________________________ ______

Infit Outfit

Item Model measure MNSQ MNSQ Item name

3 0.20 1.56 1.49 Etching the copper_________

Mean 1.00 0.97

S.D. 0.25 0.24

______________________________________________________________________________

26


Figure 2. Variable map of students’ satisfaction of the project (M=mean, S=one standard

deviation, T=two standard deviation).

27


Table 4. Summary of 61 measured students for the difficulty level

____________________________________________________

Real RMSE 0.75 Separation 1.67 Student reliability 0.74

Model RMSE 0.70 Separation 1.84 Student reliability 0.77

____________________________________________________________________________

Table 5. Summary of 8 measured items for the difficulty level

______________________________________ ___________

Real RMSE 0.24 Separation 3.98 Item reliability 0.94

Model RMSE 0.22 Separation 4.23 Item reliability 0.95

_____________________________________________________________________________

Figure 3. Construct keymap for the difficulty level.

28


Table 6. Item statistics in order of misfit for the difficulty level

_____________________________________________ _______

Infit Outfit

Item Model measure MNSQ MNSQ Item name

3 -0.65 1.51 1.42 Etching the copper________

Mean 0.99 1.00

S.D. 0.34 0.32

______________________________________________________________________________

29


Figure 4. Variable map of the difficulty level (M=mean, S=one standard deviation, T=two

standard deviation).

30

Documents

Applying the Rasch Model in KEEP - 1