Part 6. Formative Evaluation Plan

6a. Expert Review The subject matter expert (SME) for this project is Sherilyn Skiba. Sheri was chosen

because she has been teaching AP Statistics for ten years at Eastlake High School in

Libertys neighboring school district, Lake Washington SD. Sheri had already been used

as a resource for Libertys start up AP Statistics Course to help determine which

textbook would be the best choice. Sheri Skiba was contacted on July 25, 2013 and

responded back almost immediately. The survey associated with the interview of the

SME can be found within section 7a.

6b. One-to-One Evaluation This evaluation process involves testing out the project materials with several students before the project is actually assigned to a complete classroom. During the one-to-one evaluation process, there are specific things the designer is looking for such that the project is improved before actual administration. The designer is mainly seeking feedback regarding formatting and grammar errors. In addition, the designer needs information about the flow of the project. The designer may wish to ask clarifying questions of the student, thus there is designer and student interaction. Inquiry about the following would be essential feedback from each student. To obtain that feedback, the student would be presented with both a computer version and a paper copy of the ID project such that they could write comments. Three or four students would be chosen to complete the one-to-one evaluation process. Selection of a variety of students would be ideal. Each student would be interviewed individually so that previous student comments do not influence other student comments (Smith, 2005).

1. Identify any instructions that are not clear to follow such that you can obtain a result without guidance.

2. Locate and identify any misspelled words. 3. Will these instructions lead you to designing a box plot without assistance? If not,

please indicate which step needs clarification. 4. If you need instructor assistance to make a box plot, at which point in the

instructions does this occur? 5. When looking at the visuals provided in the project, do you understand what they

all mean? Please list any visuals that do not appear relevant. 6. Please identify any links that do not go to the expected location.

2 | E d T e c h 5 0 3 : I n s t r u c t i o n a l D e s i g n P r o j e c t

6c. Small Group Evaluation Throughout this evaluation stage of the ID project, the designer will have a small group of students, or learners, attempt completion of the project. At this time, all revisions should have been completed based upon the results of the one-to-one evaluation. During the small group evaluation process, the designer should refrain from interacting with the student evaluators. However, if none of the students in the group can make sense of a step in the project, then the designer may clarify. Likewise the designer should make note that the step in question needs revision (Smith, 2005).

During this evaluation process, four students would be selected for analysis of the AP Stats ID project. Students would work in a group of four to see if they could complete the activity while the designer observes and only interacts if no one in the group can make sense of a particular instruction. At that time, students should be instructed to call over the designer if no one in the group can determine the intention of the instructions.

For this particular AP Stats project, essential questions to refer to the students include: 1. Does the flowchart lead you to a box plot that you can make with paper and

pencil? If not, which steps are confusing? 2. Does the video for the calculator instructions lead you to a box plot result? 3. Does the video for the excel instructions lead you to a box plot result? 4. Which steps were unclear to more than one person in the group? 5. How should that instruction change to become more clear? 6. Did all students in the group have the skills necessary to complete the project? If

not, then what was the cause of the issue that didn't allow student completion? 7. Will these instructions lead you to designing a box plot without assistance? If not,

please indicate which step needs clarification. 8. Please identify any links that do not go to the expected location.

6d. Field Trial The field trial process is essential to determine if all revisions from the one-to-one and small group evaluations were corrected to achieve a positive result. Throughout the field trial, the designer should use a larger number of students to conduct the trial. At this time the project should actually be presented in the environment where regular instruction occurs (Smith, 2005).

1. Will these instructions lead you to designing a box plot without assistance? If not, please indicate which step needs clarification.

2. Please identify any links that do not go to the expected location. 3. Are all the students successful in all objective categories? 4. Does the video for the calculator instructions lead you to a box plot result? 5. Does the video for the excel instructions lead you to a box plot result? 6. How long did it take each student to complete the activity? 7. Is there anything the instructor noticed that need revision? 8. Please add any additional comments about this project.

3 | E d T e c h 5 0 3 : I n s t r u c t i o n a l D e s i g n P r o j e c t

Part 7. Formative Evaluation Report

7a. Evaluation Survey or Rubric The questions were designed such that a yes response means the SME agrees with the designer about that particular aspect of the project and determines success. The point scale then determines ID Project success with a higher score. The SME will be asked to rate each response on a scale of 1-5 as shown below.

1 no, I completely disagree

2 somewhat disagree

3 unsure

4 somewhat agree

5 yes, agreed

The following are the list of questions that were presented to the SME, Sheri Skiba.

1. Are these skills AP Statistic students need to have? 2. Are the examples and practice exercises relevant to this course? 3. Is the instruction provided sufficient for students to learn this topic in an online

venue? 4. Do you believe the instructional tools are consistent with how an AP Statistics

student can learn? If you do not answer a 5 for any particular question, then can you please comment on how improvement can be made?

Question Rating/Comments

1. Are these skills AP Statistic students need to have?

2. Are the examples and practice exercises relevant to this course?

3. Is the instruction provided sufficient for students to learn this topic in an online venue?

4. Do you believe the instructional tools are consistent with how an AP Statistics student can learn?

4 | E d T e c h 5 0 3 : I n s t r u c t i o n a l D e s i g n P r o j e c t

7b. Expert Review Report

Question Rating/Comments

1. Are these skills AP Statistic students need to have?

4/ The actual creating of the graphical display is a very small part of the purpose of this topic. The AP exam focuses much more on the analysis. Box plots are particularly useful when comparing two sets of data and are rarely used in isolation. Often on the AP Exam students are actually given several box plots and expected to be able to discuss and compare spread, center, shape and outliers. They should be able to discuss in detail what the quartiles represent and use that information to compare sets of data. They should also know how to create a modified box plot which clearly shows outliers. The Excel and hand methods are enrichment activities as students will most likely create the box plots with a calculator on the exam.

2. Are the examples and practice exercises relevant to this course?

4/ As noted in the comment above, the focus should not be on the mechanics of creating the graphical display but on the analysis. AP students are very quick to be able to create the box plots with any of the methods, so I would suggest that you use at least two sets of data such different seasons or different teams so they focus on the important analysis skills. It is also important for them to know when it is best to use box plots and when a different graphical display would be more appropriate.

3. Is the instruction provided sufficient for students to learn this topic in an online venue?

5

4. Do you believe the instructional tools are consistent with how an AP Statistics student can learn?

4\ I think spending this much time on mechanics isnt necessary.

7c. Designer's Response of Review Based on the above feedback from the SME, it is good to know that the Instructor Guide

was successful in clearly identifying the steps needed to complete the activity. The SME

review however, clearly noted that this much time on such a small aspect of the course

is not necessary. Moreover, it may not have been clear to the SME that this activity is

intended to occur at the beginning of the AP Statistics course as perhaps a pre-

assessment activity to prepare students for the statistics course. Another use of this

activity could be for remediation for any student that has not learned about box plots in

other courses or has forgotten how a box plot is created. Clear knowledge of the

creation and analysis of a box plot is necessary for future activities in AP Statistics such

that students use these ideas for statistical inference later in the course.