TRC212: An Instructional Product Evaluation · 2018-09-20 · Michael Puckett Instructional Product Evaluation Page 4 are plans in development to perform another evaluation incorporating

EDCI 57700-002 TRC212: Instructional Product Evaluation Fall 2017

TRC212: An Instructional Product Evaluation

Michael Puckett EDCI 577-002

Fall 2017

Michael Puckett Instructional Product Evaluation Page 2

Instructional Product Overview: TRC212 At 211 degrees, water is hot. At 212 degrees, water boils and creates STEAM. In 2017, Tennessee Robotics Club (TRC) launched a new online program called TRC212 with the goal of developing a series of online E-Learning courses focusing on STEAM topics: Science, Technology, Engineering, Art/Design, and Mathematics. Tennessee Robotics Club (TRC) is a local FIRST Tech Challenge team where team members in grades 9-12 explore real-world engineering challenges by building robots to complete mission tasks on a challenging playing surface. TRC Team members are encouraged to work together solving complex engineering challenges, building robots using engineering design principles and exploring exciting career fields in STEM. TRC team coaches are interested in learning how team members react to the online learning module and how effectively the first module meets the learning objectives. After the evaluation results are compiled and presented to the TRC team manager, improvements can be implemented in the design of future learning modules in the TRC212 learning program. If the program is successful, the TRC212 program will decrease the amount of time spent on fundamental training during meeting times.

The TRC Online Engineering Journal is the first module in the TRC212 learning program and will be used to evaluate the reaction and effectiveness of this type of training. This first module is designed to teach the process of submitting an online engineering notebook entry following the FIRST Tech Challenge Engineering guidelines. The learning module is located at http://tennesseeroboticsclub.org/engineering-notebook/. Table 1 provides a module overview of the section and content in the Engineering Journal Entry module which will be used for the instructional product evaluation. Module Section Content Let the Adventure Begin Learning Game Activity/ Kickoff Video Section 0: Course Introduction Course Instructions/ Learning Challenge 0 Section 1: Access Engineering Journal Interactive Content/ Learning Challenge 1 Section 2: Create a New Journal Entry Interactive Content/ Learning Challenge 2 Section 3: Enter Meeting Details Interactive Content/ Learning Challenge 3 Section 4: Enter Reflection and Media Interactive Content/ Learning Challenge 4 Section 5: Review, Save and Publish Interactive Content/ Learning Challenge 5 Learning Assessment Final Assessment Table 1: Overview of Module Contents


Audience and Context The intended audience for the learning module includes both new and existing new team members, coaches, and mentors of the Tennessee Robotics Club. TRC team members are between the ages of 14-18 and have at least 1 year of experience writing journal entries. Many of the team members have 6 or more years of FIRST robotics experience.

Each season beginning in May, TRC team members begin documenting team meeting details and personal reflections using an Engineering Journal documenting their team meetings and design process during the season. In robotics, teams face many challenges and discover many new lessons learned throughout the season. Engineering Journals provide a valuable tool for teams to share information with competition judges about their team’s journey and experience in the FIRST program.

Team members and coaches will be sent an email with the link to the training module. The email contains the instructions and the password to login. No prior knowledge of engineering or robotics is required for the training, only basic writing and basic computer skills to successfully complete the course. Having a good understanding of the FIRST Tech Challenge Engineering Notebook guidelines beforehand will help set the context of the FIRST Tech Challenge program. This will be available in the course introduction for participants to download and review in PDF format.

The training can be completed from anywhere the team member has access to a computer with an Internet connection. Ideally, the training will be completed in one session which should take about 30 minutes to complete. The final assessment will take about 15 minutes to complete. The training needs to be completed within the first week of the season so that journal entries can begin immediately at the completion of the first meeting. Team members will need to have the following technology to complete the TRC Engineering Journal Entry training module: Computer/ Device PC, Laptop, Android Tablet or iPad Internet Internet Access Browser(s) Internet Explorer 9.0 or higher, Google

Chrome, Safari, Firefox PDF Adobe PDF Viewer

Table 2: Technology Requirements

Measurement Instruments and Data Collection This evaluation will follow the guidelines for Level 1 and Level 2 evaluations established by the Kirkpatrick Four Level Evaluation model (Kirkpatrick & Kirkpatrick, 2006). There


are plans in development to perform another evaluation incorporating Levels 3 and 4 to measure behavior and results of the training program.

Level 1- Reaction Evaluation The first step of a successful course evaluation begins with the learner’s reaction to the training program. “Evaluating reaction is the same thing as measuring customer satisfaction” (Kirkpatrick & Kirkpatrick, 2006, p. 27). This is an important first step in the evaluation process because understanding how the learner reacted to the training program will help determine if learning is likely to occur. Ideally, a favorable reaction to the training will result in a positive motivation to learn. Secondly, gathering feedback from the learner about the training experience will help improve the program for future participants which is important in developing future modules in the TRC212 program.

Evaluation Instrument After the training is complete and the Final Assessment results in a passing score of 80%, team members and coaches will receive an email from the TRC team manager to complete an anonymous course survey online. The survey will provide the participant with the option of entering their name for follow-up, otherwise, no personal identifying information will be collected. This will help achieve honest feedback for the survey results. The reaction survey will be created using Google Forms. The reaction survey will consist of 10 questions to measure learner reaction to the course content, design and learning technology. The reaction data will be compiled into a spreadsheet form and stored on the TRC Google Drive for future reference. The numerical data will also be tabulated to create a chart to report the results of the survey in a visual form. Please see Appendix A: TRC212- Engineering Journal Entry Reaction Survey to view the Google form in its entirety. The Google Form survey will rate the responses on a Likert scale of 1 to 5 with 1 representing “Strongly Disagree” and 5 representing “Strong Agree” and varying levels in between. Each question will include a space for learners to enter additional comments to help clarify their response selection. The last question in the survey will provide a comment section for the learner to make any suggestions about the training that were not asked in the previous survey questions. The survey is short and should take about 10 minutes to complete.

Level 2- Learning Evaluation The second step in the evaluation is to measure learning which is comprised of skills, knowledge and attitudes. As Kirkpatrick (2006) points out, unless the learning objectives were met, then no change in behavior can be expected (p.42). Ideally, a


control group would be used to help measure gains in learning but since the team is small consisting of only 5 members and three coaches, the control group would not be possible. The training process is new for all team members so there is no need or benefit in doing a pre-test. Therefore, the primary evaluation instruments will consist of a post-test to measure knowledge and attitudes and a performance test to measure skills. These results will help measure what was knowledge was learned and what skills were developed.

Evaluation Instrument- Post Test Learning will be evaluated in two steps. The first step will be accomplished by a final assessment to measure knowledge learned. Each section contains a learning challenge which must be completed before moving on to the next section. The learning challenges reinforce the content for the Final Assessment. See Appendix B: TRC212- Engineering Journal Entry Learning Challenge for an example.

Evaluation Procedure- Post Test After completing each of the sections successfully, the learner will then be presented with a final assessment of 10 questions to successfully complete the training module. The final assessment will be administered online at the end of the final section. Learners must pass the test with a score of 80% or higher to receive credit for the course. Please see Appendix C: TRC212- Engineering Journal Entry Final Assessment for detailed questions and answers.

Evaluation Instrument- Performance Test The second way the learner’s knowledge will be measured is by submitting an actual journal entry for evaluation by the TRC team manager. This is an important step as it provides an opportunity to receive constructive feedback on the Engineering Journal.

Evaluation Procedure- Performance Test Upon completion of the final assessment, the team member will submit a new online engineering journal entry for evaluation. The TRC Team Manager will review the journal entry and check for accuracy and completeness of the following areas from the training sections: meeting details, reflection and media submission. Please see Appendix D: TRC212- Engineering Journal Entry Performance Test Checklist to view the performance objectives.


Reporting Process Upon completion of the reaction survey, the final assessment and the performance test, the evaluation data will be collected and tabulated into a series of charts by the TRC Team Manager and the course evaluator. The evaluation results will be presented to the TRC Team Manager at the completion of the evaluation. The reaction results will be presented in a graph. All comments will be consolidated to one page for review. Reporting the results of the reaction data will be completed by compiling the results of the survey on a 5-point scale and averaging them out over the number of completed surveys. On a scale from 1 to 5, an average result of 4 or higher would indicate a positive learner experience with the program. If the results are lower than a 4 in one of the three evaluation areas: course content, materials and platform, that would indicate an area for improvement and the TRC Team Manager can improve that section before developing future TRC212 modules. See Appendix E: TRC 212- Example Reaction Data Results for an example chart displaying reactions in the three areas. Analysis of the learning data will be completed by listing the test scores of each participant. Each section of the module contains a non-graded learning challenge but serve as learning checkpoints of understanding before a learner can continue. All sections of the module will be combined into a single final assessment. Since each question is worth one point, a passing score of 80% or higher is necessary to complete the training and would indicate that the learner did achieve a basic understanding of the material. This test score would also indicate that the team member has the basic skills required to successfully complete an engineering journal entry for the performance test. The performance checklist would measure the learning that has taken place after the training is completed. See Appendix F: TRC 212- Example Learning Data Results for an example chart displaying the performance results from the checklist. Using the results of the reaction surveys and learning test scores, the TRC Team Manager will be able to effectively evaluate the course module effectiveness and improve upon the course design for future modules in the TRC212 online program.


References Kirkpatrick, D.L. & Kirkpatrick, J.D. (2006). Evaluating training programs: The four levels (3rd ed.). San Francisco, CA: Barrett-Koehler Publishers, Inc.

Appendices Appendix A: TRC212- Engineering Journal Entry Reaction Survey Appendix B: TRC212- Engineering Journal Entry Learning Challenge Appendix C: TRC212- Engineering Journal Entry Final Assessment Appendix D: TRC212- Engineering Journal Entry Performance Test Checklist Appendix E: TRC212- Example Reaction Data Results Appendix F: TRC212- Example Learning Data Results


Appendix A: TRC212- Engineering Journal Entry Reaction Survey

Please enter your name if you would like to be contacted about your comments to help improve future training programs. Otherwise, your name is not required and the comments you enter will be anonymous. Participant Name (Optional): ___________________________________ Survey Instructions: This survey is designed to collect reactions and comments in three areas of the training program: course content, course format and technology. Please answer each question by circling the number next to your response.

1- Strongly Disagree 2- Disagree 3- Neutral 4- Agree 5- Strongly Agree

Comments: Please enter comments to make recommendations where applicable.

Reaction Area 1: Rate your reaction to the course content:

1. I found the course content to be relevant and engaging. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:

2. The material presented in each section was organized clearly and easy to understand. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:

3. I feel that I can successfully complete an Engineering Journal entry on my own. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:


Reaction Area 2: Rate your reaction to the course design

4. The guided walk-throughs were easy to follow and understand. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:

5. The Learning Challenges reinforced the content from each section for the final assessment. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:

6. The final assessment tested me on the knowledge presented in each section. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:

Reaction Area 3: Rate your reaction to the course technology:

7. The online learning module was easy to use and navigate throughout the course. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:

8. The course navigation and features worked properly in each section. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:


9. The videos and images properly displayed without delay. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:

Reaction Area 4: Overall Learner Satisfaction:

10. Overall, I am satisfied with this training course and it met my learning needs. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:


Appendix B: TRC212- Engineering Journal Entry Learning Challenge Example


Appendix C: TRC212- Engineering Journal Entry Final Assessment

Okay, you have made it to the end. Now let's check what we have learned in this training module. There are 10 questions covering the steps required to successfully enter an engineering journal entry. You must pass with an 80% score to move on. After completing the final learning assessment, you will receive instructions by email to complete a brief reaction survey. Good luck! START QUIZ

1. Select the name of the link to add a photo, drawing or code snippet to your journal entry:

a. Add New b. Add Media c. Add Form d. Preview

2. What is the correct Windows keyboard shortcut to paste content? a. Ctrl-C b. Ctrl-S c. Ctrl-V d. Crtl-P

3. Identify the correct source to find your username and password to login. a. Welcome Packet b. trc.com c. WordPress Registration Email d. FTC Website

4. What is the correct Windows keyboard shortcut to select and copy content? a. Ctrl-S and Ctrl-V b. Ctrl-C and Ctrl-V c. Ctrl-S and Ctrl-C d. Ctrl-A and Ctrl-C

5. Select the correct username format from the list below. a. mpuckett b. tnroboticsclub c. admin d. [email protected]


6. If a Team Member completes a rough sketch of the chassis design, what type of media is best for this reflection?

a. Photo b. Drawing c. Code Snippet d. Not Required

7. Identify the correct url to navigate to the TRC website to bring you to the login screen.

a. Trc.com b. Tnroboticsclub.com c. Tennesseeroboticsclub.org d. Tennesseeroboticsclub.org/wp-admin

8. Choose the task below that best describes the task goal. Remember be

SMART! a. Task 1: Build robot chassis b. Task 1: Build robot c. Task 1: Vivamus in diam turpis d. Task 1: Build robot chassis subassembly for battery mount by end of October

9. If a Team Member completes an update to the program code, what type of media is best for this reflection?

a. Photo b. Code Snippet c. Drawing

10. What is the final step to make your post live and viewable online in the Engineering Journal?

a. Save Draft b. Move to Trash c. Publish d. Preview


Final Assessment Answer Key:

1. a) Add Media 2. c) Ctrl-V 3. c) WordPress Registration Email 4. d) Ctrl-A and Ctrl-C 5. a) mpuckett 6. b) Drawing 7. d) Tennesseeroboticsclub.org/wp-admin 8. d) Task 1: Build robot chassis subassembly for battery mount by end of October 9. b) Code Snippet 10. c) Publish


Appendix D: TRC212- Engineering Journal Entry Performance Test Checklist

Use the following checklist to evaluate the completion of the performance test. This evaluation will critique the following three areas: Meeting Details and Tasks, Journal Reflection, and Media Submission. Meeting Details

1. The Meeting Details and Tasks section was entered correctly and accurately. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments: Reflection

2. The journal reflection was concise and well written providing a good summary of the meeting events. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments: Media Selection

3. The journal reflection included a media submission that accurately represented the journal reflection. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments: Completeness

4. The engineering journal entry was entered according to the training guidelines. 1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree

Comments:


Appendix E: TRC 212- Example Reaction Data Results

Appendix F: TRC 212- Example Learning Data Results

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Content Design Technology Satisfaction

Reaction Data Results

Rating Scale

3.8

4

4.2

4.4

4.6

4.8

5

Meeting Details Reflection Media Completeness

Learning Data Results

Rating Scale

Documents

TRC212: An Instructional Product Evaluation · 2018-09-20 · Michael Puckett Instructional Product Evaluation Page 4 are plans in development to perform another evaluation incorporating