Evaluation of Online InstructionET 755Zeta: Shelby Simmons, Sarah Tolson, Lindsay Tucker

Evaluation 1Site: MIT Open Courseware

Course Title: Combinatorics: The Fine Art of Counting

Evaluators: Shelby Simmons, Sarah Tolson, & Lindsay Tucker

Course URL: http://ocw.mit.edu/high-school/mathematics/combinatorics-the-fine-art-of-counting/index.htm#

Introduction:

The first course we evaluated was entitled “Combinatorics: The Fine Art of Counting” and was presented by Andrew Sutherland. Combinatorics is defined as “a fascinating branch of mathematics that applies to problems ranging from card games to quantum physics to the Internet.” This course was designed for the High School Studies Program (HSSP) from MIT, which offers “non-credit, enrichment courses to 7th–12th grade students on Sundays at MIT.” The course is designed to be engaging and appeal to students’ interests, as well as preparing students for math contests and competitions.

Sutherland, Andrew. Combinatorics: The Fine Art of Counting, Summer 2007. (MIT OpenCourseWare: Massachusetts Institute of Technology), http://ocw.mit.edu/high-school/mathematics/combinatorics-the-fine-art-of-counting (Accessed 10 Jun, 2014). License: Creative Commons BY-NC-SA

1. Course Overview and IntroductionStrengths: The intentions of the course are made clear through the syllabus and introduction from the instructor. The syllabus and instructor made the structure of the course clear, outlining each week’s lecture, providing notes, and a relevant homework assignment. The syllabus clearly states that students must have prerequisite knowledge in the area of algebra, and encourages a love for mathematics. The instructor provides a self-introduction that was appropriate and available online, which emphasized his love for Combinatorics and why/how he teaches the course.

Weaknesses: There are no explicit instructions on how to follow the course online, since lectures were offered in person. However, things are clearly labeled and accessible (including lecture notes). Not much of this course seemed to focus on the online instruction; rather it seemed to use the internet as a method of communication and sending documents between the instructor and students. There were no evident “netiquette” expectations listed, nor were there clear course and institutional policies. There is no evidence that students were introducing themselves to each other online, but it may have happened during the introductory session from Week 1. The information that is accessible online does not discuss minimum technical

skills needed by students, nor does it discuss many syllabus elements: course materials (implies that course notes may be all), assessments (introduction video says there are problems, syllabus states no grading, but he still needs to assess how well his students are grasping material—that information is not clear), nor does it provide clear information about the feedback provided by the instructor (are homework assignments due to him? Will he provide feedback individually? How? When?)

2. Learning ObjectivesStrengths: The instructor does cover what the course is about briefly in his introduction video, which could be seen as objectives for the course. The syllabus also states that the course will prepare students for mathematical courses.

Weaknesses: Learning objectives for the course are not clearly stated, so we cannot determine whether they would be measurable or consistent with course-level objectives. Even though this course is catered toward younger students and does not supply grades, it would benefit from having clear, measurable objectives for the course and each meeting, rather than simply a list of topics that will be covered. Since there are no objectives stated, it is impossible for there to be clear instructions on how to meet the requirements and expectations for the course.

3. Assessment and MeasurementStrengths: The homework assignments are the only assessment that are accessible online. The assignments seem to align with what is covered in each lecture/class and are consistent with the course activities. The homework is offered as a menu, allowing students to choose which “selections” they would like (what problems they would like to answer.) These homework assignments get more in-depth and reinforce concepts that have been covered in the lecture/lecture notes, as well as encouraging students

Weaknesses:Since there are not clear learning objectives, we cannot determine whether the assignments align with and measure the objectives. There is no grading policy for this course, but a rubric, or another tool would probably be beneficial as feedback for students. There is no criterion for assessing the work of or participation of students, and there is no evidence of opportunity for self-reflection.

to use high order thinking skills. Many of the problems offered in these assignments are relevant to students, such as determining odds for games (dice, cards, etc.), game shows (Monty Hall – The Price is Right), probability, or simply challenging questions.

4. Instructional MaterialsStrengths: The instructional materials provided online for this course are the lecture notes, which are extensive. They cover the topics as listed on the schedule. All of the instructional materials are optional as the course is not graded; however, it is clear that many of them need to be completed in order to understand the following week’s lecture and additional material to be covered. Instruction materials available online (everything but the live lecture itself) can be easily accessed for students by opening and printing (if desired) a PDF document. Throughout many of the lectures Sutherland refers to different perspectives and tries to explain some of the material in laymen’s terms.

Weaknesses: Without clear learning objectives it is impossible to determine if the materials contribute to the achievement of the objectives. Citations are not provided on the lecture notes or the homework problems. The problems could have been created by Sutherland himself, but more than likely he at minimum referred to additional texts that were not referenced to create his lectures and/or problem sets. It is explained that as students work on problem sets (activities) they should refer to their notes (instructional material), which is clearly explained; however, there is no alternative way to understanding the content via a different avenue if a student didn’t quite grasp the concept.

5. Learner Interaction and EngagementStrengths: One way that the instructor engaged the students in the course content (homework problems) was writing them as a menu. Students were able for instance for one assignment to choose an appetizer, salad, entrée, and dessert option. For another assignment that had to choose a couple of different items from the tapas “menu”. It is a creative way to engage students! There is also a competition on the final day of class where students take part in answering actual contest questions. If nothing else, most students love competition!

Weaknesses: The learning activities (homework sets) provided to students cannot promote achievement of unstated learning objectives. They are still linked to what was covered in class as well as the topic (not objective) stated on the schedule. In one of the introductory videos Sutherland discusses in class activities where students work in groups, but it is never referenced in the material provided online. It is not indicated for the course whether individual feedback was provided to students, the solutions to all of the homework sets and one class problem set were provided. No student-to-student or faculty-to-student interaction is required, facilitated, or

encouraged in any way. The recorded videos allow for students to be introduced to the instructor, but that is not an actual interaction.

6. Course Navigation and TechnologyStrengths: The course was easy to navigate using the tabs located on the left-hand side. All the links were active and all PDF files were able to be opened. The navigation was logical, consistent, and efficient.

Weaknesses: The technology necessary for this class was to have access to the internet; no other technology was used or encouraged. Technology was not used to engage students, only to allow access to information.

7. Learner SupportStrengths: Course appears to be designed to provide extensive and useful supplement to an in-person or video based course and is an excellent resource under those conditions.

Weaknesses: The course is not for credit and does not provide grades. Students are able to read solutions and self-assess, however there is no support available from the instructor outside of links to related materials (such as on Wikipedia). The course does not specify technical or academic support to learners to promote success.Actual lectures are not included online. No direct communication with an instructor or facilitator is available.

8. AccessibilityStrengths: The actual homework assignments and solutions are in PDF format and a quick check in Adobe Acrobat indicates that the PDF is accessible. The lecture notes include intuitive naming for the files and are listed in sequence. Because images are not used there are no issues with alt text for screen readers (excluding issues with link text). Most website accessibility issues are minor.

Weaknesses: Meet the author videos have clear audio and answer useful common questions, however they do not feature captions, and no transcript is available.Large file sizes for downloads, but are appropriately labeled with file type and size, although they do not contain descriptive text for links.Videos appear in pop-up boxes over main page text and do not auto-start which can cause difficulty for persons who do not have the physical dexterity to easily start the video due to involuntary movement or poor fine motor skills and those who are using a screen reader.A WebAIM WAVE evaluation of the lecture notes page found: 2 Errors, 12 Alerts, 9 Features, 43 Structural Elements, 5 HTML5 and ARIA, and 11 Contrast Errors of concern. Most of

these are minor, but reduce accessibility for some students. The actual homework assignments are in PDF format and a quick check in Adobe Acrobat indicates that the PDF is accessible. However, the PDF does not include any images that could assist students with reading disabilities. The lecture notes are the same, however they do include intuitive naming for the files but do not include file size.There is only text representation of content, not multiple representations as called for by UDL principles.

9. Course AdministrationStrengths: For students who attended the original or a similar course the materials are useful and the organization and pacing would be appropriate.

Weaknesses: Due to the design and purpose of the course, there is no gradebook or scoring system. The recommended schedule likely reflects a campus-based course and timeline.

Summary:

The instructor has provided a series of self-study experiences in combinatorics for the mathematically inclined. No lecture is available and the course is designed to assist students who are confident in mathematics and are in search of additional problems to work in preparation for competition or to interest students who are more advanced than the typical high school student (as noted in the instructor’s videos). There are formative self-assessments as homework in the course, however there is no system for assigning a score, grade, or credit for the course.

There is extensive evidence that the course content and environment is not designed for students who have cognitive or physical disabilities. There are low levels of possible engagement and little adherence to Universal Design for Learning principles of multiple representation. The course has little to offer the casual mathematician, but would be helpful in the stated context of competition or advanced practice.

Reflection:

This course on Combinatorics: The Fine Art of Counting is designed, as stated on the syllabus, to be an enrichment course for high school students who are bored with their math classes. The syllabus also states that only basic algebra skills are required as a pre-requisite; however, based on the lectures and homework problems assigned, students must also possess a desire to think critically and understand and apply more challenging mathematical concepts. Sutherland has put a lot of work into the creation

of the materials for this course, but the materials provided online are a supplement to the in class lectures. It is truly unfair to critique this course as if it were strictly online instruction. Regardless, there were still many strengths and weakness that we were able to discuss that should be common practice in both the classroom and online environments. One imperative piece of information that was missing was the learning objectives for the course as a whole. The topics that would be covered were listed, but there were no objectives to determine either the purpose of the course or what the students should be able to know or do by the end of the course. There was also not much student engagement supported by the lesson activities and materials nor was there any type of student-to-student or student-to-teacher interaction that was required or encouraged.

The overall idea of this course was to give advanced mathematical students a class to take for fun without having to worry about receiving a grade. At the same time the course helps students prepare for talent search and mathematical competition examinations. Sutherland managed the course well considering he was most likely teaching students who love math and want to work challenging problems for the fun of it. For those of us who live in the real world teaching students who are not all advanced mathematicians, it was easy to pick apart the course and find its faults. We do give Sutherland credit for his innovative and creative homework assignments written in menu style where students are able to make selections of problems as you would a menu item.

Evaluation 2Site: MIT Open Courseware

Course Title: Game Design

Evaluators: Shelby Simmons, Sarah Tolson, & Lindsay Tucker

Course URL: http://ocw.mit.edu/courses/comparative-media-studies/cms-608-game-design-fall-2010/

Introduction:

The course we evaluated was Game Design by Massachusetts Institute of Technology’s online open courseware website.  The course was presented by Philip Tan and Jason Begy, as course CMS.608 for undergraduate students and course CMS.864 for graduate students. The course focused on the design and analysis of non-digital games. Students learn the different genres and aspects of games and create their own games. A major emphasis of this course is on the evolution of game rules.

Tan, Philip, and Jason Begy. CMS.608 Game Design, Fall 2010. (MIT OpenCourseWare: Massachusetts Institute of Technology),http://ocw.mit.edu/courses/comparative-media-studies/cms-608-game-design-fall-2010 (Accessed 10 Jun, 2014). License: Creative Commons BY-NC-SA

1. Course Overview and IntroductionStrengths: The purpose of the course is stated in the syllabus: “to understand the interaction and evolution of game rules.” The syllabus briefly introduces students to the structure of the course, including two weekly lectures and a weekly lab. It also discusses how student grades will be computed. The syllabus is clear, although concise, and could use more elaboration. However, grades will be determined through a combination of student projects, class participation, and forum discussion. As discovered through the lectures, much communication was done through email and is no longer accessible—we cannot access nor visualize any resources they mention.

Weaknesses: Instructions are vague; it is unclear on how to begin the course. This may be covered during the first session of the course, but the audio recording is not available. “Netiquette” is not clearly stated, but the syllabus does mention that students will work together in teams and may be “docked for antisocial or disruptive behavior.” There is no link to current policies of the course, nor the school. Prerequisite knowledge and competencies are not stated, other than students are expected to demonstrate a proficiency in English. The course also does not discuss what technological skills and competencies students must have. There is no introduction from the facilitator, nor is there an introductory discussion board post or other evidence of students getting to know each other online. There is evidence that students could access a discussion board, but it is no longer accessible. Many course

materials are not elaborated on, nor are they available currently—some of the weekly meeting sessions have been recorded and some assignments are still accessible. There is nothing mentioned about feedback from the instructor.

3. Learning ObjectivesStrengths: Objectives are posted on the assignments page to accompany each assignment. The objectives are measureable. Directions are fairly clear and help students understand how to meet requirements for each assignment, but could be supported with some examples.

Weaknesses: Objectives for the course are not clearly stated, but some are entwined in directions to the assignments. Objectives should be made more clear and written from the students’ perspective. Cannot determine whether or not objectives are appropriately designed for the level of this course.

5. Assessment and MeasurementStrengths: The assignments that are presented, suggested from the course lectures, align with what students learned through their course readings and lecture sessions. The assignments seem to get progressively more in depth, requiring more design knowledge and responsiveness to feedback and issues that arise as students move through the course. The assignments can be work-based and relevant, depending on the route a team chose to take when designing their game.

Weaknesses: The only assessments that are accessible through the online materials provided include the three team-based project assignments to create games. If there were any quizzes, etc., they are not accessible from this site. The syllabus states that part of the students’ grade will come from their discussion posts, but there is no apparent rubric for these posts, nor are they currently accessible. We cannot determine whether or not the assignments truly assess the course objectives, since the course objectives are not explicitly stated. The course grading policy is mentioned, but is not explicit—it lacks a rubric, or similar tool, to determine exact grades. The only information available on grading is the break down that each assignment counts for 25% of a students’ grade. In turn, it says that 5% comes from team work, 10% from the design process, and 10% from responsiveness from feedback. There is no evidence that students could self-reflect about their learning.

7. Instructional MaterialsStrengths: Weaknesses:

The instructional materials (audio lectures, course text, supplemental texts, and games played in lab) contribute to the achievement of the learning objectives for the three major assignments given. All of the resources and materials used are cited and permission given for audio recordings used for guest lectures. Much of the instruction material is current, many older games are used, but that is part of the nature of the course. The instructional materials provided allow for multiple means of representation for students. They are allowed to listen to lectures, read text, as well as participate in the playing and designing of games.

Although it can be inferred, it was not clearly stated how the instructional materials are to be used for learning. It does not appear that any of the reading material or designing assignments are optional and that everything is required, including attendance to class and lab. It does say that students “should” and are “encouraged” to participate in the forum discussions, but it also says that they are “expected” to do so. This provides some confusion as to whether or not it an actual requirement of the course. The articles required for reading are linked and easily accessible, but it appears that many of the games must be purchased. It is quite possible that they are available for use in a lab setting, but as it is laid out on the available course, it seems like a lot of money would need to be spent on games that the students might only use once.

9. Learner Interaction and EngagementStrengths: The learning activities where students work in groups to design their own card and board games promote the achievement of the objectives stated for the activities. There is of course interaction between students to support active learning when they are playing the games in the lab. It is also stated that they should participate in a discussion forum about the text and group projects that they are working on. These tools are not provided on the course website, but they are required, so must be available to the enrolled students. Student-to-Student interaction is encouraged in a variety of ways including the forum, playing games with one another in lab, and being part of a team to create new games. Many different learning styles are addressed including visual, textual, auditory, and kinesthetic.

Weaknesses: There is reference in the syllabus to instructor feedback, but there is no plan stated for how that feedback will happen or the response time. There is a definite indication that students must interact; however, it is not clearly articulated (other than a weight of a final grade) as to how that will be assessed or the specific requirements.

11.Course Navigation and Technology

Strengths: Course navigation and content functions as expected. Students have access to the texts they need as well as audio recordings of lectures. Navigation is also consistent, logical, and efficient. Students have access to previous student work so that they can have prior examples to follow while designing their own games.

Weaknesses: There are not many opportunities available on the course website for students to be an active online learner. There is reference to a forum for students to write on, but it is not available for us to view.

13.Learner SupportStrengths: Site FAQ provides some technical support to learners.

Weaknesses: The course instructions and supporting documentation do not list clear academic or technical support, although some technical information is listed in the sites’ FAQ information. The instructor is not actively monitoring the course, there is no provision for the collaboration required for success in the course. Students would be unable to self-assess.

15.AccessibilityStrengths: Links open in current tab, most issues in WebAIM Wave accessibility evaluator are minor. Course materials that do not have to be purchased are in accessible formats with intuitive links. Captions/transcripts are provided for audio lectures.

Weaknesses: Course materials require access to Amazon or some other resource to purchase them. No explicit information on accessibility within course. Length of pages and detail may contribute to excessive cognitive load for some students. Proximity of some links may be difficult for students with visual or motor skill disabilities.WebAIM Wave Results (2 pages) Assignments: 3 Errors, 32 Alerts, 9 Features, 49 Structural Elements, 5 HTML5 and ARIA, 48 Contrast Errors. Study Materials: 3 Errors, 54 Alerts, 59 Features, 48 Structural Elements, 5 HTML5 and ARIA, 48 Contrast Errors.

17.Course AdministrationStrengths: Course is organized in sections with individual sessions identified. Some explanation and philosophy of grading is included. Grading is suitable for management even with large class sizes.

Weaknesses: Students must spend extensive time offline in collaborative groups in order to be successful in the graded course. No online gradebook is available. Extensive sessions/lectures/assignment info may make it difficult for some students to

navigate course or instructor to keep track of student progress during the course.

Summary:

It is clear that the instructor/designer of this course has put extensive effort into the design of the assignments and learning objects within the course. The recorded lecture/collaboration indicates that this is an in-person course with online components (also known as a blended format). The instructor addresses the goals for learning in the course and provides extensive examples in a variety of appropriate formats. While the course would benefit from explicit rubrics for discussion posts and assignments, it does include details about the objectives of the assessments and available materials include completed assignments.

Reflection:

At first this course sounds like it is going to be all fun and games, and there is definitely some opportunity for that, but there is also a lot of really hard work that students have to put into understanding the history, thought processes of game designers, evolution of game rules, and in the creation of their own games. Again, similar to the previous course, the information provided online supplements an in-class lecture and lab. Students have plenty of opportunity for interaction with one another when they are using the discussion forum, playing card and board games with each other, and working in groups to design their own games. Overall this course was very well planned with lectures (by the instructors and guests) accompanied by readings, labs, and group designing activities. For each of the activities students are given examples of previous student work to have an example to mimic and get their ideas flowing. The rubrics were missing though and there was no reference to the fact that one would be used in the grading process. By the end of the course the students would have had the opportunity to meet the course learning objectives via many different avenues and there seemed to be an appropriate blend of project based learning, interactivity, text, lecture, and discussion opportunities. With as much work and the critiquing of games that students endure throughout this course, it is our hope that they still enjoy playing games when they are finished!