Running head: EVALUATION PLAN BLENDED …...Kay Seo, Ph.D. September 8, 2019 EVALUATION PLAN BLENDED ORGANIC CHEMISTRY UNIT 2 Table of Contents Background and

Running head: EVALUATION PLAN BLENDED ORGANIC CHEMISTRY UNIT 1

Evaluation Plan for the Design of a Blended-Learning Organic Chemistry Unit

Emily Morales

University of Cincinnati

Master’s Project 8130

Kay Seo, Ph.D.

September 8, 2019

EVALUATION PLAN BLENDED ORGANIC CHEMISTRY UNIT 2

Table of Contents

Abstract ......................................................................................................................................................... 3

Introduction ................................................................................................................................................... 4

Background and Scope ................................................................................................................................. 4

Evaluation Methodology ............................................................................................................................... 6

Usability and Accessibility .............................................................................................................. 6

Content Delivery .............................................................................................................................. 8

Motivational Design......................................................................................................................... 9

Evaluation Instruments ............................................................................................................................... 10

Usability and Accessibility ............................................................................................................ 11

Content Delivery ............................................................................................................................ 12

Motivational Design....................................................................................................................... 12

Participants .......................................................................................................................................... 13




Analysis Procedures .................................................................................................................................... 16




Timeline and Conclusion ............................................................................................................................ 18

Revision Notes .......................................................................................................................................... 19

Appendix A – Evaluation Tools.................................................................................................................. 21

Appendix B – Data and Analysis ................................................................................................................ 38

References ................................................................................................................................................... 45


Abstract

The evaluation plan for this blended-learning organic chemistry course will require the

implementation of many different evaluation tools, and the patient cooperation of participants

from diverse fields of expertise, user experience, and academic backgrounds. Using one-to-one

formative evaluation practices as outlined by Dick, Carey, and Carey (2015), usability and

accessibility of the course will be ascertained by surveys filled out by information technology

(IT) experts, learning specialists (LS), and the intended end-users, learners. The IT expert will

provide feedback concerning the usability of the site and its media resources, and the LS will

issue opinions regarding the presence of motivational elements (as enumerated by Keller’s

ARCS Model [1987]). Learners will evaluate navigability, motivational elements, and content

delivery. Revisions to the course in these key areas will be contingent upon the outcome and

analysis of the evaluation tools.


Evaluation Plan for Design of a Blended-Learning Unit of Organic Chemistry

The function of this evaluation plan is to generate objective feedback concerning the

efficacy of the blended-learning design in delivering content for an organic chemistry unit. The

unit covers the three-dimensional geometry of molecules and how this geometry impacts

intermolecular interactions and physical properties. My chief concern is whether learners find the

blended-learning environment, (1) usable and accessible – with respect to the technology choices

and implementation, (2) effective in content delivery by meeting the learning objectives

supporting the above-mentioned content, and (3) motivational in design.

The evaluation plan is largely informed by formative evaluation practices outlined within

the Dick, Carey, and Carey (2015) text, The Systematic Design of Instruction. Formative

evaluation enables the collection of data and valuable feedback during the development process

of instruction for the express purpose of improving its effectiveness before its distribution to a

larger learner population (Dick et al., 2015, p. 283). Surveys and questionnaires (Appendix A)

assessing the degree to which the course met the above objectives were derived from Keller’s

ARCS checklists (1987, 2010), Dick et al., (2015), and usability evaluation checklists proffered

by Roy and Pattnaik (2014). The examination evaluating student success with the content was

provided by the textbook publisher, Norton Publishing.

Background and Scope

About four years ago, an excellent chemistry student of mine asked if I could teach an

organic chemistry course at our local, private, homeschool cooperative. As a potential nursing

major, she had heard anecdotal accounts of college students in various nursing programs she was

exploring, who simply could not pass the course - forcing many to change their programs of

study. Having taken a year of organic chemistry in my own undergraduate program (molecular

biology), I was no stranger to its difficulty – thus, I was immediately empathetic. After a


proposal for the class was accepted by the school’s administrators, I had the pleasure of teaching

organic chemistry for the first time.

Remarkably, this young woman’s concerns were not without merit. One needs to take

merely a cursory glance at the chemical education literature to realize that the perceived

difficulty of organic chemistry among undergraduates is not merely anecdotal. Joel Karty,

Assistant Professor of Chemistry at Elon University and author of an excellent organic chemistry

text, Organic Chemistry – Principles and Mechanisms, laments, “Organic chemistry is a

notorious class among undergraduates. Its perceived difficulty seems to resonate across all

majors,” (2007, p. 1209). Still other researchers have similarly recognized the fact that the course

maintains an immutable reputation for difficulty. Grove, Hershberger, and Bretz (2008)

remarked, “As students watch peers struggle, the myth that organic chemistry is an impossible,

unforgiving subject is regrettably passed on to the next generation of students” (p. 157).

Recognizing the challenge that this course presents to many would-be doctors

pharmacists, nurses, and engineers, I wondered why public and private high schools do not offer

this as a fourth-year science elective – in the same way they offer anatomy and physiology,

forensics, or environmental science. It appears the challenge to schools lies in the lack of

instructional talent –there are not enough qualified teachers to take on such an advanced course.

Knowing this, I had questioned the feasibility of delivering an organic chemistry course in a

blended-learning environment, leveraging the benefits of both the face-to-face and online

learning environs. In a theoretical blended-learning design, the more difficult concepts of organic

chemistry could be relatively easy to disseminate in an online format, while the hands-on

laboratory component could be managed face-to-face. The instructional talent necessary to

facilitate such a lab could be easily satisfied by any qualified high school chemistry teacher.

Bearing all of these factors in mind, I set to work on developing such a course. What remains to


be seen is if such a course is efficacious. Before investing a voluminous amount of time

developing an entire course, however, I am interested in evaluating first a single unit or chapter.

Evaluation Methodology

In order to evaluate the effectiveness of the blended-learning organic chemistry course

unit, it is important to employ methodologies and tools that are aligned to the project itself, and

have their support in the literature. The methodology I plan to use will be that of formative

evaluation by way of one-to-one evaluation with (1) IT experts, (2) learners, and, (3) learning

specialists (Dick et al., 2015). Time does not permit the recruitment of a subject matter expert,

nor even a small group study or field trials. Ideally, this methodology and the requisite tools will

align with the criteria being evaluated.

Briefly stated, to evaluate the usability and navigability of the website and associated

apps, I developed an instrument that assesses ease of use, efficiency, learnability, user

satisfaction, and memorability, for the use of IT experts and the learners. The degree to which

the blended-learning design could deliver course content will be measured the learner’s objective

performance on an exam developed by the textbook publisher. To determine whether the course

satisfied motivational design strategies elucidated by Keller’s ARCS Model (1987), I included

many questions from Keller’s checklists (p. 4, 5) in a survey, along with a questionnaire

informed by Dick et al., (2015).

Usability and Accessibility

Usability, according to Roy and Pattnaik (2013, p. 535) defines: the ease of learning a

new system (or software); the capability of holding the users interest; and, the ability to generate

user satisfaction. The authors add that the function of assessing usability is to determine ways to

improve the product, ascertain the user’s general preferences, help users in goal achievement,


reduce errors contained within the system, and increase the likelihood the user will use the

product again in the future.

Since my targeted learner will have to navigate through many different technologies and

applications, a usability evaluation is critical. To deliver the course, I plan to use a Weebly

website on which learners will access instructional videos (broken down into sections that align

with the course textbook), homework assignments, quizzes, exams, and laboratory projects. The

means by which I will assess usability will be by way of a one-on-one evaluation, using a Real-

Time Walkthrough (Roy & Pattnaik, 2013, p. 540), conducted by two IT experts, and a

temporally extended version of this by three of the course learners. The purpose of this one-to-

one evaluation is “. . . to identify and remove the most obvious errors . . .” and, “. . . to obtain

initial performance indications and reactions . . .” (Dick et al., 2015, p. 288).

The IT expert will be provided with a survey (Appendix A.1) and a link to the course. He

or she will be allowed no more than ninety minutes navigation time, to assess the following:

whether all links to videos and printables work

the layout of the site: is it user friendly and aesthetic?

the ability to achieve the goals of instruction from the learner’s perspective

the presence of any technological errors within the course (e.g., broken links, error

messages).

In the event the IT expert ranks some items in the course as especially objectionable, he or she

will write these down on a separate form, or discuss with me directly, in our one-on-one session.

The learners selected will similarly be provided with a survey (Appendix A.2) and link to

the course, but they will not fill out the survey until the end of the course unit – which could take

two to three weeks. The protocol for evaluation is by necessity different for the learner compared

to the IT expert, because the learner’s objectives for actually using the site will be different –


theoretically, they are accessing the site to learn the subject matter, and their interaction with the

site will well exceed merely ninety minutes. The directives to the learners will also be different

from that of the IT experts as they have never been asked to critique instruction. Dick et al.,

(2015, p. 291) remarks that learners may find it challenging to “criticize” an authority figure –

thus, they will be assured that I am seeking their honest opinions and there will be no negative

consequences for candid and open criticism.

Content Delivery

Since organic chemistry content is challenging in the best face-to-face environments, it is

especially important that the online instruction is able to meet content standards established by

the American Chemical Society (ACS) for the topics of address. Fortunately, Joel Karty, in his

exemplary text, established learning objectives that meet ACS standards. These standards by

nature are voluminous, demonstrating not only the problem of “tyranny of content” (Kennepohl,

2012, p. 671), but also demand the higher cognitive process dimension skills of application,

analysis, evaluation, and synthesis (Mayer, 2002). The combined challenge of voluminous

content and demand for higher order cognitive skills makes a blended-learning course all the

more challenging since students, in effect, will have to teach themselves.

The question then, as to whether the course content is successfully delivered will be

answered by way of an end-of-the-unit assessment (See Appendix A.3). I am hopeful that in

observing universal design for learning (“UDL”) practices, specifically the tenet of multiple

means of representation (King-Sears, Johnson, Berkeley, Weiss, Peters-Burton, Evmenova,

Menditto, Hursh, 2015), that students will successfully demonstrate content mastery. Students

will not only have the written textbook/ solutions manual, homework assignments, and

instructional videos, but face-to-face time for collaboration with their peers, as well. To

minimize the effects of cognitive overload that many students encounter with texts that often


“say too much,” the videos, each representing a single section from the text, can provide an

alternative method of instruction. The evaluation methodology for content delivery will involve

administering the exam provided by Norton Publishing to three students, allowing them only

ninety minutes to complete.

Motivational Design

C.J. Blair, chemistry professor and columnist at the Oberlin Review wrote, “If Principles

of Organic Chemistry were assigned a character based on its reputation, it would be the shark

from Jaws swimming in a pool of undergraduate minnows,” (2015, para. 1). If this statement

were not bad enough, he adds, “I can’t think of another class so notorious that its reputation

extends beyond the people who take it” (para. 2). Blair laments that nearly every non-science

major at Oberlin has an anecdote concerning a “friend” who stressed over a plummeting GPA on

account of a failed exam, or botched lab. Organic chemistry’s well-substantiated reputation

oftentimes places students in the position of feeling intimidated before they even step foot in

their class. Add to these dilemmas the problem of higher dropout rates evidenced in online

learners (Allen & Seaman [2009] as cited by Stavredes & Herder [2012, p. 155]), it is obvious

that a blended-learning organic chemistry course will demand intentional motivational design.

John Keller’s ARCS Model for motivational instructional design consolidates many

theories (Self-Efficacy [Bandura, 1977], Origin and Pawn [de Charms, 1968], Attribution

[Weiner, 1974], Self-Determination Theory [Deci & Ryan], etc.) that facilitate learner

motivation, and uses these to proffer actionable instructional strategies (Keller, 2010). When

developing the unit in organic chemistry, I was careful to employ as many of these strategies as

the technology and learning methods afforded. To determine whether these strategies were

successfully implemented, I plan to have at least one or two learning specialists, in a one-on-one

formative evaluation, systematically rate the degree to which these strategies were satisfied (See


Appendices A.4, A.5), using the Likert scale (Wakita, Ueshima, & Noguchi, 2012) in accordance

with several of the recommendations provided by Dick et al., (2015, pp. 285-287), and Keller

(1987, pp. 4, 5).

Evaluation Instruments

The actual instruments used in the evaluation plan come in the form of surveys and

questionnaires informed by the motivation and information technology literature (as designated

in their respective sections below); and a unit examination (to assess content knowledge)

provided by Norton Publishing. In the development of all the surveys a Likert psychometric five-

point scale was used, because of its reliability (Lissitz & Green [1975], and Boote [1981] as cited

by Wakita, Ueshima, and Noguchi [2012, p. 534]). IT experts and learners will be asked to rank

their experience with the website and learning materials, by checking one of five boxes that

correlate to the scale as follows:

1 = needs significant improvement

2 = needs some improvement

3 = neutral

4 = satisfactorily met

5 = very satisfactorily met

Learning specialists, using this same scale will rank the degree to which the various objectives of

the Keller’s ARCS Model for learner motivation is satisfied.

The neutral anchor of 3 was inserted so as to allocate equal psychological distance

between this category and those both sides of neutral; it seemed this was necessary so as to

provide equal psychological distance between all categories (Wakita et al., (2012, p. 535).

Wakita et al., (2012, p. 535) reported that when objectives or questions were negative, the

distance between categories can become skewed – with the width of the neutral category being


more narrow than the others. For this reason, all questions were formulated using a positive

narrative to minimize this skewing.

Dick et al., (2015, p. 291) recommends that the very assessment instruments used to

evaluate the products themselves should be evaluated formatively before being used, so that the

directions are clear to anyone using them. To ensure this, I had a proof-reader the same age as

my high schoolers go through the instruments the learners will use (Appendix A.2, A.3), along

with an adult who carefully read and made revisions on all of them, where necessary.


In the Real-Time Walkthrough, IT experts will be provided with a two-part evaluation

(see Appendix A.1). Part I is a survey that is a modified version of the Norman’s Cognitive

Walkthrough, and Cognitive Walkthrough for the Web (Mahatody, Sagar, & Kolski, 2010, pp.

747, 748). The purpose of using this modified tool is to determine whether the “user” correctly

interprets the prompts provided on the course website, and is able to follow the progression of

the organic chemistry unit based upon those prompts (buttons, links, icons, etc.). Since the IT

experts will be familiar with what constitutes excellent universal website design and user

accessibility, their input will be quite valuable. For any survey objectives that receive a “1” or

“2,” the IT expert will provide on Part II of the evaluation suggestions for improvement. I added

space for commentary from the IT experts because “. . . descriptive information rather than

quantitative data probably yields the best information about clarity for revising the instruction,”

(Dick et al., 2015, p. 290).

A separate survey (Appendix A.2), with questions very similar to the one the IT experts

complete, will be administered to three organic chemistry students after they complete the unit.

The students may not have the experience of the IT expert when it comes to providing feedback

with respect to layout or design, but as users it will be important to know whether they found the


website and its resources both usable and accessible. With the help of a high school proof-reader,

I was cautious to craft these questions so they made the most sense.

Content Delivery

It cannot be overstated the importance of a well-designed website and technological

accoutrements being aligned to specific learning objectives. In order to evaluate the efficacy of

the instructional materials in content delivery, the three students have volunteered to take a

standardized exam (Appendix A.3) developed by Norton Publishing that accompanies the course

textbook. Following instruction, the students will take the proctored pencil-and-paper closed-

book, closed-note exam, in a time period no greater than ninety minutes. A past organic

chemistry student looked through each problem on this exam, to be sure they were clear.

Motivational Design

I hope to enlist the gracious services of two students in the IDT Program (hopefully Ann

Harris for one on account of her science background), to act as learning specialists in the

formative evaluation of this project. The learning specialist(s) will be provided with a survey

(Appendix A.4) developed from Keller’s ARCS Model (1987, pp. 4, 5), and a questionnaire

(Appendix A.5) informed by Dick et al., (2015, pp. 285-287). Since in this particular evaluation I

am interested in knowing whether the design is motivational, it made sense to write survey

questions derived from Keller’s original research (1987). Ideally, the answers as to whether the

design demonstrated motivational components will be answered by the specialist(s) navigating

the website, perusing through the homework questions, labs and quizzes, and having to watch

only one or two of the videos.

As already mentioned, the questions in the learning specialist questionnaire (Appendix

A.5) were derived from Dick et al., (2015, pp. 285-287) with some slight modifications. LS will


answer Y/ N questions on motivational strategies, with space to comment in more detail. While

surveys certainly can

Participants

Participants completing the surveys, questionnaires and exams were not selected

randomly. Instead, they were chosen for their expertise in media design (IT experts),

instructional design (learning specialists), and their stakeholder interest to the organic chemistry

course materials (students). The background and expertise of the participants correlated with the

aspect of the evaluation under their review. Decisions concerning who to select as participants

for particular sections of the evaluation plan were influenced largely by Table 11.1 provided by

Dick et al., (2015, p. 286).


This aspect of the evaluation does not necessitate an evaluator with any background in

organic chemistry, as the objective is to gain feedback with respect to the usability and

accessibility of the website itself and its accompanying resources. For the Real-Time

Walkthrough, however, I will solicit feedback from three organic chemistry students and two IT

experts. The participants in both these groups have been strategically recruited, in accordance

with recommendations by Dick et al., (2015), as specified below.

Roy and Pattnaik (2013, p. 540) assert that when it comes to evaluating the design of a

website IT experts can provide, “. . . an immediate and concrete tactical analysis of the user’s

experience to make the product or website. . .” more user-friendly, and point out any flaws with

respect to links and navigation. In addition to their IT background, Dick et al., (2015, pp. 287,

288) recommends that evaluators should be familiar with the target population, someone who

can look at the courses resources “. . . through the target population’s eyes and react.” In the


recruitment of the two IT experts I was interested in both their background in IT, and their

experience working with a target population very similar to mine.

I enlisted the services of Blas Morales because of his education and experience. He holds

an MBA in Information Systems, and has been working in the IT industry for the last thirty

years, of which the most recent fourteen have been in the university environment. Currently a

business analyst and project manager at the University of Dayton, his role includes ensuring that

product functionality and features are both identified and are fully met. This includes ensuring

that end-user acceptance testing is completed and approved (with end-users often being

students). Tiffani Puckett is my second choice as an IT expert. She currently teaches classes in

computer applications, along with web and graphic design at a local, private high school. Since

she also builds and maintains websites, and posts social media for several Cincinnati-area

churches, she would have an excellent perspective as to what constitutes user-friendliness. I

anticipate that the backgrounds of both Mr. Morales and Ms. Puckett will enable them to “. . .

provide insights into the appropriateness of the material for the eventual performance context,”

(Dick et al., 2015, p. 288).

In addition to the actionable input I hope to receive from the IT experts, I am seeking

feedback from the learners. The three learners selected for this particular evaluation have diverse

experience with respect to using learning management software, with one in particular having a

great amount of experience due to enrollment in online college classes. Another has never taken

an online course before at all, and the third has minimal experience. Although none of these

students have the experience of the IT experts – they can still provide excellent insight as to the

website’s (and accompanying materials) ease of use, and navigability. Whereas I will be looking

to the IT experts for recommendations for improving course design and navigation, from the

learners I hope to solicit feedback that is more visceral and affective, reflecting their general


attitudes towards course layout and navigability (did they grow frustrated, discouraged or

hopeless?).

Content Delivery

The three organic chemistry students who will be selected represent various ability ranges

within their group as recommended by Dick et al., (2015, p. 288). One learner in particular is

extraordinary – has the necessary pre-skills for success, and highly intelligent. Another is a

struggling student with a background known for lacking the pre-skills that would have been

acquired in a solid, general chemistry course, while the third has abilities that are intermediate.

Despite all the students having some degree of motivation (otherwise they would not be enrolled

in the course), one in particular is very motivated to succeed because he wants to become a

synthetic chemist. The other two do not have the same love for chemistry, but are instead taking

the course to prepare them for college. Dick et al., (2015, p. 289) recommended using learners

with not only variable abilities and pre-skills, but varying attitudes as well.

Since I am primarily concerned as to whether students can actually learn the content of

the unit, these three learners, by virtue of their testing scores, will provide meaningful feedback

as to the feasibility of this goal. The students selected will very well represent future targeted

learners: specifically, high school students who have already taken general chemistry.

Motivational Design

This part of the evaluation demands the critical and trained eye of a learning specialist to

ascertain whether or not the design is motivational; I hope to enlist one or two current IDT

Program students. Current IDT candidates will more than likely have a familiarity with the

learning theories for motivation (Self-efficacy, self-determination, locus of control, attribution,

etc.) together with their requisite instructional design models. Because these candidates already

took the Instructional Design course from Dr. Seo, they are aware of effective prescriptives for


learning design as outlined by Dick et al., (2015), Keller (1987), and many others. These

particular participants will not be selected at random, since there is not a large, willing pool to

recruit from, but will be selected on the basis of their expertise in science, current availability

and interest, and my relationship with them.

Analysis Procedures

Many of the evaluation instruments used in the evaluation plan are subject to quantitative

analyses. Because all of the surveys used a five-point Likert scale, they lend themselves to

statistical analysis (calculating means for survey answers). The organic chemistry exam provided

by Norton Publishing similarly enables statistical analysis. The questionnaires filled out by IT

experts and learning specialists, however, necessitate more of a qualitative analysis.


The data collected from the IT experts and students will be examined and scored

separately, even though both groups are answering similar questions. From the IT expert survey

data, mean rankings for each objective will be calculated and recorded in a table (Appendix B.1),

along with comments regarding suggestions for improvement. Mean rankings will also be

recorded for the learner’s survey (Appendix B.2). So as to not “. . . overgeneralize the data

gathered from only one individual,” objectives from both groups with mean rankings of 2 or less

will undergo certain revisions, whereas those between 2 and 3 will only be considered for

revision (Dick et al., 2015, p. 292). Mean rankings of objectives of four and above will be

deemed as satisfactory.

The IT experts have been asked to provide suggestions for improvement on objectives

they rank as 1 (needs significant improvement), and 2 (needs minimal improvement). Analysis

will be conducted on these to determine the feasibility of the recommended revision.


Content Delivery

The test will serve as the evaluation tool for the efficacy of content delivery. Note that

each question on the test key (Appendix A.3) shows the section within the text to which the

question correlates. Success within a particular section in the textbook will to some extent be

determined by the mean student performance within that section (Appendix B.3). As an example

if students collectively miss 45% of the questions correlating to Section 2.2, but miss only 15%

of the questions related to Section 2.6, I could surmise that Section 2.2 might need some

revision, whereas Section 2.6 does not. For this project, sections that will be considered for

revision will be those wherein 35% of the questions were answered incorrectly by the students

(except Sections 2.3, and 2.8, due to the lack of questions in the test bank).

Again, since there will be only three exams to evaluate, care will be taken to not

overgeneralize data collected from one student only. This is particularly important in this

particular analysis, because the three students participating have varied backgrounds in pre-

skills, and intellectual capacity (Dick et al., 2015, p. 292).

Motivational Design

From the surveys filled out by the learning specialist, I will similarly calculate mean

rankings for each motivational strategy, and post in a table (Appendix B.4). Motivational

strategies scoring at 2 or below will be considered seriously for revisions; those scoring between

2 and 3 will be merely considered, and those with mean rankings at 4 or above will be deemed

satisfactory.

From the learning specialists, qualitative data will also be collected (Appendix A.5), and

analyzed (Appendix B.5). The suggestions for improvement for motivational design will be

carefully considered and evaluated as to feasibility.


Timeline and Conclusion

Table 1. Timeline of Project Completion

Task

Date

Evaluation Plan – first copy September 8th

Evaluation Plan – revision September 22nd

Complete the project undergoing analysis (completion of website,

assignments, and instructional videos)

September 22nd

IT Expert Usability and Accessibility Analysis (Appendix A.1) September 27th

Learner Usability and Accessibility Analysis (Appendix A.2, B.2) September 27th

Progress Report October 4th

Learning Specialist(s) Motivational Design Analysis (Appendices A.4, A.5) October 18th

Content Delivery Analysis (from Learner Exams [Appendix A.3]) October 18th

Evaluation Report October 25th

Artifact Revision November 1st

Portfolio completion November 15th

Presentation and defense December 6th

The final revision of this evaluation plan will be completed by September 22nd

, after

suggestions made by other IDT students are carefully weighed. Once complete, the plan will

provide the basis by which this blended-learning course will be evaluated for efficacy and

feasibility. By September 22nd

, I anticipate having all the instructional videos completed, and

uploaded to the course website, ready for evaluation by the IT expert, the learning specialists,

and actual use by the learners.

Evaluations from the IT expert and learners concerning the usability of the website and

associated technology will be collected and analyzed by September 27th

. Learning specialist(s)

evaluations regarding the presence or absence of motivational strategies in the design will be

analyzed by October 18th

. Learner’s exams will be collected, graded, and then evaluated for


success in content delivery by October 18th

. These analyses, taken together, will determine the

degree to which the project undergoes revisions and the nature of those revisions.

Once the final changes are made to the course, it will be linked to my portfolio. On

December 6th

, I plan to present and defend the instructional and technological choices I made in

the development of this course. That defense will be buttressed by the literature.

Revision Notes

The most significant revisions made to the evaluation plan were to the evaluation

instruments. Other revisions involved fleshing out in more detail evaluation methodology, choice

of participants, and analysis. These changes came about after having read through Ann Harris’s

paper; since she just completed a dissertation last year, she provided an excellent example of

what an evaluation plan should look like.

Evaluation Instruments

In the first draft of this Plan, I used a three-point survey, with the following scale:

0 = objective not met

1 = objective needs improvement

2 = objective satisfactorily met

After Ann Harris and Jose Avila suggested I consider the Likert scale, I found an excellent

journal article (Wakita et al., 2012) on its use in psychometric testing and decided that a five-

point scale would be far superior. I also changed the wording of the questions so they would all

have a positive narrative, since Wakita et al., (2012, p. 535) reported that when content items

were negative, the distance between categories of “strongly agree,” “agree,” “neither agree or

disagree,” “disagree,” and “strongly disagree,” can become skewed – with the width of the

neutral category being more narrow than the others.


I added a Part II to both the IT experts and the learner’s analysis of usability and

navigability, which enables them to comment in more detail on those objectives in which they

rated as “needs improvement,” and “needs significant improvement.” This was purposefully

done because Dick et al., (2015, p. 290) stated that descriptive feedback yields the best

information for revising the instruction, as opposed to the quantitative analysis only that survey’s

provide. Having the form close at hand, I am hoping these evaluators will take the liberty to

express and jot down concerns, and suggestions for improvement.

The revisions in the evaluation tools demanded that I likewise change my analysis of the

results. I had to change the ranges by which I would make judgements concerning the need for

revisions, and allot more reporting space for the comments coming from the questionnaires. This

was actually a good exercise, because in digging through the Dick et al., (2015) text, I found

some additional excellent directives for analysis.

Clarity Concerning the Selection of Participants

Ann Harris pointed out the importance of describing in more detail the reasons behind my

choices of participants. This prompted yet another look at the Dick et al., (2015) resource, and as

a result, I changed my mind concerning some of them. I was originally going to randomly select

three exams from the organic chemistry students to include in the analysis of content delivery,

yet Dick et al., (2015) stated that for formative evaluations, random participant selection is not

necessary; rather the evaluation data might provide greater insights if I purposely selected

students with varying academic backgrounds, motivation levels, and attitudes.

Finally, Ruona Zhang wisely asked for clarification regarding the abilities of the learners

selected for content delivery analysis. I clarified that these would be the same types of learners as

those who would be future course users: students who have had only high school level general

chemistry.


APPENDIX A

Evaluation Tools


Appendix A.1

IT Expert’s Real-Time Walkthrough Survey – Part I

For each question, rate the degree to which the objective was met, from 1 to 5:



3 = neutral



1 2 3 4 5 Website/ Technology Assets Objectives:

□ □ □ □ □ 1. Website was accessible from the link provided.

□ □ □ □ □ 2. Website’s homepage loaded quickly on the browser.

□ □ □ □ □ 3. Website interface was attractive.

□ □ □ □ □ 4. Homepage directs users to all course assets in a logical fashion.

□ □ □ □ □ 5. The amount of text on the homepage was appropriate.

□ □ □ □ □ 6. All graphical elements (images, icons) were properly presented.

□ □ □ □ □ 7. Font size big enough so that anyone with marginal vision could read the

text on links, summaries, and check-off lists.

□ □ □ □ □ 8. The web design itself followed typical LMS protocol/ design.

□ □ □ □ □ 9. All links to lesson videos worked.

□ □ □ □ □ 10. All links to written materials worked.

□ □ □ □ □ 11. Contact information to the instructor was easily found.

□ □ □ □ □ 12. The link to the Discussion Board worked as evidenced by feedback by

the website itself.

□ □ □ □ □ 13. The protocol for asking a general question using the Discussion Board

was easy to understand.

□ □ □ □ □ 14. When the “instructor” replied on your theoretical question on the

Discussion Board, I received notifications through my email.

Questionnaire modified from Norman’s Cognitive Walkthrough, and Cognitive Walkthrough for the Web

(Mahatody, Sagar, & Kolsi, 2010, pp. 747, 748)

For objectives scoring a 1 or 2, please provide a

suggestion for improvement on Part II


IT Expert’s Real-Time Walkthrough Survey Comments – Part II

In the spaces provided below, please provide suggestions for improvement for those objectives in

Part I that were rated at a 1 or 2.

Objective No: Your Rating Recommendations for Improvement

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________


Appendix A.2

Learner’s Real-Time Extended Walkthrough Survey

For each question, rate the degree to which the objective was met, from 1 to 5:



3 = neutral



1 2 3 4 5 Website/ Technology Assets Objectives:

□ □ □ □ □ 1. Were you able to access the website from the link provided?

□ □ □ □ □ 2. Did the website’s homepage load quickly on your browser?

□ □ □ □ □ 3. Did you find the website interface simple and easy to use?

□ □ □ □ □ 4.

Were the course assets, such as videos, homework, quizzes, laboratories

and answer keys in places you’d expect to find them?

□ □ □ □ □ 5.

Were you overwhelmed at the amount of text on the homepage? If too

much, indicate with a “1” or “2,” if just the right amount, “4” or “5.”

□ □ □ □ □ 6. Did the images support the text? Did you find the images necessary?

□ □ □ □ □ 7.

Font size just the right size in your browser? Too small, “1” or “2,” just

right “4” or “5.”

□ □ □ □ □ 8. Were video, homework, lab and other links located in a logical place?

□ □ □ □ □ 9. Did all the links to lesson videos work?

□ □ □ □ □ 10. Did all the links to written materials work?

□ □ □ □ □ 11. Were you able to “contact me,” easily?

□ □ □ □ □ 12. Did the link to the Discussion Board work?

□ □ □ □ □ 13.

Was the protocol for asking a general question using the Discussion

Board clear?

□ □ □ □ □ 14.

When the “instructor” replied on your theoretical question on the

Discussion Board, did you receive notifications through your email?



For objectives scoring a 1 or 2, please provide a

suggestion for improvement on Part II


Learner’s Extended Real-Time Walkthrough Survey – Part II

In the spaces provided below, please provide suggestions for improvement for those objectives in

Part I that were rated at a 1 or 2.

Objective No: Your Rating Recommendations for Improvement

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________

_______ _______ _________________________________________________


Appendix A.3

Multiple Choice Organic Chemistry Exam (Karty, 2014)

For each question below, circle the single answer that best fits.

1. When applying VSEPR theory to determine the geometry about a central atom, it is important to count the

number of electron groups. Separately consider the two atoms highlighted with an arrow in the molecule

shown below. How many electron groups must be considered for each of these central atoms?

a. C1 has two groups; C2 has two groups. d. C1 has four groups; C2 has three groups.

b. C1 has three groups; C2 has four groups. e. C1 has four groups; C2 has four groups.

c. C1 has four groups; C2 has two groups.

ANS: C DIF: Medium REF: 2.1

OBJ: Apply VSEPR theory to understand organic structure and geometry.

MSC: Understanding

2. Which of the following statements is true about carbon tetrachloride, CCl4?

a. It is polar protic with tetrahedral geometry.

b. The carbon has trigonal planar geometry.

c. None of the CCl bonds has a dipole.

d. It is miscible in water with 109.5° bond angles.

e. It is polar aprotic with tetrahedral geometry at C.

ANS: E DIF: Easy REF: 2.1

OBJ: Relate polarity to molecular geometry and physical properties.

MSC: Evaluating

3. Which of the following molecules contains a nitrogen atom that has bent molecular geometry?

a. I d. IV

b. II e. V

c. III

ANS: D DIF: Medium REF: 2.1


MSC: Understanding


4. Which of the following molecules contain(s) a nitrogen atom that has trigonal pyramidal molecular

geometry?

a. I only d. I and II

b. II only e. II and III

c. III only



MSC: Understanding

5. Which cycloalkane contains a CCC bond angle that deviates from the ideal tetrahedral bond angle

by approximately 20°?

a. A seven-membered cycloalkane d. A four-membered cycloalkane

b. A six-membered cycloalkane e. A three-membered cycloalkane

c. A five-membered cycloalkane

ANS: D DIF: Medium REF: 2.1 | 2.2


MSC: Understanding

6. The carbon atoms in the molecule below are labeled 1–8. Which

CCC bond angle in the molecule would be approximately 120°?

a. C1-C2-C3 d. C5-C6-C7

b. C2-C3-C4 e. C6-C7-C8

c. C4-C5-C6

ANS: B DIF: Medium REF: 2.1 | 2.2


MSC: Understanding

7. What are the approximate HCH bond angles expected for the carbanion whose

structure is given in the ball-and-stick representation below?

a. 180° d. 107°

b. 150° e. 90°

c. 109.5°



MSC: Understanding


8. The carbon atoms in the molecule below are labeled 1–8.

Which CCC bond angle in the molecule would be

approximately 180°?

a. C1-C2-C3 d. C5-C6-C7

b. C2-C3-C4 e. C6-C7-C8

c. C4-C5-C6

ANS: A DIF: Easy REF: 2.1 | 2.2


MSC: Applying

9. The carbon atoms in the molecule below are labeled 1–8. Which

CCC bond angle in the molecule would be approximately

109.5°?

a. C1-C2-C3 d. C5-C6-C7

b. C2-C3-C4 e. C6-C7-C8

c. C4-C5-C6

ANS: E DIF: Easy REF: 2.1 | 2.2


MSC: Applying

10. Which of the following molecules contains a nitrogen atom with linear geometry?

a. I d. IV

b. II e. V

c. III

ANS: E DIF: Medium REF: 2.1 | 2.2


MSC: Applying

11. Which cycloalkane has the greatest ring strain per-CH2-unit?





OBJ: Understand the influence of ring strain on organic structure.

MSC: Analyzing


12. Which cycloalkane contains a CCC bond angle that deviates from the ideal tetrahedral bond angle

by approximately 50°?




ANS: E DIF: Medium REF: 2.1 | 2.2


MSC: Analyzing

13. Which of the following choices correctly describes the structure of the ball-and-

stick representation with the formula H3C+?

a. A carbocation with a tetrahedral carbon

b. A carbocation with trigonal planar geometry

c. A carbocation with unknown geometry

d. A carbanion with a tetrahedral carbon

e. A carbanion with trigonal planar geometry



MSC: Analyzing

14. Which of the following cycloalkanes contains a CCC bond angle of approximately 90°?


b. A six-membered cycloalkane e. A four-membered cycloalkane


ANS: D DIF: Easy REF: 2.1 | 2.5


MSC: Understanding

15. What is the VSEPR geometry for the carbon atom of a carbonyl?

a. Linear d. Trigonal planar

b. Tetrahedral e. Bent

c. Trigonal pyramidal



MSC: Understanding

16. What is the VSEPR geometry for any carbon atom in a phenyl ring?

a. Linear d. Trigonal planar

b. Tetrahedral e. Bent

c. Trigonal pyramidal



MSC: Understanding


17. Which of the following molecules contains a trigonal planar nitrogen atom connected to two different

tetrahedral carbon atoms?

a. I d. IV

b. II e. V

c. III


OBJ: Apply VSEPR theory to understand organic structure and geometry. MSC: Understanding

18. Turn the original molecule shown below 90° in a clockwise direction on the plane of this paper. Which

choice represents the product of this manipulation?

a. Structure a d. Structure d

b. Structure b e. Structure e

c. Structure c


OBJ: Execute a prescribed rotation and draw the molecule in its new orientation.

MSC: Applying

19. Rotate the molecule below 180°, in the same way you would flip a pancake or an egg during cooking.

Which choice represents the product of the manipulation?

a. Structure a d. Structure d

b. Structure b e. Structure e

c. Structure c

ANS: B DIF: Medium REF: 2.2

OBJ: Execute a prescribed rotation and draw the molecule in its new orientation. MSC: Applying


20. Which of the following molecules possesses at least one polar covalent bond but does not have an overall

net molecular dipole?

a. CH4 d. CCl4

b. CHCl3 e. CH3CH3

c. CH2Cl2


OBJ: Differentiate between polar and nonpolar organic molecules.

MSC: Remembering

21. Which of the following intermolecular forces is responsible for the boiling-point trends in alkanes?

a. Hydrogen bond d. Dipole–induced dipole

b. Ion–dipole e. Induced dipole–induced dipole

c. Ion–ion

ANS: E DIF: Easy REF: 2.4 | 2.6

OBJ: Identify the relative strength of the common intermolecular forces as they apply to organic

molecules. MSC: Analyzing

22. Rank the following molecules based on increasing boiling point.

a. I < II < III < IV d. II < III < IV < I

b. II < I < III < IV e. IV < III < II < I

c. IV < III < I < II

ANS: B DIF: Easy REF: 2.4 | 2.6


molecules. MSC: Evaluating

23. Which functional group will engage in dipole–dipole interactions, but will not serve as a hydrogen-bond

acceptor?

a. Nitrile d. Carboxylic acid

b. Ketone e. Amine

c. Alkyl bromide


OBJ: Identify the intermolecular forces in which a functional group engages.

MSC: Remembering

24. Which of the following functional groups contains both a hydrogen-bond donor and a hydrogen-bond

acceptor?

a. Alkyl fluoride d. Nitrile

b. Epoxide e. Ketone

c. Carboxylic acid

ANS: C DIF: Easy REF: 2.5 | 2.6


MSC: Applying


25. Consider the structure of sodium benzoate, NaOC(O)Ph, the sodium salt of benzoic acid. In which of the

following solvents would you predict sodium benzoate to be soluble?

I. Water, H2O II. Pentane, CH3(CH2)3CH3 III. Diethyl ether, (CH3CH2)2O

IV. Methanol, CH3OH V. Acetone, CH3C(O)CH3

a. I only d. III and V

b. I and III e. III and IV

c. I and IV

ANS: C DIF: Medium REF: 2.5 | 2.7

OBJ: Utilize knowledge of chemical structure to identify solubility properties of an organic compound.

MSC: Evaluating

26. Identify the strongest intermolecular force.



c. Ion–ion

ANS: C DIF: Easy REF: 2.6


molecules. MSC: Remembering

27. When applying VSEPR theory to determine the geometry about a central atom, it

is important to count the total number of bonded and nonbonded electron groups.

Separately consider the two atoms highlighted with an arrow in the molecule

shown below. How many bonded electron groups must be considered for each of

these central atoms?

a. C has two groups; O has two

groups.

d. C has three groups; O has three

groups.

b. C has three groups; O has four

groups.

e. C has four groups; O has four

groups.

c. C has three groups; O has two

groups.



MSC: Understanding

28. How does the presence of the lone pair affect the geometry of the central atom in the

following molecule?

I. The lone pair is attracted to the nuclei of the three substituents, creating larger bond

angles.

II. The lone pair repels the three sets of covalently bonded electrons.

III. The lone pair has no bearing whatsoever on the VSEPR geometry at the central

atom.

IV. The bond angles are smaller than a traditional tetrahedral bond angle due to lone pair repulsion.

a. I d. IV

b. II e. II and IV

c. III

ANS: E DIF: Medium REF: 2.6

OBJ: Apply VSEPR theory to understand organic structure and geometry. MSC: Understanding


29. How many hydrogen-bond donors and acceptors are present in the following molecule?

a. One donor and four acceptors d. One donor and three acceptors

b. Two donors and four acceptors e. Two donors and two acceptors

c. Two donors and three acceptors

ANS: B DIF: Easy REF: 2.6


MSC: Applying

30. Identify the weakest intermolecular force.



c. Ion–ion




31. When mixed, which of the following pairs of compounds will exhibit both ion–dipole and ion–ion

intermolecular attractive forces?

a. I d. IV

b. II e. V

c. III


OBJ: Deduce the intermolecular forces that are possible between specified molecules.

MSC: Analyzing

32. What is the strongest intermolecular attractive force between an alcohol and a ketone?



c. Ion–ion

ANS: A DIF: Medium REF: 2.6



33. What is the strongest intermolecular attractive force possible between an alkyl chloride and an alkane?



c. Ion–ion





34. Rank the following molecules based on decreasing boiling point.

a. I > II > III > IV d. IV > III > I > II

b. II > I > III > IV e. IV > III > II > I

c. I > III > IV > II

ANS: D DIF: Easy REF: 2.6, 2.8*


molecules. MSC: Evaluating

35. Select a phrase to complete this sentence: “___________________ are induced dipole–induced dipole

interactions common to nonpolar molecules such as hydrocarbons.”

a. Electromotive forces d. London dispersion forces

b. Hydrogen bonds e. Mechanical forces

c. Ionic attractions

ANS: D DIF: Easy REF: 2.6 | 2.7


molecules. MSC: Remembering

36. Dimethyl sulfoxide (DMSO) is a polar aprotic solvent that is frequently used for organic reactions. Rank

the following sodium halide salts for decreasing solubility in DMSO.

a. NaBr > NaCl > NaI d. NaCl > NaBr > NaI

b. NaBr > NaI > NaCl e. None of these salts is soluble in DMSO.

c. NaCl > NaI > NaBr

ANS: D DIF: Medium REF: 2.7, 2.9*

OBJ: Evaluate the solvation of an ion by a protic or aprotic solvent.

MSC: Analyzing


Appendix A.4

Learning Specialist’s ARCS Model Strategies Survey Part I, page 1

For each statement, rate the degree to which the strategy was met by √ the box, from 1 to 5:



3 = neutral



1 2 3 4 5 Attention-getting Strategies

□ □ □ □ □ 1. Shows visual representations (images, photographs, models) of any

important object or set of ideas or relationships (A2.1).

□ □ □ □ □ 2. Provides worked out examples of every instructionally important concept or

principle (A2.2).

□ □ □ □ □ 3. Uses content-related anecdotes, case studies, biographies, etc. (A2.3).

□ □ □ □ □ 4. Varies the format of instruction (content presentation, practice problems,

reflection, etc.) according to the attention span of the audience (A3.2).

□ □ □ □ □ 5. Varies the medium of instruction (platform delivery, film, video, print, etc.)

(A3.3).

□ □ □ □ □ 6. Breaks up print materials by use of white space, visuals, tables, different

typefaces, etc. (A3.4).

□ □ □ □ □ 7. Shifts between student-instructor interaction and student-student

interaction (by way of labs, collaboration) (A3.6).

□ □ □ □ □ 8. Uses creativity techniques to have learners create unusual analogies and

associations to the content (A5.1).

□ □ □ □ □ 9. Embeds problem-solving activities at regular intervals (A5.2).

1

2

3 4 5 Relevance Strategies

□ □ □ □ □ 1. States explicitly how the instruction builds on the learner's existing skills

(R1.1).

□ □ □ □ □ 2. Uses analogies familiar to the learner from their past experience (R1.2).

□ □ □ □ □ 3. States explicitly the present intrinsic value of learning the content, as distinct

from its value as a link to future goals (R2.1).

□ □ □ □ □ 4. States explicitly how the instruction relates to future activities of the learner

(R3.1).

□ □ □ □ □ 5. To enhance achievement-striving behavior, provides opportunities to achieve

standards of excellence under conditions of moderate risk (R4.1).

□ □ □ □ □ 6. To satisfy the need for affiliation, establishes trust and provides

opportunities for no-risk, cooperative interaction (R4.3).

□ □ □ □ □

7.

Models enthusiasm for the subject taught (R5.3).

Survey derived from Keller, J.M. (1987). The systematic process of motivational design. Performance & Instruction,

26(9-10), 1-8. Letters next to the written strategy correlate to those in Keller’s Tables 1 to 4.


Learning Specialist’s ARCS Model Strategies Survey Part I, page 2

For each statement, rate the degree to which the strategy was met by √ the box, from 1 to 5:



3 = neutral



1 2 3 4 5

Confidence Strategies

□ □ □ □ □ 1. Incorporates clearly stated, appealing learning goals into instructional

materials (C1.1).

□ □ □ □ □ 2. Provides self-evaluation tools which are based on clearly stated goals

(C1.2).

□ □ □ □ □ 3.

Explains the criteria for evaluation of performance (C1.3).

□ □ □ □ □ 4. Organizes materials on an increasing level of difficulty; that is, structure the

learning material to provide a "conquerable" challenge (C2.1).

□ □ □ □ □ 5. Includes statements about the likelihood of success with given amounts of

effort and ability (C3.1).

□ □ □ □ □ 6. Explains to students how to develop a plan of work that will result in goal

accomplishment (C3.2).

□ □ □ □ □ 7. Attributes student success to effort rather than luck or ease of task when

appropriate (i.e. when you know it's true!) (C4.1).

□ □ □ □ □ 8. Allows students opportunity to become increasingly independent in learning

and practicing a skill (C5.1).

□ □ □ □ □ 9. Have students learned new skills under low risk conditions, but practice

performance of well-learned tasks under realistic conditions (C5.2).

□ □ □ □ □ 10.

Helps students understand that the pursuit of excellence does not mean

anything short of perfection is failure; the student learns to feel good about

genuine accomplishment (C5.3).

1

2

3 4 5 Satisfaction Strategies

□ □ □ □ □ 1. Verbally reinforces a student's intrinsic pride in accomplishing a difficult

task (S1.2).

□ □ □ □ □ 2. Allows a student who masters a task to help others who have not yet done

so (S1.3).

□ □ □ □ □ 3.

Gives verbal praise for successful progress or accomplishment (S3.1).

□ □ □ □ □ 4.

Avoids the use of threats as a means of obtaining task performance (S4.1).

□ □ □ □ □ 5.

Avoids surveillance (as opposed to positive attention) (S4.2).

□ □ □ □ □ 6. Avoids external performance evaluations whenever it is possible to help the

student evaluate his or her own work (S4.3). Survey derived from Keller, J.M. (1987). The systematic process of motivational design. Performance & Instruction,

26(9-10), 1-8. Letters next to the strategy correlate to Keller’s Tables 1 to 4.


Appendix A.5

Learning Specialist’s Questionnaire – Motivational Design – Part II

Question

Response

1. Do the materials reflect a natural progression in skills, so as to minimize frustration

on the part of the learner?

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

2. What do you perceive to be the chief motivational value of the materials?

Comments:___________________________________________________________

____________________________________________________________________

N/A

3. Do you think learners will find the materials relevant to their needs and interests in

both the short term and long term?

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

4. Do you think there are adequate cues to gather the learner’s attention?

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

5. Are there adequate steps taken to maintain the learner’s attention?

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

6. Could more be done to facilitate learner confidence? If so, describe.

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

7. Besides satisfying a requirement for granting a degree, are there other elements

within the course that might generate learner satisfaction? If so, please describe

those.

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

8. When looking through the course materials and the website, did you get the

impression that course success can be realized with hard work?

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

Questions are modified from Dick, Carey, and Carey, 2015, pp. 286, 287.


APPENDIX B

Data and Analysis


Appendix B.1

IT Expert’s Real-Time Walkthrough Survey Data and Analysis

Each question was rated from 1 to 5, accordingly:



3 = neutral



Mean

Ranking Website/ Technology Assets Objectives: Suggestions for Improvement

_____ 1. Website was accessible from the link provided.

_____ 2.

Website’s homepage loaded quickly on the

browser.

_____ 3.

Website interface was attractive.

_____ 4.

Homepage direct users to all course assets in a

logical fashion.

_____ 5.

The amount of text on the homepage was

appropriate.

_____ 6.

All graphical elements (images, icons) were

necessary.

_____ 7.

Font size big enough so that anyone with

marginal vision could read the text on links,

summaries, and check-off lists?

_____ 8.

The web design itself followed typical LMS

protocol/ design.

_____ 9.

All links to lesson videos worked.

_____ 10.

All links to written materials worked.

_____ 11.

The contact information to the instructor was

easily found.

_____ 12.

The link to the Discussion Board worked, as

evidenced by feedback from the website.

_____

13.

The protocol for asking a general question

using the Discussion Board was easy to

understand.

_____ 14.

When the “instructor” replied on your

theoretical question on the Discussion Board,

you received notifications through your email.



Items scoring 0 will require revisions; 1 will be considered; 2 will be deemed satisfactory.

For objectives scoring a 1 or 2, IT experts

provided suggestions for improvement.


Appendix B.2

Learner’s Extended Real-Time Walkthrough Survey Data and Analysis

Each question was rated from 1 to 5, accordingly:



3 = neutral



Mean

Ranking Website/ Technology Assets Objectives: Suggestions for Improvement

____ 1.

Were you able to access the website from the

link provided?

_____ 2.

Did the website’s homepage load quickly on

your browser?

_____ 3.

Did you find the website interface simple and

easy to use?

_____ 4.

Were the course assets, such as videos,

homework, quizzes, laboratories and answer

keys in places you’d expect to find them?

_____ 5.

Were you overwhelmed at the amount of text on

the homepage? If too much, indicate with a “1”

or “2,” if just the right amount, “4” or “5.”

_____ 6.

Did the images support the text? Did you find

the images necessary?

_____ 7.

Font size just the right size in your browser? Too

small, “1” or “2,” just right “4” or “5.”

_____ 8.

Were video, homework, lab and other links

located in a logical place?

_____ 9.

Did all the links to lesson videos work?

_____ 10.

Did all links to written materials work?

_____ 11. Were you able to “contact me,” easily?

_____ 12.

Could you tell whether the link to the Discussion

Board worked?

_____ 13.

Was the protocol for asking a general question

using the Discussion Board clear?

_____ 14.

When the “instructor” replied on your theoretical

question on the Discussion Board, did you

receive notifications through your email?

Questionnaire modified from Norman’s Cognitive Walkthrough, and Cognitive Walkthrough for the Web (Mahatody,

Sagar, & Kolsi, 2010, pp. 747, 748)

Items scoring 0 will require revisions; 1 will be considered; 2 will be deemed satisfactory.

For objectives scoring a 1 or 2, IT experts

provided suggestions for improvement.


Appendix B.3

Content Delivery Analysis from Student Scores on Norton-published Exam

Section

2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8

Questions

on Exam

1 to 18 5 to 13,

18, 19

N/A 20 – 22 15 – 17,

23 - 25

21, 22,

24, 26 -

35

25, 35,

36

N/A

Mean

Scores

N/A N/A


Appendix B.4

Data Analysis of Learning Specialist’s ARCS Model Strategies Survey Part I, page 1

For each statement, the motivational strategy was rated accordingly:



3 = neutral



Mean

Ranking

Attention-getting Strategies

______

1. Shows visual representations (images, photographs, models) of any important

object or set of ideas or relationships (A2.1).

______

2. Provides worked out examples of every instructionally important concept or

principle (A2.2).

______

3.

Uses content-related anecdotes, case studies, biographies, etc. (A2.3).

______

4. Varies the format of instruction (content presentation, practice problems,

reflection, etc.) according to the attention span of the audience (A3.2).

______

5. Varies the medium of instruction (platform delivery, film, video, print, etc.)

(A3.3).

______ 6. Breaks up print materials by use of white space, visuals, tables, different

typefaces, etc. (A3.4).

______ 7. Shifts between student-instructor interaction and student-student interaction

(by way of labs, collaboration) (A3.6).

______ 8. Uses creativity techniques to have learners create unusual analogies and

associations to the content (A5.1).

______

9.

Embeds problem-solving activities at regular intervals (A5.2).

Relevance Strategies

______

1.

States explicitly how the instruction builds on the learner's existing skills

(R1.1).

______

2.

Uses analogies familiar to the learner from their past experience (R1.2).

______

3.

States explicitly the present intrinsic value of learning the content, as distinct

from its value as a link to future goals (R2.1).

______

4.

States explicitly how the instruction relates to future activities of the learner

(R3.1).

______

5.

To enhance achievement-striving behavior, provides opportunities to achieve

standards of excellence under conditions of moderate risk (R4.1).

______

6.

To satisfy the need for affiliation, establishes trust and provides opportunities

for no-risk, cooperative interaction (R4.3).

______

7.

Models enthusiasm for the subject taught (R5.3). Survey derived from Keller, J.M. (1987). The systematic process of motivational design. Performance &

Instruction, 26(9-10), 1-8. Letters next to the written strategy correlate to those in Keller’s Tables 1 to 4.


Data Analysis of Learning Specialist’s ARCS Model Strategies Survey Part I, page 2

For each statement, the motivational strategy was rated accordingly:



3 = neutral



Mean

Ranking

Confidence Strategies

______ 1. Incorporates clearly stated, appealing learning goals into instructional materials

(C1.1).

______ 2.

Provides self-evaluation tools which are based on clearly stated goals (C1.2).

______ 3.

Explains the criteria for evaluation of performance (C1.3).

______ 4. Organizes materials on an increasing level of difficulty; that is, structure the

learning material to provide a "conquerable" challenge (C2.1).

______ 5. Includes statements about the likelihood of success with given amounts of effort

and ability (C3.1).

______ 6. Explains to students how to develop a plan of work that will result in goal


______ 7. Attributes student success to effort rather than luck or ease of task when

appropriate (i.e. when you know it's true!) (C4.1).

______ 8. Allows students opportunity to become increasingly independent in learning and

practicing a skill (C5.1).

______ 9. Have students learned new skills under low risk conditions, but practice

performance of well-learned tasks under realistic conditions (C5.2).

______ 10.

Helps students understand that the pursuit of excellence does not mean anything

short of perfection is failure; the student learns to feel good about genuine


Satisfaction Strategies

______ 1. Verbally reinforces a student's intrinsic pride in accomplishing a difficult task

(S1.2).

______ 2. Allows a student who masters a task to help others who have not yet done so

(S1.3).

______ 3.

Gives verbal praise for successful progress or accomplishment (S3.1).

______ 4.

Avoids the use of threats as a means of obtaining task performance (S4.1).

______ 5.

Avoids surveillance (as opposed to positive attention) (S4.2).

______ 6. Avoids external performance evaluations whenever it is possible to help the

student evaluate his or her own work (S4.3). Survey derived from Keller, J.M. (1987). The systematic process of motivational design. Performance & Instruction,

26(9-10), 1-8. Letters next to the strategy correlate to Keller’s Tables 1 to 4.


Appendix B.5

Qualitative Data - Learning Specialist’s Questionnaire on Motivational Design – Part II

Question Response

1. Do the materials reflect a natural progression in skills, so as to minimize frustration

on the part of the learner?

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

2. What do you perceive to be the chief motivational value of the materials?

Comments:___________________________________________________________

____________________________________________________________________

N/A

3. Do you think learners will find the materials relevant to their needs and interests in

both the short term and long term?

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

4. Do you think there are adequate cues to gather the learner’s attention?

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

5. Are there adequate steps taken to maintain the learner’s attention?

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

6. Could more be done to facilitate learner confidence? If so, describe.

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

7. Besides satisfying a requirement for granting a degree, are there other elements

within the course that might generate learner satisfaction? If so, please describe

those.

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

8. When looking through the course materials and the website, did you get the

impression that course success can be realized with hard work?

Comments:___________________________________________________________

____________________________________________________________________

Y/ N

Questions are modified from Dick, Carey, and Carey, 2015, pp. 286, 287.


References

Blair, C.J. (2015, November 13). Organic chemistry’s reputation built on unnecessary anxiety.

The Oberlin Review. Retrieved from http://oberlinreview.org/9148/opinions/organic-

chemistrys-reputation-built-on-unnecessary-anxiety/

Dick, W., Carey, L., & Carey, J.O. (2015). The systematic design of instruction. Pearson Higher

Education.

Grove, N.P., Hershberger, J.W., & Bretz, S. L. (2008). Impact of a spiral organic curriculum on

student attrition and learning. Chem. Educ. Res. Pract., 9, 157-162.

Karty, J.M., Gooch, G., & Bowman, B.G. (2007). Teaching a modified Hendrickson, Cram, and

Hammond curriculum in organic chemistry: curriculum redesign to turn around student

performance. J. Chem. Educ., 84 (7), 1209-1216.

Karty, J. (2014). Organic Chemistry: Principles and Mechanisms. WW Norton & Company.

Keller, J.M. (1987). The systematic process of motivational design. Performance & Instruction,

26(9-10), 1 – 8. doi: 10.1002/pfi.4160260902

Keller, J. M. (2009). Motivational design for learning and performance: The ARCS model

approach. Springer Science & Business Media.

Kennepohl, D. K. (2012). Teaching science at a distance. In M. G. Moore & W. G. Anderson

(Eds.), Handbook of distance education, (pp. 670-683). Mahwah, NJ: Lawrence Erlbaum

Associates.

King-Sears, M. E., Johnson, T. M., Berkeley, S., Weiss, M. P., Peters-Burton, E. E., Evmenova,

A. S., Menditto, A., & Hursh, J. C. (2015). An exploratory study of universal design for

teaching chemistry to students with and without disabilities. Learning Disability

Quarterly, 38(2), 84-96.


Mahatody, T., Sagar, M., & Kolski, C. (2010). State of the art on the cognitive walkthrough

method, its variants and evolutions. Intl. Journal of Human–Computer Interaction, 26(8),

741-785.

Mayer, R. E. (2002). Rote versus meaningful learning. Theory into practice, 41(4), 226-232.

Roy, S., & Pattnaik, P. K. (2014). Some popular usability evaluation techniques for websites. In

Proceedings of the International Conference on Frontiers of Intelligent Computing:

Theory and Applications (FICTA) 2013 (pp. 535-543). Springer, Cham.

Wakita, T., Ueshima, N., & Noguchi, H. (2012). Psychological distance between categories in

the Likert scale: Comparing different numbers of options. Educational and Psychological

Measurement, 72(4), 533-546.

Documents

Running head: EVALUATION PLAN BLENDED …...Kay Seo, Ph.D. September 8, 2019 EVALUATION PLAN BLENDED ORGANIC CHEMISTRY UNIT 2 Table of Contents Background and