Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Enhancing Your Canvas Course
Formative Evaluation Report
Instructional Designer: Bryan Hauf
11/28/2016
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 1 of 20
Introduction The self-paced e-learning course developed intends to build faculty competency in designing Canvas
courses that facilitate ease-of-use no matter the format. Major themes include: 1) organizing materials
based on learner needs, 2) creating easily navigable course structure and sequence given LMS best
practices and Quality Matters Standards, and 3) developing activities and materials that facilitate and
support learner efficacy using Web usability/accessibility guidelines and Quality Matters Standards.
Two strategies were used to formatively evaluate the course:
1) Subject Matter Expert (SME) review comprising one reviewer to validate content, structure,
appropriateness, and overall design of the learning experience.
2) Small Group Evaluation comprising three members from the target audience to assess the
effectiveness of the training module.
Formative Evaluation Strategy #1: Subject Matter Expert Review
Introduction A subject matter expert review was conducted after completing the development phase of the
instructional solution. A colleague, an instructional designer of 15 years, performed the SME review of
the course given her expert level knowledge in all topics instructed. The review was performed by the
SME without observation over the span of 3 days from her home office.
An oral communication was performed to initiate the review. The following information was delivered
to the SME:
1. A brief introduction of the course.
2. Instructions for the SME to review each lesson, including the pre and post course activities. The
SME does not have to complete activities, but evaluate their appropriateness and quality.
3. Comments and feedback should be provided page-by-page using the template to be furnished
over email.
4. The SME is to review the following in the course:
a. Content quality
b. Content flow and structure
c. Ease of navigation
d. Readability (includes spelling and grammar)
e. Assessment Activities and criteria
f. Learning outcome alignment
g. Visual design of course
An email communication was sent to the SME linking to a document with instructions and a template to
recording findings, feedback, and recommendations. A copy of this document is located in Appendix A.
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 2 of 20
Summary of Findings The following table organizes the Subject Matter Expert’s comments and feedback/recommendations
and my, the instructional designer’s, response for addressing each item.
Page/Object Feedback/Recommendations Instructional Designer Response
Homepage Video is nicely done and very
cheerful to listen to.
In this sentence, remove the
word above “Click the Start Here
link above when finished to
begin this course.”
Revise sentence as recommended.
Lesson 0: Course Information In this sentence “Directions on
where to go next after reading
can be found on the bottom of
this page” remove the words
after reading.
Revise sentence as recommended.
Lesson 0: Course Structure & Activities Is it a requirement to write your
objectives with the phrase
“Given” for each? I am
wondering if this is the clearest
way to express the outcomes. It
makes perfect sense on Lesson 4,
but not so much on 1, 2, 3. How
about “Using various scenarios”
(because the will be using the
scenarios, correct?);
“Referencing research based
standards” (because they will be
referencing or referring to the
research based standards)
It may not be necessary to call it
a “self-paced web-based e-
learning module”. You could
remove web based. Is “learner-
usable” really a word phrase?
How about user friendly, which
you use later on the page.
Revise sentence to “self-paced elearning
module”.
Replace “learner-usable” with “user-
friendly”.
Revise learning outcomes 1-3 to reflect
recommendation. Will need to update
within each lesson and key assessment.
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 3 of 20
Lesson 0: Technology & Skill Requirements
Content the course instructor
with Subject Line "Canvas
Practice Course" if you do not
have one. Should be Contact.
need to have = should have
Replace “Content” with “Contact”.
Replace “need to have” with “should
have”.
Lesson 0: Pre-Course Self-Assessment Consider emphasizing “is not” in
Q2 and Q3 so participants do not
misread the “is not.”
Question 2 - Is it missing an
answer option? There does not
appear to be an option that is
“not” a QM standard
requirement - you have three
possible answers and all the
other MC questions have 4.
Answer option is not missing. All are
standards, but the learner should be
looking for the “essential” or “required”
standard. Update question stem to
emphasize.
General Comments on Introduction Module: The page design is easy to read and beautifully laid out. The structure adds to the usability of the content. I like the use of subtle icons and the graduated bands of color headers that do not overpower the content. Is there a purpose for double labeling these items? Is it for clarity? Consider removing ( ) and making the explanatory words part of the Practice Activity (Practice Exercise): Key Assessment (Evaluation): Lesson Wrap-up (Summary):
Remove parenthetical clauses as suggested.
Page/Object Feedback/Recommendations Instructional Designer Response
Lesson 1: Overview Same comment use of the word
“Given” in the objective.
Revise learning outcome 1 to reflect
recommendation.
Lesson 1: Best Practices for Organizing Your Course
Should the learning best
practices table have references?
Consider changing this wording:
The file should be identifiable
without having to click for the
content.
“The name of the file should
References are provided below the table
in article. Will explore adding academic
references instead of 2nd
party articles.
Add whitespace if possible, or break
table into two separate lists.
Like how the sentence is currently
worded. No plan to revise.
Add reference to a Faculty Focus article
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 4 of 20
clearly identify it’s contents.”
Im going to ask for a reference
on this statement on your audio,
not because I disagree, but
because it is a large sweeping
statement.
“Making sure that assignments
are always due on the same day
of the week and modules always
begin on the same day of the
week helps strengthen your
course structure and allows
learners to stay on task through
routine.”
Elearning best practices table -
consider adding white space for
readability.
summarizing the findings from “A Guide
to Creating Modular Courses”.
Add additional reference to University
course shell template design.
Lesson 1: Canvas Tools for Organizing Content and Navigation
On files under exploring Canvas -
is the use of spaces in file names
now acceptable due to
technology improvements? In
Canvas it is OK, but in general is
it still a best practice not to use
spaces?
For readability change the quick
check 1 answers. Do not use
Roman numerals. Use the
terminology as the answer
selections.
Remove roman numerals and include
terminology directly within answer
choices.
Lesson 1: Practice Activity It is great that Ross and Rachel
have moved beyond TV comedy
to course design. :-)
Reminder - if you have an
assessment question, using a
term that is not used in the
content of the course, even if it
seems obvious, is not a
suggested practice. Specifically,
“self-descriptive” is not used in
“Self-descriptive” is mentioned on the
content pages within the comparison
tables. Will not revise.
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 5 of 20
the content. (q1)
Lesson 1: Key Assessment Revise learning outcome 1 to reflect
recommendation.
Lesson 1: Wrap-up Nice use of infographic as
something learners can take
away from the course.
General Comments on Lesson 1: The use of a contrasting color with the continued same page design adds visual interest. Consider using less ( ) in the text. I would consider changing the wording on activities that are optional. I would call these practice activities (which I think you do later). I think if you call them optional, participants will mostly skip them. Consider saying this: Complete the exercise Lesson 1: Practice Activity. This activity is highly recommended for successful completion of the Key Assessment. So users will not automatically skip them. If you are worried about labeling optional and required, do so in one place - on the “How each Lesson is structured” page call them optional there.
Remove word “optional” from practice activities and only include it in introduction module, as recommended.
Page/Object Feedback/Recommendations Instructional Designer Response
Lesson 2: Overview Good use of color again.
Same comment use of the word
“Given” in the objective.
Revise learning outcome 2 to reflect
recommendation.
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 6 of 20
Lesson 2: Canvas Tools for Creating a Custom Homepage
“When students access the
course the first couple of times,
they need a homepage that
conveys the guidelines offered
by Quality Matters.” Suggest re-
writing this sentence. How
about: “When students access
the course the first couple of
times, they need the information
and guidance as suggested by
the Quality Matters guidelines.”
Revise sentence as recommended.
Lesson 2: Practice Activity On the mac, links, for example, in
this: “Taking a look at the
Example #1 Custom Course
Homepage, the longer page,
what recommendations would
you make to the instructor to
improve the page? Provide 2-3
suggestions in a brief narrative
(approx. 3-5 sentences)” the link
is not underlined and the
contrast is not enough to easily
find the link; see screenshot
image.
Adjust contract and force underlines to
make it visible as a link.
Lesson 2: Key Assessment – Develop a Course Homepage
The rubric is very detailed and
complete. Great!
In this sentence the rubric is
double linked: “Using the
Homepage Example #1 and
Homepage Example #2 provided
in this lesson and the grading
rubric.
View in a new window as your
guide.”
In this sentence “Incorporating
strongly recommended
standards is strongly
encouraged.” remove one of the
“strongly”.
Revise learning outcome 2 to reflect
recommendation.
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 7 of 20
General Comments on Lesson 2: Consider adding a story to the course. You have the right idea with the comment below, but the introductions to each module need a bit more warmth. I suggest changing this introduction into some kind of story that flows throughout the course: “Have you ever been given a complex task without guidance on how to begin or complete it?” Turn this into a student story that is used as a “case” throughout the modules. For example: “Today, Annette accessed her Canvas course to complete her course media project.” And then she could not complete this complex assignment because there were not enough details in it, blah, blah. I would even add an image of Annette as a reminder that the users that courses are being organized and designed for are people. Annette looking confused: http://www.webnode.me/confused-looking-person.html You have this kind of warmth in the assessment as where you use names (like Rachel and Ross) in the questions.
Rewrite introductions to each module to add a bit more warmth, as recommended. Use an Avatar to personalize message.
Page/Object Feedback/Recommendations Instructional Designer Response
Lesson 3: Canvas Tools for Creating Assessments
Nice use of thinglink and a
previous course asset to
demonstrate the course is
designed using best practices.
Accessibility information on
thinglink should probably be
included.
Accessibility information will be added
where thinglink is used, if it is available.
Lesson 3: Practice Activity In the practice activity Q1,
suggest using alignment since it
is the terminology used in the
previous explanation: “Suggest
that the assessment be revised
since it is not aligned with the
learning outcome.” Mastery
should probably not be used
unless mastery is defined in the
course.
Adjust practice activity to reflect
recommendation for using the word
“alignment”.
Lesson 3: Key Assessment – Create an Assignment
Nice use of video with captions.
The movement in the video adds
interest and gets attention.
Another very complete and
usable rubric, great job.
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 8 of 20
Page/Object Comments Feedback/Recommendations
Lesson 4: Overview The more I read the verb
leverage, the more I wonder if it
is measurable. I a slightly lest
subjective verb, even though you
are using a rubric.
“Leverage Canvas tools for
creating usable content and
instruction.”
Change “the same principles and
standards can be applied for
content design.”
the same principles and
standards can be applied to
content design.
Change sentence as recommended.
Lesson 4: Web Design Usability Guidelines
On the bulleted list for Usability
Guideline 1, be consistent with
“.” at end of sentences.
On the quick check, this
statement is unclear: “It is fine to
just accommodate the
accessibility of learners.”
Add periods to make bullet list set
uniform.
Change question stem to
“Accommodating solely accessibility
guidelines makes content usable for
learners.”
Lesson 4: Key Assessment – (Re)-Design a Learning Activity
May want to reconsider how
much is each rating of the rubric
for this assessment. IF each item
is worth 3 points, go ahead and
make each bulleted point a
rating.
Adjust rubric so each bullet point is
worth 3 points each.
Page/Object Comments Feedback/Recommendations
End of Course: Course Evaluation Change this for clarity: “Course
activities enhanced my
understanding of the skills that
are described in the course.”
And this
Adjust item 2 in survey to reflect new
statement recommended.
Add “for this learning activity” to the end
of question 9.
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 9 of 20
“The delivery method used was
appropriate.” appropriate for?
Synopsis All of the recommendations provided by the Subject Matter Expert are revisions that have been
identified as immediately actionable within the course. These revisions include grammar, spelling errors,
and sentence revisions for clarity identified throughout each lesson; adding sources to content within
lesson one; adapting the lesson 4 rubric will also be completed immediately; and updating the story
within the lesson overviews to add warmth.
Formative Evaluation Strategy #2: Small Group Evaluation
Introduction A small group evaluation was conducted after completing the Subject Matter Expert review to gain
feedback from members of the target audience on the effectiveness of the course. Three individuals
with varying backgrounds participated in the evaluation. Participant #1 has a strong education
background with experience teaching online and using e-learning practices. Participants #2 and #3 have
introductory level knowledge of the content presented. Each participant was passively observed in their
work office to document behaviors, struggles, issues, successes, questions, and comments as they
completed assigned lessons. In order to reduce time spent on the evaluation, each participant was only
required to complete a portion of the course:
Participant #1: Course Introduction, Lesson 4
Participant #2: Course Introduction, Lesson 1, Lesson 2
Participant #3: Course Introduction, Lesson 4
Meetings were scheduled 2 weeks ahead of time. Participants were given access to the course 2 minutes
prior to the review. Lessons were assigned at the start of the evaluation. The following information was
delivered to the SME prior to beginning the review:
1. A brief introduction of the course
2. Expectations from participant
a. Participant can ask questions if unable to progress or needs clarification
b. Must complete pre- and post-course assessments and Course Evaluation
3. Lesson to review
4. Instructional designer’s role as a passive observer
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 10 of 20
Summary of Findings Notes taken by instructional designer are located in Appendix B.
Participant #1
Page/Object Comments Instructional Designer Response
Course Information Learning objectives: move given statement to end of sentence
As per SME review, given statements will be adjusted.
Course Structure page
Practice activity description is not consistent with other bullets.
Make bullet consistent by revising the first sentence to include the word “Using”.
Pre-course Assessment
Answers are visible Adjust setting so answers are not visible; only display score.
Lesson 4 Content Page 1
Before accordion, “is” is missing from the sentence. Emphasized text color in overview text is distracting. Accessibility section: Last two bullets need to start with a verb.
Add “is” to the sentence before accordion. Change color emphasis to a neutral grey color. Change to all directional information in course. Change bullets to start with a verb for consistency.
Lesson 4 Content Page 2
Remove comma before WordPress. Convert Word documents to PDF.
Remove comma before WordPress. Documents will not be converted to Adobe PDF. Find as Word documents since they are viewable with the Box integration in Canvas.
Lesson 4 Practice Activity
Answer choices don’t appear to be randomized. Links in Quiz activities are not opening in a new tab causing the learners to have to re-enter the quiz from where they left off. Feedback not displaying in Question 3.
Randomize matching answer choices. Adjust links to open in a new tab/window. Add feedback for Question 3.
Lesson 4 Key Assessment
Since learners are self-scoring, may be beneficial to have an interactive rubric for the learners.
Turn Rubric documents into checklists that learners can download and interact with.
Participant #2
Page/Object Comments Instructional Designer Response
Course Dashboard Text is a tad confusing on dashboard. As per SME review, given statements will be adjusted.
Course Information Instructor intro does not indicate instructor’s role. Links to other activities in text is a bit confusing with having the navigation arrows. Purpose of Canvas Practice Course is not clear.
Include role of instructor on page. Modify course navigation instructions to indicate the purpose of linking to items that are delivered in sequence. Clarify instructions and purpose of having a Canvas Practice Course and knowing if you have such a course.
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 11 of 20
Lesson 1 Content Page #2
List in text before accordion is in a different order than accordion items.
Reorganize list to be in same order as accordion items.
Lesson 1 Practice Activity
“instructions” header is duplicated in quiz. Question 2: Naming convention is not defined in content. Missed that this question was multiple answer.
Remove one of the instructions header across all practice activities. Update content to define naming convention. Emphasize question stem to indicate that learner should select all that apply.
Lesson 1 Key assessment
Change instructions to directions or remove header.
Same issue as practice activities. Adjust so only one “instructions” header is on page.
Lesson 2 Content Page 2
Instructions for annotations were a bit confusing.
Clarify instructions for how to interact with the ThingLink examples and the key used for annotations.
Participant #3
Page/Object Comments Instructional Designer Response
Course Information Links to other activities in text is a bit confusing with having the navigation arrows. Learning outcomes include the word “faculty”. Comes across as being critical of this audience. “Your First Tasks” is confusing since it links to a task and then progress using the next buttons. Font appears to be compressed. The word “final” before grade indicates a grade is given when the activities are7 not graded.
Modify course navigation instructions to indicate the purpose of linking to items that are delivered in sequence. Replace word “faculty” to “learners” across all outcomes. Do not link items. Just list the first tasks learners are to complete and direct them to click on the “Next” button. Font is standard to the LMS. Will add instructions for how to zoom in using browser functions. Remove word “final” from grade.
Pre-course self- assessment
Question 3 response b. Specific and descriptive criteria is unclear.
Will not change since this is the purpose of the pre-course self-assessment. Learners will know the terms after completing the content.
Lesson 3 Overview Content seems dense and a lot info to consume.
Clarify that learners are not expected to memorize content.
Lesson 3 Content Page #1
Accordion: Sentences are long and complicated.
Review sentences and adjust to be less complex if needed.
Lesson 3 Key Assessment
Time for completing the assessment may be better at the beginning so it is easily identifiable how much time is needed right from the start.
Move time and grading statement to top of the page.
Synopsis All of the issues identified through observing each participant are immediately actionable within the
course. These revisions include fixing grammar errors and typos; improving navigation instructions;
adjusting assessment settings and instructions; adapting content to be less dense and easier to read by
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 12 of 20
rewriting complicating sentences; improving the usability of pages by adjusting colors of directional
elements. Long-term revisions may be identified after learners complete the course in full without
observation by analyzing the post-course evaluation, pre- and post-course evaluations, and achievement
of learning outcomes in key assessments.
Summary The practicum project allowed me to demonstrate improvement since beginning the educational
technology program by applying instructional design and technology practices final time. By
incorporating many of the lessons learned throughout the program, I was able to analyze the target
audience while also considering the diverse needs of learners, leverage instructional media principles to
provide variety in content delivery, and use integrated course design principles to align assessments and
instruction with the performance outcomes all in the pursuit of developing an effective instructional
solution. But that isn’t to say there weren’t additional lessons to learn.
In progressing through the project, I learned to be realistic with the time available. The breadth of the
course developed was perhaps too much for the time allotted. By having to rush through the
development phase, more time was spent on earlier lessons leaving some gaps in content coverage in
the final lesson. I should have used the opportunity to focus on developing a product that is still a
weakness, such as a self-contained, self-paced course geared towards corporate professional
development. In hindsight, the content may have been better if delivered as separate content packages
instead of one large course given the volume of information presented in each. In the future, I would
like to expand my skillset by learning how to use instructional authoring programs such as Articulate
Storyline and Adobe Captivate to provide me with more work opportunities in the future.
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 13 of 20
Appendix A
Subject Matter Expert Evaluation Template Directions for Subject Matter Expert
The SME will review the course for the following:
Content quality
Content flow and structure
Ease of navigation
Readability (includes spelling and grammar)
Assessment Activities and criteria
Learning outcome alignment
Visual design of course
Use the table below to record feedback and recommendations on a page-by-page basis.
Page/Object Comments Feedback/Recommendations
Homepage
Lesson 0: Course Information
Lesson 0: Course Structure & Activities
Lesson 0: Technology & Skill Requirements
Lesson 0: Pre-Course Self-Assessment
General Comments on Introduction Module:
Page/Object Comments Feedback/Recommendations
Lesson 1: Overview
Lesson 1: Best Practices for Organizing Your Course
Lesson 1: Canvas Tools for Organizing Content and Navigation
Lesson 1: Practice Activity
Lesson 1: Key Assessment
Lesson 1: Wrap-up
General Comments on Lesson 1:
Page/Object Comments Feedback/Recommendations
Lesson 2: Overview
Lesson 2: QM Standards for Learner-Centered Course Introductions
Lesson 2: Canvas Tools for Creating a Custom Homepage
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 14 of 20
Lesson 2: Practice Activity
Lesson 2: Key Assessment – Develop a Course Homepage
Lesson 2: Wrap-up
General Comments on Lesson 2:
Page/Object Comments Feedback/Recommendations
Lesson 3: Overview
Lesson 3: QM Standards & Best Practices for Assessments
Lesson 3: Canvas Tools for Creating Assessments
Lesson 3: Practice Activity
Lesson 3: Key Assessment – Create an Assignment
Lesson 3: Wrap-up
General Comments on Lesson 3:
Page/Object Comments Feedback/Recommendations
Lesson 4: Overview
Lesson 4: Web Design Usability Guidelines
Lesson 4: Canvas Tools for Creating Usable Learning Activities
Lesson 4: Practice Activity
Lesson 4: Key Assessment – (Re)-Design a Learning Activity
Lesson 4: Wrap-up
General Comments on Lesson 4:
Page/Object Comments Feedback/Recommendations
End of Course: Post-Course Assessment
End of Course: Course Evaluation
General Comments on End of Course Activities:
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 15 of 20
Appendix B: Passive Observation Notes
Participant #1
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 16 of 20
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 17 of 20
Participant #2
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 18 of 20
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 19 of 20
Participant #3
EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013
Page 20 of 20