080412 Edweb Project Report

7/31/2019 080412 Edweb Project Report

1/19


2/19

Project Title: policyIQ University Learning Materials for All policyIQ AudiencesProblem or NeedGoals and ObjectivesAnalysis of your learners, content, and settingDesign of an Instructional Plan

Prototype Materials and RationaleFormative EvaluationSummative Evaluation

Problem or NeedUsers of the application (policyIQ)need easy access to learning materials that aresuitable for their specific needs and circumstances. policyIQ is a content managementapplication built on a .net platform over a relational SQL database. Our clients utilizepolicyIQ to improve data integrity, to achieve centralization and standardization of theircontent, to manage shared development of content using the automated workflowcapabilities, to maintain version control, to access powerful and reliable reporting

capabilities and to communicate and share information with relative stakeholders. Thetypical applications of policyIQ manage policies, procedures, certifications,specifications, contracts, compliance with various laws and regulations, internal andexternal audit documentation and the automation of various processes using the onlineforms management functionality.

Most of the learners are expected to implement this content management system as afunction of their current role or responsibility and have very little time to devote tolearning about the application and the implementation of the system. Our organizationoffers support via three primary channels to facilitate the implementation process for ourclients: 1) the policyIQ team works very hard to continuously improve the user interfaceso that it is intuitive, 2) the team develops and maintains a variety of learning materials

targeting the diverse users and their needs and 3) we train and support our companysconsultants so that they may provide on-site support of the implementation. The secondchannel here is the focus of this project; to bring together a variety of learning materials(access to synchronous and asynchronous sessions, written materials, job aids, acommunity of users and expert assistance) into one well-organized and easy to useinterface.

In the absence of this product-specific learning materials platform, the users of theproduct may or may not seek out or locate the materials that meet their specific needsand they may or may not have the experience or technological savvy to successfullyimplement the application; thereby risking exposure in all of the areas mentionedpreviously (Our clients utilize policyIQ to improve data integrity, to achieve centralization

and standardization of their content, to manage shared development of content using theautomated workflow capabilities, to maintain version control, to access powerful andreliable reporting capabilities and to communicate and share information with relativestakeholders).

This interface, dubbed policyIQ University, will reside on the web and will provideaccess to a variety of materials that may be accessed via the web, CD, teleconference,or face to face interaction.


3/19

To be clear, the scope of this project is a relatively wide span: it is an interface providinguser-friendly access to a complete set of learning materials. The intention is provideusers a fast and convenient means to their desired learning material. The product user(learner) may choose to print out a cheat sheet and begin working in policyIQ in a matterof a few minutes, or the user may decide to participate in a series of synchronous orasynchronous courses totaling a few hours.

Back to Top

Goals and ObjectivesThe learners (a. k. a. product users) will be able to record, organize, track, report on and

share business information, with confidence in the data integrity, in a more streamlined

and automated fashion resulting in improved efficiencies in the work place.

Stakeholder Results

Basic Users can demonstrate:

How to login

How, generally, to locate information

Where they might find critical updates (using

Pages to Read)

Content creators in addition to the above, can demonstrate:

How to create and edit content

How to move content through the organizations

workflow using policyIQ

How to publish content

How to review their own published work

Those in management oroversight roles

in addition to the above, can demonstrate:

How to access reports specific to their

organization

How to create custom reports

Performance of annual Account Reviews of a clients site will indicate that users areappropriately creating content; linking relative content, files and web links; and are

utilizing reports to meet auditor or oversight requests. Clients will more consistentlydocument handoff (workflow) procedures. Semi-annual account survey will yield anoverall usability rating at satisfied or very satisfied. In addition to these indicators, thepolicyIQ team will see a measurable decrease in the direct client contact with thesupport desk, the overall client population will increase their rate of expansion to bothmore users and to more uses and policyIQ sales as a result of existing client referralswill increase.


4/19


5/19

Most of these learners work in e-environments on a regular basis and are accustomed tousing various electronic systems. Most are self-directed. I would estimate that 5% ofparticipants require personalized additional support.

Learners are motivated to improve their work environments, become more efficient and

to guide their teams to follow suit (most are project leaders of some sort). Factors thatderail success: 1) implementers who do not follow best practices (recommended duringcourses), 2) lack of documentation of procedures, practices, handoffs supportingsustainability of the implementation (turnover in these organizations is high; requiringdocumentation of implementation decisions in order to maintain uninterruptedmaintenance and flow of information). The learners in this environment are verymotivated to meet the needs highlighted above and will be motivated to gather moreinformation and a better understanding if the materials are well-organized and suit theirparticular learning style. The policyIQ team has been working to provide a variety oflearning materials from self-directed to instructor led, electronic written manuals to videodemonstrations, and quick cheat sheets to support from a subject matter expert.

These are typically adult learners in a corporate environment. We work to supportvarious learning styles by including audio, visual, reading materials and practice orhands-on exercises.

Meeting business deliverables is the greatest incentive for desired performance. Theexercises may pose both incentive for desired and disincentive for undesiredperformance (by nature of the built-in evaluation). We provide a product that is one toolin our clients toolboxes which helps them to increase efficiencies, comply with laws andregulations, achieve cost savings and communicate business information to their variousstakeholders. We do not profess methodology to achieve these various goals, butprovide guidance on the practices and use of the products functionality that has beenparticularly useful to our clients. We are very careful to under-promise and over-deliver,

where our products features and functionality are concerned. We are also careful toprovide our product and support services with the utmost integrity; which means that wedo not make claims that we cannot assuredly support. We communicate what ourproduct can do and share case studies and examples of what has been accomplishedusing our product; this is the extent to which we provide incentive for our clients to bothpurchase our product, to implement the application according to best practices and toparticipate or engage with our various learning materials.

Learners will need high speed internet and Internet Explorer to practice and applylearning within their policyIQ application. The courses will not necessarily require highspeed internet access (perhaps downloaded and run locally). Learners typically willcomplete the entire course in one sitting and begin to apply lessons learned almost

immediately (within the week, typically). Typically, they will continue to use theinformation on a regular basis (between daily and monthly).

Learners supervisors are in various roles across various functions (CFOs; SeniorManagement in Finance, Internal Audit, HR, IT, etc).

Many users of the application choose to begin using without training and are generallysuccessful. Participating in the learning experience is not forced. Successfulimplementation of the application is not typically voluntaryonce all of the effort to


6/19

choose an application has been exerted, they typically expect to get up and running assoon as possible. There is a lot of motivation to be successful and most will participatein some sort of training.Back to Top

Design of an Instructional PlanAs the scope of this project has shifted from the original narrow presentation of a singleasynchronous course to a platform for accessing all policyIQ learning materials, thisinstructional plan takes on a new look as well. I started with the plan for a specificcourse addressing the management of content (workflow) in the policyIQ product andreporting on that information. I have similar outlines for the other courses provided bythe policyIQ team. I see the development of an instructional plan for this product as amulti-phase effort with the initial focus on the plan for the interface, the second phasefocusing on the continuous improvement of the instructional plans for each of thecourses (first synchronous, then asynchronous), and a third phase (which might bemanaged simultaneously with the second phase) focusing on the improvement of othertraining materials.

I need help with articulating my instructional approach as it relates to learning theory.This course was my first exposure (that I can recall) to learning theory and I need todevelop a better understanding to make a confident statement about my approach.

This project, the design and implementation of policyIQ University, will incorporate allfive of the common instructional values:1. Learner Centered

a) Students can choose whether they want to participate in instructor-ledsynchronous training or review recorded sessions

b) Students can move ahead, go back, or jump into training at the level that theyfeel is appropriate

c) Students can choose to access the online Help guide to read more about a topicwhen they wish

2. Sociala) From website, can access Live, Instructor-led Training page which will include

registration for weekly Q&A sessionb) Button within recorded sessions will direct to window to direct a question to

choice of audiences: Ask the Expert or policyIQ Community Board (thismessage board is not yet up and running, but will be soon

3. Contextuala) Demo will review common uses of application for management of policies,

procedures, controls, compliance, formsfamiliar to the majority of policyIQstudents

b) Students will have an opportunity to practice and enter their own examples andinformation

4. Activea) Students will have an opportunity to practice and enter their own examples and

informationb) Students can engage in discussionsc) Students can move around and select the sessions that they are interested in

reviewing


7/19

d) Students can (will eventually be able to) test their skills and submit results forCPE credit

5. Supportivea) Ask the Expertb) policyIQ Community Boardc) Register for regularly scheduled Live Q&A

d) Select Support to expose menu or directly link toi) Online Helpii) Frequently Asked Questionsiii) Contact Us

(1) Link to Submit a Support Question(2) Email Address(3) Toll-free phone number

Back to Top

Prototype Materials and Rationale1. Navigation Design Rationale:

a) Sparse or rich navigation:i) I believe that somewhere in the middle is best for my project. Im leaning a

little more toward sparse navigation.ii) I plan to create a course shell used to present and teach learners, our clients,

how to use our software application, policyIQ. The application accomplishessome complex tasks, sometimes with multiple pop-up windows open at thesame time and sometimes the pop-ups contain scrolling information. Thesevisuals are not something that I can change. Because of this, I think that Ishould try to minimize additional distractions and keep navigation of theeLearning course as simple as possible so that focus on the applicationsfunctionality can be maintained.

b) List the navigation mechanisms you think your site will employ.

i) Menu:(1) I would like to include a primary menu with controls for exiting the training

session, accessing support materials, accessing other referencematerials, going Home to a page that allows students to access all typesof trainings offered (Im sure Im leaving some things out heremore tocome).

(2) The layout will also include a navigation area containing a menu thatallows learners to launch to other topic sections or chapters of eachcourse. I want this navigation bar to be a constant on the screen so thatlearners can choose to shift to another section when they would like.(a) Within this menu, Id like to include a submenu with options to review

the Demonstration, navigate to the Practice exercise(s), navigate

to the Evaluation and Review and Submit Evaluation Results(important for CPE credits). (Im not married to these terms and willlikely change them, but this explains where Im headed at this point.)

ii) Play/PauseControls: Because the course contains several cannedrecordings, Id like learners to have the option to pause, rewind, fast-forwardand play the portions that they are most interested in reviewing.

iii) Next/Previous Buttons: While it will be possible to use the menu to go to anearlier or later section, Id like to include Next/Previous buttons for tworeasons: redundancy (it may be natural for learners to look for them in such


8/19

an interface) and transitioning from a practice/interactive portion back torecorded sections might be more intuitive with these buttons.

iv) Im not sure what this might be calledId like to have a control appear,maybe a menu, in the main screen when a demonstration (recording) ends.Id like the learner to be presented with choices:(1) What would you like to do next?

(a) Participate in the practice exercise (this launches them to that portion(b) Go directly to the Evaluation (launches to evaluation)(c) Go to next demonstration(d) Save work completed so far and exit (important for CPE credit)(e) Exit

v) Im also not sure about how the Submit Evaluation Results stuff will work. Itwill be important for all results to be sent to an approver so that CPE creditcan be awarded. I think it can maybe be sent automatically following anevaluation when participants achieve at/above a certain percentage of correctresponses.(1) Relative to this evaluation stuff, the participants will need to register their

names, company, contact information (perhaps fill out a mini-form at the

top of the evaluation) in order to be awarded CPE credits.2. Five Common Instructional Values

a) Learner Centeredi) Students can choose whether they want to participate in instructor-led

synchronous training or review recorded sessionsii) Students can move ahead, go back, or jump into training at the level that they

feel is appropriateiii) Students can choose to access the online Help guide to read more about a

topic when they wishb) Social

i) From website, can access Live, Instructor-led Training page which willinclude registration for weekly Q&A session

ii) Button within recorded sessions will direct to window to direct a question tochoice of audiences: Ask the Expert or policyIQ Community Board (thismessage board is not yet up and running, but will be soon

c) Contextuali) Demo will review common uses of application for management of policies,

procedures, controls, compliance, formsfamiliar to the majority of policyIQstudents

ii) Students will have an opportunity to practice and enter their own examplesand information

d) Activei) Students will have an opportunity to practice and enter their own examples

and information

ii) Students can engage in discussionsiii) Students can move around and select the sessions that they are interested in

reviewingiv) Students can (will eventually be able to) test their skills and submit results for

CPE credite) Supportive

i) Ask the Expertii) policyIQ Community Boardiii) Register for regularly scheduled Live Q&A


9/19


10/19

i) Information related to live sessions is grouped in several places (took care toinclude redundant navigation)

ii) Information related to recorded sessions is grouped in several placesiii) Help/Support information is groupediv) Courses are grouped into a menu/list

5. Typography

a) Alignment Text throughout the site is left justified. Rather than a traditionalheading, I have included tabs that also have left-justified text. I haveintentionally offset the heading from the body text to have a more realistic tabappearance.

b) Line length While this site has a limited amount of text, the text that is includedhas a limited line length for simplified reading. Boxes have been added aroundtext areas to break up the visual stage and help readers to focus on the short textdescriptions and information.

c) White space The site incorporates ample white space. Where 14 point text isused, there are 17 points of leading space. Where 12 point text is used, thereare 14 points of leading space.

d) Typefaces The typeface used throughout the site is Arial. This was changed

from Century Gothic; a font that may not be universally recognized. Arial is amore standard font and is easy to read.

e) Type size the font sizes used throughout the site are somewhat large. It maybe necessary to reduce all fonts, across the board for appropriate appearance onthe web. Here are the fonts as they are presented in PowerPoint:i) Main Navigation Bar and Support tab are in 14 pointii) Headings (in tabs) are in 16 pointiii) Main sections in large, left area of screen directing to live and recorded

training are in 14 pointiv) Secondary sections on right side of the screen incorporate 12 point textv) Body text (very little body text is used) is in 14 pointvi) Navigation through specific courses in recorded training menu uses 14 point

for course types/sections and 12 point for course titles and chapters withinthose titles

vii) The support/resources navigation bar at the foot of the pages and the legal-ese are in 10 point

f) Emphasis Apart from varying the size of text, very little emphasis is used in thesite.i) The headings and main navigation bar at the top have a different color text

than the body of the site.ii) A bold font is used to identify the current page that the user is in (bold Home

when in the Home page, for example).iii) Rolling over a selection that one might launch to will cause that item to be

underlined. If it is chosen and the user is moved to the new page, the

underlined item then becomes bold (this might not come across as plainly asId like in PowerPoint).

iv) In the recorded training viewer, the course chapter that is currently playing isoffset using color differentiation (red text versus the dark gray for other text).

v) The course types/sections that are not active are in a lighter shade of gray.vi) Spacing and indentation is used to differentiate between the categories of

services (in Suite of Services) and the more specific service type listed below.It is also used to differentiate between Course Types, Courses, and Chapterswithin a Course in the training player.


11/19

g) Graphic Text the only graphic text that I anticipate using on the site, at thistime, is in the logo. I plan to use anti-aliasing to yield a crisp image.

6. Categorization of Visuals / Graphics I am not certain if I am making good use ofvisuals in this interface. The actual course that will initially be presented in theinterface will be a simulation of how our application works. The course willincorporate callouts, highlights and the like to emphasize or draw attention to what is

being presented or emphasized at that point in time.a) Decorative

i) While it is not currently visible, I plan to have an image on my Home page,only for the purpose of motivating the learner. This image will be of a studenton the computer or something of the like.

b) Representativei) Within the actual asynchronous course, the technique for carrying out

activities will be simulated for viewers to observe. The visuals that will beadded will be in the form of callouts adding text to emphasize what theimages are demonstrating.

ii) There are buttons in the player pages that are fixed on the screen and therollover will lead to display of text describing the feature or action that will

take place by selecting the button: The 7 player buttons include Previous,Stop, Rewind, Play, Pause, Fast Forward, Next.

iii) Four Buttons for navigation to various learning tools are presented in theplayer pages. Rolling over these buttons will display the text descriptions ofthe actions that may be taken: Play Video Demonstration, Go to PracticeExercise, Ask for Help, Test Your Skills

c) Organizationali) The interface in which the courses will be displayed has a number of

organizational visuals intended to provide a path for various types of learnersand to guide them to the appropriate training forum and courses. Theseinclude the organization of courses into levels or forums (such as beginning,intermediate, advanced versus recorded, live online, live in-person).

ii) The course list is presented in a hierarchy that is similar to a table ofcontents.

d) InterpretiveThe interface that Ive built for my EdWeb does not include anyInterpretive visuals, but the courses that will eventually be presented through thisinterface will. Occasionally, complex concepts are presented in a simplifiedvisual model in PowerPoint and then demonstrated in the application.

e) Transformational While this interface does not include Transformationalvisuals, they are used in some of the training sessions. For example, to illustratethe many-to-many linking capability within the relational database and how itmight apply to the client, an animated model of the relationships and their linksmay be created in PowerPoint (or in another application now!).

Back to Top

Formative Evaluation1. Goals of formative evaluation

a) Achieve management buy-in for implementation of such a system in ourorganization

b) Learn whether all of the redundant navigation is helpful or confusing (this may bedifficult without a more functional prototype)


12/19


13/19

(2) For the policyIQ University and its contents to have any utility for thelearner, a basic understanding of what is policyIQ and how they may beapplying the product is required.

b) With regard to the specific Content Management session presented in theinterface as an examplei) Objectives the participant will be able to:

(1) Navigate, confidently, throughout the policyIQ product(2) Understand the variety of possible uses of the workflow stages within the

Create And Edit and Approve modules(3) Create a Page of content and move it through the workflow(4) Generate reports and to analyze information from various perspectives

ii) Prerequisites(1) A basic understanding of what is policyIQ and how the learner may be

applying the product is required.4. Learning environment

a) Learners are typically in one of two categories:i) Mid-upper level employees in an organization, usually in the Accounting,

Finance, or IT departments. They are usually very familiar with their business

information and accustomed to a very different, manual way of managing theinformation.(1) These learners are usually met with competing priorities and have limited

time and energy to commit to learning and implementing a newapplication (ironically, implementation of the application typically frees upa significant amount of their timeits getting there thats the challenge).

(2) Some of the learners have experience with databases and understandbasically how they work. Others are very concrete and struggle tounderstand that information is managed relationally and redundant copiesof documentation are unnecessary; still information can be accessiblefrom many different pathways.

(3) This learning community is also split into those that are good at

developing and implementing a vision which efficiently utilizes staff attheir various levels and strengths; and those that are afraid to relinquishcontrol, keep everything close to the vest, and manage all informationthrough a bottleneck of one or two trusted stewards.

ii) Associates or consultants that go into organizations to help them with theirpolicies & procedures, contracts, compliance with laws and regulations andchange management initiatives.(1) This group is usually better at developing the big picture vision and

managing the implementation of the application like a project.(2) This group is typically flexible and has experienced various database and

other software applications.(3) These learners are usually expected to know more than they do and must

get up to speed in a very short period of time so that they are qualified tohold the authoritative/project manager role.

iii) Both groups of learners are typically subject to the technology thats beenafforded to them. Internet connection speed and access to a quiet workspace can be challenges for both groups.

5. Media characteristicsa) Typically, all training and learning materials are delivered over the web in the

form of written materials, asynchronous (recorded) sessions, and synchronousinstructor-led sessions that may be accessed online.


14/19

b) Users have access to training PowerPoint presentations that they can tailor fortheir specific audiences.

c) Guidance is provided for the development of an electronic written manual orguide with information on the navigation, implementation, and procedures thatare specific to each organization.

d) Online, instructor-led training is also provided in a separate setting for

consultants who may be hosting live (in-person) training for clients.e) In any case, the clients particular application (web-based) may be presented

along side the training materials. policyIQ requires internet access via InternetExplorer (version 6 or better).

6. Questions to be answered (most important)a) Learning Effectiveness

i) How clear is the narration?ii) Is the content in the right order?iii) Is there an appropriate amount of practice? (within each subchapter; not built

into prototype yet)iv) How is the quality of feedback? (within practice and test your skills sections;

not built into prototype yet)

v) How is the quality of learner interactions? (with content everywhere and inlessons where they can post to a message board or ask a policyIQ expert;not built into prototype yet)

vi) Do the graphics make sense?b) Learner Interest/Motivation

i) Is the content interesting?ii) Are the levels of learner challenge appropriate?iii) Will the users be willing to use?iv) From the learners perspective, what is the perceived value of learning?v) How much time is spent learning?vi) Is it the appropriate amount of learning material (too much, too little, just

right)?

c) Content Qualityi) Is this the right material?ii) Whats missing?iii) Is it overdone (too much noise)?

d) Technical Qualityi) What recommendations do you have to improve the visual quality of the

interface?ii) Are sounds used effectively throughout the interface? (not built into

prototype)iii) Is media used effectively?

e) Ability to Implementi) Is the interface easy to use?

ii) Would the user be willing to use the site?iii) Is this application a good fit for the learning need?iv) Will such an application be supported by policyIQ/Resources management?v) Is the proper support in place or feasible?

7. Parts of instruction to be evaluatedthe prototype does not function as envisionedfor the actual site.a) While it may be difficult to move smoothly through the various pages in the

prototype, it is important that I learn whether the plannednavigation is intuitivei) Focus on the Home screen and all components in the this view


15/19

ii) Focus on the interface that has the courses listed in tabsiii) Focus on the player with the actual courses

8. Stages and subjectsa) Stage I (Initial reaction): Small group of content experts (a team of my

colleagues) to perform initial evaluation of overall interface plan and content plan.b) Stage II: Small group of members of the target audience (this might be a stretch,

politically, and I may need to resort to using a small group of consultants) toperform more in depth evaluation of content (they will also be invited to share anyother reactions).

c) Stage III: Following the build of links to outside interfaces (Help content,Message Board), Small group of content experts (a team of my colleagues) toperform a comprehensive evaluation.

d) Stage IV: A small group of members of the target audience/consultants (maybe2) to perform a field test of site

e) Stage V: A small group of members of the target audience/consultants (maybe 2)to perform a field test of site to review changes made

9. Data gathering tools and methodsa) Stage I: Interview with questionnairewill interview all members of the group at

once (really giving them a tour while asking questions) and will also give (inadvance) each of them a set of questions to address and respond to in writing.May also give them full access following the presentation to go back through siteand take any additional notes.

b) Stage II: A questionnaire will be sent with access to the site to perform an indepth evaluation of the content (they will also be invited to share any otherreactions).

c) Stage III: A questionnaire will be sent with access to the site to perform acomprehensive evaluation. Will provide instruction to use the interface toindependently arrive at specific targets and take any additional notes.

d) Stage IV: Interview with questionnairewill interview all members of the groupat once (really giving them a tour while asking questions) and will also give (in

advance) each of them a set of questions to address and respond to in writing.Will also give them full access following the presentation with instruction to usethe interface to independently arrive at specific targets and take any additionalnotes.

e) Stage V: Interview with questionnairewill interview all members of the groupat once (really giving them a tour while asking questions) and will also give (inadvance) each of them a set of questions to address and respond to in writing.Will also give them full access following the presentation with a timeframe tointeract and take any additional notes/make any additional recommendations orobservations.

10. Reports of evaluationa) At the conclusion of each stage of evaluation, I will evaluate the evaluation and

document:i) A summary of the processii) An outline the changes made based on the evaluationiii) A description of the resources (time, money, etc) spent on the evaluation

processiv) An evaluation of this stages value in the overall product improvementv) Any reactions or recommendations for changes to future evaluation stages or

projects based on the learning from this stage of evaluation


16/19

Formative Evaluation - Implementation of Stage I:1. The question you selected to be answered.

a) I was primarily interested in learning if the navigation was user-friendly andprovided complete access to the various resources that our organizationprovides. I asked many questions that would approach this topic from variousperspectives. Secondarily, I asked questions about the quality of the site

(content, technical, visual) and ability to implement.2. The test subject(s) involved in the formative evaluation. Test subjects need not be

listed by name unless they agree to be named. Full names of K-12 students shouldnot be provided. For expert review, the expert should be identified to add credibilityto the results.a) I was not permitted to go to members of my target audience at this early stage.

While I understand that this would be valuable, my organization has not agreedto implement this project for the business and would not expose it to our targetaudience at such a preliminary stage of development. As such, I was able toengage two subject matter experts from my work team to perform this evaluation.i) Caroline Villella, Operations Director, oversees all projects that touch our

clients (the target audience). She is abundantly aware of the many tools and

resources that we provide via a variety of decentralized sites.ii) Chris Burd, Product Director, is an expert on the functionality of the software,

shares responsibility for delivery of live training sessions, acts as a liaisonrepresenting all users requests and feedback to the development team andis also aware of the many pathways that our clients must travel to arrive atthe desired resources.

3. The evaluation process followeda) Stage I: Interview with questionnaire

i) Interviewed both members of the group at once (really giving them a tourwhile asking questions).

ii) I gave each of them (in advance) a set of questions to address and respondto in writing.

iii) I also gave them full access to the site following the presentation to go backthrough and take any additional notes.

4. Any deviation from the evaluation process or any unexpected event that may haveoccurred during the evaluation.a) I followed the process exactly as I had planned. In order to do so, I had to

develop the written questions (see sample appended to this write-up), an outlinefor the session which, in summary, went as follows:i) 5 minutes Introduction and background on the degree program, its

objectives and how Ive chosen to focus my EdWebii) ~10 minutes Part I: Show flat view of Home in EdWeb and walk through

first 10 questionsiii) ~5-10 minute tour of the EdWeb prototype

iv) 5 minutes Part II: Following tour, respond to 4 questions addressingnavigation, completeness, learner motivation.

v) 5 minutes Part III: took some time to address remaining questions andinvited them to continue exploring prototype on their own while addressingfinal questions regarding navigation, learner motivation, feasibility (support inour organization and technical)

b) I did not pick a single burning question as the plan described. When workingthrough the plan for practical application, I found that I wrote the Goals of theFormative Evaluation with the details of my prototype and EdWeb on top of


17/19

mind; however, I wrote the Questions to be Answered from more of a text bookperspective and they were really out of context. I realized, as I was preparing forthe implementation, that I do still think the questions I had written are important,but I should have organized them so that they were more aligned with my goals(as my goals are aligned with the need that I am aiming to fulfill). My EdWebreally encompasses the entire platform that will bring together all learning

materials that are offered by my organization including the content offered insynchronous and asynchronous forums. Therefore, the focal point of myquestions was adjusted accordingly to the sites navigation, quality andcompleteness.

5. A summary of the data / results of the formative evaluation.a) The subjects responded to 10 questions regarding navigation that required

evaluation of the visual layout and language without hands-on experience andprior to any explanation. Following a guided tour of the site, they responded toquestions addressing quality, completeness and feasibility of implementation.They gave feedback on the navigation, made suggestions for improving clarity,noted some additional resources that might make the site more complete and

judged the implementation of the project as feasible and supported by the

organization. I also received positive feedback on the evaluation process andconduction of the evaluation session.

6. Your analysis of the data / results, including implications for changes to the design(instructional, visual, interface) of your EdWeb.a) The results were generally positive. They responded accurately to navigation

questions and pointed out that one of the responses might not have beenobvious without inside information. The subjects did help me to identify someelements that could make the site more complete. I also received the all-important green light; a generic message that my management group issupportive of my continued pursuit of this project.

7. A description of the changes you plan to make to your EdWeb, including rationale forNOT making changes suggested by the formative evaluation data / results.

a) I will include on screen text or hover-over text that describes what is thesmartercommunity. This link in the Home interface might not be intuitive tonewcomers to policyIQ. The smartercommunity is the forum for connecting withother policyIQ users.

b) I believe that I will NOT add case studies or testimonials to this site as suggestedby one of the evaluation subjects. Our organization has a website for marketingour product which includes case studies (policyIQ implementation successstories of our clients). I believe that policyIQ University will be a link from thepolicyIQ website to access training and support materials. I do not feel that it isnecessary or appropriate to make room for this sales-y material on the trainingsite, which is intended for the use of our current clients (not prospects).

c) A section of the Home interface is intended to attract those learners seeking

training that suits their level of understanding. I will:i) Rebuild the next screen to include a variety of learning materials, not just

training sessions for that specific level. I have drawn notes on a screen printto guide further development of the window to include access to(1) Written materials such as cheat sheets and job aids(2) Asynchronous/recorded training sessions(3) Synchronous (live) online training sessions(4) Information regarding synchronous in-person training services


18/19

ii) Consider different approaches to training materials by level where levelcould relate to usage (policies and procedures versus Sarbanes Oxley) or tothe users policyIQ role/authority level (read-only access, Reporting User,Manager, Project Manager, Global Administratoreach of these roles hassome beginner tasks up through advanced tasks).

d) I will add a link(s) to request access to a policyIQ practice site/sandbox.

e) I will keep in mind that the written descriptions that have yet to be developedneed to build accurate expectations. Depending on the window, I may be able tobuild recommendations for suggested order of exploration through lessons.

f) I will need to put some resources into continued development of the actuallearning content and then gather feedback on its quality, completeness, learnermotivation and so on.

g) I need to put some thought and pen to paper on a plan for putting all of thislearning material behind a secure login; as our organization does not share ourlearning materials with non-clients/non-employees.

h) I would like to build a better project plan and present it to my team so that I maytake advantage of this early management support. It has been difficult for me tobe clear about the steps or progression of the development of my EdWeb.

i) The outcome: I need to develop some clarity of exactly what the end productwill be.

ii) Labor:(1) How much of that end product will I be producing myself (will I have the

skill-set to create the product of my EdWeb vision)?(a) What are the major milestones of development?

(2) At what stage of development will I be at the end of my skill-set and whatwill be left to produce?(a) What are the major milestones of development?

(3) What resources will be needed to finish development?(a) From my team?(b) Procure outside of my team?

iii) Time/Schedule:(1) When do we expect to meet the various milestones?(2) When do we expect the product development to be complete?(3) When do we expect the product implementation to be complete?

iv) Cost:(1) I need to build a budget for this product implementation.

(a) What hard/software is needed?(b) What labor is needed?(c) Investment in additional training/research?(d) Marketing/launch/kick-off expenses?(e) Formative and Post-Implementation Evaluation expenses?(f) Other? What am I forgetting?

8. Lessons learned from this implementation that you will apply to future formativeevaluations you may conduct in IT 5680 or for other projects.a) The evaluation session was extremely useful and went off without a hitch. The

most significant lesson learned was that I need to be focused on the product orend-result that I am seeking and take care to align this goal with the programlearning development. My questions for the formative evaluation were not insynch with the goal. I wrote them based on the examples in the learningmaterials and momentarily (or a little longer than momentarily) lost sight of thegoal. This is really a two-pronged approach that I am working through:


19/19

development of the interface and development of the content. The first phase ofdevelopment of the content is complete. The development of the interface is thenext priority (now) and refinement of the content is next.

Back to Top

Summative EvaluationThis section was originally scheduled for completion in IT5680. I interrupted my studiesfrom 2008 through 2011. Upon returning to the program, this portion was no longer apart of the curriculum in INTE 5680.Back to Top

Documents

080412 Edweb Project Report