LMSFocusGroupReportBackgroundUT Dallas’ current eLearning system, Blackboard Vista 8.2, is an end-of-life system and will no longer be supported by Blackboard after 2013. With this end in mind, the university is conducting a search for a new learning management system. UTD will publish a Request for Proposals in early Spring 2011. In October and November 2010, as directed by the LMS Committee, the Office of Educational Enhancement, assisted by SOM eLearning, conducted a series of focus groups to solicit feedback from faculty, staff and students about the use of the current eLearning system and about their needs to determine what features would be necessary and desired in any learning management system. FocusGroupsA total of eight (8) focus groups were held - 3 for faculty users, 3 for student users, 1 on accessibility and 1 for back-end users. See the below tables for attendance and demographic information of attendees. At the beginning of each focus group (except for the back-end users group), a brief survey was administered individually to solicit quick feedback about specific tools used. A comments area was provided. See Appendix A for the survey and survey results. GuidingQuestionsFor the focus group sessions, a set of “guiding questions” was used to direct the conversations. This semi-structured approach allowed the facilitator to guide the discussion and to prevent the focus groups from becoming “gripe” sessions. See Appendix B for the Guiding Questions. ObservationsInterestingly, many of the features that faculty expressed a desire for in a new system already exist in the current eLearning. Faculty were either unaware of them or had not received training. Faculty emphasized that migration from eLearning to new system should be easy. Students generally wished that all faculty used or were required to use eLearning. “It would be convenient.” Not all of the students were taking full online course; thus their faculty tended to use eLearning for posting syllabi and lecture notes and homework assignments. Some students took online quizzes. Student disappointment with eLearning was more from lack of use by faculty and from stability. Students related that other students don’t like eLearning because it often didn’t work and they need to feel confident that the system will work.

FocusGroupResultsCurrentusage(fromsurveys)SeeAppendixAUsabil i ty• “eLearningispowerfulbutforapoweruser”(F)• “thelearningcurveismoreworkthanit’sworth”(F)• navigationiseasy(S)• couldlookmoremodern,moreupdated(S)Tools

CommunicationMail • Toolshouldbeadefaultinallcourses(F)

• Needsearchbynameandkeyword(F)• Ifmailforwardedout,needtobeabletoreplybackinto(F)• Windowdoesn’texpand(F)• ConnectUTDemailandeLearningmail(F)(S)

Discussions • Notintuitivetocreateanewpost(F)• Ingradeddiscussions,movefromstudenttostudent(notbacktolist)(F)

• Needtoreplyprivatelytoapost(F)• gradingisclunky(F)• doesn’tthatitopensnewintoseparate,smallerwindows(F)• needstobeeasytouse(likeFacebookmessaging)(S)• in“showall”shouldstillbethreaded(S)• “read”shouldapplyonlytoonesactuallyread(notallnew)(S)• searchableandtagablethreads(S)

Calendar • linktoOutlookcalendar/Googlecalendar(F)• colorcodeentriesbycourse(userassignedcolors)(F)

Announcements • needabilitytoedit,deleteannouncementsbyTAs(F)• needabilitytohideannouncements(F)

Chat • wouldbeessentialtoonlinecourse(S)• chatnotsufficient(movedtoSkype)(S)

CollaborationGroups • useful(F)

• manualcreationofgroupsissomewhathard(F)• nicebutnotcrucial(S)• groupcontactcouldbeexpanded(S)

AssessmentAssignments • Easyformattingw/olaunchinghtmleditor(F)

• Sequentialgradingofassignments-studenttostudent(noreturntolist)(F)

• Easyuploadofmultipleorlargefiles(eg.video)(F)• Somewhatdifficulttosubmit(S)• Somefiles(MSAccess)hardtodownload(S)

Quizzes • Improvegraphicability(embedding,formats)(F)• Changesettingswhilequizisopen(F)• Equationeditorisn’tgood;integrate“MathType”(F)• Systemnot“dynamic”,thoughworkswellforMCtests(S)

Gradingforms • Abilitytoaddcommentswitheachitem(F)

LearningModules • Morevisualway(F)

CourseManagement • Desired-centrallocationfromwhichtopushemail,announcements,

examstoallsections(F)Filemanager • Shouldworkmorelikeadropbox(F) • Fasterdownload/uploadingoffiles(F)Syllabus • IntegratewithCoursebook(F)

• Hypertexttocalendar(F)Icons • Morechoices(F)Editpages • Makedefault“Seeall”(F)MyFiles • Nice,filesalwaysthere(S)GradebookColumns • Rearrangingisdifficult(F)

• Needpercolumnsettings(F)Formulas • Formulaeditingisdifficult(F)General • Addcategories(F)

• Havedefaultgradingschemes(F)• MakeitmorelikeExcel(F)

IntegratedTools• Library,libraryreserves(F)(S)• Videoserver,YouTube,iTunesU,etc(F)• GoogleDocs(F)• Turnitin-useGrademark(onlinegrading)(F)• MuchLearning(calculus)(S)• Skype(S)Addit ionaltoolsorfunctionality• Onlineeditingofdocuments,regardlessofformat(F)• Bettertexteditor(F)• Notificationsystem(sendremindersaftermissedexam,sendnewannouncement,newdiscussion,etc,RSSfeeds)(F)

• Pushnotificationforupdates(S)• Abilitytoembedvideoseasily(F)• Wikis(F)• LaTex(S)• SmartTutor(S)• Webconferencing(S)• Resumefunctiononquizzes(S)• Consistentlook/structurebetweencourses(S)• Pictureroster(S)Accessibli ty• Facultyandstudentsconfusedthiswithnavigationandstability

OverallImprovementsLogin • toomanyclicks(F)(S)

• shouldbemorelikeshareddrives(accessiblewhenlogintocomputeroncampus)(F)

Mobile • needmobileapp(F) • pushnotifications,RSSfeeds(F)(S)Public • needabilitytomakepartspublic(resources,etc)(F)(S)Migration • easyportingtonewsystem(F)Listings • makedefault-latestentryontop(foranything)(F)MyCourses • elearningcouldbemoreintegratedwithotherUTDsystems • abilitytoopenmultiplewindows • medialibrarythatallowsstudentstopost/upload(S) • sitemap(S)

ResultsfromBack-endUsersFocusGroupDeliveredintegrationw/PS • functionalitydelivered-50%(MH)

• APIhooks(SH)• Tightintegration(NTH)

Structure/hierarchy • Detaillevel-product/service-howmuchcontroldowehave(MH)

• Conditionalcourseexistence(MH)• Roles-admin,support-differentlevels(MH)

Authentication • LDAP(MH)-Shibboleth(SH)• Pointofentrycontrol-singlesignon(NTH)

Hardware • Nospecifichardwareaslongasbelowmet• Enterpriselevel(MH)

Database • Noflatarchitecture-Oracle/MSSQL(MH)• Mustbeabletoquery(MH)• Publishedschemes(SH)

Externalintegrationcontrol • Weknowwhereourdataresides(MH)• Weknowwhohasaccesstoourdata(MH)

Archiving • Comprehensivebackupstrategy(MH)• Capabilityforrestoringdeletedcontent(MH)• Dailybackup(SH)

Application • Web-based(MH)• Supportmultiplebrowsers/OS(MH)• Highavailability(MH)

Migration • MigrationsupportfromVista(SH)Security • SSL-formobile(grades)(MH)Hosting • MeetUTD/staterequirements(MH)IntegrationControlofcontent • Courseobject(MH)

• Individualobject(MH)• Gradebook(MH)• Memberenrollment(MH)• Term-based(MH)

Support • Whatkind(MH)• Multiplelevels(MH)• Process(MH)• SLAs(MH)

Hosting • DatastoragemeetsFERPA,UTD,statepolicies(MH)

Concern-timelineforvendorstogetthroughthevettingprocessinInformationSecurity.

AppendixA

StudentFocusGroupSurveyWhat is your status?

Freshman Sophomore Junior Senior Graduate Student What is your major / school? How long have you been using eLearning? How comfortable are you using eLearning?

Very comfortable Comfortable Neutral Uncomfortable Very uncomfortable

Which course tools in eLearning do your instructors use?

• Announcements • Assessments

o Quizzes o Surveys o Self-tests

• Assignments o Individual assignments o Group assignments

• Calendar • Chat / Whiteboard • Discussions

o Graded discussions o Journals o Blogs o Threaded

• Goals • Gradebook • Grading forms (rubrics) • Groups • Learning Modules • Mail • Posting syllabus • Web links • Who’s Online

Which student tools do you use?

• My Files • My Progress

• My Grades • Notes

Please use the back of this survey for any other comments you’d like to add.

StudentFocusGroupSurvey

Whatisyourstatus?[2]Freshman [1]Sophomore [2]Junior [1]Senior [8]GraduateStudentWhatdepartmentareyouin?

• [2]ArtsandHumanities• [1]NaturalSciencesandMathematics• [4]SchoolofManagement• [1]EconomicPublicPolicySciences• [0]BrainandBehavioralSciences• [4]EngineeringandComputerSciences• [1]Undeclared

HowlonghaveyoubeenusingeLearning?• [9]1year• [1]3years• [1]5years

HowcomfortableareyouwitheLearning(Pleasemarkyourresponse)[3]VeryComfortable[6]Comfortable[5]Neutral[]Uncomfortable[]VeryuncomfortableWhattoolsareyoucurrentlyusingineLearning(Markallthatapply)[10]Announcements [2]Assessments [1]Goals[1]Quizzes [9]GradeBook[]Surveys [3]GradingForms(Rubrics)[1]Self‐Tests [5]Groups[4]Assignments [6]LearningModules[7]IndividualAssignments [11]Mail[7]GroupAssignments [1]Calendar [12]Syllabus[4]Chat/Whiteboard [4]Discussions [3]Gradeddiscussions [3]WebLinks[2]Journals [3]Who’sOnline[4]Blogs[8]ThreadedWhichstudenttoolsdoyouuse? [6]MyFiles [11]MyGrades []MyProgress [2]Notes

Pleaseusethebackofthissurveyforanyothercommentsyou’dliketoadd.

• IhaveneverhadasemesterwhereeveryprofessorusedeLearning.Asofnow,Ihave2usingeLearning,2usingtheirownpersonalwebsites,andoutoneusingthelibrary’seReserves.Forthistypeofsoftwaretobeeffective,useneedstobemandatorybyprofessors.Asofrightnow,itissimplyaninconveniencetousethesoftware.Currently,themathandsciencedepartmentshavestartedusingcomputerizedtesting.Thissoftware,muchlearning,whileprimarilyatesting/quizzingsoftware,shouldhavetheabilitytopostnotes/announcements/gradesitwouldbemuchbetterforstudentsiftherewasonetypeofsoftwareusedthroughouttheuniversity.Iimagineitwouldalsosavequiteabitofmoneyonlicensingfees(themathdepartmenthasalreadypurchased6000copiesofthissoftware.It’salreadyverydifficulttogetaprofessortolearnonenewpieceofsoftware–havingonecentralsoftwarewouldgreatlyhelpitsadoption.

• eLearningisfairlysimpletouseandIlikeitbutinallhonestyeLearningdoesn’tlookverycool.

• Libraryintegration.IamalsoworkingatUTSouthwesternMedicalcenter(Research)AcademicInformationSystems.WeuseUPIC(PeopleSoftnowOracle)towriteeLearningandwehaveELM(EnterpriseLearningManagement).Iwritetechnicaltraining.Wearelookingforsolutionsaswell.Suggestions:Saiki,MoodleRooms

• IuseeLearningasaninstructor(TA).VerylittleuseofeLearningasastudent

• Iamcurrentlydoinganindependentstudy,soIamnotina“class”thatuseseLearning.However,IamaTAforaprofessorwhousesitonlyalittle(topostassignmentsandsyllabus).Iamassignedtoteachnextsemesterthough,andIwillbeusingitalot–theclasswillbea“hybrid”(halfon‐line,halfinperson)class.

FacultyFocusGroupSurvey

Whatisyourroleintheuniversity?[28]Faculty

[2]Staff

Whatdepartmentareyouin?[3]ArtsandHumanities[3]NaturalSciencesandMathematics[10]SchoolofManagement[5]EconomicPublicPolicySciences

[3]BrainandBehavioralSciences[3]EngineeringandComputerSciences[1]InterdisciplinaryStudies

HowlonghaveyoubeenusingeLearning?[2]Never[7]1year[8]2years[2]4years

[5]5years[1]8years[1]9years

HowcomfortableareyouwitheLearning(Pleasemarkyourresponse)[12]VeryComfortable[14]Comfortable

[1]Neutral []Uncomfortable[2]Veryuncomfortable

WhattoolsareyoucurrentlyusingineLearning(Markallthatapply)[22]Announcements[8]Assessments[13]Quizzes[5]Surveys[8]Self‐Tests[11]Assignments[15]IndividualAssignments[11]GroupAssignments

[13]Calendar[7]Chat/Whiteboard[17]Discussions[9]Gradeddiscussions[4]Journals [5]Blogs[21]Threaded

[13]Filemanager[3]Goals[26]GradeBook[4]GradingForms(Rubrics)[11]Groups[14]LearningModules[13]SelectiveRelease

[22]Mail[22]Syllabus[1]SCORM[9]Templates[17]WebLinks[11]Who’sOnline

Pleaseaddanyothercommentsbelow.• I’minterestedinopensourceproductslikeSakai• Itisagreatresource.MyusedependsonthecourseIamteachingandthelevelofthe

course(undergraduateorgraduate)• IhavehelpedinpastwebCT/eLearninginitiativesandappreciatethischanceforusto

shareourexperiences.Pleasedon’thesitatetoaskifIcanhelpinanyway.IfeellikethecurrenttoolweareusingforcoursesisdifficultforstudentsandinstructorstouseeasilyandIamexcitedtoexploreotheropportunitiesforustoimproveourdistancelearningandenhancelearninginallUTDcourses.

• 1.WhengradingassignmentssubmittedineLearning,itshouldbepossibletogostraightfromonepapertothenextwithouthavingtogobacktothemainscreenforthat

assignment.2.WhenIookingatalistofsubmittedassignmentsandsortitbystudentname,Ifindtwoproblemsa.Thelistissortedbyfirstname,notlast.B.WhenIlookatapaperandthengobacktothelist,thesortingrevertsbacktodefault.Resortingtakesforever.Ihave120studentsperclasssubmittingatleast10paperseach,in3courses.3.Studentssimplydon’tknowhowtouseeLearningfeatures.Theydon’tknowhowtoeditasubmittedassignment,forexample.Theydon’tunderstandthedifferencebetweenapublicdiscussionandemail.Amoreaccessiblesystemmightcauselessconfusioninstudentswithmorelimitedcomputerskills.4.LinkstoTurnItInandCPRwouldbehelpful.Integration,Imean.5.IfaTApostsanannouncement,Ihavenoaccesstoeditordeleteit.Ishouldbeabletochangeallannouncements.Similarly,studentscanchangethecalendarforeveryone,andIcan’tintervene.Ihadastudentpostsomeofherassignmentsthere,andIcouldn’tdeletethem.6.Inassignments,HTMLEnablerdoesnotworkformoststudents,whichcauseswidespreadpanic.7.TheeLearninghelpnumberisnothelpful.Oncampussupportcouldn’tbebetter,butIhearmanybitterstudentscomplainabouttheremotehelpdesk.

• Ihaveanywherefrom12to17sectionsofonecourse.ThemostglaringomissionsineLearningisthelackofcentralizedlocationwhereIcanstorefiles,quizzes,assessments,etc.thatpertaintoallthesections.IwouldalsoliketosendmailandpostannouncementsfromoneplacewhereIcansendittoallthesectionssimultaneously.Likewise,Iwouldliketopostassessmentsinoneplacewheretheycanbeaccessedbyallsections.Withalloftheabove,IstillwouldliketoretaintheindividualsectionsbecausewhenitcomestopostinggradesinOrion,IcanjustdrawthemfromtheindividualsectionsineLearning.BunchingallthesectionsintoamegasectionwouldnotworkformebecauseforgradesIstillneedtoretainindividualsections.Also,differentsectionsaretaughtbydifferentTAs,whoneedtohaveaccessonlytotheirsections.

• Keepitsimple!Agreaternumberofinstructorsandstudentswilluseit.BasicBlackboardworkedverywellattheUniversityofArizona–EllerCollege(Walteristhecontactifyouwouldliketocallhim)

• IhavenotfoundthecurrenteLearningsystemtobeusefulsoIdonotuseit.Iwouldhopewecouldfindamoreeffectiveprogramfortheuniversity.

• Iwouldlovetoseeamobileapplicationforthesystem–particularlydiscussionsandquizzes.

• Keephopingforportfoliodevelopments

AppendixBFocus Group Guiding Questions Let’s talk about your Current usage

• What is your level of expertise with eLearning? • What do you use eLearning for? • What do you like about eLearning?

Let’s talk about Usability

• What do you think about eLearning’s interface? Explain. • What improvements do you need?

Let’s talk about Tools in a Learning Management System

• What do you like about the tools you are currently using in eLearning? • What do you not like about these tools? • Which tools are essential? (need to be part of whatever system we use?)

Let’s talk about Communication

• How do you communicate electronically with your students? • How do you use eLearning to communicate with your students? • What is missing from eLearning that would enable you to more effectively communicate

with your students?

Let’s talk about Collaboration • How do your students collaborate electronically? • How do you use eLearning for collaboration? • What is missing from eLearning that would enable you and your students to more

effectively collaborate? Let’s talk about Assessment

• How do you assess your students electronically? • How do you use eLearning to assess your students? • Are you able to assess your students effectively using eLearning? • How could you better assess your students in an online format?

Let’s talk about Learning Modules

• How do you organize your course (in eLearning)? • What content tools in eLearning do you use? • How would you rate each of these content tools and why?

Let’s talk about Course Management

• What course management tools in eLearning do you use? • How would you rate each of these course management tools and why? • What course management tools would you like to see made available?

Let’s talk about Gradebook

• What do you need in an LMS gradebook? • What do you like about the eLearning gradebook? • What do you not like about the eLearning gradebook? • What features would you like to see added (or changes made to) the gradebook?

Let’s talk about Integrated Tools

• What integrated tools do you use (i.e. Respondus, Turnitin, eInstruction-clickers)? • What external tools would you like to see integrated with an LMS?

What additional tools or what functionality would you like to see in an LMS? (eg portfolios, wikis, etc) Let’s talk about Accessiblity

• Do you feel eLearning is accessible? • Why/why not?

Let’s talk about Overall Improvements

• How can eLearning (an LMS) be better?