The UniProt Knowledgebase Course Design

1

The UniProt Knowledgebase

UniProtKB

Resources for training trainers

Version: 18 November 2020

Course Design Considerations for trainers

PROFESSIONALGUIDE

A Professional Guide to Course Design

2

CoursedesignConsiderationsfortrainers

Overview This Professional Guide introduces a structured approach to course design, highlighting the importance of articulating learningoutcomes commensurate with the cognitive complexity of the target learning, prior to devising learning experiences and coursecontent.Thespecificfocushereisonface-to-faceactivities,buttheguidanceisalsorelevantforthosedesigningonlinecourses.

Teaching Goals & Learning Outcomes ThisGuideoutlinesaseriesofstepsthatcanhelptrainerstodeviseanddeployeffectivecourses.OnreadingthisGuide,andengagingwiththereflectiveexercises,youwillbeableto:

• listfivekeyphasesofcurriculum&coursedevelopment;• explaintheprimaryroleoflearningoutcomes;• writelearningoutcomesforacourse;• identifytheBloom’s-levelaccomplishmentsthatdifferenttypesoflearningexperiencearelikelytosupport;• describetheroleoflearningoutcomesinselectingrelevantcontent;• distinguishdifferenttypesofassessment&theirroleinsupportinglearnerprogressiontowardslearningoutcomes;and• summarisethebenefitsofcourseevaluation.

1 Introduction

Teaching and training, core elements of academic life, can beenormously rewarding but also quite challenging. Instructors areoftenrequiredtoperformundervariousconstraints,andfrequentlyhave to accommodate, engage andmotivate student cohortswithvery different backgrounds and aptitudes in limited time-frames.Thiscanbedauntingforexperiencedteachers,butespeciallysoforthosewho’rerelativelynewtoteachingandtraining.Formal education enterprises generally begin with curriculum1

design: this involves specifying i) its purpose or Teaching Goals(TGs); ii) its duration; iii) theKnowledge, Skills andAbilities (KSAs)intendedtobeachieved,expressedasasetofLearningOutcomes(LOs); iv)howlearnerswilldemonstrateachievementofthoseLOs;v) the materials, Learning Experiences (LEs) and assignmentsinstructorswillusetosupportlearnerprogressiontowardstheLOs;andvi)theassessmentsforevaluatingstudentlearningandteachingeffectiveness. Emphasis is placed on teaching and learning, andmechanismsforcollectingevidencethatlearnershavechangedovertime2.Inthislattersense,theconceptof‘curriculum’differentiatesformalteachingfromtraining,asformalprogrammesusuallyaffordtime for learners both to be able to progress and todemonstratetheirprogression.Bycontrast,trainingcoursesaremuchshorter(measuredindays

or weeks, rather than years); they hence necessarily focus onacquiring specific KSAs in limited time-frames, generally withoutconsideration of learner progression beyond the course. However,theessential featuresofeffectivecurricula (i.e., those thatachievetheir statedLOs for themajorityof learners)pertain to instructiononanytime-scale;theyarethusalsorelevantforshortcourses,andprovide important considerations for those involved in, orembarkingupon,coursedesign(whetherface-to-faceoronline).With this in mind, this Guide outlines key steps of curriculum

development – and the role of Bloom’s taxonomy3 – that can be

used to inform thedesignofeffective courses. TheGuide isbasedon Guidelines for curriculum and course development in highereducationandtrainingbyTractenbergetal.2,and isthefirstoftheGOBLET-ELIXIRresourcesfortrainingtrainers.

2 About this Guide

This Guide provides an overview of five design phases thatfacilitate course design. Exercises and Reflections are provided tohelp readers consider how to incorporate the concepts discussedhereintotheireverydaypractices.Throughoutthetext,keyterms–renderedinboldtype–aredefinedinboxes.Additionalinformationisprovidedinsupplementaryboxesandfigures.

KEYTERMS

Curriculum:theinventoryoftasks involvingthedesign,organisation&planning of an education or training enterprise, including specifica-tion of learning outcomes, content,materials& assessments,& ar-rangementsfortrainingteachers&trainers

Learning Outcomes (LOs): the KSAs that learners should be able todemonstrateafter instruction,thetangibleevidencethatthe teach-inggoalshavebeenachieved;LOsarelearner-centric

Learning Experience (LE): any setting or interaction in or via whichlearningtakesplace:e.g.,alecture,game,exercise,role-play,etc.

Teaching: usually, instruction delivered over long time-scales via aseriesof courses (in schools, colleges,universities,etc.)designed tocontribute to a formal programme that, if completed successfully,yieldsanaccreditedqualificationinagivenfield(e.g.,adegree)

TeachingGoal (TG): the intentionsof an instructor regarding thepur-poseofacurriculum/course/lesson/activity/setofmaterials;TGsareinstructor-centric(alsotermedinstructionalobjectives)

Training: instructiondeliveredviashortcoursesdesignedtoexpandorbuildknowledge&practicalskillsinagivenfield,oftenconductedintheworkplaceortrainingcentreto‘up-skill’membersofaworkforce


3

3 Formal curriculum design

3.1 Different types of curriculum To set the scene for our considerations of course design, we

examine some of the foundations for effective curriculumdevelopment, drawing heavily on the curriculum- and course-development guidelines developedby Tractenberg et al.2. Notablehereisthefactthatdifferenttypesofcurriculumhavebeendefined:i.e., intended, implemented, attained and hidden curricula4,5.Recognisingtheexistenceofdifferentcurriculumtypes(or,perhaps,different curricular outcomes) is important because, while theintended curriculum is the starting point, it may not be thecurriculumactuallyattained:i.e.,whatyouaimedtoteachandwhatstudentsactuallylearnedmaynotbethesame.Toimproveoutcomes,differencesbetweentheintendedandat-

tainedcurriculaneedtobeminimised.Theonlywaytodiscovertheattained curriculum is to find outwhat learning actually occurred.Thisrequiresactionableevaluation,toassesswhethertheTGsandLOs were achieved, to identify weaknesses in the implementationand to highlight improvements needed to remediate them. Theseconsiderationsarekeytodevelopingeffectivecurriculaandcourses.

3.2 Structuring curriculum design Curriculumdesignbenefitsfrombeingsystematic:structuredap-

proacheshelptoorchestrateandclarifywhatwillbetaught,whyitwill be taught andhow; theyalsoaffordopportunities to evaluatewhat does and doesn’t work, and hence what needs to change,ultimatelyleadingtoimprovementsinlearningoutcomes6,7,8.Sever-al different frameworkshave been devised to facilitate the designprocess,buteach ismotivatedbythesameunderlyingphilosophy:tohelpformulateprogrammesthatpromotemeaningfulandendur-ing learning. Ifwe’re tounderstandwhetherwe’ve really achievedthis, we must i) determine the purpose of the programme (whatneedsitaddresses,whyit’sbeingdeveloped,whatlearnerswillgainfrom it, why it’s important), ii) define the intended LOs, and iii)developassessment and evaluationmechanisms thatwill allow ustomeasurewhethertheprogrammesuccessfullymetitsgoals.Notsurprisingly, thesameprinciplesapplyto coursedesign.The

processmayseemdaunting,butforthesakeofsimplicity,wefocushereononemodel:i.e.,thatproposedbyNicholls9.Beforediscuss-ing this further, however, it’s helpful to consider another very im-portanttoolusedinteachingandlearning–Bloom’staxonomy3.

Bloom’s taxonomy of cognitive complexity Learning taxonomies are useful tools that can help both to

formulate and clarify LOs, and to arrange them on a scale ofincreasingcomplexity.Bloom’staxonomy3,probablythemosteasilyunderstoodandwidelyusedtoday, featuresasix-levelhierarchyofcognitive complexity, ranging fromRemember (being able to recallfactsandbasicconcepts)toEvaluate(beingabletodefendopinionsordecisions),asillustratedinFigure1.As can be seen from the Figure, each Bloom’s level is

accompanied by a set of active verbs that express expected,measurablelearnerbehavioursatthatlevel:e.g.,achievingthelevelUnderstandmeanstobeabletoclassify,selectorexplainapieceofinformation:here,classify,select,explainareobservable,assessablebehavioursthatcanbereadilyencapsulatedincoherentLOs(moredetails on Bloom’s assessable or active verbs are given in otherGuidesfromtheProfessionalGuideseries).Typical illustrationsofthetaxonomy, likethat inFigure1,depict

successive cognitive levels, suggesting that learners must achieve

one level before advancing to the next, implying a developmentaltrajectoryfromlower-tohigher-ordercognitiveskills.However,thisstructure shouldn’t be regarded as completely rigid; indeed,Andersonetal. 10publishedarevisedversionin2001inwhichtheyplacedSynthesise(theabilitytocreatenewororiginalwork)atthetop of the hierarchy in place of Evaluate. Notwithstanding theminutiae, it is perhapsmore fruitful to regard the taxonomy as acontinuumorspectrumofcognitivelevels,whereeachmergesintothe next, providing a structured tool to help express measurable,assessableLOs,inwhichthecognitivelevelsaremadeexplicit.The cognitive aspects embodied in LOs are important. Teaching

shouldpromotemorecomplexbehavioursthanjustrecallorrecog-nition (unless remembering is the intendedLO), andpush learnersto achieve greater cognitive complexity8,12-15. This can be done byembedding development in learning activities and materials, andensuringthatLOsreflectthelowesttothehighestlevelsofcognitivecomplexityrealisticallyachievableoncompletingthoseactivitiesorhaving engagedwith thosematerials (realistic aims are key, espe-ciallyforshortcourses:e.g.,expectinglearnerswithnopriorsubjectknowledgetoachievethelevelEvaluate,say ina1-daycourse,willguaranteefailureandfrustrationforlearnersandinstructorsalike).

Figure1.Bloom’staxonomyofcognitivecomplexity,&asampleofassociatedverbsthatcanbeusedtocreateLOs.Theverbsreflectadevelopmentaltrajectory,fromlower-tohigher-orderthinkingskills;theyshouldbeconcretelyobservable&relatetohowachievementof

targetLOswillbeassessed. KEYTERMS

Actionable:supportiveofadecision,orthetakingofsomeactionbyalearner,instructororinstitution

Assessment: the evaluation or estimation of the nature, quality orabilityofsomeoneorsomething

Attainedcurriculum:whatlearnersactuallyacquire&candemonstratehavingfollowedtheimplementedcurriculum

Hiddencurriculum:unintendedcurriculareffects:unofficialnorms,be-haviours& values that are transferred (not necessarily consciously)bytheschoolcultureorethos;thisrecognisesthatschoolinghappensinbroadsocial&culturalenvironmentsthatinfluencelearning

Implementedcurriculum:ortaughtcurriculum,howthe intendedcur-riculumisdeliveredinpractice:i.e.,theteaching&learningactivities,&theinteractionsbetweenlearners&teachers,&amonglearners

Intendedcurriculum:theformalspecificationofKSAsthatstudentsareexpected to achieve&be able todemonstratehaving followed theimplementedcurriculum


4

4 Nicholls’ five phases of curriculum design

Thebackdropforourconsiderationsofcoursedesign isNicholls’paradigm for curriculum development, illustrated in Figure 2. Itsfive-phasestructurehasbeenbrieflysummarisedbyTractenbergetal.2,asfollows:

1. SelectLOs;2. SelectordevelopLEsthatwillhelplearnersachievetheLOs;3. SelectordevelopcontentrelevanttoLOs;4. DevelopassessmentstoensurelearnersprogresstowardLOs;5. EvaluatetheeffectivenessofLEsforleadinglearnerstoLOs.

Figure2.Nicholls’phasesofcurriculumdesign&theirdependencies.Foreachphase,keyconsiderationsareshown(diamonds).Wherethesearen’tsatisfied,thatorpreviousphasesshouldberevisited(redarrows),otherwiseitissafetomovetothenextphase(s)(greenarrows).Whenall

considerationsaresatisfied,thecurriculumorcoursecanbecharacterised,withconcreteevidence,assuccessful(star).

As can be seen from Figure 2, the model’s phases are inter-dependent;allareultimatelydependentonthefirst–definingLOs.

he phases are iterative: thismeans that LOs influenceMoreover, tlater decisions, but later decisions may also reflect backwards,thereby providing opportunities to check for alignment of eachphase to the target LOs (inotherwords, toensure that successivephases are mutually consistent with, and supportive of, the LOs).Thus,theroleofLOsispivotal:theymusthavespecificcharacteris-ticstofunction,andsupporteachoftheotherphasesastheydo.Figure2 illuminatesan importantfeatureofthemodel:thatLOs

are the startingpoint,anddriveall decision-making. This is just astrue for coursesas it is forprogrammes7.Missing from themodel,however, is the dependence of LOs on a hierarchy of cognitivecomplexity that establishes a developmental trajectory, like thatseen inBloom’s taxonomy.Wereflecton this crucialpoint, and itsrelevance for course design, in the discussion of each of the fivephasesbelow.

4.1 Define intended LOs Justasforcurricula,Phase1ofcoursedesignbeginswithstating

theLOs(asalreadynoted,LOsareexplicitstatementsoftheKSAs–and requisite level of cognitive complexity – that learners areexpectedtoachieve,andbeabletodemonstrate,oncompletionofaperiodofinstruction).TohelpformulateLOs,it’simportanttotakeastepbackandthinkaboutwhatyouaimtoachieve(i.e.,whatareyour TGs and the KSAs you intend to be achieved?), how you

proposetogetthere,andhowyou’llknowyousucceeded.Messick11encapsulatedthisprocessintheformofthreesuccinctquestions:

1. WhatKSAsarethetargetsofinstruction(andassessment)?2. Whatlearneractions/behaviourswillrevealtheseKSAs?3. Whattaskswillelicitthesespecificactionsorbehaviours?

These questionswere originallyposed in1994 in the context ofassessment.TheirfocusonKSAs–theLOs–thusguidesnotonlythecreation of relevant tasks (to reveal the target KSAs) but also therationaldevelopmentofappropriateassessments:i.e.,theyprovideaframeworkfor,andclarify,whattoassess.Thequestionscanthussupportall phases of coursedevelopment, startingwith the selec-tionofintendedKSAsstatedinasetofLOs.WritingcoherentLOs ischallenging:theymustcontainappropri-

ate (Bloom’s) verbs (Figure1)thatexpressmeasurable,observableandassessableactions,accuratelydescribingwhatsuccessful learn-erswillbeabletodo–andatwhat levelofcognitivecomplexity–afterinstruction.Variouscharacteristicsof,andprinciplesforarticulating,LOshave

beenpublished16,17:someofthesearelistedbrieflyintheboxbelow(furtherinformationandadditionalguidanceonhowtowriteeffec-tiveLOsisgiveninotherGuidesfromtheProfessionalGuideseries).Given their detail and complexity, and the importance of aligningthe instructional inputs you devisewith the outcomes for learnersyou intend, it can be hard to know where to start. This possiblyexplains why it may feel easier to begin developing a course byselectingitscontentratherthanfirsttryingtounderstanditsimpactonstudentlearning.Nevertheless,ensuringthattargetLOsmeet,orare consistent with, the characteristics outlined in the box belowhelpstopromotebetteralignmentofinstructionalinputsandlearn-eroutcomes.In short,whendefiningLOs, thekeyquestion toask is, arethey

Specific, Measurable, Achievable, Realistic and Time-bound – i.e.,are theySMART? If theydon’t satisfy this test, they should be re-vised; onlywhen they meet these criteria is it safe to progress toPhase2,asshowninFigure2.Ultimately,LOsprovidethenecessarystructureandcontextfordecision-makingby instructors(and learn-ers),hencetheirprimaryroleincoursedesign.

Learningoutcomes

LOsshould:

• bespecific&welldefined:LOsshouldconciselystatethespecificKSAsthatlearnersshoulddevelopasaresultofinstruction;

• be realistic: LOsmust be attainable given the context and re-sources available for instruction, and consistentwith learners’abilities,developmental levels,prerequisiteKSAs,andthetimeneededvs.timeavailabletoachievethem;

• relyonactiveverbs,phrasedinthefuturetense:LOsshouldbestatedintermsofwhatsuccessfullearnerswillbeabletodoasaresultofinstruction;

• focusonlearningproducts,notthelearningprocess:LOsshouldnot statewhat instructorswill do during instruction, butwhatlearnerswillbeabletodoasresultofinstruction;

• be simple, not compound: LOs shouldn’t include compoundstatementsthatjointwoormoreKSAsintoonestatement;

• be appropriate in number: LOs should be deliverable and as-sessablewithinthetimeavailableforinstruction;

• support assessment that generates actionable evidence: here,actionablemeans supportive of a decision, or taking some ac-tionbyalearnerorinstructor.


5

EXERCISES

1Thinkofa courseyoucurrently run,plan to runorhave run in thepast.AreitsintendedLOsstated?Ifnot,trytojotafewdown.

2Nowconsider,areyourLOsSMART? Ifanyof themdon’tmeet theSMART criteria, try revising them as follows: select an active verbthat can (in principle) be observed & assessed, & complete thesentence,“Attheendofthiscourse,learnerswillbeableto...”(ifithelps, review the verbs listed in Figure 1). It’s important to focushereonwhat learnerswillbeable todoat theendof instruction:e.g., will they be able to describe its content? Explain a concept?Implementanalgorithm?Solveaproblem?Evaluateresults?

3Todeterminehowwellyou’vestructuredyourLO,visittheIntendedLearning Outcome Advisor: https://web.cs.manchester.ac.uk/iloadvisor & paste your phrase into the input box. Press the‘SUBMIT’button.Howwelldidyoudo?

4 Consider revising your phrase if the Advisor identified any issues.ConsiderwritingfurtherLOs;testeachusingtheAdvisor.

4.2 Select LEs that will lead to the LOs Phase 2 involves identifying the most appropriate LEs to lead

learners to the intended LOs. It’s important to appreciate thatdifferent LEs can lead learners to demonstrate different Bloom’s-level accomplishments: e.g., lectures differ from problem-sets –solving problems helps learners to work with, and manipulate,information rather than passively listening to it; similarly, labexercises differ fromwriting computer programs –writing originalcode affords learners the opportunity to create something newrather than simply following instructions. Some example LEs arelistedinTable1,togetherwiththeBloom’slevelandthekindsofTGandLOthateachmaysupport.Having defined SMART LOs in Phase 1, Phase 2 thus hinges on

choosing the most appropriate LEs to best lead learners towardsthem: if LOs include, for example, being able towrite a computerprogram, thentheLEsmustallow learnerstoapplytheknowledgethey’veacquiredandtodemonstratethatthey’vewrittenapieceoffunctionalcode:i.e.,LEsandLOsmustbealigned(iftheyaren’t,this

Table1Samplelearningexperiences&thehighestBloom’slevelthateachmaysupport.ExamplesofthekindsofteachinggoalthatsuchLEsmayunderpin&thekindsoflearningoutcomestheymaypromotearealsoshown.

Learningexperience

HighestBloom’slevelssupported

ExampleTG(s)ThisLEwillallowmeto…

ExampleLO(s)Learnerswillbeableto…

Lecture,webinar

Remember,Comprehend

Inspirelearners,ignitelearners’enthusiasm,clarify/explainaconcept,provideanoverview,givecontext,summarisecontent

• listthekeypointsofthelecture/webinar• summarisetakehomemessage(s)

Exercise,practical

Apply,Analyse Helplearnersdigestcoursematerials,solvetypicalproblems,applyknowledge,showhowtodothingswithappropriateguidance,giveanideaofhowatoolworks

• followasetofinstructionsorprotocol• calculateasetofresultsoroutcomesfromagivenprotocol

Flippedclass Apply,Analyse Teachlearnershowtoformulatequestions,helplearnerstomemorisenewinformation&con-cepts,oranalyse&understandcoursematerials

• summarisethecontentmaterial• askappropriatequestions

Peerinstruction

Synthesise,Evaluate Preparelearnerstodefendanargument,givelearnersopportunitiestoexplainthings,therebyhelpingtodevelopcriticalthinking&awareness

• explainhowtheysolvedanexercise• evaluateothers’choices/decisions• diagnoseerrorsintheexercise-solvingtask

Groupdiscussion

Synthesise,Evaluate Givelearnersopportunitiestopracticequestioning,developnewideas&criticalthinking

• communicatetheirownideas• defendtheirownopinions

Groupwork Synthesise,Evaluate Promotecollaborativework&peerinstruction,provideopportunitiesforgiving/receivingfeedback,&digestingcoursematerials

• providefeedbackontheirpeers’work• shareideas• explaintheadvantagesofteam-work

Problem-solving

Synthesise,Evaluate Promotelearnerabilitiestoidentify&evaluatesolutions,developnewideas,makedecisions,evaluatedecisioneffectiveness,troubleshoot

• diagnosefaultyreasoningoranunderper-formingresult

• correcterrors

canleaddirectlytoagapbetweeninstructionalinputsandintendedoutcomes,whichisonereasonwhycourseevaluationtodetectsuchmisalignments is so crucial). If LEs don’t satisfy this criterion,alternativeLEsshouldbefound,ortheLOsshouldberevisitedandrevisedbeforeprogressingtoPhase3(asshowninFigure2).

KEYTERMS

Exercise: an activity designed to help learners to mentally put intopracticelearnedskills&knowledge

Flipped class: a learner-centredapproach inwhich students are intro-ducedtonewtopicspriortoclass;classtimeisthenusedtoexplorethosetopicsingreaterdepthviainteractiveactivities

Group discussion: an in-class, learner-centred approach in which stu-dents discuss ideas, solve problems&/or answer questions, guidedbytheinstructor

Groupwork: a learner-centredapproach inwhich studentsareorgan-isedintogroups(&perhapsassignedspecificroles)&aregiventaskstoperformcollaboratively

Lecture: a didactic approach inwhichoral presentation is used tode-scribe&explainconcepts&toimpartfacts

Peer-instruction: an interactive, in-class, learner-centred approach inwhichgroupsof twoormore studentsbrieflydiscussaquestionorassignmentgivenbytheinstructor

Practical: an activity to put into practice learned skills & knowledge,generallyinalabsetting

Problem-solving: a learner-centred approach in which students arerequired to systematically investigate a problem by building or de-termining the best strategy to solve it (usingwhat is known to dis-coverwhatisnotknown)

Webinar:alecturedeliveredonline


6

Sometimes, it may be necessary to use specific LEs: e.g., if acourse is lecture-based, it may not be possible to choose alter-natives;oraparticularteachingscenariomaynotallowforanidealLE. Even if you’re not in a position to select themost appropriateLEs,theLOsmuststillbeconsistentwiththegivenLEs;andifthoseLEswon’thelp learnerstoachievetheLOs,thentheLOsshouldberevised– i.e., LOsare still themost important feature to consider.Thekey is todetermineexactlywhat specific LEs can contribute tolearningorhowtheywillhelpmovelearnerstowardstheLOs.

REFLECTIONS

1ConsidertheLEslistedinTable1.2Areanyoftheseusedinyourcoursefromtheexerciseabove?3Whatkindsofachievementor learningare theymost likely topro-

mote?IsthisconsistentwithyourstatedLOs?4DoesyourcoursehaveanyLOthatcan’tbeachievedthroughtheLEs

thatthecoursecurrentlyuses?4.3 Select content relevant to the LOs WithLOsandLEsaligned,Phase3 involves findingthemostap-

propriatecontent tosupport learners toachievethe intendedLOs.Regardlessofwherecontent isdrawnfrom,whatmattersishowitsupportstheLOs.Tothisend,contentselectionshouldbejudicious:it shouldn’t try to be all-encompassing8, but should consider thetarget Bloom’s levels in the LOs, and the preparation of learners.Oncecontentthat’sconsideredtobethecoreofacoursehasbeenidentified, additional auxiliary materials can also be selected, andoffered, say, as ‘further reading’.Using LOs todrive content selec-tion inthiswaytherebybothprovides focus(avoiding thetempta-tiontosqueezeasmuchaspossible intoacoursetoensurecover-age18,19),andincreasesthelikelihoodofaccomplishingthoseLOs.Sometimes, specific contentmay be deemed essential20. Never-

theless,thismuststillsupporttheLEsandpromoteachievementoftheLOs; if itdoesn’t,thenadditionalcontentand/orLEs shouldbeconsidered that will, or the LOs themselves should be revised, topreventmisalignmentofinstructionalinputsandlearningoutcomes.Overall,theroleofLOshereistohelpfocusonrelevantcontent,andavoidmaterialthat’seithernon-essential/toobroadortoonarrow.

REFLECTIONS

1Thinkofa courseyoucurrently run,plan to runorhave run in thepast.Consideritscontent.Howfocusedisit?

2IsthecontentconsistentwithyourLEs?DoesitsupportyourLOs? EXERCISES

1Consideracourseyoucurrently run,plan to runorhaverun in thepastforwhichtheintendedLOsarestated.ReviewtheLOs,&iden-tifywhichpartsofthecoursecontentsupporttheLOs.

2Mapout the relationshipsbetween theLOs, theLEsviawhich theywillbedelivered&thosespecificitemsofcontent(e.g.,itemAsup-portsLO1,&willbedeliveredusingalecture).

3Isthereanypieceofcontentthatdoesn’tsupportanyLO(s)?4IsthereatleastonepieceofcontentforeachLO?5IsthereatleastoneLEforeachpieceofcontent?

4.4 Develop assessments to ensure progression Inanycourse,learnersbenefitfromhavingopportunitiestoshow

thatthey’reprogressing.Generally,this isdoneusingvarioustypesofassessment,includingtests,feedbacksurveys,andsoon.Nicholls’modelincludestwodifferenttypesofassessment,toevaluate

i) learning, to detect changes in learner performance duringinstruction, to identify their strengths, and diagnose theirweaknesses– i.e.,formativeassessment (orfeedback).Thiscaninformdecisionsabouthowtomodifyinstructiontobet-ter promote learning; it can also inform learners aboutchangestheymayneedtomaketoimprovetheirlearning;

ii) instructional outcomes, to verifywhether learners achievedthe stated LOs after instruction – i.e., summative assess-ment.Thiscanhelpto informdecisionsbothabout learners(e.g.,rankingtheirperformances)andaboutthecourse(i.e.,whetheranyofitsLOs,LEs,etc.needtoberedesigned).

Formative and summative assessments21 are important fordeterminingwhether andwhat learning has occurred. Summativeassessment doesn’t generally yield information about learners’progress: it sums up what learning has been achieved afterinstruction relative to the intended outcomes (via written tests,practical tasks, or other measurable activities), and gives valuabledataaboutlearningattainmentatthelevelofindividualsandentirelearnercohorts.Thiscanbetrickyforshortcourses,butmaybenec-essaryforthosethatgivecreditsoroffercertificatesofcompletion.Formativeassessmentsareappliedthroughoutacourse;planned

thoughtfully, they can improve the performance of learners andinstructors.Ideally,theyshouldbeusedoften(say,every15-20min-utes), thereby also yielding opportunities to change pace andrefocus learners’ attention. It may be hard to conceive how tointegrate such assessments into a course, but they need not becomplex or time-consuming (just informative about learning at agivenpoint)–seeboxonpage7forsomesimpleexamples22.Ifformativeassessmentsareusedfrequently,andleadtospecific

decisions by learners and instructors, then instruction (or learnerpreparation) canbemodified tobetterdevelop the target LOs23,24.Suchassessmentscanbedesignedtoanticipate,andidentifywhichlearners are experiencing, commonmisunderstandings, and gaugetheirreadinesstomoveahead;theycanalsohelplearnerstoidenti-fytheir ownstrengthsandweaknessesbyencouragingreflectiononwhat they do/don’t know or are/aren’t confident about, honingtheirabilitiestoself-assess12.Furthermore,ideasforhowtoaddressanyissuesthey’vefound,orforfurtherlearning,canbebuiltin(e.g.,“if you chose option C, youmight want to re-read the handout”);formativeassessmentscanthereforealsosupportself-instruction.Using actionable formative and summative assessment can help

to ensure that LOs, andprogress towards them, are explicitly sup-ported.LOsclarifywhatspecificallyneedstobeassessedandwhy.

REFLECTIONS

1Thinkofa courseyoucurrently run,plan to runorhave run in thepast. Consider the assessments you use& how you assess learnerprogression.Howmuchformativeassessmentdoyouuse?

2 Ifyouusesummativeassessment,doesitpaintaconsistentpictureof learning levels for your course? To what extent are your testsalignedwiththeKSAs&Bloom’slevelsexpressedintheLOs?

KEYTERMSContent:aspecificsubjectortopicitem(e.g.,DNA,RNA,proteins,a

biochemicalpathway,Rprogramming)thatisthetargetoflearningFormative assessment: formal or informal assessments (or feedback)

made during learning so that instruction or practice can be bettertargetedforlearnerstobeabletofullyachievetargetLOs

Summative assessment: assessmentsmade after a period of instruc-tioninordertomonitorwhetherLOshavebeenachieved


7

Simpleformativeassessmentstrategies

Themosteffectiveway to teststudentunderstanding is todoso inclass: it’s importanttoseizethemoment,anddealwithpotentialmis-understandingsassoonastheyarise.Thesein-classtipsmaybehelpful:

• reflection:towardstheendofatrainingsession,asklearnerstoreflect on, and write down, a list of new concepts and skillsthey’ve learned.Ask them to considerhow they’d apply theseconceptsorskillsinapracticalsetting;

• agreedsignals:gauge learners’ satisfactionwitha trainingses-sionbyaskingthemtouseagreedsignals(e.g.,raisingcolouredpost-it notes to indicate that the pace is too fast/ slow, etc.).This engages all learners, and allows you to check their confi-dencewiththecontentanditsdelivery,eveninlargegroups;

• 3-2-1: at the end of a training session, ask learners to note 3things they learned, 2 things theywant to knowmore about,and1questiontheyhave.Thisstimulatesreflectionontheses-sion,andhelpstoprocesstheirlearning;

• misconception check: present some common or predictablemisconceptions about a concept you’ve covered. Ask learnerswhethertheyagreeordisagree,andtoexplainwhy;

• diagnostic questions/questionnaires (which may be anony-mous):asklearnerstonoteonethingtheydidn’tunderstandorthattheymissed,andonethingthatwasverycleartothem,ormake them complete amultiple choice quiz, then display anddiscusstheanswerswiththem.

These,andmanyothersimpleassessments,maybefoundinBriggs’onlinelistof21waystocheckforstudentunderstanding22. EXERCISES

1Consider thematerials of the course youwere just thinking about.Focusonasetofthosematerialscorrespondingtoasinglelesson.

2 Analyse the flow of the lesson, & identify points where you couldintroduceformative-feedbacksessions.

3Foreachportionofthelessonbetweentwosetsoffeedback, imag-ine(&writedown)aquickactivity–anexercise,aquestion,areflec-tion–thatlearnerscouldcarryoutthatwouldhelpyouunderstandwhetherthey’refollowingthelesson&thatlearningisoccurring,orwhetherthey’relaggingbehindorarelost.

4.5 Evaluate course effectiveness Thefinalstepistoperformanactionableevaluationbycollectingqualitativeandquantitativecoursedatatoassessitseffectivenessinleading learners to the statedLOs. Summativeassessments canbeusefulhere:e.g.,iftheyrevealuniformlylowlevelsofachievement,itmayindicatethatfuturerevisionsareneededtoensurethatLEs,assessments and LOs are aligned; it could also flag problemswiththe assessment – it’s vital to understand whether performancereflects student learning levelsor issueswith theassessment itself(poorlyworded test questions, ambiguous response options, etc.),and,ifnot,totakestepstoensurethatitdoes.Even ifa fullquantitativecourseevaluation isn’tpossible, itmaystillbepossibletosolicitactionableevidenceofitsimpactviashort-orlong-termfeedbacksurveys.It’stemptingtousecompletionratesor learner satisfaction as proxies for success; however, althoughsimpletocollate,theseoutcomesaren’tinformativeaboutlearners’growthor courseeffectiveness. Satisfaction surveysoftenusepre-and post-course questionnaires25,26 to collect demographic details,and solicit learners’ self-evaluations and reactions to a course.Reviewing learners’ perceptions can help to identify whether the

conditionsforlearningwerepresent,butalonewillnotshedlightonwhetherthe intendedLOswereachieved: learnerperceptionsmaybe affectedby factors unrelated to course effectiveness, and theirself-assessmentsmaybebiased(e.g.,less-skilledlearnersnotorious-lyoverestimatetheirabilities–theDunning-Krugereffect27).Therearemanyevaluationmethods, eachwithadvantagesand

disadvantages;all concur thatmultiple featuresneed tobeconsid-ered. One framework collates learner reactions, their actuallearning, changes in their behaviour, and the impactof the courseontheirorganisation(seeboxbelow).Suchfeaturescanbeexplicitlytargeted in short- or long-term surveys: e.g., end-of-course,summative quizzes can be used to test achievement of intendedLOs; longer-term questions (say, 6-12 months after a course) canfocus on the extent to which learners have put their acquiredKSAsinto working practice; and so on. Overall, it’s important tochoose an appropriate evaluationmethod for your circumstances,and, alongside learner reactions to the course, for yourevaluationtoconsiderwhatresultsyouexpectedtoachieve,whethertheLOswereachieved,andwhetherlearners’on-the-jobpracticeschanged.

Methodstoevaluatetrainingeffectiveness

Evaluationmethodsprovidesystematicframeworksforanalysingtheeffectivenessoftrainingcourses.ThefirstapproachwasintroducedbyRaymondKatzell in 1956 and later popularisedby Kirkpatrick28,29. TheKirkpatrick-Katzellmethodproposesafour-levelstrategy:

• Level1:Reaction–whatdolearnersfeelaboutthetraining?• Level2:Learning–whatdidlearnersretainfromthetraining?• Level3:Behaviour–didlearnersputtheirlearningintopractice

on-the-job(didtheirworkingbehaviourchange)?• Level4:Results–didtheirbehaviouralchangeshaveanoverall

impactontheirorganisation(e.g.,greaterproductivity)?

Surveyingeachlevelhelpscollectqualitativeandquantitativedatatoevaluate training effectiveness. Note: this isn’t a real taxonomy, asthere’snoevidencethatoutcomesatsuccessivelevelsarelinked(e.g.,no correlation has been found between the reaction and learningstages30).However,iflevel2resultsindicatethatlearningdidn’toccur,those from level 1may identify aspectsof a course thatdisappointedlearnersandhelpunderstandwhatshouldbeimproved.

Manyotherapproachestotrainingevaluationhavebeendeveloped,eachfocusingonslightlydifferentlevelsorstages31.

Actionable evaluation is essential for identifying aspects of acourse that may benefit from intervention, leading to concretedecisionsaboutwhatneedstoberemediatedandwhy32.Designingmeaningfulevaluationsrequiresthought;insomecases,itmayhelpto appoint independent evaluators or advisory boards, as externalreviewerscangiveobjectiveappraisals,andmayalsohelptoframethecourseagainstnational/internationalstandards.Ultimately,theevaluationshould identifywhatworks inacourse,andwhatneedsremediation,tobettersupporttheachievementoftheintendedLOs.

REFLECTIONS

1Thinkofa courseyoucurrently run,plan to runorhaverun in thepast. Do you evaluate course effectiveness? If so, how do you dothis?Howdoyoudetermine itseffectiveness in leading learnerstoyourintendedLOs?

2Consider theKirkpatrick-Katzell framework.What level(s)doesyourevaluationincorporate?

3Whatadjustmentswouldyouneedtomake,ifany,inordertoalignyourevaluationwiththeKirkpatrick-Katzellstrategy?


8

5 Documenting and sharing

When developing a course, it’s helpful to document the designprocess, including details of how it was conceptualised, theassumptions and decisions made along the way, the assessmentcriteria,etc.,andtosharethatdocumentationwith instructorsandlearners(thiscanbedoneviacommunitymailinglists,throughblogposts, collaborative repositories, social channels, using GitHub,Slack,etc.).Bywayofexample,TheCarpentries33provideinstructornotes34 for most of their courses. These are collectively-writtendocuments that reflect on the strengths and weaknesses of thecourse design (and itsmaterials), what did/didn’twork, suggestedimprovements, tips for teaching, challenges encountered, learnerfeedback, and indications ofwhere alignment of LOs/LEs/content/assessments failed, and why. Best practices like this help coursedesigners and communities of trainers to understand what wasintended,whatwasdone,andwhythosemightbedifferent,iftheyare.Thisisparticularlyvaluableforcourseswhosematerialsarenotdevelopedbyitsinstructors.Suchdocumentationcanthusfacilitatereflectionandpromotegoodpractice,andcanhelpnewinstructorsprepare to deliver the course. Ifmade available to learners, it canhelpthemtounderstandwhattheycanexpect fromacourse,andmakeinformeddecisionsaboutwhetheritwillhelpthemtoachievetheir learning goals; it may also help them to better gauge theirperformance,andtoidentifywhatwillhelpthemperformbetter–itmaythereforealsoimprovelearningoutcomes.Creatingandsharingdocumentationwithinstructorsandlearners

fosters the development of communities of best practice, and cansupportbothlearningandthesuccessofacourse.

6 Conclusions

Coursedesignisprefacedbydeterminingthepurposeofthepro-gramme,analysingthecontextinwhichthecoursewillbedeliveredandwhowillbenefit.Fundamentaltothisprocessistoidentifytheneedsthecoursewilladdress,itstargetaudience,andtheprerequi-siteKSAs that learnersmusthave inorder toprofitmost fromthecourse. Identifying the target audience and learner prerequisiteshelps to define criteria for selecting participants, should a limit beneededtomaximisecourseeffectiveness. Oncesuch‘situationalanalysis’hasbeencompleted,thestarting

point,andpivotalreferenceforallsubsequentstagesofthecourse-designprocess, is toarticulateSMARTLOs.Aspartof thisprocess,Bloom’s taxonomy is useful for defining LOs that aremeasurable,and whose cognitive complexity increases along a developmentaltrajectory; and Messick’s questions help both to select KSAs andLOs,andtoensurealignmentofinstructionandassessment.Crucial-ly, informedchoicesneed tobemadeabout theLEs (and thecon-tenttheyuse)thatbestalignwiththegoalofachievingspecificLOsin the time available. General topic areas (life sciences, computerscience,datascience,etc.)mayprovidetheoverarchingframework,but your goals for learners should drive how content is selected,taughtandassessed.Intermsofassessmentapproaches,formativeassessment is generallymore relevant for short courses, but sum-mative assessmentmay be necessary for accredited courses. Ulti-mately, it’s importanttounderstandwhateachtypeofassessmentcontributestothecourse-designprocessandtobuildyourpracticetomaximisetheeffectivenessofeach.Evaluatingcourseeffectivenessshouldbethefinalstageofaro-

bustdesignprocess,andthefirststeptowardscoursere-designand

improvement, should the coursebedelivered regularly. Todo thisrigorously,it’shelpfultoemployamulti-levelevaluationstrategy,inwhich learner satisfaction is just one strand (used alone, learnersatisfactionisnotareliablemetricofsuccess).Coursedesignrequiresthoughtandtime.Successfulcoursessup-

port learners as theydevelop fromentry-level performance to theminimumperformancelevelforachievingthetargetLOs.Ifdesignedspecificallytosupport learnersandLOs,andevaluatedagainstthatobjective, the instruction that’s delivered and learning that’s in-tendedaremorelikelytomatch.Thiscan’tguaranteesuccessforalllearners,butdoescreatetheoptimalcircumstancesforsuccess.Structuredapproachesbenefitcoursedesignbyleveragingwhat’s

alreadyknownaboutlearning,andprovidingaframeworkfordeci-sion-making. The process can be challenging, but investing in it islikely topaydividends.Ultimately, everything in thedesign shouldlead to,andsupportdevelopmentof, theLOs that learners shouldpossess,andbeabletodemonstrate,oncompletionofacourse.Foreaseofreference,theapproachoutlinedherecanbedistilled

intothesimplesetofrecommendationssummarisedinTable2.

Table2.Recommendationsforsupportingcoursedesign.

Recommendation Benefits

1)Followastructuredparadigmforcoursedesign.

Leverageswhat’sknownabouteducation&learning;providesaframeworkfordecision-making.

2)FocusonLOsfirst,toinformallothercourse-designdecisions.

LOsprovidecontextfordecision-makingbyinstructors&learners.

2.1)LeverageLOstodeter-mineappropriateLEs.

LOshelptochooseLEsthatsup-portlearnerstoachievetheLOs.

2.2)LeverageLOstoselectcontentthatpromotesachievementoftheLOs.

LOshelptofocusonrelevantcontent,&toavoidmaterialthatisnon-essential&/ortoonarrow.

2.3)Assesslearners’progresstowardsLOs&theachieve-mentofLOsusingformative&summativeassessment,respectively.

LOsclarifywhatspecificallyneedstobeassessed&why.

3)Plan&executeanactionablecourseevaluation.

Helpsidentifywhatworks&whattoremedyinthecoursetobettersupportachievementofLOs.

4)Document&sharethecoursefeatureswithlearners.

Helpstosupportlearning&promotessuccessofthecourse.

TAKEHOMES

1 Coursedevelopmentisamulti-step,iterativeprocess;

2 Thefirststepistoidentifyintended(SMART)LOs;

3 Identification of key content only becomes relevant after LOs &suitableLEshavebeenidentified;

4 Bloom’staxonomycanfacilitatethearticulationofmeasurableLOs,&supportthedevelopmentofcognitivecomplexitythroughthem;

5 Messick’squestionshelptoalignKSAs,instruction&assessment;

6 For successfuloutcomes, learners& the coursemustbeevaluated(itisn’tsufficientsimplytonote‘completers’orlearnersatisfaction);

7 Asuccessfulcourseguideslearnersfromentry-levelperformancetotheminimumperformancelevelforachievingitstargetLOs;

8 Inacoursedesignedtosupportlearners&LOs,theinstructionthat’sdelivered&learningthat’sintendedaremorelikelytoalign,therebyprovidingtheoptimalcircumstancesforlearnersuccess.


9

7 References & further reading

1 Cedefop.(2008)TerminologyofEuropeaneducationandtrain-ingpolicy–aselectionof130keyterms,2E.PublicationsOffice,Luxembourg.doi:10.2801/15877

2 Tractenberg RD et al. (2020) Guidelines for curriculum andcourse development in higher education and training.SocArXiv,doi:10.31235/osf.io/7qeht

3 Bloom BS. (1956) Taxonomy of educational objectives: theclassificationofeducationalgoals.Handbook1:Thecognitivedomain.DavidMcKay,NewYork,USA.

4 Attained curriculum. In UNESCO IBE Glossary of Curriculum-Related Terminology: www.ibe.unesco.org/en/glossary-curriculum-terminology/a/attained-curriculum

5 Hidden curriculum. In The glossary of Education Reform:www.edglossary.org/hidden-curriculum

6 Fink LD. (2013) Creating significant learning experiences: anintegrated approach to designing college courses, 2E. Jossey-Bass,SanFrancisco,USA.

7 Diamond RM. (2008) Designing and assessing courses andcurricula(3E).Jossey-Bass,SanFrancisco,USA.

8 NilsonL.(2016)Teachingatitsbest:aresearch-basedresourceforcollegeinstructors,4E.Jossey-Bass,SanFrancisco,USA.

9 NichollsG. (2002)Developing teachingand learning inhighereducation.Routledge,London,UK.

10 Anderson LW et al. (2001)A taxonomy for learning, teachingandassessing:a revisionofBloom’s taxonomyofeducationalobjectives.AddisonWesleyLongman.NewYork,USA.

11 MessickS.(1994)Theinterplayofevidenceandconsequencesin the validation of performance assessments. EducationalResearcher,23(2),13-23.

12 AmbroseSAetal.(2010)Howlearningworks:7research-basedprinciplesforsmartteaching.Jossey-Bass,SanFrancisco,USA.

13 Weinstein Y & Sumeracki M. (2019) Understanding how welearn.Routledge,London,UK.

14 Roediger III HL & Karpicke JD. (2006) The power of testingmemory:basicresearchandimplicationsforeducationalprac-tice.Perspect.Psychol.Sci.,1(3),181-210.

15 JensenJLetal.(2014)Teachingtothetest...ortestingtoteach:examsrequiringhigher-orderthinkingskillsencouragegreaterconceptualunderstanding.Educ.Psychol.Rev.,26(2),307-329.

16 Creating learning outcomes: irds.stanford.edu/sites/g/files/sbiybj10071/f/clo.pdf

17 National Institute for Learning Outcomes Assessment (2016)Highereducationquality:whydocumenting learningmatters.Urbana,IL:UniversityofIllinoisandIndianaUniversity,author.

18 McKeachieWJ&SvinickiM. (2006)McKeachie’s teachingtips:strategies, research and theory for college and universityteachers,12E.HoughtonMifflin,Boston,USA.

19 LujanHL&DiCarloSE. (2006)Toomuch teaching,notenoughlearning:whatisthesolution?Adv.Physiol.Educ.,30,17-22.

20 DeVeauxRDetal.(2017)Curriculumguidelinesforundergrad-uateprogramsindatascience.Annu.Rev.Stat.Appl.,4,15-30.

21 WalvoordBE. (2010)Assessment clear and simple: a practicalguideforinstitutions,departmentsandgeneraleducation,2E.Jossey-Bass,SanFrancisco,USA.

22 Briggs S. (2014)21Ways to check for student understanding.www.opencolleges.edu.au/informed/features/21-ways-to-check-for-student-understanding/#ixzz30n5SFAYV

23 WiliamD& ThompsonM. (2008) Integrating assessmentwithinstruction:whatwillittaketomakeitwork?In,Thefutureof

assessment: shaping teaching & Learning. CA Dwyer(Ed.).Routledge,London,UK.

24 Trumbull E & Lash A. (2013)Understanding formative assess-ment:insightsfromlearningtheoryandmeasurementtheory.WestEd,SanFrancisco,USA.

25 Jordan KL et al. (2018) Analysis of Software and Data Car-pentry’s pre- and post-workshop surveys. Zenodo, doi:10.5281/zenodo.1325464

26 GurwitzKTetal.(2020)Aframeworktoassessthequalityandimpact of bioinformatics training across ELIXIR.PLoS Comput.Biol.,doi:10.1371/journal.pcbi.1007976

27 KrugerJ&DunningD.(1999)Unskilledandunawareofit:howdifficulties in recognizingone'sown incompetence lead to in-flated self-assessments. J. Pers. Soc. Psychol., 77(6), 1121–1134.doi:10.1037//0022-3514.77.6.1121

28 Kirkpatrick DL. (1959) Techniques for evaluation training pro-grams.J.AmericanSocietyofTrainingDirectors,13,21-26.

29 Kirkpatrick DL & Kirkpatrick JD. (2006) Evaluating TrainingPrograms: the four Levels, 3E. Berrett-Koehler Publishers Inc.SanFrancisco,USA.ISBN10:1576753484

30 Gessler M. (2009) The correlation of participant satisfaction,learning successand learning transfer:anempirical investiga-tionof correlationassumptions inKirkpatrick's4-levelmodel.Int.J.ManagementinEducation,3(3-4),346–358.

31 Deller J. (2020) Training evaluation models: the completeguide. kodosurvey.com/blog/training-evaluations-models-complete-guide

32 Worthen BR & Sanders JR. (1987) Educational evaluation:alternative approaches and practical guidelines. LongmanGroupLtd.,London,UK.

33 TheCarpentries:carpentries.org34 The Unix Shell: instructor notes: swcarpentry.github.io/shell-

novice/guide

8 Acknowledgements & funding GOBLET Professional Guides build on GOBLET’s Critical Guide

concept, using layout ideas from the Higher Apprenticeship spe-cificationforcollege-levelstudents inEngland.ThecontentshereinareanadaptationofGuidelines for curriculumandcoursedevelop-mentinhighereducationandtrainingbyTractenbergetal.,20202.This Guidewas developedwith the support of a donation from

EMBnettotheGOBLETFoundation.

9 Licensing & availability This Guide is freely accessible under creative commons licence

CC-BY-SA2.5.Thecontentsmaybere-usedandadaptedforeduca-tionandtrainingpurposes.TheGuideisfreelyavailablefordownloadviatheGOBLETportal

(www.mygoblet.org) and the F1000Research BioinformaticsEducation and Training Collection (f1000research.com/collections/bioinformaticsedu?selectedDomain=documents).

10 Disclaimer EveryefforthasbeenmadetoensuretheaccuracyofthisGuide;

GOBLETcannotbeheldresponsibleforanyerrors/omissionsitmaycontain, and cannotaccept liability arising from relianceplacedontheinformationherein.


10

About the organisations

GOBLET GOBLET(GlobalOrganisationforBioinformaticsLearning,Education&

Training;www.mygoblet.org)wasestablishedin2012asanot-for-profitfoundation to unite, inspire and equip bioinformatics trainers world-wide;itsmission,tocultivatetheglobalbioinformaticstrainercommuni-ty,setstandardsandprovidehigh-qualityresourcestosupportlearning,educationandtraining.GOBLET’sethosembraces:

• inclusivity:welcomingallrelevantorganisations&people• sharing:expertise,bestpractices,materials,resources• openness:usingCreativeCommonsLicences• innovation:welcomingimaginativeideas&approaches• tolerance: transcendingnational,political,cultural,social&dis-

ciplinaryboundaries

Forgeneralenquiries,[email protected].• Attwoodetal.(2015)GOBLET:theGlobalOrganisationforBio-

informaticsLearning,Education&Training.PLoSComput.Biol.,11(5),e1004281.

• Corpas et al. (2014) The GOBLET training portal: a globalrepository of bioinformatics training materials, courses &trainers.Bioinformatics,31(1),140-142.

ELIXIR ELIXIR is an intergovernmental organisation that brings together life

science resources (databases, software tools, training courses, cloudstorage, etc.) from across Europe. The aim is to create a singleinfrastructure, making it easier for scientists to find and share data,exchangeexpertise,andagreeonbestpractices:elixir-europe.org.

Since 2015, the ELIXIR Training Platform and GOBLET have workedclosely to promote and develop standards and best practices inbioinformatics training. Together, they have built a Train-the-Trainer(TtT) programme, which comprises a standard curriculum, associatedtraining materials and well-trained instructors. To date, hundreds ofscientistshavebenefitedfromthisprogramme.

EMBnet EMBnet,theGlobalBioinformaticsNetwork, isanon-profitorganisa-

tion, founded in1988toestablishandmaintainbioinformaticsservicesinEurope;today,itsupportsanddeliversservicesacrossaninternationalnetwork:www.embnet.org.EMBnethas alwayshada keen interest inEducation and Training (E&T), and has delivered tutorials and coursesworldwide. Perceiving a need to unite and galvanise international E&Tactivities, EMBnetwas a principal founder of GOBLET. Formore infor-mationandgeneralenquiries,[email protected].

About the authors

Allegra Via (orcid.org/0000-0002-3398-5462)§ AllegraViahas20years’experienceinacademia;her

current focus is on educational psychology andcurriculum development. She is Instructor/InstructorTrainer in the Carpentries, the ELIXIR-IT Training Coord-inator,andleadsELIXIR’sTtTprogrammeSheisAdjunctProfessor at SapienzaUniversity andUniversity of Bologna, andexpertconsultantofSapienza’sTeachingInnovationandQualityworkinggroup.

.• ViaAetal.(2017)Anewpan-EuropeanTrain-the-Trainerpro-

grammeforbioinformatics.Brief.Bioinform.,bbx112.• ViaAetal. (2013) Bestpractices inbioinformatics training for

lifescientists.Brief.Bioinform.,14(5),528-537.

Patricia M Palagi (orcid.org/0000-0001-9062-6303)‡ Patricia Palagi has more than 21 years’ experience

teaching bioinformatics, proteomics andprogramming inSwissuniversitiesand internationaltrainingprogrammes.Since2013,shehasledtheSIBTrainingGroup;sheistheELIXIR-CH Training Coordinator, leads ELIXIR’s TtTprogramme,andisInstructorinELIXIR’sTtTcourses.

• PalagiPM(2020)LessonslearnedwhenmovingSIBTrainingcoursesfromin-persontoonline–anythinginbioinformaticscanbetaughtonline,butnotanyhow.F1000Research,9,1148.

• GarciaLetal.(2020)Tensimplerulestomaketrainingmateri-alsFAIR.PLoSComput.Biol.,16(5),e1007854.

Jessica M Lindvall (orcid.org/0000-0002-5042-8481)¥ JessicaLindvallisAssociateProfessorinBioinformaticsat

Stockholm University, and joint head of facility for theSciLifeLab Bioinformatics platform in Sweden. She is theELIXIR-SETrainingCoordinatorandanELIXIRTtTinstructor.

• Tractenberg RE et al. (2019)The Mastery Rubric for Bioinfor-matics: a tool to support design and evaluation of career-spanningeducationandtraining.PLoSONE,14(11),e0225256.

• GurwitzKTetal.(2020)Aframeworktoassessthequalityandimpact of bioinformatics training across ELIXIR. PLoS Comput.Biol.,16(7),e1007976.

Rochelle E Tractenberg (orcid.org/0000-0002-1121-2119)#Rochelle Tractenberg is a biostatistician and cogni-

tive/educational scientist by training. She has worked inhigher education teaching students and instructors since1991,alsoengagedinbiomedicalresearchandtrainingsince2002,creatingtheMasteryRubricconstructin2005.• TractenbergRE (2020)TheAssessmentEvaluationRubric:Pro-

motinglearningandlearner-centeredteachinginhighereduca-tion.doi:10.31235/osf.io/bvwhn

• Tractenberg RE et al. (2019) TheMastery Rubric for Bioinfor-matics: a tool to support design and evaluation of career-spanningeducationandtraining.PLoSONE,14(11),e0225256.

Teresa K Attwood (orcid.org/0000-0003-2409-4235)† TerriAttwood isProfessoremeritaofBioinformatics,with

more than 25 years’ experience teaching bioinformatics inBSc,MScandPhDprogrammes,and incoursesandsummerschools, in the UK and abroad. She wrote the first bio-informaticstext-book;herthirdbookwaspublishedin2016:

• Attwood TK & Parry-Smith DJ. (1999) Introduction toBioinformatics.PrenticeHall.

• Higgs P & Attwood TK. (2005) Bioinformatics & MolecularEvolution.Wiley-Blackwell.

• Attwood TK, Pettifer SR & Thorne D. (2016) Bioinformaticschallenges at the interface of biology and computer science:MindtheGap.Wiley-Blackwell.

Affiliations §Sapienza University, Rome (IT); ‡SIB Swiss Institute of

Bioinformatics, Lausanne (CH); ¥Stockholm University, Stockholm(SE);#GeorgetownUniversity,WashingtonDC(USA);†TheUniversityofManchester,Manchester(UK).

TheGOBLETFoundation,CMBIRadboudUniversity,NijmegenMedicalCentre,GeertGrooteplein26-28,6581GBNijmegen(NL)

Documents

The UniProt Knowledgebase Course Design