Draft Learning Activity Reference Model - Pedagogy · Web viewSenge, P. M. (1990) The Fifth Discipline. The art and practice of the learning organization, London: Random House Bandura,

LARM – Pedagogy – ijf – 08/05/2023 - p1

Learning Activity Reference Model – Pedagogy

Isobel Falconer*, Gráinne Conole†, Ann Jeffery†, and Peter Douglas‡

*University of Dundee; †University of Southampton; ‡Intrallect Ltd

1 Background 12 What is the Learning Activity Reference Model (LARM)? 23 What the LARM can do for you 3

3.1 Learning Activity Authoring 33.2 Learning Activity Delivery 5

4 Creating a Learning Activity 94.1 Description of the characteristics of a learning activity 114.2 Glossary of teaching approaches 144.3 Glossary of Teaching Techniques 204.4 Template for designing a learning activity 274.5 Example of a completed learning activity template 32

1 BackgroundThe Learning Activity Reference Model (LARM) has been developed by the JISC-funded LADiE learning design project1, as part of the JISC e-frameworks programme, to encourage people to design learning activities using appropriate technologies. The aim is to develop a better understanding of the range of learning activities which practitioners might wish to create and match this to the relevant technology services and tools, whilst also creating a map of the learning design domain space.

The LARM is described in three separate guides. Each guide is intended for a different audience:Teachers/Practitioners: the Pedagogy guide which has teaching and learning as its primary focusTechnologists/Implementers: the Implementation Guide which describes how to configure learning activities from existing environmentsDevelopers/Vendors: the Services Guide which defines the reference model so that those creating new educational technology applications can ensure they can be used through the LARM.

This document represents the first of these - the Pedagogy Guide and it is designed for use by teaching practitioners who need to design and implement learning activities. It offers guidance on how to create a learning activity, on effective use of tools and resources in implementing activities, and a language and structure by which teaching 1 www.elframework.org/refmodels/ladie


practitioners and learning technologists might discuss the development and implementation of learning activities.

This guide will be in the form of a “best practice guide” to highlight recommended modes of design and implementation. For example, if you want to design and implement an activity, you can use this guide to identify the types of services available, how they have been used in exemplar activities, and the type of information that your technical support staff will need to know. The guide presents the information in two different structures: first according to the structure of the LARM; and second by working through an example of a learning activity, guided by a template.

2 What is the Learning Activity Reference Model (LARM)?A reference model such as the LARM provides a process for designing and implementing effective learning activities, from initial design, through requirements specification, to analysis of the technologies, specifications and standards necessary to meet those requirements. The LARM:

identifies common requirements of reusable learning activities based in effective practice;describes how these requirements can be met using existing and developing technologies, specifications and standards using a web services approach.

This LARM Pedagogy Guide concentrates on the first of these, on designing effective learning activities, together with an indication of the techniques and technologies that might be used to implement the activities.

The LARM Implementation and Services Guides are more technical and concentrate on how learning activities are orchestrated and run using web services. Web services are a means of linking computing applications or tools (services) and providing access to them via a common, web-based interface.

The idea underlying web services is that flexibility, adaptability and responsiveness to teachers’ needs can be achieved if different types of services or tools (eg. discussion boards, assessment tools, voting systems) can be called up and linked together as the need arises and new services develop. This contrasts with current typical experience where the constraints of proprietary, monolithic virtual learning environments (VLEs) act as barriers to change, development, and sharing of information.

The services, and common web standards to support the services, are currently being developed. The LARM is a contribution to this development. In future, the common web standards base will mean that learning activities designed with the LARM can be reused and shared across different institutions and proprietary platforms.


Figure 1 shows the role of the reference model in mapping requirements between the practitioner creating a learning activity and the web services needed to implement it. In essence a practitioner will create a learning activity which will draw on a number of services to enable the learner to complete the task set; for example a discussion tool is needed for taking part in a group debate, an assessment tool for formative assessment or submission of a completed assignment to an e-portfolio.

3 What the LARM can do for youThe Learning Activity Reference Model (LARM) coordinates a number of technological tools and services that you might want to use in creating and implementing a learning activity. Some of these exist at present, while others are currently being developed and will exist in the future. This section describes the most commonly used of these tools and services, what they will do for you, and how they fit together as a system, while section 4 supports you in making the decisions involved in planning a learning activity.


Figure 2. LARM Architecture

Figure 2 shows how the tools and services can be coordinated as a system that can help you create and run a learning activity. It makes two distinctions:

between learning activity authoring (top), and learning activity delivery (bottom); and between components that are central to creating and running the learning activity

(centre), the tools and services that the activity calls on (eg. wikis, blogs, quizzes) (right), and fairly static information components that may be institutionally based (eg. the student records system) (left).

These components, and the relationship between them, are things you will need to think about in creating and running your activity.

3.1 Learning Activity AuthoringA Learning Activity can be defined as:

an interaction between a learner or learners and an environment (optionally including content resources, tools and instruments, computer systems and services, ‘real world’ events and objects) that is carried out in response to a task with an intended learning outcome2

2 Beetham, H. (2004). Review: developing e-learning models for the JISC practitioner communities: a report for the JISC e-pedagogy programme, JISC


When designing a learning activity there are a number of things you need to think about, in particular: the context within which the activity will occur (type of institution, level, types of

students, etc), suitable teaching approaches, and appropriate tasks which will enable the students to achieve the intended learning

outcomes of the activity. Section 4 contains a template to help you work through these issues, along with a glossary with suggestions for teaching approaches and techniques.

If you want to reuse or share your activity, it will need to be stored in such a way that it can easily be moved or transferred from one system to another, for example from your institution to a second institution that uses a different VLE. Traditionally, practitioners use schemas such as paper-based lesson plans to detail the steps involved in a specific learning activity. For electronic learning activities, standard formats such as IMS Learning Design are being developed which are designed to ensure that electronic learning activities are easily transferable. Tools to help you create such Learning Activity Content Packages (LACP) in standard formats for reuse are becoming available (eg. Course Genie, Reload). The standard format is the IMS content package.

3.1.1 Learning Activity EditorThe learning activity editor can be used both to plan (ie design) a new learning activity or edit and adapt an existing learning activity. At present you probably draft your design on paper using some fairly standard tools to help you, such as a word processor. In future tools such as Course Genie, or Reload will be available to help. They will be able to

Create a new learning activity. For example, you will need to be able to input text, audio, video and other resources, link these to interactive tools such as a discussion board or a multiple choice quiz, and specify the sequence of sub-tasks. LAMS is an example of a tool that does this.

Access and import existing activities from learning activity repositories. For example, you can use an activity that you, or someone else, created earlier.

Edit a learning activity. For example, you can edit an activity that was devised for a different course, so that it fits your context. Or you might have created an activity for the course last year and want to retrieve and update it for this year’s students

Access and import existing resources, such as pictures or video clips, from a learning object repository (ie a database of existing learning activities). You may already do this by copying pictures from the Web into your teaching materials, or by providing links to the pictures on the Web.

Save the learning activity that you have created, either into your Learning Activity Player for playing with the students, or back into the repository for future use.


3.1.2 ResourcesLearning activities can incorporate a range of resources – such as additional reading material, interactive learning packages or lists of relevant web links. Resources span the distinction between authoring and delivery and may be called upon in either phase. You might want to:

use and customise learning activities from a learning activity repository; import text, pictures, animations etc from a learning object repository into your

activity; link to books from a digital library, or material from the world wide web; draw on items from a question bank in a multiple choice quiz.

The learning activity editor will provide a means of searching these repositories for relevant resources. The learning activity editor and the player will both be able to access materials in the repositories when requested. The repositories may be based within your institution or outside it.

3.2 Learning Activity Delivery

3.2.1 Learning Activity PlayerThe learning activity is delivered to anyone with permission to access it by the Learning Activity Player. This might be some sort of VLE, which automatically delivers the resources (for example briefing materials or learning materials) and services (for example a discussion forum or a MCQ tool) that you have specified in your learning activity to your students in the right order at the right time. Or it might be you, telling the students when to perform different tasks, as might happen in a blended learning environment such as that in the worked example in section 4.5. In either case, the player has to coordinate the necessary services, including initialising any services that need to be set up, for example by defining access permissions (such as communicative tools like discussion boards, email). A LARM compliant player would be able to:

Import a learning activity content package in standard format (eg. one that you have created in the learning activity editor, or one that you want to reuse from a repository)

Set up the activity, including making sure the required services (tools) are in place, assigning students to groups in the activity, defining roles (eg. conference moderator) and assigning people to them;

Orchestrate the activity so that resources, tools and tasks appear in the correct order and at the right time.

Access the resources and services required by the activity and let you and the students interact with these via your web browser (eg. Internet Explorer, Firefox, Safari);

Let you alter the activity as it is running, for example if it seems necessary to change the composition of the discussion groups, or you find that students are not understanding a concept and you want to put in some extra material;


Request ordering and archiving of records from the services (tools) used in the activity (eg. a record of discussions, or record of a specific discussion thread)

Track student progress through the activity and sub tasks (eg. you might want to track students through a multiple choice quiz to identify conceptual problems)

Send files, resources and records generated through the activity to a file storage system and/or ePortfolio and provide access to them based on authentication and access permissions. For example, you may want to access discussion records after the activity has finished for evaluation or institutional audit; the students might want to access a particular discussion thread, saved in their ePortfolio, as evidence of their contribution.

Accept work submitted by the student, notify the marker that it has been submitted, and provide access to it for the marker. Then accept feedback (mark, comments, etc) from the marker, notify the student, and provide student access to the feedback.

3.2.2 ServicesYour might want to use a variety of services (ie tools) in your learning activity. Commonly used services and their key functionalities are included here:

3.2.2.1 Email ServiceIn any learning activity you may want to keep in touch with individual students by email, or encourage them to email each other for peer support, for example in an activity where students get together in pairs to conduct an interview. A LARM email system will:

let all authenticated users send and receive messages let users store and arrange messages by time sent, sender, recipient,

subject, message thread let users attach files of all formats to messages allow archiving and storage of selected messages

3.2.2.2 Asynchronous DiscussionMany learning activities include some form of discussion forum or online conference, which enables distance students to participate in discussion and/or campus-based students to carry on discussion in between timetabled face to face meetings. They are used especially with constructivist or situative teaching approaches (see glossary). Asynchronous discussion has two main advantages when compared to synchronous discussion: students do not all need to be logged in at the same time, thus increasing flexibility; and students have time to think about their contributions, which can increase the quality of discussion. It has disadvantages, when compared to a synchronous discussion, in terms of the lack of ‘buzz’ or focus associated with a real time event. You probably already have an asynchronous discussion service on your VLE, or are using a separate service such as FirstClass. In a LARM asynchronous discussion:


participants can post messages to the forum and view other people’s messages;

you can divide the students up into groups, each with their own subconference, to allow small group discussion of particular issues. Group membership will be set up by the learning activity player, but you may want to change it during the activity, for example if you find that the groups are too small to generate an active discussion;

you can moderate the conference, or appoint a student or tutor to moderate it, to facilitate the discussion and to ensure that ‘netiquette’ is adhered to. For example, the moderator can: promote discussion by asking stimulating questions; summarise the discussion so far and steer it in a future direction; intervene to prevent the discussion straying too far off the point or to correct misapprehensions; and remove or edit any offensive messages. The moderator can also require that messages from some students receive approval before appearing in the forum.

participants can include links to web based resources or attach files in any format to their messages, for example if they want to share a picture or audio resource, or drafts of their work, with the rest of the group.

participants can order the messages by author, date, last unread, or subject, and can follow subject threads. This is particularly useful if the conference is very active and is generating a lot of messages;

participants can generate individual emails from messages to the forum, for example if you want to email a student privately about something they have said in the forum;

participants can save records of the discussion, for example into the ePortfolio

3.2.2.3 Synchronous DiscussionLike asynchronous discussions, synchronous discussions take place online but in real time. You may already be familiar with text-based systems such as Chat services in your VLE, or MSN Messenger. Synchronous systems also include web-based video conferencing, whiteboards, and audio/video systems such as Skype3. Synchronous discussion systems are currently not well specified in the LARM, but all systems will let participants send and view messages. They may also include ways of taking turns to speak, sharing files, and have voting systems. Text based systems frequently include extra ways of expressing emotions and body language, such as emoticons, nudges and winks.

Synchronous discussions may be useful with constructivist or situative teaching approaches. They let students at a distance participate in discussions, and have an immediacy and focus that asynchronous discussion lacks, but they give less opportunity for reflection than asynchronous discussion.

3 Skype is a system that allows free telephone calls over the internet using a “peer to peer voice service” www.skype.com


3.2.2.4 WikiWikis are publicly editable web sites that are increasingly being used for collaborative activities. They are currently not well specified in the LARM, but systems are likely to let you set access permissions of various levels (eg. read, edit, etc) for groups of students. They typically contain a shared work space, an asynchronous discussion area, and an archive of previous versions of the work. For example, LA16 uses a modified wiki for an engineering design project. Students discover resources, file them and discuss them in the wiki, and use the shared work space to prepare a presentation on their findings for the rest of the class.

3.2.2.5 Multiple Choice Quiz (MCQ)Learning activities frequently include a multiple choice quiz so students can test their understanding of methods or concepts and tutors can identify areas of misunderstanding. The worked example in section 4.5 uses a MCQ in this way. A LARM MCQ will:

display quiz questions, let students choose answers, and will provide students with feedback and scores;

let you set it up with questions, answers and feedback responses. You can also ask it to generate random quizzes from a question bank which you could write yourself or draw from a question bank repository;

provide you with information about individual student responses to the questions, or about class responses (eg. as a histogram of class responses), so you can identify individual or general areas of difficulty and adjust your teaching accordingly;

allow for different pathways depending on the answers to earlier questions

3.2.2.6 Concept MappingConcept maps are frequently used to help students organise their ideas or create new ideas, often following a brainstorming session. They can also be used to help students create a file storage system that makes sense to them. as was done in an engineering design poject where student groups created a filing system according to an agreed concept map, and stored their resources in it. This made it easier for them to locate stored resources and cut down on the need for search tools. A LARM concept mapping service will: let you and/or the students create, view and edit a map let you store files together with the concept map

3.2.3 Information StorageYour learning activities may need to draw on, or save records in, various information storage services


3.2.3.1 Student Information ServiceThis will probably be your institution’s central student record system. It will contain information for authenticating students and letting them log in to the learning activity. It will also contain personal information, so that the system can deliver the learning activity in a suitable format and with appropriate resources. For example, a blind student might receive audio files instead of text files. Another example is an activity about health and safety, where students from England and from Scotland require different resources because of differences in legislation (LA8).

3.2.3.2 ePortfolioStudents may want to save the output of learning activities into their ePortfolio to use later as evidence of their achievements. A LARM ePortfolio service will: store students’ work and marks for future reference; save files and hyperlinks that students have found useful, and let the

students classify and order them; let the students access files and hyperlinks for an indefinite period of time

after the activity is finished.

3.2.3.3 Archive areaMany activities will require saving student work or resource files for future reference, such as evaluation or quality control. These will be linked to the other services used in the activity and to the learning activity player.

4 Creating a Learning ActivityThe aim in this section will be to give you as practitioners suggestions for creating pedagogically effective learning activities, together with a framework for putting together your ideas and for discussing them with the technical support staff who may be helping you to implement the activity (as outlined in the implementation layer of the LARM).

When creating a learning activity you will need to think about three things: the context in which the learning activity is taking place, the teaching approaches used in the learning activity, and the tasks the students and you will perform in order to complete the learning activity. Section 4.1 shows the many components and options within this classification. It is backed up by the glossaries in sections 4.2 and 4.3 which describe in more detail different types of teaching approaches and techniques that you might use. You can use this table and glossary as a source of ideas for new activities.

Many learning activities, such as the one described in section 4.5 consist of several different tasks, roles and interactions, which may follow on from each other or may run in parallel

The learning activity template in section 4.4 follows the structure of section 4.1, and is designed to act as a prompt to the things you will need to think about when designing a


learning activity. It is followed by a worked example of a completed template in section 4.5. At present you may well need to call on your technical support staff in implementing your learning activity, as you may not have a VLE in your institution that already satisfies all your requirements. The template will provide a starting point for discussion with technical staff and help them identify what your requirements are and what needs to be done to enable your learning activity.


4.1 Description of the characteristics of a learning activity4

Teaching Approaches and Techniques are further explained in sections 4.2 and 4.3 respectively

Context Task taxonomyContext Learning

outcomesTeaching approaches

Type (What)

Technique(How)

Interaction Roles(Which)

Tools & resources

Assessment

AimsInstitutional context

HEFESchoolCommunityInformal

QA process Timetable Subject Pre-requisites LocationEnvironment

Audio-based Computer-basedField-based Lab-basedLecture-basedSeminar-basedSimulatorVideo Work-based

Learner needsAccessibilityLang support

CognitiveKnowledgeDraw Finding out/discover List PronounceRecallReciteRecogniseReproduce SelectSpecifyState

ComprehensionClarify Describe reasons ExplainIdentifyIdentify causes ofIllustrateQuestionUnderstand

ApplicationApply

AssociativeBehaviourist Didactic Elaboration theory Instructional system designIntelligent tutoring systemsTraining needs analysisCognitiveActive learning Constructivist-based design Cognitive apprenticeship Cognitive scaf folding Dialogue/ argumentation E-moderating framework Enquiry-ledExperiential learning

Assimilative Listening Reading Viewing Information Handling

Analysing ClassifyingGathering ManipulatingOrderingSelecting

AdaptiveModellingSimulation

CommunicativeCritiquing Debating DiscussingPresenting

ProductiveComposing CreatingDrawing Producing

Assimilative Scanning Skim readingInformation HandlingBrainstormingBuzz words Concept mappingCrosswordsDefiningMindmapsWeb search

AdaptiveModelling

CommunicativeArticulate reasoningArguingCoachingDebateDiscussionFishbowlIce breakerInterviewNegotiation

WhoClass based Group based IndividualOne to many One to one

MediumAudio Face to faceOnlineText messaging Video

TimingAsynchronous Synchronous

Coach Deliverer Group leaderGroup participant Facilitator Individual learnerMentorModerator Pair person Peer assessor Presenter RapporteurSupervisor

HardwareComputerHeadphones Lab equipment PDA ProjectorRecorders Specialist subject equipment VideoWebcams

SoftwareBibliographic software Database Digital image manipulation software Libraries Microsoft exchange Mind mapping software NVIVO Project manager

Not assessedDiagnosticFormativeSummative

4 Table derived from the DialogPlus Taxonomy (project website http://www.nettle.soton.ac.uk/toolkit/Default.aspx)

http://www.nettle.soton.ac.uk/toolkit/Default.aspx


Learner disability CultureMotivation

TimeDifficultySupport Learning styles SkillsAbility to learn Commercial awarenessComputer literacy CreativityCritical analysisCritical readingCriticism Data modellingDecision makingE-literacy Foreign languages Group/team workInference and synthesis of information Information handling Information literacyInterpersonal competence ITLibrary Listening and comprehension Literacy Logical argumentMaking notes Management of change Negotiating Numeracy

Assemble CalculateConstructDemonstrateHypothesizeInferInvestigateProduceSelectSolveTranslateUseWrite

AnalysisAnalyse Break downCompare Compare and contrast Critique Differentiate between Distinguish between List component parts ofPredictSelect

SynthesisArgue Design Explain the reasons for GeneraliseOrganiseSummarise

EvaluationJudgeEvaluateGive arguments for and againstCriticise

Goal-based scenariosLearning cycle Problem-based Project-based Reflective practitioner Social constructivist

SituativeActivity theoryActivity basedAction researchApprenticeshipCollaborative learning Communities of practiceConceptualisation cycleReciprocal teaching Systems theoryVicarious learning

Re-mixing SynthesisingWriting

ExperientialApplying ExperiencingExploringInvestigatingMimickingPerforming Practicing

On the spot questioningPair dialoguesPanel discussion Peer exchangePerformanceQuestion and answerRounds Scaffolding Short answerSnowball Socratic instruction Structured debate

ProductiveArtefactAssignmentBook reportDissertation/thesisDrill and practiceEssayExerciseJournaling Literature reviewMCQ PortfolioPresentation ProductPuzzlesReport/paperTestVoting

ExperientialCase studyExperimentField tripGameRole playScavenger huntSimulation

Search engines SpreadsheetSPSSText, image, audio or video viewerVirtual worlds Word processor

ModelsAccess grid Blogs CAA tools Chat Discussion boards Electronic whiteboardsEmailInstant messaging ModellingSimulation Voice over IPVideo conferencingVLEsWikis

Resources


Oral communication Oral presentation Planning and organisingPracticalProblem solvingReadingReferencing ResearchSelecting and prioritising information Self managementSelf reflectionStudy skills Summary skillTime management and organisation SynthesisWriting styleWritten communication

FeedbackReflect

AffectiveAppreciateAwareness ListenResponsive

Aesthetic AppreciationCommitmentEthical awareness Moral awareness

PsychomotorDrawExercise Jump MakePerformPlayRunSwim Throw


4.2 Glossary of teaching approachesThis glossary explains the teaching approaches listed in section 4.1, Description of the characteristics of a learning activity. Terms are listed in the order given in the table. Each term has a very brief description, followed by sources of further information. Teaching techniques are described in section 4.3.

Further details of teaching approaches and theories can be found at the Theory into Practice website ( http://tip.psychology.org/ ) and in the DialogPlus toolkit (http://www.nettle.soton.ac.uk/toolkit/Default.aspx )

4.2.1 Description of a Learning ActivityA Learning Activity is:

an interaction between a learner or learners and an environment (optionally including content resources, tools and instruments, computer systems and services, ‘real world’ events and objects) that is carried out in response to a task with an intended learning outcome5

Learning activities are achieved through completion of a series of tasks in order to achieve intended learning outcomes. We have defined the components which constitute a learning activity as:

The context within which the activity occurs includes the subject, level of difficulty, the intended learning outcomes and the environment within which the activity takes place. Learning outcomes are mapped to Bloom’s taxonomy of learning outcomes and grouped into three types: cognitive, affective and psychomotor and are what the learners should know, or be able to do, after completing a learning activity; for example they might be required to be able to: understand, demonstrate, design, produce or appraise.

The learning and teaching approaches adopted. These are grouped according to Mayes and de Freitas’ three categories – associative, cognitive and situative6.

The tasks undertaken, which specifies the type of task, the (teaching) techniques used to support the task, any associated tools and resources, the interaction and roles of those involved and the assessments associated with the learning activity.

5 Beetham, H. (2004) Review: developing e-learning models for the JISC practitioner communities: a report for the JISC e-pedagogy programme, JISC6 Mayes, T. and S. de Freitas (2004). Review of e-learning frameworks, models and theories: JISC e-learning models desk study, JISC

http://www.nettle.soton.ac.uk/toolkit/Default.aspx

http://tip.psychology.org/


4.2.2 Associative Teaching ApproachesAssociative Learning & Teaching approaches focus on learning through association and reinforcement, and observable outcomes. Learning is viewed as acquisition of external knowledge or skills. A large number of e-learning models and tools derive from the associative perspective.

4.2.2.1 BehaviouristKnowledge is seen as a collection of facts which the teacher knows and must instil into the student, based on a theory of stimulus-response pairs. The teacher will stimulate the students to learn the facts. Success is judged by the ability of the students to reproduce the facts7.

4.2.2.2 DidacticThe didactic model is based on transmission of knowledge, explicit instructional goals, objectives, content, and expectations from teacher to learner. The traditional lecture is an example of didactic teaching

4.2.2.3 Elaboration theoryThis approach suggests starting from simple concepts and building on them to bring the learners to mastery of the more complex. It provides a model for sequencing and organizing courses of instruction8.

4.2.2.4 Instructional system designBased largely on the work of Gagne, this approach is based on building up knowledge and skills through a series of systematic steps to achieve specified learning objectives9.

4.2.2.5 Intelligent tutoring systemsIntelligent tutoring systems (ITS) have embedded intelligence, so personalised material can be presented to users. The system presents the student with a problem, compares the student’s answer with its own and provides feedback based on the differences. It then updates its records of the students’ skills and presents an appropriate new problem, repeating the cycle10.

7 Gagne, R. M. (1992) Principles of instructional design. Harcourt Brace Jovanovich, 4th edition8 Reigeluth, C.M., (1999). The elaboration theory: Guidance for scope and sequence decisions. In C.M. Reigeluth (ed.), Instructional-design theories and models: A new paradigm of instructional theory, volume ii. (pp. 425-459). Hillsdale, NJ: Lawrence Erlbaum Associates.9 Reference: Gagne, R. M. (1992) Principles of instructional design. Harcourt Brace Jovanovich, 4th edition.10 Books: Sleeman, D. H. & Brown, J. S. (Eds.). (1982). Intelligent Tutoring Systems. New York: Academic Press.


4.2.2.6 Training needs analysisTraining needs analysis is a work based approach which analyses the needs of organisations/teams/individuals, identitifies gaps and specifies training11.

4.2.3 Cognitive teaching approachesCognitive Teaching approaches view learning as a process of transormation in internal cognitive structures. Learning is characterised by processing and transmitting information through communication, explanation, recombination, contrast, inference and problem solving.

4.2.3.1 Active learningActive learning requires that students do things and think about what they are doing12.

4.2.3.2 Cognitive apprenticeshipLearning cognitive skills through guided experience, and reflection, working with an expert13.

4.2.3.3 Cognitive scaffoldingCognitive scaffolding provides a framework that helps the learner to organize prior knowledge and internalize new information. The teacher collaborates with students in activities that are just outside what they could achieve on their own. Once the student has reached the new level of understanding the scaffolding can be removed. Scaffolding might include models, cues, prompts, hints, partial solutions, think-aloud modelling and direct instruction14.

4.2.3.4 Constructivist based designThis approach draws on the work of Bruner and others who believe that learning is an active process where learners construct new ideas based on their existing or previous knowledge and understanding15.

11 Bee, F. & Bee, R.(2003) Learning Needs Analysis and Evaluation. Chartered Institute of Personnel & Development 12 Stiles, M.J., and Orsmond, P., Managing Active Student Learning with a Virtual Learning Environment., in Educational Development Through Information and Communications Technologies, Fallows, S.J. and Bhanot, R., Kogan Page, 200213 Seitz, R., “Short paper: Cognitive Apprenticeship” http://chd.gse.gmu.edu/immersion/knowledgebase/strategies/constructivism/CognitiveApprenticeship.htm 14 Van Der Stuyf, Rachel R. (2002) “Scaffolding as a Teaching Strategy” http://condor.admin.ccny.cuny.edu/~group4/Van%20Der%20Stuyf/Van%20Der%20Stuyf%20Paper.doc 15 Bruner, J., (1960) The Process of Education. Cambridge, Massachusetts: Harvard University Press.

http://condor.admin.ccny.cuny.edu/~group4/Van%20Der%20Stuyf/Van%20Der%20Stuyf%20Paper.doc


http://chd.gse.gmu.edu/immersion/knowledgebase/strategies/constructivism/CognitiveApprenticeship.htm

http://chd.gse.gmu.edu/immersion/knowledgebase/strategies/constructivism/CognitiveApprenticeship.htm


4.2.3.5 DialogueDialogue is a structured process that uses talk to advance participants’ deep knowledge, understanding and belief. It is concerned with solving problems and developing new ideas. It goes beyond the simple sharing of ideas, or traditional question-answer or listen-tell teaching approaches16.

4.2.3.6 E-moderating frameworkSalmon’s e-moderating framework describes five levels of competence of online teachers and learners and how their skills can be developed17.

4.2.3.7 Enquiry-ledAlso know as enquiry-based learning (EBL), this approach requires that content, teaching methods and assessment all encourage students to research, discover and construct their own knowledge and meanings18.

4.2.3.8 Experiential learningExperiential learning, based on the work of Piaget, Lewin, Kolb and others, requires that learners reflect on experience, devise, and subsequently test, general rules19.

4.2.3.9 Goal-based scenariosGoal-based scenarios, such as simulations or role play, use skills based learning to achieve specified learning outcomes20.

4.2.3.10 Learning CycleA recurring cycle within which the learner tests new concepts and modifies them as a result of reflection and conceptualization. Kolb’s learning cycle suggests four stages (experience, reflection, abstraction and experimentation)21.

16 Alexander, R .J. (2005) Towards Dialogic Teaching: Rethinking Classroom Talk, York: Dialogos17 Salmon, Gilly (2000) E-moderating: the key to teaching and learning online London: Kogan Page18 Peter Kahn and Karen O'Rourke (2004) Guide to Curriculum Design: Enquiry Based Learning http://www.heacademy.ac.uk/resources.asp?process=full_record&section=generic&id=359 19 Kolb, D. (1984) Experiential Learning, New Jersey: Prentice Hall, Inc.20 Schank, Roger, C (1992) Goal-Based Scenarios, http://cogprints.ecs.soton.ac.uk/archive/00000624/00/V11ANSEK.html21 KOLB D A (1984) Experiential Learning: experience as the source of learning and development New Jersey: Prentice-Hall (0 13 295261 0)

http://www.heacademy.ac.uk/resources.asp?process=full_record&section=generic&id=359

http://www.heacademy.ac.uk/resources.asp?process=full_record&section=generic&id=359


4.2.3.11 Problem-basedLearners investigate a specific scenario either individually or in groups & propose solutions or determine what skills and/or information they would need to manage or solve the problem(s)22.

4.2.3.12 Project-based learningProject based learning is a "systematic teaching method that engages students in learning knowledge and skills through an extended inquiry process structured around complex, authentic questions and carefully designed products and tasks"23.

4.2.3.13 Reflective practitionerReflective teaching means thinking about what you are doing in your teaching practice, why you do it that way, and what effect it has on students’ learning. Crucially, reflective practitioners will encourage their students to self-evaluate their own learning in a similar way24.

4.2.4 Situative Teaching ApproachesSituative learning views learning as social participation, emphasising interpersonal relationships in imitation, modelling, and joint construction of meaning.

4.2.4.1 Action researchAction research involves the pursuit of change and understanding. Action, and critical reflection about the action, are integrated, often in a cyclic process, so that the knowledge gained is directly relevant to the issues being studied. It is frequently done by a group often with an outside facilitator25.

4.2.4.2 Activity theoryActivity theory is based in a theory of ‘doing’ where human subjects work on an object in order to obtain an outcome using tools. The object might be material (eg. a book) or less tangible (eg. an organisational structure). The tools might be external (eg. a computer) or internal (eg. a plan). Communities may form around shared tools or objects. Activities are thus carried out in a social context26.

22 Savin-Baden, M., (2000). Problem-based Learning in Higher Education, Buckingham: Open University Press.23 Project Based Learning Handbook (2002) Buck Institute for Education http://www.bie.org/pbl/pblhandbook/intro.php#history 24 Schön, D. A. (1990) Educating the Reflective Practitioner : Toward a New Design for Teaching and Learning in the Professions. Jossey-Bass.25 Book: Marquardt, M. J., (1999) Action Learning in Action, Consulting Psychologists Press.26 Activity theory pages, http://carbon.cudenver.edu/~mryder/itc_data/activity.html

http://www.bie.org/pbl/pblhandbook/intro.php#history


4.2.4.3 ApprenticeshipApprenticeship can be described as "a social theory of learning in which young learners (newcomers) are conceptualised as 'legitimate peripheral participants' who learn by participating first peripherally and gradually more fully in communities of practitioners"27.

4.2.4.4 Collaborative learningCollaborative learning is "an instruction method in which students at various performance levels work together in small groups toward a common goal. The students are responsible for one another's learning as well as their own. Thus, the success of one student helps other students to be successful." 28

4.2.4.5 Communities of practiceCommunities of practice emphasise social participation, viewing learning as construction of community, identity, meaning and practice29.

4.2.4.6 Conceptualisation cycleA three-stage cycle of conceptualisation (exposure to new ideas), construction (ideas are applied to meaningful tasks) and dialogue (during performance of tasks, where understanding of ideas is tested during conversation: this is where learning takes place)30.

4.2.4.7 Reciprocal teachingReciprocal teaching entails the teacher and/or learners take turns leading a dialogue. There are four key activities: predicting, questioning, summarising and clarifying31.

4.2.4.8 Systems theorySystems theory focuses on organisational learning and on modelling the development of learners in response to feedback32.

27 Fuller, Alison and Unwin, Lorna (2003)Creating a Modern Apprenticeship: a critique of the UK's multi-sector, social inclusion approach. Journal of Education and Work, 16 (1), 5-2528 Gokhale, A. A. (1995)Collaborative Learning Enhances Critical Thinking, Journal of Technology Education Vol.7,(1)29 Wenger, Etienne (1999) Communities of Practice. Learning, meaning and identity, Cambridge: Cambridge University Press30 Mayes, T (1995) “Learning Technology and Groundhog Day” in Strang, W. Simpson, V. B. & Slater, D. (eds) Hypermedia at Work: practice and theory in higher education, Canterbury, UK: University of Kent Press. 31 Palincsar, A.S. and Brown, A.L. (1984) Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 2, pp. 117-175


4.2.4.9 Vicarious learningVicarious learning entails learning by observing and modelling behaviours, attitudes, and emotional reactions. In learning activities it frequently involves students observing their peers participating in discussions as learners33.

4.3 Glossary of Teaching TechniquesThis glossary explains the teaching techniques listed in Section ...: Description of the characteristics of a learning activity. Terms are listed in the order given in the table. Each term has a very brief description, followed by sources of further information. Teaching approaches are described in the previous section.

4.3.1 Assimilative techniques

4.3.1.1 ScanningScanning is rapid reading to obtain specific information from a piece of text, eg. a particular reference in an essay34.

4.3.1.2 Skim readingSkimming is a fast reading technique used to obtain the overall sense of a piece of text and to identify the main ideas before deciding whether to read it more slowly and in detail35.

4.3.2 Information Handling techniques

4.3.2.1 BrainstormingBrainstorming generally takes place in a group situation. A problem or idea is defined and all members of the group make suggestions related to the topic. All ideas are written down, however unlikely they may seem. They are evaluated only after the session is complete.

4.3.2.2 Buzz words Buzz words can be used in a form of brainstorming session, for example when students are asked to list the words they connect with ‘learning’, ‘policy’, or some other concept.

32 Senge, P. M. (1990) The Fifth Discipline. The art and practice of the learning organization, London: Random House33 Bandura, A. (1977). Social Learning Theory. New York: General Learning Press.34 BBC skillswise: scanning http://www.bbc.co.uk/skillswise/words/reading/techniques/scanning/index.shtml35 BBC skillswise: skimming http://www.bbc.co.uk/skillswise/words/reading/techniques/skimming/index.shtml


4.3.2.3 Concept mapping (also called mind maps)Concept mapping is a way of representing, exploring, and sharing knowledge and information as a diagram showing a connected web of ideas36. For example. the activity in section... used concept mapping for students to share and organise their ideas about the UK print industry. Concept mapping is often used by students for taking notes. It is easily done on paper, but a number of software tools are also available.

4.3.2.4 CrosswordsCrossword puzzles, in which words corresponding to numbered clues are found and written into squares are often used by teachers to reinforce definitions, simple facts and vocabulary.

4.3.2.5 DefiningDescribing the meaning of a concept or term.

4.3.2.6 Mindmaps (see concept mapping)

4.3.2.7 Web searchSearching the world wide web for information about a specified topic using a search engine such as Google.

4.3.3 Adaptive techniques

4.3.3.1 Modelling Describing or representing a system, often with a view to understanding its behaviour. The model may be material, as in a clay model, or abstract, as in a mathematical model.

4.3.4 Communicative techniques

4.3.4.1 Articulate reasoningStudents explain their reasoning via writing, speaking etc.

4.3.4.2 ArguingA verbal dispute.

4.3.4.3 CoachingCoaching provides support and encouragement to students to find their own solutions to problems or issues. Coaching is often used with constructivist teaching approaches such as problem based learning.

36 BBC keyskills: mind maps http://www.bbc.co.uk/keyskills/extra/module4/1.shtml


4.3.4.4 DebateA structured discussion of opposing points of view.

4.3.4.5 DiscussionAn extended verbal interaction focusing on a particular topic.

4.3.4.6 FishbowlA discussion technique where an inner group of students discuss a topic while the rest of the students observe the discussion. At the conclusion of the discussion the whole group comes together to evaluate it37.

4.3.4.7 Ice breakerIcebreakers are short, fun, activities used to facilitate introductions and warm-ups, to introduce the topic of a meeting or training or to facilitate team building. They can also be used within established groups to facilitate discussion on a chosen topic.

4.3.4.8 InterviewA conversation between two or more people in which the interviewer(s) ask questions of the interviewee to elicit information.

4.3.4.9 NegotiationA discussion that is intended to produce agreement.

4.3.4.10 On the spot questioningThe teacher asks questions and selects students at random to give answers.

4.3.4.11 Pair dialoguesTwo students discuss an issue together and then present to the class

4.3.4.12 Panel discussion A discussion on a specified topic among a small group of people in front of an audience.

4.3.4.13 Peer exchangeSharing of ideas and work between peers (eg. between students without the presence of the teacher, encouraging them to discuss the material between themselves)

37 http://academy.fiu.edu/aatarticles/ClassroomManagement/fishbowldiscussion.html


4.3.4.14 PerformanceA live presentation, by a student or group of students, in front of an audience

4.3.4.15 Question and answerStudents are put into teams or pairs. One half ask questions and the other half give answers

4.3.4.16 Rounds This is a simple technique that encourages participation. The facilitator states a question and then goes around the room inviting everyone to answer briefly. This is not an open discussion. This is an opportunity to individually respond to specific questions, not to comment on each other's responses or make unrelated remarks38.

4.3.4.17 ScaffoldingThe teacher provides a temporary framework, or scaffold, that helps the learner perform a task that is just outside what they could achieve on their own. as the learner becomes adept, the scaffolding is removed. Scaffolding might include models, cues, prompts, hints, partial solutions, think-aloud modelling and direct instruction39

4.3.4.18 Short answerStudents are asked to provide very short written answers to questions. These may be only one or two sentences, or up to about 100 words. Short answers can be used to boost student confidence and writing skills, building up to longer essays.

4.3.4.19 Snowball Students share their ideas on an issue in pairs. The pairs then group into fours to take the discussion further; the fours combine into eights, etc.

4.3.4.20 Socratic instruction Socratic instruction helps students to discover and understand concepts through questioning rather than telling. Typically the teacher will ask an open question of the students and then ask further questions based on their answers. The technique seems particularly popular in teaching law.

38 http://www.consensus.net/ocac6.html39 Van Der Stuyf, Rachel R. (2002) “Scaffolding as a Teaching Strategy” http://condor.admin.ccny.cuny.edu/~group4/Van%20Der%20Stuyf/Van%20Der%20Stuyf%20Paper.doc




4.3.4.21 Structured debateA structure that enables an online debate between students. The teacher poses an issue. Each student is obliged to stake out a position. All positions can be posted in the same document if everybody wants the convenience of being able to see all positions at once. Then to each position, each student attaches (i.e., hypertext links) pro or con arguments. For convenience, these also may be put in a common pro or a con document. Students then critique the arguments by attaching (linking) various comments, two to four participants engage with each other on provocative or divisive issues with an eye to challenging themselves and the audience to examine their assumptions and unconscious beliefs.

4.3.5 Productive techniques

4.3.5.1 ArtefactA material object made or used by humans

4.3.5.2 AssignmentA task whose outcome or performance is used for summative assessment

4.3.5.3 Book reportA critical account of a book

4.3.5.4 Dissertation/thesisA relatively long (5,000 – 100,000 words) piece of writing produced by independent study or research

4.3.5.5 Drill and practiceRepetitive performance of a small task to improve specific skills, for example spelling or arithmetic.

4.3.5.6 EssayA relatively short piece of writing (up to about 5000 words) written from the author’s viewpoint

4.3.5.7 ExerciseA task (usually written) performed to increase skill or understanding

4.3.5.8 Journaling A personal record of a process reflecting on who, what, where, when and/or why, used either for reflection or for enabling a rapid re-start of the activity.


4.3.5.9 Literature reviewA critical survey of publications on a given topic

4.3.5.10 Multiple Choice Quiz (MCQ)A series of well defined questions presented with a range of possible answers. Students have to choose the correct answer(s). MCQ are frequently computer marked and/or generated: a variety of MCQ authoring tools are available and are included in many VLEs.

4.3.5.11 PortfolioA collection of pieces of work that has been put together to provide evidence of a student’s efforts, abilities and achievement. ePortfolios enable students to gather, organise, and store, their work in a variety of digital formats (text, audio, video, graphics etc).

4.3.5.12 PresentationA publicly available account, usually based around visual representations and given live or accessed as a web page. Presentation software such as Powerpoint is often used to help generate visual aids.

4.3.5.13 ProductSomething created by a person or process

4.3.5.14 PuzzlesA problem with a correct solution that the student must work out

4.3.5.15 Report/paperA written document describing the findings of a student or group of students on a particular topic or problem

4.3.5.16 TestA way of measuring a student’s knowledge, skill or abilities, often as summative assessment. A test is usually more exacting than a quiz, but less exacting than an exam.

4.3.5.17 VotingA question is posed and several possible answers suggested. The students vote for the answer they think is correct. Voting is often used in real-time sessions to increase interactivity and assess student understanding of the concepts being discussed. Many synchronous communication systems have voting functions; lecture theatres may be equipped with personal response voting systems; or students can vote by show of hands, or text messaging from their mobile phones.


4.3.6 Experiential techniques

4.3.6.1 Case studyAn in depth description and analysis of an individual example of a problem, topic or issue, and its context, used to gain understanding of complex systems.

4.3.6.2 ExperimentA set of controlled actions, and observation and analysis of the consequent outcomes, designed to arrive at the solution to a problem or to create new knowledge. Experiment is the basis of an empirical approach to science and is fundamental to many science lessons.

4.3.6.3 Field tripA group excursion to a location outside the normal teaching environment (eg. seashore, city centre, museum) for firsthand examination of a topic

4.3.6.4 GameAn activity with a set of rules, usually undertaken in pairs or by a group. Because games are generally perceived as recreational, they are often used in teaching contexts to increase student motivation while practicing skills, gathering knowledge or developing concepts. A multitude of computer mediated games are available which allow individual and/or multiplayer play.

4.3.6.5 Role playThe students pretend to be other individuals (eg. doctors, planning officials, historical characters) in order to develop understanding of complex social processes.

4.3.6.6 Scavenger huntA game in which individuals or groups are given a list of items to find, often as a race.

4.3.6.7 SimulationMimicking some, or all, of the behaviour of a system using a different (usually cheaper and/or more easily controlled) system. Computer simulations of complex systems which allow for student interaction with the system (eg. virtual laboratories) are becoming increasingly available.


4.4 Template for designing a learning activityThis template is designed to act as a prompt for the things you need to think about in designing a learning activity, and as a basis for discussion with your technical support staff who may be helping you implement the activity. Section 4.5 contains an example of a completed template.

Key words in red and underlined are further described and characterised in section 4.1 ‘Description of the characteristics of a learning activity’ and the glossaries in sections 4.2 & 4.3

Authors and Institutions

Summary of learning activity

(you may prefer to fill this section in last)

What are your aims in running this activity?

Institutional context: FE/HE/School etc Are there assessment

requirements? Are there timetabling

issues? What is the quality

assurance process?

What subject (discipline) and level are you teaching?

Are there any academic prerequisites?


What is your learners’ environment? (eg. Classroom, distance, etc)

What are your learner’s needs? Include things like: Accessibility Language support Disability Culture/nationality Motivation Other needsDo any individual students have specific requirements?

Learning style: how do learner needs/preferences affect your teaching and delivery?

Is there anything else you feel is important about your learners’ context?

What are the intended learning outcomes of the activity?

What teaching approach is best suited to your intended learning outcomes, learners’ needs and institutional context?

You may want to use several approaches within a single activity

(suggestions for teaching approaches and techniques are given in the glossary)


What task types and techniques are involved in the activity

(suggestions for task types are given in the glossary)

Who will be involved in the activity? What will their roles be?

(people may have more than one role)

Who? Role?

What tools and resources do you need to run the activity?

What happens during the activity?

Describe the sequence of actions and interactions

Include information on: What would you (the

teacher) do to set up the activity?

What would you do to run the activity?

What would the students do?

What interactions will you have with the students? The technological systems?

What interactions will the students have with you? Each other? The


technological systems?Continue on a separate sheet if necessary

Would you assess the activity? If so, how?

Would you evaluate the activity? If so, how?


What support would you need to run the activity? (eg. help using video equipment)

How long does the activity take?How difficult is the activity?


4.5 Example of a completed learning activity templateThe activity in this example consists of several tasks, and types of student and lecturer role and interaction. Some of the tasks run concurrently while others follow on from previous tasks in sequence. The activity is typical in having a strong student discussion component, in using a multiple choice quiz to enable students and the lecturer to assess their understanding, and in collecting ideas via a brainstorming session and organising them using concept maps.

Key words in red and underlined are further described and characterised in section 4.1 ‘Description of the characteristics of a learning activity’ and the glossaries in sections 4.2 & 4.3

Authors and Institutions John Edmonstone, Cardonald CollegeAnn Jeffery, University of Southampton

Summary of learning activity

(you may prefer to fill this section in last)

A 1st year HN Journalism class (Scotland) studying the UK print industry. Introductory session on structure of industry. The session would begin with a brief exposition on industry, followed by a brainstorm on categorising newspapers within the industry framework. The brainstorm would have the purpose of building dialogue and enabling validation of prior learning. as as well delivery an agreed model of the industry. A testing activity would follow to establish understanding and then research based task.

What are your aims in running this activity?

Introduce conceptual and practical map of UK print sectorCreate peer group connectionsIntroduce students to VLE useBegin reinforcement of transferable research skills

Institutional context: FE/HE/School etc Are there assessment

requirements? Are there timetabling

issues? What is the quality

assurance process?

FE (in Scotland). 20 in whole class, divided into small groups for chat - 5 maximum.

What subject (discipline) and level are you teaching?

HNC/D Journalism


Are there any academic prerequisites?

Higher/AS level English needed. 2 x Highers/AS levels. ICT helpful or ICT support.

What is your learners’ environment? (eg. Classroom, distance, etc)

Classroom/computer lab based plus VLE support but also 25% online Distance Students using VLESelf study

What are your learner’s needs? Include things like: Accessibility Language support Disability Culture/nationality Motivation Other needsDo any individual students have specific requirements?

Motivation OK as vocational. High level of motivation so can do more things than with other subjects

Some visual, some collaborative, auditory and pragmatist as embedded in practice

Is there anything else you feel is important about your learners’ context?

What are the intended learning outcomes of the activity?

Unit outcome:Describe and explain the UK print market

What teaching approach is best suited to your intended learning outcomes, learners’ needs and institutional context?

You may want to use several approaches within a single activity

Blended learning. Conceptualisation. Social/dialogic, Vocational, Community of Practice/Reciprocal, Collaborative learning/ Situative


(suggestions for teaching approaches and techniques are given in the glossary)

What task types and techniques are involved in the activity

(suggestions for task types are given in the glossary)

Exposition at beginning. Brainstorming. Validate via concept map/presentation. Brainstorm in a group in class. Groupbuilding first to establish group cohesion. Potential discussion forum. Chat/webcams and record conversation in VLE (or chat history)

Who will be involved in the activity? What will their roles be?

(people may have more than one role)

Who? Role?

lecturer/tutor Lecturer/facilitator

Summariser

Students Individual learner

Group learner

What tools and resources do you need to run the activity?

LCD projector. Network link. Chat/bulletin boards

Websites, tests/activities/ newspapers/workshops

What happens during the activity?

Describe the sequence of actions and interactions

Include information on: What would you (the

teacher) do to set up the activity?

What would you do to run the activity?

What would the students do?

What interactions will you have with the students? The technological systems?

Write and prepare course materials in paper and electronic formats. Load course materials into VLE (Blackboard)

With students I:Introduce to session – expositionBrainstorm using examples of newspapers – trying to fit these into a structure plan for industry (based on concept mapping)Exposition on structure using PowerPoint Diag in VLEStudents access VLE – carry out self-testStudents begin research task on VLE (to be completed and submitted online). I support students individually.

Students take part in brainstorm session Students complete online task and research task


What interactions will the students have with you? Each other? The technological systems?

Continue on a separate sheet if necessary

I would interact with students face to face one to many at outset in exposition and brainstorm. I would support students one-to-one in accessing the VLE. I would access their self-test results online to check learning. The students would feedback their research to me individually online and I would feedback to them online and individually next session. The students will interact with me both individually and collectively in the initial classroom situation. They will interact with one another in the brainstorm, through myself as moderator. They will also be able to question me at any point. They students will also interact with the VLE – both in the classroom and after the session - in carrying out self-test and then ongoing research task.

Would you assess the activity? If so, how?

Record of debate into task worksheet to test individual understanding. Also self-assessment and more support in self-assessment depending on context/supplementary materials. The online test would assess initial understanding (although I would be doing this informally during brainstorm too.) The research activity would be a further assessment.

Would you evaluate the activity? If so, how?

I would informally evaluate the activity in the classroom discussion and by eliciting class feedback at the end. The online test would be a form of evaluation too.

What support would you need to run the activity? (eg. help using video equipment)

No support other than network support generally available in college

How long does the activity take?How difficult is the activity?


Documents

Draft Learning Activity Reference Model - Pedagogy · Web viewSenge, P. M. (1990) The Fifth Discipline. The art and practice of the learning organization, London: Random House Bandura,