Multimodal Methodologies Part I and II

Jeff Bezemer, Carey Jewitt and Gemma MossMO

DE m

ultim

odal

met

hodo

logi

esFO

R RE

SEAR

CHIN

G DI

GITA

L DA

TA A

ND

ENVI

RON

MEN

TS

One

http://mode.ioe.ac.uk

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

• An understanding of the origins and scope of multimodality

• Explications of multimodal frameworks and key concepts

• Knowledge of the different ways in which multimodality has been taken up by a range of perspectives

• An outline of the procedures of multimodal analysis

The aim of today is to provide:

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

• Why take a multimodal approach?• What concepts and methods can be used to

analyse interaction in digital environments?• How can we get at how the use of different

digital technologies shapes understandings and experiences?

• What challenges does multimodality raise for researching digital data and environments?

Focusing questions:

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

Part I • 9.15 Introduction • 9.30 Origins, Scope and Key concepts (Carey Jewitt)• 11.15 Multimodal perspectives: video data analysis (Jeff

Bezemer)

12.45 lunch

Part II • 16.00 Presentation and data workshop: Analyzing multimodal

digital texts (Gemma Moss)

Program:

Multimodality: origins, scope and key

concepts

MO

DE m

ultim

odal

met

hodo

logi

esFO

R RE

SEAR

CHIN

G DI

GITA

L DA

TA A

ND

ENVI

RON

MEN

TS

http://mode.ioe.ac.uk

6

7

Concepts: the level of modeMODEMEANING FUNCTIONSSEMIOTIC RESOURCESEMIOTIC (MEANING) POTENTIALMATERIALITYAFFORDANCEPROVENANCEMODAL LOGICSFUNCTIONAL SPECIALISM

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

December 2011, Norway

Concepts: Level of modal-interaction MULTIMODAL ENSEMBEL INTER-MODAL RELATIONSCOHERENCEDISSONANCEMODAL DENSITYMODAL INTENSITYEQUAL-UNEQUALDOMINANT – SUBORINATEELABORATIONEXPANSIONREPETITION-EMPHASISSALIENCE

December 2011, Norway

Concepts: level of process of sign makingSIGN MAKERINTERESTMULTIMODAL ORCHESTRATIONMULTIMODAL DESIGNTRANSFORMATIONTRANSDUCTIONTRANSMODAL MOMENTTEXTUAL CYCLEMODAL ORIENTATION CONTEXT

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

• Social Semiotic Multimodal Analysis• A Systemic Functional Approach (Multimodal

Discourse Analysis)• Multimodal Interactional Analysis • Multimodal conversation analysis

Four main perspectives on multimodality:

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

• fine grained analysis: the need to understand the details of texts and interactions

• focus on meaning making as social and semiotic• the demand to broaden conceptions of

communication beyond language• shared interests, and shared histories• externally material focus

Similarities

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

• Historical origins, influences and directions• Extent of emphasis on context• Degree of emphasis on modes and modal

systems• Attention to the agentive work• The scale of attention

Differences

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

• Organizational geography• Temporality of events• Recognizing events: patterns and customs• Detecting incongruities and breakdowns• Sequences of interaction, before, after and

immediately following one another

Conversation analysis & multimodality: analytical focus

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

Systemic Functional Linguistics (Anders Bjorkvall, 2009 in HMA)

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

Social Semiotic Multimodal Analysis

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

Analyzing Multimodal Interaction Norris (2004)

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

Norris (2009) Modal Denisity in HMA

MODE multimodal methodologiesFOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

• Sensory anthropology (e.g. David Howes)• Socio-cultural approaches and multimodality

(e.g. Barbara Rogoff, Roger Saljo)• Eye-tracking (Jana Holsvana)• New literacy Studies (Kate Phal, Jennifer Rowsell)• Historical analysis with a multimodal focus (e.g.

Grosvenor)

Other approaches

mode@ioe.ac.uk

http://mode.ioe.ac.uk