Newmedia4

Introduc)on to New Media Lecture 4

Mixed media art and comprehension

What perhaps has been most surprising and consequen4al about new technologies has been not so much what such innova4ons were originally designed to do, as what users have chosen to use them to accomplish. Harrison & Barthel, 2009

Topics

•  The levels of visual media and examples •  New media, crea)on and art

•  Comprehension of new media

Ques)ons

•  Draw a concept map of comprehension and interac)on with new media (with mixed media types that allow par)cipatory interac)ons)

•  For essay: Mixed media percep)on – how we may perceive one new media applica)on?

The levels for visual media

•  Lev Manovich The Language of New Media (2001)

•  Manovich is looking new media at five levels: – Numerical representa)on & programmability – Modularity at code level – Automa)za)on of opera)ons in crea)on, manipula)on and access

– Variability and branching type of interac)vity – Transcoding at the cultural interface level

Lev Manovich The Language of New Media (2001)

Digital images

•  Computer‐based image is discrete, since it is broken into pixels. This makes it more like a human language (but not in the semio)c sense of having dis)nct units of meaning).

•  Computer‐based image is modular, since it typically consists from a number of layers whose contents oSen correspond to meaningful parts of the image.


Digital images •  Computer‐based images are typically compressed using lossy compression techniques, such as JPEG. Therefore, presence of noise (in a sense of undesirable ar)facts and loss of original informa)on) is its essen6al, rather than accidental, quality.


Digital images •  Variability and automa6on, these general principles of new media, also apply to images.

•  For example, using a computer program a designer can automa)cally generate infinite versions of the same image which can vary in size, resolu)on, colors, composi)on and so on.


Technologies have disappeared behind the metaphor of pain)ng, enabling others to use them within the familiar frame of tradi)onal ar)s)c prac)ce (from Lehmann, 2009).

Digital images •  Computer‐based image consists from two levels, a surface appearance and the underlying code (which may be the pixel values, a mathema)cal func)on or HTML code).

•  In terms of its “surface,” an image par6cipates in the dialog with other cultural objects. The surface‐code pain can be related to signifier — signified, base — superstructure, unconscious — conscious pairs. So, just as a signifier exists in a structure with other signifiers of a language, a “surface” of an image, i.e. its “contents” enters in dialog with all other images in a culture.

•  In terms of its code, an image exist on the same conceptual plane as other computer objects.


Digital images: image interfaces


•  An image acquires the new role of an interface ‐ image becomes image‐interface.

•  In this role it func6ons as a portal into another world

•  Rather than staying on its surface, we expect to go “into” the image.

•  Image can func)on as an interface because it can be “wired” to programming code; thus clicking on the image ac)vates a computer program (or its part).

An image‐interface may be interac)vely manipulated

image as image‐interface

•  The new role of an image as image‐interface competes with is older role as representa6on.

•  Therefore, conceptually, a computer image is situated between two opposing poles: an illusionis6c window into a fic6onal universe and a tool for computer control.

•  The task of new media design and art is learn how to combine these two compe6ng roles of an image.


image‐instruments •  computer images also func6ons as image‐instruments.

•  If image‐interface controls a computer, an image‐instrument allows the user to remotely affect physical reality in real 6me.


Ini)a)ng Social art ac)vity

Mixed body parts

•  I came up with the idea of new narra)ve using parts of bodies in urban environment and trace the dimension of urban hybrid being, thus research how different people perceive and par)cipate.

•  For this experiment common tag besides #narra)veecology is mixedbodies and then for par)cular images ‐ head, foot, torso, arm.

•  In flickr you can easily organise photos in a batch (rotate, add tags, geo loca)on to all needed pictures at once and send them to the group sets ;).

Mixed body parts

It is not easy to trigger par)cipa)on

The flickr group is created and you are welcome to add your body parts.

Hyperlinked images

•  A computer image is frequently hyperlinked to other images, texts, and other media elements.

•  Rather than being a self‐enclosed en)ty it points, leads to, directs the user outside of itself towards something else.

•  Hyperlinked image, and hypermedia in general, “externalizes” Pierce’s idea of infinite semiosis


Nelson: ”hyperfilm” and “hypermedia”

•  Nelson introduced the terms hyperfilm and hypermedia:

•  Films, sound recordings, and video recordings are also linear strings, basically for mechanical reasons. But these, too, can now be arranged as non‐linear systems – for instance, laFces – for educa6onal purposes, or for display with different emphasis…

•  The hyperfilm – a browsable or vari‐sequenced movie – is only one of the possible hypermedia.


A database of images as a cultural unit

•  From a single image which represented the “cultural unit” of a previous period we move to a database of images as “cultural units” .

•  Today the problem is no longer how to create the right image, but how to find already exis)ng one.


hhp://mayron.net/liam/pub/mayron_disserta)on.pdf

A database of images as a cultural unit

Sunsets by )me: )me graphs (Photographs mapped by )me.) hhp://www.flickr.com/photos/krazydad/4992355/in/set‐140323/

The deepest "dip" in the wave formed by the images is the Summer Sols)ce.

A database of images reveals paherns of interac)on with space

hhp://liSlab.com/think/fabien/2006/12/13/tracing‐the‐visitors‐eye/

A trace consists in an ordered set of geotagged images taken by one person in one day and downloaded to Flickr.

There might be a poten)al to define and confirm paherns of how tourists navigate the urban space

A database of images as a cultural unit reveals niches

•  Invisible Ci)es maps geocoded ac)vity from online services such as Twiher and Flickr, both in real‐)me and in aggregate to another: an immersive, three dimensional space crea)ng hills and valleys represen)ng areas with high and low densi)es of data.

hhp://www.chris)anmarcschmidt.com/invisibleci)es/

Storytelling in databases is collabora)ve

Iden)fying narra)ve paherns have always been one of our main strategies for understanding the world

hhp://jilltxt.net/txt/Walker‐AoIR‐3500words.pdf

Storytelling in databases reveals niches

Film

•  Cinema erases any traces of its own produc6on process, including any indica6on that the images which we see could have been constructed rather than recorded

•  It denies that the reality it shows oSen does not exist outside of the film image

•  It pretends to be a simple recording of an already exis6ng reality


the shiS to digital film

•  digital film = live ac6on material + pain6ng + image processing + composi6ng + 2D computer anima6on + 3D computer anima6on

•  film obtains the plas6city which was previously only possible in pain6ng or anima6on ‐ "elas6c reality.”

•  The result: a new kind of realism, which can be described as "something which looks is intended to look exactly as if it could have happened, although it really could not."


Future for digital films

•  cinema)c forms which are not linear narra6ves: music video, CD‐ROM‐based games, computer mul)media

•  non‐narra6ve films •  New Temporality: Loop as a Narra)ve Engine (Lola, run) (adopted from peep show), arranging the trajectory of every character through space as a loop

•  The loop gave birth not only to cinema‐ computer programming involves altering the linear flow of data through control structures, such as "if/then" and "repeat/while"; the loop is the most elementary of these control structures.


A game

•  A game is an ar6s6cally simplified representa6on of a phenomenon.

•  The game designer simplifies deliberately in order to focus the player's aRen6on on those factors the designer judges to be important.

•  A game is collec)on of parts which interact with each other, oSen in complex ways. It is a system in which interac6ve element is a crucial factor

•  The subjec6ve reality and objec6ve reality are intertwined – it is a subset of reality. A game is a safe way to experience reality

•  The player's fantasy is the key agent in making the game psychologically real.

The Art of Computer Game Design by Chris Crawford

A game •  While a story is a vehicle for represen)ng reality, not through its facts per se, but through the cause and effect rela)onships suggested by the sequence of facts, •  the game presents a branching tree of sequences and allows the player to create his own story by making choices at each branch point,

•  a game is dynamic. The Art of Computer Game Design by Chris Crawford

A game •  Game‐playing requires two components: a game and a player.

•  A game acknowledges the player's existence and reacts to the player's personality and transforms the nature of the challenge from a passive challenge (defined by someone) to an ac)ve challenge (defined by me)

•  Interac)on injects a social or interpersonal element into the event and transforms the challenge of the game from a technical one to an interpersonal one.


A game •  Conflict is fundamental to all games •  If an intelligent agent from the game ac)vely blocks the player's ahempts to reach his goals, and purposefully responds to the player’s ahempts, the conflict between the player and the agent is inevitable.

•  Without ac)ve response, there can be no interac)on. Thus, expunging conflict from a game inevitably destroys the game.

•  It is possible to include coopera)ve elements by shiSing the conflict


spa)al narra)ve

•  In general, spa)al montage would involve a number of images, poten6ally of different sizes and propor6ons, appearing on the screen at the same 6me.

•  Its up to the filmmaker to construct a logic which drives which images appear together, when they appear and what kind of rela)onships they enter with each other

•  The same principle made computer programming possible: a computer program breaks a tasks into a series of elemental opera)ons to be executed one at a )me.


Spaces of cultural par)cipa)on

•  Television programs or website interfaces: – User prac)ces are structured by pre‐exis)ng socially and ideologically defined spaces within which ac)ons are performed

– User ac)ons may nego)ate and transform the very conven)ons and limits of a given mediated space

– The social power to construct such spaces and to define the frameworks for ac)on is not shared equally within a society


Example: Interac)ve television

•  Loops controlled by sms

•  Akvaario program and Sheep TV in Finland.

•  Repe))ve ac)on can be basis for new rou)ne

A story between stories: algorithmic and audience control of video segments in an experimental interac)ve television programme. Chris Hales; Teijo Pellinen; Markus CastrénDigital Crea)vity 2006, Vol. 17, No. 4, pp. 234–242

Interac)ve television

•  Online video‐sharing sites such as YouTube, are examples of how the corporate media industry invests in digital technologies to redefine the rela6onship between the spheres of produc6on and consump6on: –  by seducing the audience to contribute to a website, new sources of revenue are created;

–  users gain access to encrypted live streaming; –  users contribute their own content without monetary compensa)on;

–  and whenever they do go online and interact, users produce data valuable for targeted marke)ng and mass customiza)on.

Example: Youtube •  YouTube and Google Video create a space of par)cipa)on in which users maintain more control over the space –  Liberty: any one who owns the technological means can record and upload a video film and share it online

–  Restric6ons: ‘protocols’ and copyright regula)ons –  Commercial: markets for customized adver6sing; the adver)sers with data reveal any individual user’s online ac)vi)es, his or her cultural preferences, and the communi)es in which he or she par)cipates; ‘consumers’ of a ‘tube of plenty’; viral videos

–  Interac6vity: ‘snippets’: ‘prerecorded, rerecorded, )nkered, and self‐produced audiovisual content’ that is not finished invites users to appropriate, 6nker with and respond to it; ‘channel’ to which other users can subscribe

Example: Youtube •  Nine prototypes of users (YouTube 4 You, Miller 2007): –  the ‘recorder/sharer’, –  the ‘historian/enthusiast’, –  the ‘home movie maker’, –  the ‘video blogger’, –  the ‘instructor’, –  the ‘reporter’, –  The ‘performer’, –  the ‘aspiring film director’ –  the ‘online business’

Find one example for each type from Youtube videos

You know your meme: Drama)c Chipmunk and other varia)ons in Youtube

hhp://knowyourmeme.com/

‘FormaFng’ in reference to the adapta)on of interna)onally circula)ng television programs, in which the format details how a program should be produced but at the same )me allows producers to adapt it to the local culture.

Loca)ve interac)ve videos •  Can be focusing on: – Geographical loca)ons ‐ places with loca6ve film content might bring forth interac6ons and ac6vi6es with the film or between people (see Michael Naimark projects – Aspen moviemap hhp://www.naimark.net/projects/aspen.html)

– Narra)ves – story from place, anecdote; user‐configured perspec)ves can be chosen from the pool of thumbnails of videos as a pool of the space (see projects from Medialab Europe hhp://medialabeurope.org/ )

– Aesthe)c, landscape tradi)ons –  Social use of space In search of Oldton hhp://www.oldton.com/our_oldton.html

Chris Hales •  Structure for the interac)ve videomaking process:

–  Building projects of the place. One method: walk around and find a place of hanging around, make some basic films, interviews of the place, which other places you hang around – go to the second place and repeat the process.

–  Constraints will be needed: ‐ broad range of subject to fixed loca)ons; fix theme and film it in wider range of loca)ons (eg. movement etc.). ‐ geographical architecture of city (add visitors’ viewpoints or interviewing local residents, this could work with the older genera)on: memory‐based approach) ‐ social ac)vi)es related to the place ‐ all the factors involved in the place

–  Examples: hhp://csw.art.pl/new/99/7e_heldl.html –  hhp://www.youtube.com/watch?v=4va2dwRl6dU –  See Smartlab projects

hhp://smartlab.uel.ac.uk/2projects/index.htm

Criteria defining par)cipatory culture •  ‘Low barriers to ar)s)c expression and civic engagement’

•  ‘Strong support for crea)ng and sharing one’s crea)ons with others’

•  ‘Informal mentorship whereby what is known by the most experienced is passed along to novices’

•  The belief ‘that their contribu)ons maher’ •  The belief ‘members feel some degree of social connec)on with one another’ (Jenkins et al. 2006)

Jenkins, Henry, Ka)e Clinton, Ravi Purushotma, Alice J. Robison, and Margaret Weigel. 2006. Confron)ng the challenges of par)cipatory culture: Media educa)on for the 21st century. MacArthur Founda)on. hhp://www.digitallearning.macfound.org/ay/cf/% 7B7E45C7E0‐A3E0‐4B89‐AC9C‐E807E1B0AE4E%7D/JENKINS_WHITE_PAPER.PDF.

Concept: Produsage •  Bruns (2006) has coined the term ‘produsage’ to describe a new form of produc6on arising in the transi)on of media users from audience and consumers to ‘produsers’ enabled by new technologies.

•  ‘Produsers tend to collaborate’ (Bruns, 2006) because they are simultaneously consumers of content produced by others which they then use as raw material for their own content construc6on to be consumed and subsequently acted upon by others.

Bruns, A. (2006) ‘Towards Produsage: Futures for User‐led Content Produc)on’, in F. Sudweeks, H. Hrachovec and C. Ess (eds) Proceedings: Cultural AItudes towards Communica4on and Technology 2006, pp. 275–84. Perth: Murdoch University.

New media for crea)on and art Art explores and expresses our aesthe6c rela6on to our environment and ourselves.

•  Ar)s)c opportuni)es are expanding into: individual expressiveness and collec6ve ac6vism (Harrison & Barthel, 2009).

•  Using social soSware everybody can simultaneously be an ar6st and the audience, and this is the ground where both may be mutually inspiring for par6cipa6on.

Wielding new media in Web 2.0: exploring the history of engagement with the collabora)ve construc)on of media products Teresa M. Harrison; Brea Barthel (2009)

Community’s par)cipa)on in art •  Five modes of crea)ve control are (Brown, 2005): –  inven6veness –  interpre6ve acts of self‐expression – curatorial acts of selec)ng organizing and collec)ng

– observa6onal arts experience that one chooses – ambient arts experiences that are encountered unconsciously

Brown, A. (2005) Presenta)on of Neighborhood Survey Results for Philadelphia–Camden Cultural Par)cipa)on Benchmarking Project

Par)cipa)on inside? User ac)vi)es between design and appropria)on Mirko Tobias Schäfer

Implementa)on of crea)ve user ac)vi)es into commercial applica)ons

Accumula)ng and aggrega)ng

Mixing and remixing media for art

•  Portwiture” – the applica)on that aggregate Twiher and Flickr accounts of user, making visual representa)on of last Twiher posts of the user.

Par)cipa)ng for art: archiving and accumula)on

Mikrogalleri.es – the project of ZKM Media Museum in Karlsruhe. “Visitors to the museum or anyone with an Internet connec)on can upload images which are manipulated and published with other user‐supplied images”. The project goes further than Flickr: it does not only network the data of digital cameras as par)cipatory images, but also the cameras itself and with them their physical and ar)s)c context, place or space. The museum’s website supplies direc)ons for crea)ng a display of Internet images that can be projected at remote loca)ons and on the Internet. As ZKM proclaims, this network of par)cipatory produc)on is a further step from ‘user generated content’ to ‘user created/shared media,’ a preview to the new genera)on of the web: the Internet of Things” (Yang & Roch, 2007).

Cosntruc)on and sharing: Rhizome.org

Swarming for art

•  Twee6ng Colors by Brian Piana ‐ a public webpage consisted of “ver)cal color bars created by special tweets from Twiher users” (Piana, 2009). Any Twiher user can add bars by pos)ng special tweet as shown in the direc)ons.

•  Mobotag, by Marta Lwin, “reveals the hidden layers of a city through an ac)ve exchange of loca)on based media and text messages via the cellphone” (Lwin, 2009). It is a collabora)ve phone tagging of the city. Part virtual graffi), part walking tour, “mobotag” creates a spontaneous and easy way for tagging a neighborhood via the cellphone. Par)cipants may respond with their mobile media “in the crea)ve expression and mapping of their neighborhood” by sending and view messages, images, videos and sounds. A unique geocoding feature of Mobotag is uncovering other messages exis)ng in people’s local area.

Swarming for art

Hybrid & augmented reality art The object is oversized 3‐dimensional computer cursor (pointer). Placed in a public space and embedded with GPS device it could be touched and moved even further the special “screen based” territory.

Coordinates were mapped in Google Maps.

Par)cipants could also upload photos of the cursor at the website. (Campion, 2009).

hhp://www.urbancursor.com/

Urban Cursor is a project facilita)ng “social interac)on and play”.

Cogni)ve informa)on‐processing model

•  We need to consider, how humans process informa)on in order to predict the effect of new media and develop new media applica)ons

Percep)on •  Inputs is received from perceptual organs:

‐  Eyes: Visual input (graphics, text) (visual percep)on)

‐  Ears: Auditory input (sound, speech) ‐  Skin: Somato‐sensory input (hap)c percep)on) ‐  Tongue: Gustatory input ‐  Nose: Olfactory input

hhp://www.medical‐look.com/human_anatomy/systems/Sensory_organs.html

Technology can extend human senses

•  McLuhan: any technology can extend human senses and bodies

•  In which way technology can extend the sense organ? How does using such technology change our behavior? –  Sensor technology – Wearable displays –  Seamless interfaces –  Ergonomics

•  Robots to simulate human senses

Percep)on Perceptual informa)on is kept 1 sec in perceptual registers of the sensory memory, this is an unconscious process

Memory, stored schemata, and past experience play an important role in percep)on.

Interface and interac)on

Percep)on •  Percep)on is a media)onal process ‐ “mental processes” (cogni)ve ac)va)on from memory) are needed to enable the pick up of the relevant informa)on

•  Cultural interface and cogni6ve processing in working memory influence what we perceive when using new media.

•  Selec6ve aRen6on requires us to select those items which will enable us to succeed in our task from within an informa)on‐rich environment.

•  Divided aRen6on refers to the parallel or concurrent informa)on processing opera)ons mobilized simultaneously in order to accomplish the task.

Seven records of eye movements by the same subject. Each record lasted 3 minutes. 1)  Free examina)on. Before

subsequent recordings, the subject was asked to:

2)  es)mate the material circumstances of the family;

3)  give the ages of the people; 4)  surmise what the family had

been doing before the arrival of the "unexpected visitor;”

5)  remember the clothes worn by the people;

6)  remember the posi)on of the people and objects in the room;

7)  es)mate how long the "unexpected visitor" had been away from the family

Yarbus, A. L. (1967).Eye movements during percep)on of complex objects, in L. A. Riggs, ed., `Eye Movements and Vision', Plenum Press, New York, chapter VII, pp. 171‐196.

Working memory

•  Working memory is based on the classic defini)on of this memory structure (Baddeley, 1986; Paivio, 1986) which the authors believe to be: –  (a) composed of two different systems for short‐term informa6on processing (one for graphic informa)on and one for verbal informa)on) and an aRen6onal component which selects the informa6on to be processed,

–  (b) endowed with a limited capacity to process new, complex informa)on,

–  (c) capable of having parts of its capacity freed up by certain specific configura)ons of the mul)media presenta)on

Components of working memory

Informa)on processing theory explana)ons

– Cogni)ve theory of mul)media learning (Clark & Paivio, 1991; Mayer, 2001)

– Cogni)ve load theory (Sweller, 2003) – Model of text and graphics comprehension (1993; Schnotz & Bannert, 2003);

Clark & Paivio (1991): Dual‐coding theory

•  Informa)on is processed through one of two generally independent channels.

•  One channel processes verbal informa)on such as text or audio. The representa)ons of informa)on processed by this system are known as logogens.

•  The other channel processes nonverbal images such as illustra)ons and sounds in the environment. The representa)ons of informa)on processed by this system are known as imagens.

•  Referen6al processing is ac6va6on that goes across the two representa6onal systems. Informa)on processed through both channels has an addi6ve effect on recall.

Clark & Paivio (1991): Dual‐coding theory

Clark, J.M., & Paivio, A. (1991). Dual Coding Theory and Educa)on. Educa)onal Psychology Review, 3, 149‐210.

Cogni)ve theory of mul)media learning (R.Mayer)

•  Subjects learning in a mul)media se�ng mobilize three successive cogni6ve processes: – selec6on (permi�ng the construc)on of the text and graphic base),

– organiza6on (permi�ng the crea)on of models based on the textual and graphic material)

–  integra6on (permi�ng the establishment of connec)ons between the corresponding events in the models based on the verbal and graphic material).

How do these cogni)ve processes influence new media ac)vi)es such as remixing/percep)on of remixes or percep)on of aggregated and interac)ve new media?

Cogni)ve theory of mul)media learning (R.Mayer)

A benefit can be gained from the dual coding of the informa)on

Cogni)ve load (Sweller) •  Cogni)ve load is the total amount of cogni)ve ac)vity imposed on working memory at an instance in )me.

On the table is plate, in the leS of plate there is a cup. On the right of the plate are spoon and fork.

Chunking informa)on in working memory: Low and high interac)vity

Processing in working memory: unconscious and conscious

•  Joint ac)on of two dis)nct processes in informa)on processing (Sweller, 2003): – an automa6c process of "similarity processing" which permits the automa)c retrieval of informa)on from long‐term memory,

– and a controlled process for the processing of differences, the role of which is to process new or contradictory informa)on.

Cogni)ve cost of informa)on processing

•  Sweller (2003) believes that split aRen6on occurs when subjects have to process mul)ple sources of informa)on which they have to integrate at the mental level in order to infer the meaning from the presented material.

•  The cogni)ve load theory has been used to suggest that the processes of spliFng aRen6on and mentally integra6ng the material are cogni6vely costly

Principles derived from Cogni)ve theory of mul)media learning

•  The modality effect: elimina)ng the split ahen)on effect improves learning by reducing the working memory load (Jeung, Chandler & Sweller, 1997)

•  The redundancy effect: The addi)on of redundant informa)on impairs learning. The modality effect should be used only if it is necessary to process disparate informa)on in order to reduce working memory load.

•  The element interac6vity effect: the effects of modality and redundancy are observed if and only if the material for processing is cogni)vely demanding, if it possesses a large number of interac)ng elements.

Principles derived from Cogni)ve theory of mul)media learning

•  The isolated interac6ng elements effect: when a learner has to process a very complex situa)on, processing can be facilitated by ini)ally isola)ng the elements of which it consists of

•  The exper6se reversal effect: when dealing with advanced learners, almost all the effects described above cease to operate.

Does interface output influence how users interact with it?

•  When the system uses text or speech, users tend to use speech.

•  When the system uses graphics then users tend to use poin6ng gestures on the touch screen

•  It seems that each modality has a kind of influence power and when the modali6es are combined their influence powers are combined too.

hhp://www.limsi.fr/Individu/jps/research/daS/doc/09.A_INTERACT‐bellik.23august09.pdf

Does interface output influence how users interact with it?

hhp://www.limsi.fr/Individu/jps/research/daS/doc/09.A_INTERACT‐bellik.23august09.pdf

If the output is mixed media, How do users interact with the systems?

Essay ques)ons

•  Mixed media percep)on – how we may perceive one new media applica)on? – For this task choose one new media applica)on that is largely based on mixed media informa)on.

– How can you explain the effect of new media on people, using this new media applica)on as an example?

Education

Newmedia4