Imagineering Creative Technologies

http://www.infm.ulst.ac.uk/~paul/

Magee Campus

1

Imagineering Creative Technologies

Professor Paul Mc Kevitt

Intelligent Systems Research CentreSchool of Computing & Intelligent Systems

Faculty of Computing & EngineeringUniversity of Ulster, Magee

Derry/Londonderry Northern Ireland


Magee Campus

2

Background• Intelligent MultiMedia & MultiModal Systems• Integration of natural language, speech/audio,

graphics/VR, vision• Semantic representation, fusion & synchronisation• Affective computing• Integration & reuse of existing software (Thórisson, 2007)• Relations with cognitive science• Deployment on mobile devices

(Dell Axim, Smartphones, Nintendo DS)• Applications in arts: music, drama, dance, design,

computer games, film, TV/radio, art, storytelling


Magee Campus

3


Magee Campus

4


Magee Campus

5


Magee Campus

6


Magee Campus

7


Magee Campus

8


Magee Campus

9

CHAMELEON:a platform for developing intelligent multimedia applications

Tom Brøndsted, Paul Dalsgaard, Lars Bo Larsen, Mike Manthey, Paul Mc Kevitt(CPK, Aalborg University, Denmark)

• Speech input/output, image processing & laser

• IntelliMedia Workbench campus information system application

• Semantic representation with frames (Minsky, 1975)

• Blackboard for semantic representation

• Bielefeld DACS distributed communications platform (Fink et al., 1996)

• Implemented with C, C++, DACS, Infovox TTS, grapHvite

• Applications in multimodal systems development


Magee Campus

Architecture of CHAMELEON

10


Magee Campus

IntelliMedia Workbench

11


Magee Campus

CHAMELEON Semantic Representation

• Semantic Frames (Minsky, 1975)

12

Input frame:

[SPEECH-RECOGNISER UTTERANCE: (Point to Hanne's office) INTENTION: instruction! TIME:timestamp]

Output frame:

[LASER INTENTION: description (pointing) LOCATION: coordinates (5, 2) TIME: timestamp]

Integration frame:

[NLPINTENTION: description (pointing) LOCATION: office (tenant Hanne) (coordinates (5, 2)) UTTERANCE: (This is Hanne's office) TIME: timestamp]


Magee Campus

CONFUCIUS:automatic conversion of natural language to 3D animation

Minhua (Eunice) Ma, Paul Mc Kevitt

• Lexical Visual Semantic Representation (LVSR)Use of Jackendoff’s (1990) Lexical Conceptual Structure (LCS)

& Schank’s (1977) scripts

• Use of Fellbaum’s (1998) semantic classes in WordNet

• Implementation with Java, VRML, WordNet, Connexor parser, FreeTTS (Java)

• Applications in education, computer games, animation production,

digital storytelling

13


Magee Campus

Architecture of CONFUCIUS

14


Magee Campus

MultiModal semantic representation

15


Magee Campus

CONFUCIUS examples

16

• Contact verb example (e.g. hit, collide, scratch, touch)

“The car collided with a wall.” Collision detection Nonspeech sound effects No human role involved

• Human action verb examples“Ipke gave Brigitte a loaf of bread.”

“Bernie put a cup of coffee on the table.” H-Anim Site node locative tags of object (on tag for table object)

“Paul pushed the door.”

“Gert ate the bread.”


Magee Campus

17

“Paul sat on the chair.”

“Teri was waving goodbye.”

“Helge went to the gym.”

“Gert put on his hat.”

States“Anke is sad.”

“Anke is ashamed.”


Magee Campus

18


Magee Campus

SceneMaker:automatic visualisation of screenplays Eva Hanser, Joan Condell, Tom Lunney, Paul Mc Kevitt

• Automatically generates affective virtual scenes from screenplays/play scripts

• Realistic visualisation of emotional aspects

• Extraction of genre/moods from screenplays

• Use of OCC (Ortony-Clore-Collins) affective model,ConceptNet, Open Mind Common Sense (OMCS), WordNet-Affect

• Enhance believability of virtual actors and scene presentation

• Multimodal representation with 3D animation, speech, audio & cinematography

• Applications in assisting directors, actors, drama students, scriptwriters

19


Magee Campus

Architecture of SceneMaker

20


Magee Campus

MemoryLane:a mobile digital storytelling companion

Sheila Mc Carthy, Heather Sayers, Mike McTear, Paul Mc Kevitt

• Multimodal stories (speech, text, video) from life-cached memory data (poems, songs, photos, home videos)

• Extensive HCI requirements analysis with focus on older people• Multimodal preferences and abilities of users determines

story modalities and user interface configuration

• Prudent on sensitive life-cached data

• Creates new stories on the fly

• Applications in medical informatics and digital care

• Implemented with Microsoft Visual Studio

• Deployed on Dell Axim X51v PDA

21


Magee Campus

Architecture of MemoryLane

22


Magee Campus

MemoryLane Prototype

23


Magee Campus

AmbiLearn:an ambient intelligent multimodal learning environment for children

Jennifer Hyndman, Tom Lunney, Paul Mc Kevitt

• Educational games & virtual learning environments

• Child - Computer Interaction (ChiCI)

• AmbiLearn architecture & TreasureLearn application

• Implementation with DevKit Pro (Nintendo homebrew)

• Deployment on mobile device (Nintendo DS)

• Application to collaborative learning for children

24


Magee Campus

Architecture of AmbiLearn

25


Magee Campus

PlayPhysics:an emotional games learning environment for teaching Physics

Karla Munoz Esquivel, Tom Lunney, Paul Mc Kevitt

• Virtual Learning Environment (VLE) for teaching Physics

• Incorporating serious games & intelligent tutoring systems (ITS)

• Modelling of emotions & personality of student

• Use of Bayesian Networks for modelling student knowledge

• Affective feedback through audio-visual effects

• On-line web application with Student, Lecturer, Head of Department, Administrator views

• Implemented with Elvira Bayesian Network tool, Flash, Java, Tomcat Apache Server, MySQL database, ConceptNet, 3D Studio Max & Maya

• Applications in educational software & computer games

26


Magee Campus

Olympia architecture

27


Magee Campus

PlayPhysics interface (astronaut game)

28


Magee Campus

TeleMorph & TeleTuras:bandwidth determined mobile multimodal presentation

Tony Solon, Kevin Curran, Paul Mc Kevitt

• Multimedia limitations of mobile networks, mobile device capabilities

• User preferences on modalities

• Transmoding between modalities with cross-modality adaptation

• Use of Fuzzy (logic) Inference system (FIS)

• TeleMorph platform & TeleTuras tourist application

• Implemented with MATLAB

• Applications in streaming media, computer networks, multimodal presentation

29


Magee Campus

30

Architecture of TeleMorph Fuzzy Inference System (FIS)

hi veryHiaboveAvgavgbelowAvglo veryLonull


aboveAvgavgbelowAvgveryLonull lo< Avg > Avg

hi veryHi

Packet Loss

bpsCurr,vCurr, vLowest, vHighest

aiCurr, aiLowest, aiHighestitCurr, itLowest, itHighestaCurr, aLowest, aHighesttCurr, tLowest, tHighest

18 Inputs

pktLoss

adaptFreq







IfAndThenRule 1

Rule7464

IfAndThen




Aggregation

Defuzzification

Implication – Apply 7,464 Rules

AdaptFreq

5 outputs

v - video qualityai - audio & image qualityit - image & text qualitya - audio qualityt - text quality

bpsCurr, vCurr, etc...


Magee Campus

31

TeleTuras GUI


Magee Campus

32

Available media qualities in 6 sample test scenarios


Magee Campus

Adaptation trace data from test scenario 6

33


Magee Campus

MediaHub:Bayesian decision-making in an intelligent multimedia distributed platform hubGlenn Campbell, Tom Lunney, Aiden McCaughey, Paul Mc Kevitt

• Interpret/generate semantic representations of multimodal input/output

• Fusion & synchronisation of multimodal data

• Decision-making with Bayesian Causal Probabilistic Networks (CPNs)

• Implemented with Psyclone platform, HuginLite Bayesian modelling tool, Java, NetBeans IDE

• Applications in decision-making for key problems in multimodal systems: anaphora resolution, domain knowledge awareness, multimodal presentation, turn-taking, dialogue-act recognition & parametric learning

34


Magee Campus

Architecture of MediaHub

35


Magee Campus

Decision-making layers

36


Magee Campus

psyProbe view of MediaHub Whiteboard

37


Magee Campus

Alternative `Turn-taking' Bayesian network in Hugin

38


Magee Campus

SoFI:SOng Form Intelligence for repairing streaming musicacross wireless bursty networks

Jonathan Doherty, Kevin Curran, Paul Mc Kevitt

• Large packet loss dropouts of 5 - 15 s.• Use audio pattern recognition & matching to repair gaps• K-means clustering classifier determines self-similarity matches• Match current sections of song received with previous

sections played• Typicality of Western Tonal Format (WTF)• Use of MPEG-7 Audio Spectrum Envelope (ASE) multimedia

content description standard giving features for extraction• Implemented with G-streamer for steaming media, Ices2, Icecast2

for audio broadcast & MPEG-7 Audio Spectrum Envelope (ASE)• Applications in repair of streaming music

39


Magee Campus

Architecture of SoFI

40


Magee Campus

Spectrogram representation

41


Magee Campus

42


Magee Campus

43


Magee Campus

Steganoflage:strengthening steganography in digital imagesAbbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt

• Hiding/embedding data in transmission medium• Use of 1D hash algorIthm coupled with

2D iFFT (irreversible Fast Fourier Transform)• Encryption of digital documents in 2D spatial domain• Use of steganography to add security layer resistant to

noise & JPEG compression• Exploitation of Jarvis kernel (Jarvis et al., 1976) to generate error

diffusion signal• Use of Wavelet-based Inverse Halftoning via De-convolution

(WInHD) to recover approximate original signal• Implemented with MATLAB & PHP• Applications in security, forensics, forgery, digital watermarking

44


Magee Campus

Architecture of Steganoflage

45


Magee Campus

Securing confidential data

46

The traditional method of securing individuals’ data with the notion of security by obscurity that Steganography introduces

App

licat

ion


Magee Campus

National & international collaboration

• InstitutoTecnológico de Estudios Superiores de Monterrey, Campus Ciudad de México (ITESM-CCM), Mexico

• Aalborg University, Denmark• CADIA, Reykjavik University, Iceland• University of Bielefeld, Germany• LIMSI (CNRS), Orsay, Paris, France• Sheffield University, England• Stanford University, California, USA

47


Magee Campus

48


Magee Campus

49


Magee Campus

50

Noel Sharkey thinks AI is an illusion


Magee Campus

51


Magee Campus

Intellectural Property (IP)Steganoflage:

United Kingdom Patent Application [No. 0819982.0]:“Method for Skin Tone Detection”,University of Ulster,Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt.

SoFI:United Kingdom Patent Application [No. 0908879.0]: “A system and method for streaming music repair and error concealment”,University of Ulster,Jonathan Doherty, Kevin Curran, Paul Mc Kevitt.

52


Magee Campus

Industrial collaborationInterTradeIreland Fusion project, 2008/10:Nvolve Ltd., Letterkenny, Co. Donegal, Ireland, Michael McNeill, School of Computing & Information Engineering, Coleraine; Prof. Paul Mc Kevitt.

Enterprise Ireland Innovation Voucher, IV-2008 1047, 2008/09:``Navigatour'',Backseat Tour Guides Ltd., Letterkenny, Co. Donegal, Ireland,Anton Hutton, School of Creative Arts & EyeSpyFX; Prof. Paul Mc Kevitt.

53


Magee Campus

University of Ulster, Office of Innovation, Proof-of-Principle (PoP), 2008/09:``Identity cards employing steganography``,Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt.

University of Ulster, Office of Innovation, Proof-of-Principle (PoP), 2009:``System and method for streaming music repair and error concealment``,Jonathan Doherty, Kevin Curran, Paul Mc Kevitt.

Invest NI Proof-of-Concept (PoC) project, 2009/10:``Secure digital watermarks (SDW):secure data hiding in image frames using steganography``,Joan Condell, Kevin Curran, Paul Mc Kevitt.

54


Magee Campus

Creative People

55

• Abbas Cheddad• Conrad Deighan• Jonathan Doherty• Eoghan Furey• Eva Hanser• Jennifer Hyndman• Sheila McCarthy• Karla Munoz-Esquivel• Dr. Glenn Campbell (Nvolve Ltd., Letterkenny)• Dr. Minhua (Eunice) Ma

(University of Derby)• Dr. Tony Solon (Bath)

• Dr. Joan Condell• Dr. Kevin Curran• Anton Hutton

(Creative Arts & EyeSpyFX)• Dr. Tom Lunney• Aiden McCaughey• Prof. Paul Mc Kevitt• Dr. Michael McNeill

(Coleraine)• Prof. Mike McTear

(Jordanstown)• Dr. Heather Sayers


Magee Campus

Conclusion & future work• Imagineering creative technologies

• MultiModal semantic representation

• Affective computing

• Deployment on mobile devices

• Applications in creative industries

56

Documents

Imagineering Creative Technologies