1. Minimal Displaysor Breaking the Pixel SpellIphone andBlack Suprematic Square(Kazimir Malevich 1915)

2. Holy Grail: non-flat & super-resolution screens?Cave painting (Lascaux, la salle des Tauraux)not new!Why dont we call painting, graffiti,make-up, pre-operative marking, oreven dressing spatial augmentedreality?Lack interesting attributes of the digital realm!Content update / copying / deleting / backup / undo 3. screen (n.)A covered framework, partition, or curtain, either movable orfixed, which serves to protect from the heat of the sun or of a fire,from rain, wind, or cold, [] conceal, shut off the view, orsecure privacy; as, a fire-screen; []; as, a screen upon whichimages may be cast by a magic lantern; in general, and shelter ormeans of concealment.Definition of 'screen', The Century Dictionary and Cyclopedia, 1911 (1889), as cited inErkii Huhtamo, Elements of Screenology: towards an Archaeology of the Screen Canadian Avant-Garde Film in the 1990s by Gerald Saul 4. display (v.)late 13c., "unfurl" (a banner, etc.), from OldFrench desploiir (Modern French dployer) "unfold, unfasten,spread out" (of knots, sealed letters, etc.), from Latin displicare "toscatter," from dis- "un-, apart" (dis-) + plicare "to fold" (ply ). 5. Origins of the frame: portability? writing? Portability demand a frame? Writing content is digital :coordinate system (matrix, grid) 6. Why images would need a frame? delimits / separate semantic and functional spaces. Legibility. An information tile to build complex information surfaces (rearranged, removed)This is a first MOVE TOWARDS the DIGITAL REALM(not at the level of the pixel though)La Tribune des Officesde Johann Zoffany,1772-1778, RoyalCollection, Windsor. 7. Where we are now: a window within a window within a window Stuck in this paradigm ! Smartphones/glass = modern peeping box? Position of information tiles on real space is not yetrelevant enoughOuroboros[2008]boxedEgo [2006]Can we give meaning to information by properly placing itIN THE WORLD (FRAME=WORLD?) 8. What have we been creating and what we really want? Obsession with RECTANGULAR screens has not augmented reality but perverted it! Position on space not (yet) relevant to content Shape not relevant anymore (what happened to skeumorphs?)A clear technological challenge vs. a clear purpose?.will not bring about the ultimate display - in Sutherland sense!?? 9. tricks attempting to integrate the frame in the real world Art / Media Arts Ubiquitous displays and projection mappingMicrosoft IllumiRoomPere Borrell del Caso, Escaping Criticism,1874, oil on canvasPablo Valbuena, augmented sculpture 10. Ouroboros [2008]a shared virtual space, a world-scale tunnel builtby chaining video-conferencing cameras andprojectors in a closed loop around the world. 11. interaction with the content is key interaction needs to happen INSIDE the frame (screen as interface) virtual world CO-LOCATED with the real (same coordinates system)VideoDromescreen as interfaces or portals 12. KHRONOS PROJECTOR [2005] Deformable screen / fixed in space Screen as a membrane separating worlds Screen as a controller 13. KHRONOS PROJECTOR (media arts), video excerpt Fixed screen / deformable (passive haptic feedback) Screen as a membrane between real and virtual 14. Precise control possible [2008]Note: a physical attribute makes manipulation more precise, even if this isPASSIVE FORCE FEEDBACK 15. Volumetric data visualization & interaction [2006] Rigid screen / moving on space (proprioception): position on space relevant Screen as a controller, the locus of interaction 16. Technical details: pose estimation markers- retro-reflective paper to reflect IR light- used both for pose estimation and control (drag, slice, zoom...)- Off-the-shelf, camera-projector setup 17. Application to Neurosciences: twitterBrain [2013~] Common virtual support to query academic publications Real time communication between researchers Proprioception, spatial memory, procedural memoryMultimodal virtualpresence of dataExtension: portable memoryblock to store personal data(music, books, etc)Room, public space as virtualbookshelfA shared physical database 18. Spatialized database for neurosciences Spatialized academic database Spatialized social network big data visualization techniquesAugmented Memories [by J. Puig, 2012] 19. Parallax Augmented Desktop [2007] display behaves as a 3d volume (proper sensory motor patterns arerecreated by the computer) Again, continuity of real and virtual space 20. Tele-Ping Pong [2005] An attempt to use high speed robotics tosimulate a worm-wholefailed!Too much delay (we are not yet there) 21. Ok, ok, but what about pixels?The Connection MachineBurroughs B205 in Batman - The Movie (1966)Obviously, logical pixels can be represented by on/off lights...Is this the origin of the pixelated display?Blinkenlights paneland hardware registers 22. pixel (n)First published in 1965 to describe the picture elementsof video images from space probes to the Moon and Mars.Georges Seurat, detail (La parade, 1889):chromo-luminarisme or pointillisme Psychophysics Well posedtechnologicalchallenge!A one size fits all kind of solution 23. BEYOND PIXELSMINIMAL DISPLAYS 24. minimal = ambient, alsoiconographicWORKSHOP at CHI 2013:Steimle J., Benko H., Cassinelli A., Ishii H., Leithinger D., Maes P., Poupyrev I.: Displays Take New Shape: An Agenda for FutureInteractive Surfaces. CHI13 Extended Abstracts on Human Factors in Computing, ACM Press, 2013. 25. Laser Aura: externalizing emotions [2011]minimal display inspired by manga graphical conventions 26. minimal displays by no means imply small:Laserinne [2009~] 27. Chromatographic display [2014~]A rare exception:CHROMATOGRAPHIC WRITINGSYSTEM (Benin and Edo people ofsouthern Nigeria have )All writing systems consist on SHAPES?Why COLORS have not being used more extensively for writing?Nineteen Eighty-Four (enciphered in colour), byHyo Myoung Kim Technically difficult.. But not today! Perfect minimal/ambient display, and extremely wearable Perfect to represent emotions? 28. Colors & instant qualia Towards a more complex color language? Temporal or spatial arrangement relevant Continuum spectrum, but can be discretized and ordered Color and sound association may be idiosyncratic, but canbe powerful (subitization) Synesthesia 29. Face and colors: interactive makeup [2013~] inter-personal and minimal display Enhanced emotional communication?Color language? 30. How to make AugmentedReality REALZero-delay / Zero-mismatchMultimodal Spatial Augmented Reality& Sense of PRESENCEwithMinimal displays? 31. Generating a sense of co-located presence for data Now: projection mapping (image) Soon: sound, vibration, temperature Required: zero-delay, zero-mismatch interaction Perhaps photorealistic rendering not required 32. Haptic interactionwith virtual 3dobjectsembedded inthe real worldforcefieldVirtual Haptic Radar [2008]- Importance of multi-modal immersion (or partial immersion) - 33. Haptic Radar for extended spatial awareness [2006]In collaboration with Professor Eliana Sampaio (CNAM, Paris) Qualitative results were extraordinary (semi-structured interviews & ANOVAanalysis of anxiety trait/state).Presently: Quantitative measures (using a simulator calibrated magnetic compass, andvirtual reality environment) Production? 34. Haptic Radar Games / Simulator: minimal tactile cues [2013~](Simulator using tilted compensated magnetic compass) Accot-Zhai steering law for constrained motion on a corridor orcars on a road, when only local information is known. Games for the blind / haptic feedback for video games 35. Multimodal interfaceLocation-based social network(virtual public spaces) 36. The importance of real time in HCI 1000fps 30ms UI 30fps 200msComputeror Game 37. High Speed Gesture UI for 3d Display (zSpace) [2013]- Importance of instant feedback to make Augmented Reality REAL!! 38. OmniTouch (Microsoft research)Smart Laser Sensing [2003~]What? no delay, no misalignment projection on mobile, deformable surfaces Real time sensing Smart sensing Laser Sensing DisplaySmart sensingvs.Vision basedHow?Skin Games 39. Low level image processing by intelligent lightingubiquitous display Artificial fluorescence,AR surveying (distances,angles, depth...)medical imaging (IR,polarization...)image enhancement(contrast compensation,color..). 40. Markerless laser tracking (I/O interface) [2003]Camera-lessactive trackingprinciple2004 2004 41. scoreLight: a human sized pick-up head [2009] artificial synesthesia real-time interaction new interfaces for musical expression(in collaboration with Daito Manabe) 42. scoreLight [2010]- zero-delay/mismatch interaction generate a sense of intentionality - 43. Skin GamesBody as a controller (kinect) & body as a displaysurface 44. Spatial AR surveying907055250240(AR for interior design)(Laser level)Smart Laser LevelAR surveying [2013~] 45. Interaction possible 46. Laser GUI: minimal interface [2013~]Cameraless interactive display (no calibration needed) 47. Text projection-mapping at around 200 fps! 48. Laser Web Cam (minimal display!) Semi- automatic detection of projection surfaces.Per-pixel sensing What was required: calibration of LASER projector and camera. 49. Rethinking the metaphor ofthe windowThe Physical CloudReal objects as handles or data containersMulti-modal ARFUNCTION PROJECTOR 50. Physical Cloud?(b) Data on Objects(objects as intuitive handles or icons : beyond PUI)(a) Data on 3D spaceExperiments on psychology space perception andorganization The body as a bookshelfor body mnemonics(*) Shared Memory Palace(interpersonal spatializeddatabase) The city, public spaces,etc. as a 3d bookshelfExperiments onpsychology background picture asorganizationalscaffold for files andfolders (Takashita-kun) Association byperceived affordances(Invoked computing)How to interact with justcoordinates in space? Propioceptive interfaces(VSD, Krhonos, Virtual HR) Deformable workspacemetahor Spidar Screen Virtual BookShelf(Takashita-san) Real time projectionmapping twitterBrain (Philippe, Jordi)How to interact withobjects that do nothave I/O interfaces? Invoked Computing /function projector Objects with memory ofaction (thermal camera) On the flight I/O interfaces(LSD technology)Research ConceptEnabling Technology(*) Jussi Angesleva, 2004Memory Blocks (2011~)Laser Sensing Display(create the sense of presence / mixed reality) 51. Invoked Computing (Function Projector) [2011]Towards afunctionprojectorMultimodal Spatial Augmented Reality (MSAR)Towards a Function Projector capable of projecting affordances! 52. Invoked Computing (Function Projector) [2011]Augmented Reality as the graphic front-end of Ubiquity.And Ubiquity as the killer-app of Sustainability.Bruce Sterling (Wired Blog on Invoked Computing) 53. Physical affordances as services?rechargeablehammererasablebook...utopian or dystopian future? 54. How? Visual and Tactile + high speed interaction [2012] 55. Saccade Display + laser sensing [2012~]Real-world special effectsReal world shader?example of a contextaware displayWhy laser? Stronger persistence of vision effect Very long distances! (on a car, etc)[+DIC] 56. Light pet [2014~] Can sense, measure Draw, print Signal (danger), indicate Play Robot made of light Extension of the self Minimal interactive display 57. 3D slow serial display / printer [2014~]2D pneumaticbasedprototype3d fractalmatrix.. Another option: manipulating matter (this is along the lines of radical atoms) 58. Conclusion The vision: Physical Cloud (computing) Real space as an opportunity to organize data (vs. the cloud) Real objects as handles to trigger computing functions Enabling technology? Minimal, context aware displays (less is more!) Engineered Intuitive physics (generalization of tangible interfaces) Intentional stance (live agents instead of control knobs) Real time, zero-delay, zero-mismatch Spatial AR will produce a paradigm shiftmaking the digital analog again Possible instantiation: the light-pet vs. pixelated screens an alternative vision to the pixelated screen (including HMD) avatar robot made of light communication enhancement (laser aura) enhanced spatial awareness (signaling, etc) 59. Thanks!For more:www.alvarocassinelli.com