Digital Cinema and New Media Artsat Calit2

KeynoteRetreat of the Interdisciplinary Film and Digital Media Program

University of New MexicoAlbuquerque, NM

May 1, 2008

Dr. Larry SmarrDirector, California Institute for Telecommunications and

Information TechnologyHarry E. Gruber Professor,

Dept. of Computer Science and EngineeringJacobs School of Engineering, UCSD

California’s Institutes for Science and Innovation A Bold Experiment in Collaborative Research

UCSBUCLA

California NanoSystems Institute

UCSF UCB

California Institute for Bioengineering, Biotechnology,

and Quantitative Biomedical Research

UCIUCSD

California Institute for Telecommunications andInformation Technology

Center for Information Technology Research

in the Interest of Society

UCSC

UCDUCM

www.ucop.edu/california-institutes

Calit2--A Systems Approach to the Future of the Internet and its Transformation of Our Society

Calit2 Has Assembled a Complex Social Network of Over 350 UC San Diego & UC Irvine Faculty

From Two Dozen DepartmentsWorking in Multidisciplinary Teams

With Staff, Students, Industry, and the Community

Integrating Technology Consumers and ProducersInto “Living Laboratories”

Over 300 Grants and 200 Companies

Calit2 Has Facilitated Deep Interactions With the Digital Arts on Both Campuses

“Researchers Look to Create a Synthesis of Art and Science

for the 21st Century”

By John MarkoffNYTimes November 5, 2005

Ruth West, UCSD “Ecce Homology”

Bill Tomlinson, Lynn Carpenter UCI “EcoRaft”

SPECFLIC 1.0 – A Speculative Distributed Social Cinema by Adrienne Jenik

Two New Calit2 Buildings Provide Laboratories for “Living in the Future”

• “Convergence” Laboratory Facilities– Nanotech, BioMEMS, Chips, Radio, Photonics– Virtual Reality, Digital Cinema, HDTV, Gaming

• Over 1000 Researchers in Two Buildings– Linked via Dedicated Optical Networks

UC Irvinewww.calit2.net

Preparing for a World in Which Distance is Eliminated…

$100M From State for New Facilities

Calit2 is Experimenting with Open Reconfigurable Work Spaces to Enhance Collaboration

Photos by John Durant; Barbara Haynor, Calit2

The Calit2@UCSD Building is Designed for Prototyping Extremely High Bandwidth Applications

1.8 Million Feet of Cat6 Ethernet Cabling

150 Fiber Strands to Building;Experimental Roof Radio Antenna Farm

Ubiquitous WiFiPhoto: Tim Beach, Calit2

Over 10,000 Individual

1 GbpsDrops in the

Building~10G per Person

UCSD Campus has one 10G

CENIC Connection for ~30,000 Users

24 Fiber Pairs

to Each Lab

Calit2 Was Designed with Advanced Multimedia Facilities

High Definition Studio

• Audio Spatialization• Motion Capture• Gaming Lab

Autostereo4K VTC

3D TV

4K on OptIPortal

VirtuLab

Center for Research on Computing and the Arts

StarCAVE AnalysisAnd Calibration

Meyer Sound - Calit2 Partnership:StarCAVE Audio Hardware, Acoustical Analysis,

State-of-the-Art Sound for Cinema and Scientific Visualization Syst

Source: Peter Otto, Calit2, CRCA, UCSD Dept of Music

Virtual Audio Environment Research:A General Model for 3D Spatial Processing of Sound Sources

• Objectives– Accurate and Convincing Sound Source

Localization using Loudspeakers– Acoustical Space Modeling

• Two Nested rooms – Listening Room as the Inner Room– Virtual Acoustic Space as the Outer Room– Speakers as Openings on the Perimeter of the Inner Room

• Processing Per Audio Sample ~44,100 times a second– 2D spatial Processing/Sample (4 Speakers Each Order)

– 1 Direct, 4 First order Reflection, 12 Second Order– 3D spatial Processing/Sample (8 Speakers Each Order)

– 1 Direct, 6 First Order Reflection, 30 Second Order

3 meters

12 meters

Shahrokh Yadegari, UCSD Dept. of Theatre and Dance, CRCA

Sculptures created from a variety of computer controlled modeling and fabrication processes. They each begin with the same seed of 3D object data that is transformed by a variety of algorithmic and modeling manipulations. The resulting sculptures are the intersection between material properties, object data space and constructive processes.

Calit2 Has a Variety of Computationally Controlled Sculpture Machines

Calit2 New Media Arts Gallery:A Space for Interactive Exhibits

http://gallery.calit2.net

Calit2@UCSD Designed Three ExperimentalNew Media Arts Spaces into Atkinson Hall

New Media Arts Wing

Digital Auditorium:Building a Global Collaboratorium

Sony 4K Digital Cinema Projector

24 Channel Digital Sound

Gigabit/sec Each Seat

Calit2@UCSD Auditorium Has 200 Seats

September 26-30, 2005Calit2 @ University of California, San Diego

California Institute for Telecommunications and Information Technology

Borderless CollaborationBetween Global University Research Centers at 10Gbps

iGrid 2005THE GLOBAL LAMBDA INTEGRATED FACILITY

Maxine Brown, Tom DeFanti, Co-Chairs

www.igrid2005.org

100Gb of Bandwidth into the Calit2@UCSD BuildingMore than 150Gb GLIF Transoceanic Bandwidth!450 Attendees, 130 Participating Organizations

20 Countries Driving 49 Demonstrations1- or 10- Gbps Per Demo

First Trans-Pacific Super High Definition Telepresence Meeting Using Digital Cinema 4K Streams

Keio University President Anzai

UCSD Chancellor Fox

Lays Technical Basis for

Global Digital Cinema

Sony NTT SGI

Streaming 4K with JPEG 2000 Compression ½ gigabit/sec

100 Times the Resolution

of YouTube!

Calit2@UCSD Auditorium

Calit2@UCSD Multipurpose Room—A “Black Box” Theatre Space

• 100-seat Reconfigurable Space• Performances/Seminars

Pulitzer Prize-WinningRoger Reynolds:

Calit2Composer-in-Residence

Sanctuary Project

Source: Peter Otto, Calit2, CRCA, UCSD Dept of Music

The Salk Institute in La Jolla, CAon October 11, 2008

Filling the 30’ x 30’ “Hole” in the Calit2 BuildingWith Next Generation Virtual Reality

• Varrier Autostereo Virtual Reality– High Res Stereo Without Glasses!– 65 High Resolution LCD Tiles– 45 Mpixels/eye of Visual Stereo

• StarCAVE– 32 HD Projectors!– 15 Meyer Sound Speakers+Subwoofer

OptIPortals: Toward Gigapixel Tiled Display Walls Providing Scalable Digital Interactive Pixel Spaces

200 Megapixel Calit2@UCI Apple Tiled Display WallDriven by 25 Dual-Processor G5s

50 Apple 30” Cinema Displays

Source: Falko Kuester, Steve Jenks, Calit2@UCINSF Infrastructure Grant

Data—One Foot Resolution USGS Images of La Jolla, CA

HDTV

Digital CamerasDigital Cinema

www.ferragallo.com/digitalpaintings.html

High Resolution Digital Media for Art Creation:Tree of Life by Roger Ferragallo (546 Megapixel)

Source: Falko Kuester, HiPerSpace, Calit2@UCSD

UCSD is Pioneering the Fusion of Engineering with the Arts

Multi-Spectral Imaging Diagnostics:Discovering the “Painting Behind the Painting”

Source: Maurizio Seracini, Director CISA3

Digital Clinical Charts –Collaboration with the San Diego Museum of Art

Source: Maurizio Seracini, Director CISA3

Cultural Heritage Virtual Reality:Photo Texture Mapping onto High Resolution Laser 3D Scanning

Leica Scanstation2 3D Laser Scanner Set up for Scanning Inside Florence's

Palazzo Medici

In This Image you are Looking at the Michelozzo Courtyard (the Doorway in the Very Back of the Image Leads into the Garden).

This Image Is 6392 Pixels Wide X 4192 High.

Source: Calit2 Team Led by Maurizio Seracini, Director CISA3

Exploring Cultural Heritage Virtual Realities Using Large Scale OptIPortals

A Real-Time Rendered 3D Point Cloud Model (1 Billion Points) of Palazzo Medici

Being Navigated via a Wireless Sony Game Controller

Falko Kuester and Maurizio Seracini,

CISA3

The New Field of Scalable Cultural Analytics:Finding Patterns in Massive Amounts of Cultural Data

Software Studies Initiative, Calti2@UCSD

Interface Designs for Cultural Analytics

Research Environment

Jeremy Douglass (top) & Lev Manovich

(bottom)

Second Annual Meeting of the

Humanities, Arts, Science, and Technology Advanced

Collaboratory(HASTAC II)

UC Irvine May 23, 2008

Calit2@UCI200 MpixelHIPerWall

Lev Manovich on High PerformanceCultural Analytics

www.youtube.com/watch?v=xtbzVuDqSas

Digital Analysis of an Artist’s Lifetime Production151 Paintings by Rothko

Source: Software Studies Initiative

Software Studies InitiativeFilmHistory.viz

1100 Films in Cinemetrics Database

Source: Jeremy Douglass, Lev Manovich, Calit2, UCSD

CineGrid 4K Digital Cinema Projects: “Learning by Doing”

CineGrid @ iGrid 2005 CineGrid @ AES 2006

CineGrid @ GLIF 2007CineGrid @ Holland Festival 2007

CineGrid Founding Members• Cisco Systems• Keio University DMC• Lucasfilm Ltd. • NTT Network Innovation Laboratories • Pacific Interface Inc.• Ryerson University/Rogers Communications Centre• San Francisco State University/INGI• Sony Electronics America • University of Amsterdam • University of California San Diego/Calit2/CRCA• University of Illinois Chicago/EVL • University of Illinois at Urbana-Champaign/NCSA• University of Southern California/School of Cinematic Arts• University of Washington/Research Channel

The Founding Members of CineGrid are an extraordinary mix of media arts schools, research universities, and scientific laboratories

connected by 1GE and 10GE networks used for research & education

Rogers Communications CentreRyerson University’s Rogers Communications Centre

Linking to CA*net4 and CineGrid• In the Heart of Toronto - Canada’s Largest Media Centre

– Creating Digital Cinema/Visualization Lab– School of Image Arts and School of Radio and Television Arts

– 1300 Undergraduate Students

Connection to Calit2 Achieved Dec 18, 2006!

Calit2 and the Venter Institute Test CineGrid™ with HDTV Movie by Independent Film Director John Carter

Coast-to-Coast Screening of New HD Movie—Live Demonstration of 21st Century

Entertainment Delivery (June 14, 2006)

JCVI

Sony HDTV JH-3

Rockville, MD

Calit2 Auditorium

StarLight

Chicago

The CineGrid Node at Keio University, Tokyo Japan

SXRD-105 4K Projector

Imagica 4K Film Scanner

Sony 4K Projectors Olympus4K Cameras

NTT JPEG2000 Codec

CineGrid @ iGrid2005: Six Hours of 4K Projected in Calit2 Auditorium

4K Scientific Visualization

4K Digital Cinema

4K Distance Learning

4K Anime

4K Virtual Reality

Source: Laurin Herr

Audio Engineering Society (AES)/LucasFilm Trans-Pacific CineGrid 4K Demonstration, October 8, 2006

Keio/DMC Tokyo

CineGrid International

Networks

LucasFilmTheater

San Francisco

UCSD USC

SyncNTT JPEG2000 Servers

Sony 4K

Audio

CineGrid CaliforniaNetworks

Audio Server

Mixer

Sync

DVTS Sony DV

NTT JPEG2000

CODECand Server

Olympus 4KCamera

4k Video (500mbps Streams)

Over 3 L2 GE VLANs Plus 24 Channel Audio

Over Another GE Source: Peter Otto, Calit2

4K Film Shoot in the Canals of Amsterdam,

CineGridHolland Festival 2007

Swimming Fiber the Last 500m to the Muziekgebouw: CineGrid @ Holland Festival 2007

CineGrid @ Holland Festival 2007As Seen in the Calit2@UCSD Auditorium

E ra la Notte, J une 2007 (Live!)

CWave core PoP

10GE waves on NLR and CENIC (LA to SD)

Equinix818 W. 7th St.Los Angeles

PacificWave1000 Denny Way(Westin Bldg.)Seattle

Level31360 Kifer Rd.Sunnyvale

StarLightNorthwestern UnivChicago

Calit2San Diego

McLean

CENIC Wave Cisco Has Built 10 GigE Waves on CENIC, PW, & NLR and Installed Large 6506 Switches for

Access Points in San Diego, Los Angeles, Sunnyvale, Seattle, Chicago and McLean

for CineGrid MembersSome of These Points are also GLIF GOLEs

Source: John (JJ) Jamison, Cisco

Cisco CWave for CineGrid: A New Cyberinfrastructurefor High Resolution Media Streaming*

May 2007*

2007

Spinoff:Lucasfilm’s Skywalker Sound Internships

For UCSD Undergraduates

Source: Peter Otto, Calit2, CRCA, UCSD Dept of Music

Hybrid Computational Infrastructure for Multi-User Virtual Environments, While Deriving 4K Digital Cinema

• Assess Impact on Understanding of the Social-Cultural Milieu• Utilizing Emerging Technologies:

– Intel Larrabee Multi-Core CPU, – IBM/Sony Cell processors– IBM Networked Mainframe

• Three Key Project Areas:– Future Cinema– Multi-User, Extensible Virtual Worlds– Assets, Dynamics and Behavior Computation for Virtual Worlds and Games

Center for Next Generation Digital MediaSheldon Brown, Director, Also Director CRCA, Prof. Dept of Visual Arts

$2.4 Million Gift from IBM, $300K from Intel, Prototype 4K computing from Sony, National Science Foundation Node for Center for Hybrid Multicore ResearcH

Ogre3D Scene graph

Open Source Libraries – Needs Work for Adding Data Level Parallelism

The Scalable City Next Stage Technology Infrastructure

Abstract Physics –Use Multiple Physics Libraries

(ODE, Bullet, etc.) Replace Computational

Bottlenecks in these Libraries with Data Parallel Operations

CGAL Computational Geometry Library

Intel OpenCV Real-Time Computer Vision

Fmod Sound Library

Cell Processors Compute

Dynamic Assets

Input Data

Output Data

Data Paralleln threads + SIMD

Thread Barrier

ERSATZ ENGINE

Input Data

Output Data

Convert Assets to Data Parallel Meshes After Physics Transformation, Boosts Rendering ~33%

Source: Sheldon Brown,

CRCA, Calit2Dept of Visual Arts

4K/Stereo High Resolution Cinema Development with Same Asset Pipeline as Game Environment

Source: Sheldon Brown, CRCA, Calit2, Dept. of Visual Arts

Calit2@UCI Game Culture & Technology Lab

• Unique Partnership with Discovery Science Center and Santa Ana Unified School District

• Complementary K-8th Grade Science and Learning Games in Line with CA Teaching Standards

• Developed a Gaming Undergraduate Degree Concentration – Most Sought After Minor at UCI

• Offering “Joystick Corridor” Internships and Highly Developed Workforce

• International Gaming Research Partnership Developed with Daegu City, Korea

EcoRaft-Enabling Social Change

Bill Tomlinson, Interactive Animation Lab, Calit2@UCI

Mixed Reality Games - Informal Science Education for K-6 Students and Families

eMedia Studio: Interactive TelepresenceDance/Media Performances

Entangled Photons(November 2004)

• Calit2 Irvine building opening• Five dancers in two sites; interactive visuals• Inspired by quantum entanglement concept, Einstein’s

"spooky action at a distance"

ÖÖTÖÖ(June 2006)

• eMedia Studio at UC Irvine connected to theatre mainstage at UC Santa Cruz

• 25 dancers; live interactive video from both sites processed at eMedia Studio

• Depicted dream states through structured movement improvisation, visuals and music

Songs at a Distance(April 2007)

• eMedia Studio connected to Loewe Theatre at New York University; 40 performers

• Live movement analysis linked to interactive composition systems for visuals and music

http://embodied.uci.edu

Becoming Dragon-A Mixed Reality, Durational PerformanceCombining Virtual Reality, Motion Capture, Second Life

Source: Micha Cárdenas, CRCA, Calit2365 hours, from December 1-17th, 2008

Academic Research “OptIPlatform” Cyberinfrastructure:A 10Gbps Lightpath Cloud

National LambdaRail

CampusOpticalSwitch

Data Repositories & Clusters

HPC

HD/4k Video Images

HD/4k Video Cams

End User OptIPortal

10G Lightpath

HD/4k TelepresenceInstruments

WAAG, AmsterdamCalit2, UCSD

Scalable City on the Global OptIPuter

Gridjam—Performance Art in the New World ofDedicated End-to-End Optical Networks

Visualization: Jack Ox and Dave BrittonMusic: Alvin Curran, composer and musician, Del Sol Quartet and Anthony Braxton playing.Organizations and Venues

University of New Mexico’s College of Fine Arts, with resources of the Art Research Technology and Science Laboratory and the Center for High Performance Computing.Calit2@UC San Diego,University of Alberta, University of Amsterdam in collaboration with De Waag Society, The Laboratory for Creative Arts and Technologies (L-CAT) at LSU

The Desert Organ stop inside the Virtual Color Organ™ . This is the VR space where Gridjam will be visualized-

Sound objects modeled by Ox of Curran’s collected

sound files

A Visual Artist, Composer, Musicians, Scientists and Technologists Collaborate to Produce a Real-time, Globally Distributed Performance in Virtual Space

Source: Jack Ox, UNM

OptIPuter Establishes Room-to-Room Telepresence on a Global Scale

Covise, Phil Weber, Jurgen Schulze, Calit2CGLX, Kai-Uwe Doerr , Calit2

www.calit2.net/newsroom/release.php?id=1219

January 15, 2008No Calit2 Person Physically Flew to Australia to Bring This Up!

January 15, 2008 Melbourne, Australia

Calit2@San Diego

Just in Time OptIPlanet Collaboratory:Live Session Between NASA Ames and Calit2@UCSD

Source: Falko Kuester, Calit2; Michael Sims, NASA

View from NASA AmesLunar Science Institute

Mountain View, CA

Virtual Handshake

HD compressed 6:1

From Start to This Image in

Less Than 2 Weeks!

Feb 19, 2009

NASA Interest in Supporting

Virtual Institutes