1

Telepresence: Telepresence: An Umbrella Research TopicAn Umbrella Research Topic

Jim GrayMicrosoft ResearchGray@Microsoft.comhttp://research.Microsoft.com/~Gray/

2

NSF: Nerve Center of ScienceNSF: Nerve Center of ScienceIf it’s not broke, don’t fix it.If it’s not broke, don’t fix it.But….But….

US Science is the engine of progressBUT…..

Best and brightest are spending increasing

time fundraising Seems excessive to me. Venture capital community is

richer and more generous

than NSF

3

Outline (ambitious!)Outline (ambitious!) Microsoft Research (census) Tele-Presentations (Gordon Bell, Jim Gemmell) Microsoft Research initiative on Telepresence What if you could record everything you see & hear? The architecture revolution:

processing moves to transducers

4

Microsoft Research -- 1991Microsoft Research -- 1991 Founded in 1991 Goal:

pursue strategic technologies for Microsoft

Original research groups:– Natural Language Processing– Operating Systems– Programming Languages

Overall size < 20 at the end of 1992

5

Microsoft Research -- 1998Microsoft Research -- 1998 280 Researchers in 25 areas

– Operating systems to Statistical Physics Research lab locations:

– Redmond, Cambridge, San Francisco Internationally recognized research teams

– Hundreds of publications, presentations– Leadership roles in professional societies,

journals, conferences

6

MS Research AreasMS Research Areas Operating systems, languages, compilers,

virtual machines, networking, wireless computing, fault-tolerance, large scale servers, security

Natural language, speech, vision, graphics, decision theory, information retrieval, UI, collaboration, statistics, signal processing

Cryptography, statistical physics and discrete mathematics

7

Growing FastGrowing Fast

Grew 4x from ‘94 to ‘97 Decided in ‘97 to grow by a 3x in 3 years

– 200 in FY97 => 600 in FY00, primarily in Redmond

Major impact on MS products– Virtually all MS products shipped today use

technology from MS Research Critical role in MS growth

– Pioneering research in software that allows computers to see, hear, speak and understand

8

Microsoft Research Microsoft Research PhilosophyPhilosophy

University organizational model– Flat structure, critical mass groups

Open research environment– Aggressive publication of research results

in literature and on world wide web

– Frequent visitors, daily seminars

– Over 70 visiting professors and interns in 1997

– Over 110 visiting researchers in 1998

9

Some Key Senior ResearchersSome Key Senior Researchers Systems

– Rick Rashid, Butler Lampson, Gordon Bell

– Anoop Gupta, Roger Needham, Chuck Thacker Databases & Data Mining

– David Lomet, Jim Gray, Usama Fayyad Graphics

– Jim Kajiya, Jim Blinn, Alvy Ray Smith, Michael Cohen Speech & Language

– Karen Jensen, George Heidorn, X.D. Huang, Alex Acero, Hsiao-Wuen Hon, Scott Meredith

10

Some Key Some Key Senior ResearchersSenior Researchers

UI Design, Intelligent Systems, IR

– George Robertson, Linda Stone, Susan Dumais, David Heckerman, Eric Horvitz, Jack Breese

Computer Vision & Signal Processing

– Steve Shafer, Rick Szeliski, P. Anandan, Rico Malvar Cryptography & Theory

– Yacov Yacobi, Jennifer Chayes, Christian Borg, Michael Freedman

Languages & Compilers

– Daniel Weise, Chris Fraser, Amitabh Srivastava, Luca Cardelli, David Hanson, Charles Simonyi, Todd Proebsting

11

Microsoft ResearchMicrosoft Research 1997 BusinessWeek Poll of Academia:

– Voted #7 lab (overall) in Computer Science

– Voted #3 industrial research lab (after Bell Labs and IBM Research)

– Voted #2 most desirable lab to work (after Stanford)

12

Outline (ambitious!)Outline (ambitious!) Microsoft Research (census) Tele-Presentations (Gordon Bell, Jim Gemmell) Microsoft Research initiative on Telepresence What if you could record everything you see & hear? The architecture revolution:

processing moves to transducers

13

Gordon Bell on Gordon Bell on Tele PresentationsTele Presentations

http://research.microsoft.com/barc/GBell/http://research.microsoft.com/barc/GBell/

14

Motivation:Motivation:TelepresentationsTelepresentations

• Presenter and/or audience telepresent

NOT: meeting or collaboration settings

Forget the nasty social issues!

Mostly one-way

15

TelepresentationTelepresentationElementsElements Slides Audio Video Script,

text comments, hyperlinks,etc.

16

Telepresentations:Telepresentations:The EssentialsThe Essentials

Slide and audio a must Add some video

(low quality) to make us feel good

Storage and transmission costs low

17

Telepresentations:Telepresentations:The Killer AppThe Killer App

Increased attendance & lower travel costs

Practical and low-cost NOW e.g. ACM97 - 2,000 visitors in real

space, 20,000 visitors on Internethttp://research.microsoft.com/acm97

18

Today’sToday’sExperimentExperiment

Would you like to pause, rewind, browse? Do you wish you could have seen this

– At home?– At another time?

How much does a present speaker add? How much would you pay for real presence?

19

Outline (ambitious!)Outline (ambitious!) Microsoft Research (census) Tele-Presentations (Gordon Bell, Jim Gemmell) Microsoft Research initiative on Telepresence What if you could record everything you see &

hear? The architecture revolution:

processing moves to transducers

20

Changing role of computationChanging role of computation Past: Computers for:

– computing (Cray)– business data processing (IBM)– “document” creation (PC)

Future: Computers for: – understanding & learning– communicating– consuming & entertaining

Requires new User Interface to machines

FlowsFlows

22

Making “Flows” a RealityMaking “Flows” a Reality Computer Graphics

– Creating realistic looking environments, people

Computer Vision– Analyzing posture, gaze, gestures

Speech input/output Natural Language

– Analysis, IR Implicit requests for information

Building life-like human Building life-like human characterscharacters

Recognizing gesturesRecognizing gestures

Live videoLive videoArea of Area of motionmotion H flowH flow V flowV flow

25

Generating life-like speech Generating life-like speech from textual datafrom textual data Data-driven stochastic speech

– Natural sounding– Rapid, automatic customizability

Examples– Synthetic voice w/ transplanted speech

contours

26

AT&T Voder, 1962, by Homer Dudley– Daisy (Inspiration for HAL’s voice in 2001)

Microsoft Research Whistler, 1997– Scarborough Fair

Artificial singingArtificial singing

27

Analyzing languageAnalyzing language Language recognition shipped in Word 97 General purpose text-critiquing,

summarization, Japanese word-breaking

28

Inside The Office Inside The Office Grammar CheckerGrammar Checker

29

Understanding language: Understanding language: MindNetMindNet

A huge language knowledge base

Automatically created from dictionaries

Words (nodes) linked by relationships

Millions of links Recently added

(Encarta) encyclopedia knowledge

Is_amouthLocn_of

MindNet -- “Going to the birds”MindNet -- “Going to the birds”

face

peck limbgoose

creaturemake

Is_a

Typ_obj sound

Typ_subj

preen

Is_a

Part feather

Not_is_a plant

Is_agaggle Is_a

Is_a

PartPart_of

catch

Typ_subj_ofTyp_obj

claw

wing

Is_a

turtle

beak

Is_a

Is_a

strike

Means

hawk

Typ_subj

opening

Is_a

chatter

Means

Typ_subjTyp_obj

Is_a

Is_a

clean

smooth

billIs_a

duck

Is_a

Typ_0bj_of keep

animal

quack

Is_a

CausePurpose

bird

meat

egg

poultry

Is_a

supplyPurpose Typ_obj

Quesp

hen

chicken

Is_a

Is_a

leg

arm

Is_a

Is_a

Is_a

Typ_subj_of

fly

31

Changing balance between Changing balance between user & software systemsuser & software systems

Yesterday:– Applications were single programs running in

isolation– Users used to (more or less) understand systems

that they used Today:

– Componentized applications operate in concert – Sophisticated users understand only small

percentage of systems they use

32

Tomorrow’s Systems and Tomorrow’s Systems and ApplicationsApplications Users will not be able to predict

– where computations will be performed, – when they will be performed or – by what software components

Gap between system capabilities and user understanding will grow to the point that the only way user will be able to use system is through assisting agents

33

Examples of user agents & Examples of user agents & implicit actionsimplicit actions Lumiere (Office 97)

– Monitoring user and program events to provide user help and assistance

Implicit queries– Inferring information needs from browsing

Lookout/SpamKiller– Monitoring mail activity to auto-categorize it

User ModelingUser Modeling Models of a user’s informational goals

– User’s query (when available…)– User’s background– Acute and long-term search activity– Acute actions with objects and documents– Program data structures

Explicit and implicit information access and display

35

Outline (ambitious!)Outline (ambitious!) Microsoft Research (census) Tele-Presentations (Gordon Bell, Jim Gemmell) Microsoft Research initiative on Telepresence What if you could record everything you see &

hear? The architecture revolution:

processing moves to transducers

36

Some Tera-Byte DatabasesSome Tera-Byte Databases Kilo

Mega

Giga

Tera

Peta

Exa

Zetta

Yotta

The Web: 1 TB of HTML TerraServer 1 TB of images Several other 1 TB (file) servers Hotmail: 7 TB of email Sloan Digital Sky Survey:

40 TB raw, 2 TB cooked EOS/DIS (picture of planet each week)

– 15 PB by 2007 Federal Clearing house: images of checks

– 15 PB by 2006 (7 year history) Nuclear Stockpile Stewardship Program

– 10 Exabytes (???!!)

37

Kilo

Mega

Giga

Tera

Peta

Exa

Zetta

Yotta

A novel

A letter

Library of Congress (text)

All Disks

All Tapes

A Movie

LoC (image)

Info CaptureInfo Capture You can record

everything you see

or hear or read. What would you do

with it? How would you

organize & analyze it?

Video 8 PB per lifetime (10GBph)Audio 30 TB (10KBps) Read or write: 8 GB (words)

See: http://www.lesk.com/mlesk/ksg97/ksg.html

38

Kilo

Mega

Giga

Tera

Peta

Exa

Zetta

Yotta

A novel A letter

Library of Library of Congress Congress (text)(text)

All Disks

All Tapes

A Movie

LoC (image)

All Photos

LoC (sound + cinima)

All Information!

39

Michael Lesk’s PointsMichael Lesk’s Points www.lesk.com/mlesk/ksg97/ksg.htmlwww.lesk.com/mlesk/ksg97/ksg.html

Soon everything can be recorded and kept

Most data will never be seen by humans

Precious Resource: Human attention Auto-SummarizationAuto-Search

will be a key enabling technology.

40

Outline (ambitious!)Outline (ambitious!) Microsoft Research (census) Tele-Presentations (Gordon Bell, Jim Gemmell) Microsoft Research initiative on Telepresence What if you could record everything you see &

hear? The architecture revolution:

processing moves to transducers

41

Put Everything Put Everything in Future (Disk) Controllersin Future (Disk) Controllers(it’s not “if”, it’s “when?”)(it’s not “if”, it’s “when?”)

AcknowledgementsAcknowledgements::

Dave PattersonDave Patterson explained this to me a year ago explained this to me a year ago

Kim KeetonKim Keeton

Erik RiedelErik Riedel

Catharine Van IngenCatharine Van Ingen

Helped me sharpen these arguments

42

Remember Your RootsRemember Your Roots

43

Technology Drivers: DisksTechnology Drivers: Disks Disks on track 100x in 10 years

2 TB 3.5” drive Shrink to 1” is 200GB Disk replaces tape?

Disk is super computer!

Kilo

Mega

Giga

Tera

Peta

Exa

Zetta

Yotta

44

Data GravityData Gravity Processing Moves to TransducersProcessing Moves to Transducers

Move Processing to data sources Move to where the power (and sheet metal) is Processor in

– Modem– Display– Microphones (speech recognition)

& cameras (vision)– Storage: Data storage and analysis

45

It’s Already True of PrintersIt’s Already True of PrintersPeripheral = CyberBrickPeripheral = CyberBrick You buy a printer You get a

– several network interfaces– A Postscript engine

cpu, memory, software, a spooler (soon)

– and… a print engine.

46

Tera Byte Backplane

TODAY– Disk controller is 10 mips risc engine

with 2MB DRAM– NIC is similar power

SOON– Will become 100 mips systems

with 100 MB DRAM. They are nodes in a federation

(can run Oracle on NT in disk controller). Advantages

– Uniform programming model– Great tools– Security– economics (CyberBricks)– Move computation to data (minimize traffic)

All Device Controllers will be Cray 1’sAll Device Controllers will be Cray 1’s

CentralProcessor &

Memory

47

Basic Argument for x-DisksBasic Argument for x-Disks Future disk controller is a super-computer.

– 1 bips processor– 128 MB dram– 100 GB disk plus one arm

Connects to SAN via high-level protocols– RPC, HTTP, DCOM, Kerberos, Directory Services,…. – Commands are RPCs– Management, security,….– Services file/web/db/… requests– Managed by general-purpose OS with good dev

environment Apps in disk saves data movement

– need programming environment in controller

48

The Slippery SlopeThe Slippery Slope

If you add function to server Then you

add more function to server Function gravitates to

data.

Nothing = Sector Server

Everything = App Server

Something =

Fixed App Server

49

Why Not a Sector Server?Why Not a Sector Server?(let’s get physical!)(let’s get physical!)

Good idea, that’s what we have today. But

– cache added for performance– Sector remap added for fault tolerance– error reporting and diagnostics added– SCSI commends (reserve,.. are growing)– Sharing problematic (space mgmt, security,…)

Slipping down the slope to a 2-D block server

50

Why Not a 1-D Block Server?Why Not a 1-D Block Server?Put A LITTLE on the Disk ServerPut A LITTLE on the Disk Server

Tried and true design– HSC - VAX cluster– EMC– IBM Sysplex (3980?)

But look inside– Has a cache – Has space management– Has error reporting & management– Has RAID 0, 1, 2, 3, 4, 5, 10, 50,…– Has locking– Has remote replication– Has an OS– Security is problematic– Low-level interface moves too many bytes

51

Why Not a 2-D Block Server?Why Not a 2-D Block Server?Put A LITTLE on the Disk ServerPut A LITTLE on the Disk Server Tried and true design

– Cedar -> NFS– file server, cache, space,..– Open file is many fewer msgs

Grows to have– Directories + Naming– Authentication + access control– RAID 0, 1, 2, 3, 4, 5, 10, 50,…– Locking– Backup/restore/admin– Cooperative caching with client

File Servers are a BIG hit: NetWare™– SNAP! is my favorite today

52

Why Not a File Server?Why Not a File Server?Put a Little on the Disk ServerPut a Little on the Disk Server Tried and true design

– Auspex, NetApp, ...– Netware

Yes, but look at NetWare– File interface gives you app invocation interface– Became an app server

Mail, DB, Web,….

– Netware had a primitive OS Hard to program, so optimized wrong thing

53

Why Not Everything?Why Not Everything?

AllowAllow Everything on Disk Server Everything on Disk Server(thin client’s)(thin client’s)

Tried and true design– Mainframes, Minis, ...– Web servers,…– Encapsulates data– Minimizes data moves– Scaleable

It is where everyone ends up. All the arguments against are short-term.

54

The Slippery SlopeThe Slippery Slope

If you add function to server Then you

add more function to server Function gravitates to

data.

Nothing = Sector Server

Everything = App Server

Something =

Fixed App Server

55

Disk = NodeDisk = Node has magnetic storage (100 GB?) has processor & DRAM has SAN attachment has execution

environment

OS KernelSAN driver Disk driver

File System RPC, ...Services DBMS

Applications

56

Technology Drivers:Technology Drivers: System on a ChipSystem on a Chip

Integrate Processing with memory on one chip– chip is 75% memory now– 1MB cache >> 1960 supercomputers– 256 Mb memory chip is 32 MB!– IRAM, CRAM, PIM,… projects abound

Integrate Networking with processing on one chip– system bus is a kind of network– ATM, FiberChannel, Ethernet,.. Logic on chip.– Direct IO (no intermediate bus)

Functionally specialized cards shrink to a chip.

57

Technology Drivers: What if Technology Drivers: What if Networking Was as Cheap As Disk IO?Networking Was as Cheap As Disk IO? TCP/IP

– Unix/NT 100% cpu @ 40MBps

Disk– Unix/NT

8% cpu @ 40MBps

Why the Difference?Host Bus Adapter does

SCSI packetizing, checksum,…flow controlDMA

Host doesTCP/IP packetizing, checksum,…flow controlsmall buffers

58

Technology Drivers: Technology Drivers: The Promise of SAN/VIAThe Promise of SAN/VIA:10x in 2 years:10x in 2 years http://www.ViArch.org/http://www.ViArch.org/

Today: – wires are 10 MBps (100 Mbps Ethernet)

– ~20 MBps tcp/ip saturates 2 cpus– round-trip latency is ~300 us

In the lab– Wires are 10x faster Myrinet, Gbps Ethernet,

ServerNet,…

– Fast user-level communication tcp/ip ~ 100 MBps 10% of each processor round-trip latency is 15 us

59

Gbps Ethernet: 110 MBps

SAN: SAN: Standard InterconnectStandard Interconnect

PCI: 70 MBps

UW Scsi: 40 MBps

FW scsi: 20 MBps

scsi: 5 MBps

LAN faster than memory bus?

1 GBps links in lab.

100$ port cost soon

Port is computer

RIPFDDI

RIPATM

RIPSCI

RIPSCSI

RIPFC

RIP?

60

Technology Drivers:Technology Drivers:

100 GBps Ethernet replaces SCSI100 GBps Ethernet replaces SCSI Why I love SCSI

– Its fast (40MBps)– The protocol uses little processor power

Why I hate SCSI– Wires must be short– Cables are pricey– pins bend

61

Functionally Specialized CardsFunctionally Specialized Cards Storage

Network

Display

M MB DRAM

P mips processor

ASIC

ASIC

ASIC

Today:

P=50 mips

M= 2 MB

In a few years

P= 200 mips

M= 64 MB

62

Technology DriversTechnology Drivers

Plug & Play SoftwarePlug & Play Software RPC is standardizing: (DCOM, IIOP, HTTP)

– Gives huge TOOL LEVERAGE– Solves the hard problems for you:

naming, security, directory service, operations,...

Commoditized programming environments – FreeBSD, Linix, Solaris,…+ tools– NetWare + tools– WinCE, WinNT,…+ tools– JavaOS + tools

Apps gravitate to data. General purpose OS on controller runs apps.

63

Basic Argument for x-DisksBasic Argument for x-Disks Future disk controller is a super-computer.

– 1 bips processor– 128 MB dram– 100 GB disk plus one arm

Connects to SAN via high-level protocols– RPC, HTTP, DCOM, Kerberos, Directory Services,…. – Commands are RPCs– management, security,….– Services file/web/db/… requests– Managed by general-purpose OS with good dev

environment Move apps to disk to save data movement

– need programming environment in controller

64

SummarySummary Microsoft Research (census) Tele-Presentations (Gordon Bell, Jim Gemmell) Microsoft Research initiative on Telepresence What if you could record everything you see &

hear? The architecture revolution:

processing moves to transducers