An Interview With Pranav Mistry

8/6/2019 An Interview With Pranav Mistry

1/3

An interview with Pranav Mistry, the

genius behind Sixth Sense

Pranav Mistry is the MIT grad student behind Sixth Sense, a tool that connects thephysical world with the world of data. He and his advisor at the MIT Media Lab,PattieMaes, unveiled Sixth Sense at TED2009, and the Sixth Sense demo premiered yesterdayon TED.com and in both places, it has fired peoples imaginations. The TED Blogspoke with Pranav this morning, to ask him some questions that have arisen on TED.com

and at the TED office. From the interview:

Why choose a projector versus goggles?

We actually thought a lot about this. At MIT, lots of research has been done with glasses theres even research going on to put information in your contact lenses. But thisparticular project has an important aspect: We want this thing to merge with the physicalworld in a real physical sense. You are touching that object and projecting info onto thatobject. The information will look like it is part of the object.

Read the full interview with Pranav Mistry, after the jump >>

Watch the Sixth Sense demo on TED.com >>

TED Blogs interview with Pranav Mistry, 3/11/2009

What was your role on the project?

I took this from the idea, the concept, and developed the software and the hardware.Pattie is my advisor. She helps me brainstorm, What should we do next? From thebeginning, I started working on the concept of merging the physical world and the digital
http://blog.ted.com/2009/03/11/sixth_sense_pranav/http://blog.ted.com/2009/03/11/sixth_sense_pranav/http://www.ted.com/index.php/speakers/pranav_mistry.htmlhttp://www.pranavmistry.com/projects/sixthsense/index.htmhttp://www.media.mit.edu/http://www.media.mit.edu/http://www.ted.com/index.php/speakers/pattie_maes.htmlhttp://www.ted.com/index.php/speakers/pattie_maes.htmlhttp://www.ted.com/index.php/talks/pattie_maes_demos_the_sixth_sense.htmlhttp://blog.ted.com/2009/03/sixth_sense_pranav.php#morehttp://www.ted.com/index.php/talks/pattie_maes_demos_the_sixth_sense.htmlhttp://blog.ted.com/2009/03/11/sixth_sense_pranav/http://blog.ted.com/2009/03/11/sixth_sense_pranav/http://www.ted.com/index.php/speakers/pranav_mistry.htmlhttp://www.pranavmistry.com/projects/sixthsense/index.htmhttp://www.media.mit.edu/http://www.ted.com/index.php/speakers/pattie_maes.htmlhttp://www.ted.com/index.php/speakers/pattie_maes.htmlhttp://www.ted.com/index.php/talks/pattie_maes_demos_the_sixth_sense.htmlhttp://blog.ted.com/2009/03/sixth_sense_pranav.php#morehttp://www.ted.com/index.php/talks/pattie_maes_demos_the_sixth_sense.html


2/3

world, like with my earlier project,Quickies, merging physical sticky notes with digitaldata.

Why choose a projector versus goggles?

We actually thought a lot about this. At MIT, lots of research has been done with glasses

theres even research going on to put information in your contact lenses. But thisparticular project has an important aspect: We want this thing to merge with the physicalworld in a real physical sense. You are touching that object and projecting info onto thatobject. The information will look like it is part of the object.

Also, a projector opens up interaction and sharing. Say you and I are walking down thestreet in New York, talking, and suddenly I get mail we both want to see. I can show it ona wall, and we can take a decision together right there. Its opening information to share.At a coffeeshop, we can use the whole table as an interactive surface.

Wheres the battery?

The project itself contains a battery inside, with 3 hours of battery life. The other thing is,Im making a small solar panel. Im trying that out right now because I want to go withsustainable energy and so you dont always need to be charging. Whenever youreoutside, youll be charging, and with the system, you can be outside more.

How does the software know what you want the system to do next?

The software works on the basis of computer vision. Theres a small camera acting asyour eye, your third eye, your digital eye, connecting you to the world of digitalinformation. Processing is happening in your mobile phone, and basically works oncomputer vision algorithms that we developed ourselves, taking advantage of some open-source code but mainly writing code ourselves here at the lab. We had to write a lot of

algorithms from scratch here because there was nothing that did what we wanted. Wewrote 50,000 lines of code.

The software recognizes 3 kinds of gestures:+ multitouch gestures, like the ones you see in Microsoft Surface or the iPhone whereyou touch the screen and make the map move by pinching and dragging.+ what I call freehand gestures, like when you take a picture [as in the photo above]. Or,you might have noticed in the demo, because of my culture, I do a namaste gesture tostart the projection on the wall.+ iconic gestures, drawing an icon in the air. Like, Whenever I draw a star, show me theweather. When I draw a magnifying glass, show me the map.

You might want to use other gestures that you use in everyday life. This system is verycustomizable. Because its my choice, I do it my way in the demo, but I dont want theuser to change their habits. I want the Sixth Sense to change for them.

Have you thought about using this device for gaming?

Definitely. We can do all the kinds of gaming that exists now, but not only that, we canuse the physical world inside the game. You can play with physical stuff, invent some
http://www.pranavmistry.com/projects/quickies/http://www.pranavmistry.com/projects/quickies/http://www.pranavmistry.com/projects/quickies/http://www.pranavmistry.com/projects/quickies/


3/3

new games. Maybe you can hide something in the physical world open a book andhide something in the pages.

Weve been using Minority Report as shorthand to explain the device, or the

heads-up screen in Robocop. But was this device influenced by science fiction

Im not a very big fan of science fiction. I think that Im a very big fan of living in thephysical world. Im good with digital technology, but I start to miss the physical world. Imiss riding my bike, talking to friends. Technology now separates us from the physicalworld more and more. Even social networking sites are taking us away from the physicalworld.

At the lab, we like making things that we can touch, we can feel, we can take with uswherever we want to go, that we know how to interact with. The digital world has powerbecause it has dynamic information, but its important that we stay human instead ofbeing another machine sitting in front of a machine.

Whatever science fiction movies we watch now, we can make the technology real in twodays. What we can do is not important. What we shoulddo is more important.

What would you have said on the TED stage, if you and Pattie had had one more

minute?

We have lots of applications. Any application now on your computer or mobile phone,you can use in Sixth Sense. In fact, I was going to come on the stage and do a live demo,but then we decided to use the demos I had filmed. We were worried about technicalproblems and the low light. But we have more to show.

The demo has gotten a lot of attention. Whats life been like since your demo at

TED?As soon as the talk finished, so many people rushed up to us. Before TED I was working,easy, three months straight in my lab. I was thinking after TED I would take a one-weekbreak. But even today I havent taken that one-week break. We are getting, every day, 50mails, we should do this, we should do that.

Theres some really interesting comments, not only from people interested in makingcomputer interfaces, but people asking, Why cant we use this system for people whohave accessibility problems, blind people, deaf people? The camera can act as a thirdeye for the blind person, and tell them what it sees. It could be an ear for a deaf person.Ideas are also coming from developing countries in part because of the low cost. It

cost me $350 to build Sixth Sense in the lab, but the price will come down.
http://www.media.mit.edu/http://www.media.mit.edu/http://www.media.mit.edu/

Documents

An Interview With Pranav Mistry