6 NTT Technical Review

—Dr. Kobayashi, what is the Human Interaction Pro-ject?

The Human Interaction Project that I belong to wasestablished just under a year ago to perform new testson human interfaces. It encompasses a wide varietyof research topics from robots to image processingthat might initially appear to have no consistenttheme. But in this project, we treat these technologiesas communication-support platforms with possibleinterrelationships to study how interfaces betweenpeople and media should really be as our main theme.

—What particular theme are you pursuing within it?

As the name implies, our Human CommunicationGroup is researching environments that make it easi-er for people to communicate with each other. For thelast few years, this group has been researching waysof facilitating human-to-human conversation by skill-fully using media to interpret various types of signalsthat exist in real space. While past research on com-munication and human interfaces mainly aimed toimprove the quality of voice and images, research ofthe Human Communication Group revolves around

human senses and has been given the name“Kimochi-communication” to distinguish it fromconventional approaches. This form of communica-tion includes the transmission of decisions andacknowledgements in addition to feelings. So whyare we researching communication centered abouthuman (user) senses? Because the ability to feel thatone is conveying and understanding intention isexpected to be extremely important in future commu-nication media. We refer to this as a “sense of mutu-al understanding,” and we feel that the quality ofmutual understanding in addition to voice-and-image quality will play a significant role in commu-nication media of the near future.

—Could you tell us about any trials that are now tak-ing place?

Yes. In one trial, we are asking subjects to engagein multiple conversations and collecting data onphysical characteristics such as pulse and body tem-perature while recording those conversations. Thisdata can then be compared with the content of con-versations along the time axis and analyzed accord-ingly to investigate, for example, what kind of phe-nomena might occur before a person speaks and whatto do when something that a person wanted to say

Exploring human interfaces based on novelconcepts

Toward Communication Systems That Can Convey Feelings

Minoru KobayashiSenior Research EngineerHuman Communication GroupHuman Interaction ProjectNTT Cyber Solutions Laboratories

The Human Communication Group of the Human Interaction Project at NTT Cyber SolutionsLaboratories is researching new human interfaces with a view toward establishing deeper meansof communication. In addition to developing advanced technologies, it pursues the essence ofcommunication with original ideas and aims to uncover sophisticated ways of expressing andsupporting human communication. We asked Dr. Minoru Kobayashi, a senior research engineerand the driving force behind the Human Communication Group, to tell us about communicationsystems that can convey feelings, a central theme of this group.

R&D Spirits

R&D Spirits

Vol. 3 No. 8 Aug. 2005 7

could not be said. In this way, we believe that we canestablish a basis for developing media that steer top-ics toward the person that has just spoken and thuspromote smooth conversation.

We are also developing haptic displays as part ofour research into tactile interfaces (Fig. 1). In one ofour haptic displays, “pressing” a three-dimensionalcomputer graphic on a table generates a gush of airfrom below to produce a sense of resistance just likewhen touching a soft surface. In contrast to generalhaptic interfaces that connect the human body by armmechanisms or wires in a somewhat unpleasant man-ner, the use of air in our haptic display results in avery “clean” medium that feels comfortable to theuser. This research has demonstrated that the act oftouching can be an effective means of supportingcommunication just like seeing and hearing.

—What are the ultimate goals of your research?

We have three ultimate goals. The first is to make iteasier to express what one is thinking. For example, amedium that could make it easy to “draw” what oneis saying in a picture would make it difficult for mis-understandings to develop in a conversation. The sec-ond is to support the recording of communications. Inother words, equipment that could make it easy toremember when one thought about something wouldprevent a great idea that could not be jotted downfrom being forgotten. And finally, our third goal is todetect lapses in mutual understanding. For example,

there are many words that might be given differentmeanings and used differently depending on a per-son’s value system and background knowledge. Here,a medium that could detect such an instance mightalso be able to convey nuances and/or emotions notexpressed by the word itself. It could make conserva-tions all the more satisfying. These are our ultimategoals, and finding solutions to reach them is our chal-lenge.

Working to eliminate breakdowns incommunication

—Dr. Kobayashi, how did you develop an interest inhuman interface research?

In graduate school, I selected man-machine inter-faces as a research theme, and for my master’s thesis,I constructed an information-browsing interfaceusing books as a metaphor and implemented it on acomputer. It would therefore appear that I have beenon this path from the very start, but in fact, it wasslightly different. The first time I encounteredhuman-interface research was during my undergrad-uate years when I occasionally worked part-time inthe creation of computer-aided-design (CAD) sys-tems. At that time, general-purpose CAD applica-tions were not commercially available and companiesor organizations that wanted to use them had to havethem developed individually. It was also a time whenthe mouse had not yet become standard equipment on

Fig. 1. Haptic interface using air jet.

R&D Spirits

8 NTT Technical Review

personal computers, and those of us students whowere working at this part-time job would often dis-cuss the merits of using a mouse or a tablet for a par-ticular system. Come to think of it, this work itselfwas a form of interface research, but I would say thatmy deep interest in interface research began afterreading “The Human Interface” from the MIT MediaLaboratory, a book that fellow part-time workers whowere already graduate students introduced me to.

—Could you tell us something about your researchhistory?

Well, soon after entering NTT, I became involvedwith the development of video-based groupware aspart of NTT’s research on computer-supported coop-erative work (CSCW). Here, I proposed the Clear-Board concept (Figs. 2 and 3) after being greatlymotivated by the TeamWorkStation system devel-oped by Hiroshi Ishii, Kazuho Arita, and MasaakiOhkubo. The TeamWorkStation system provided ajoint work-support environment through video con-ferencing based on the concept of an open and sharedworkspace. It made geographically remote users feelas if they were using the same whiteboard. This sys-tem had a big impact as it could reflect the writingmotion of participants in real time while overlappingthe video of participants’ desks on the screen. Unfor-tunately, the shared workspace and the video spacefor displaying users’ faces were separate in Team-WorkStation, and it was not possible to make use ofgestures and eye movement in a natural manner. Toeliminate this drawback, ClearBoard unites theshared workspace and user video space, making themone and the same.

ClearBoard introduces an entirely new concept inallowing people to converse and draw from both sidesof a transparent glass panel. In developing Clear-Board, I first thought of creating an L-shaped displayto combine the user video screen and joint work-space, but in this case, the space between the verticaland horizontal portions of the L-shaped display couldnot be represented. Then, after giving this problem alot of thought, I finally came to the conclusion that Icould fold over the L-shaped display. In other words,I realized that a transparent glass panel was the wayto go! With hindsight, this seems simple, but the evo-lution of this concept was a significant breakthroughin human-interface research. In short, while my initialgoal was just to connect what was originally two dif-ferent spaces, I eventually found on making that con-nection that the reactions of the other user could berecognized and understood much better than expect-ed. Understanding what the other party is looking atmakes it clear what that user is interested in and whatshould be done next in the session. We have namedthis property “gaze awareness” and this term isbecoming standard usage in media research.

After completing a major portion of the ClearBoardresearch, I had the opportunity to study abroad atMIT Media Laboratory, where I received my secondmaster’s degree. There, I took on research dealingwith the creation of an interface for simplifyingaccess to aural information by mapping of the auraltime axis to space. A simple metaphor of this processis a radio that is turned on and emitting sounds whilemoving about. Mapping of this type can connect auralinformation with spatial recording. It can also be usedto understand aural data in its entirety and to listen tovarious locations simultaneously. I worked on devel-oping this peculiar but interesting form of technolo-gy.

On returning to Japan, I did some work in virtualreality and developed an interface called GAVA thatused a box-type display and computer graphics toprovide face-to-face communication. This systemmade it possible to convey the other person’s eyemovements, which increased that person’s sense ofpresence. I therefore discovered that eye movementcould also play an important function in virtual com-munication.

—What have you personally been pursuing throughyour research?

The first is to find ways of eliminating break-downs in communication. At first glance, computerFig. 2. TeamWorkStation.

R&D Spirits

Vol. 3 No. 8 Aug. 2005 9

research and communications research might appearto be very distant fields from the viewpoint of peo-ple’s lives. Communication media, however, havecome to play a big role in modern society, and theycan sometimes give rise to quarrels and disputes dueto miscommunications and may even be a source ofdepression and crime. Against this background, Icame to believe that communication media might beone field that could help people in their lives in a waydifferent from that of medicine and pharmacology. Tobe honest, I initially only wanted to create communi-cation media that simply felt good, but I graduallycame to attach more significance to communicationmedia as my research continued.

Furthermore, though it might sound like bragging,I also considered how a research laboratory should berun. Research in a private company tends to connectnear-term results with clear and obvious needs, but Ihave always made it a personal rule to take on morechallenging research. This is because I believe that aresearch laboratory is essentially a place where diffi-cult problems should be directly confronted. Here, tobe able to continuously output things that people feelare amazing and interesting and that they want to use,we must face difficult issues head-on while realizingthat failure is part of the process. If NTT is to be aplace that promotes communication media research,it must be an institution that can tolerate detailedresearch in a step-by-step and steady manner even ifresults are not quick to materialize. It is in that spiritthat I have been carrying out my research up to thepresent.

—Dr. Kobayashi, what is currently going on inhuman-interface research overseas and in Japan?

Communication media research is now beingundertaken by many companies and universities asbroadband networks continue to expand. But I don’tbelieve that there are many corporate research labora-tories that are taking on somewhat advanced researchthemes dealing with intangible things like “feelings”as in our research projects. While there is some com-petition at least in part, this is mostly coming fromuniversities. In fact, human-interface research is com-paratively active in Canadian and American universi-ties and there are even some areas in which they aretaking the lead.

—Are you collaborating with other research institu-tions?

At this point in time, we are not, but we do welcomecollaboration with overseas organizations. We havehad contacts with several research institutions. Look-ing back at past inventions dealing with human inter-faces, many of them arose from an encounter withdifferent values, so it is important that collaborationbe pursued with people having a different sense ofvalues. Of course, various sets of values can also befound within NTT and we are pursuing serious dis-cussions and research in-house as well. But if we con-fine ourselves to only one research institution, it can-

Tying up with overseas organizations inpursuit of more breakthroughs

Fig. 3. ClearBoard.

R&D Spirits

10 NTT Technical Review

not be denied that there may not be enough energy tobring about major breakthroughs, and it is my desireto incorporate more values from the outside fromhere on.

—What has been the outside reaction to NTTresearch achievements in human interfaces?

In a newspaper article distributed at the 2002 Com-puter-Human Interaction (CHI) conference com-memorating the 20th anniversary of that society, thepresentation that Hiroshi Ishii and I gave on Clear-Board in 1992 was selected as a “most memorableevent” in one person’s opinion that presentation led toa continuous stream of inquiries from many countriesaround the world indicating that ClearBoard has hada big impact in various ways. But, sad to say, theClearBoard concept was never commercialized, andthe only remaining unit of ClearBoard equipment thatanyone can operate can be found in the city ofKumamoto in Japan as it was obtained by the Con-temporary Art Museum in Kumamoto the year beforelast. In all honesty, I would rather see it displayed inan NTT showroom than in a museum, but seeing chil-dren and the elderly getting a kick out of using thisClearBoard equipment gives me much pleasure.

We also held a demonstration of our haptic displayas a type of tactile interface at a SIGGRAPH confer-ence. There, more than 2400 people tried out this dis-play one after another and we received many positivecomments about it.

The attraction of working at NTTLaboratories having close ties to business

—For the future, how do you plan to expand yourresearch?

Our research is still in its initial stage. Even withour development of ClearBoard and discovery ofgaze awareness, our work in this area has not been anongoing process since then. This is not really a prob-lem of technology or ideas. Rather, as a communica-tion tool, the ideal human interface is always chang-ing as human lifestyles themselves continue tochange. To give a simple example, the appearance ofcellular phones brought major changes to everyone’slifestyle, and as a result, a number of research themesthat preceded the cellular phone era quickly becameobsolete. In our research too, we must always bethinking about our next move while keeping an eyeon an ever-changing society. With this in mind, our

ultimate objective is to establish a communicationenvironment with mutual understanding so that peo-ple can communicate without misunderstandings andbe more productive. It is in this way that I would liketo connect communication media with human happi-ness.

—What is it about research and development workthat you find interesting?

That’s a difficult question. Sometimes I myself findit unbelievable that I put so much energy and effortinto research. Perhaps it’s the sheer pleasure of dis-covering something of great significance. For exam-ple, the instant that I hit upon the ClearBoardmetaphor was very exciting and it’s a moment that Iwill never forget. Of course, there is no guarantee thatsuch momentous events will occur often during one’slife, but just the same, I cannot stop wanting to expe-rience that sensation again and again. I think that any-one that has been involved with the research processfor a long time can sympathize with this desire. Bythe way, I do have several specific visions for thefuture, but, unfortunately, they are not yet at the stagewhere I can talk about them freely.

—What do you think is the most important attributefor a researcher?

I feel that researchers must be obsessive about theirwork if they are to have long and fruitful careers.Research, I believe, is an activity where all of one’sattention comes to be focused on one thing. Althougha private company must be concerned with obtaininga return on its investments, work that only chasesafter greater efficiencies without that obsession withresearch can only weaken not only the researcher butalso the research theme and research institution.Researchers should therefore reflect on their activi-ties on a daily basis to ascertain whether they aredoing something that they feel is important. But I alsobelieve that each researcher must make a contributionto the company, while pursuing his or her obsession.

—Finally, what is the attraction of working at NTTLaboratories?

At NTT Laboratories, there are many people thathave the mind of a researcher—that are well-informed about their field, are confident in their abil-ities, and are continuously moving forward with theirresearch from one day to the next as leaders in their

R&D Spirits

Vol. 3 No. 8 Aug. 2005 11

field. That is especially attractive to me. In addition,as all NTT researchers fall under one corporate orga-nization, we tend to follow a basic rule of mutualcooperation, which makes it easy to establish collab-orative relationships and projects. Such an environ-ment encourages people to ask for each other’s helpand even stimulates friendly competition. I believethat the ability to work in such a setting is very mean-ingful.

And if I may add one more thing, the fact the NTTLaboratories is a research center for a major telecom-munications company is significant to me. On enter-ing NTT to pursue communications research, thepressure that I felt from the communications-businessside made me all the more energetic. We never feltanything like that at a university research laboratory.Although research is basically an endeavor that takestime, the pressure coming from the business sideforces us to feel the urgency for results that are usefulin the business world. This can lead to results that cancontribute to the world at large. That is also why Ihave enjoyed working at NTT Laboratories.

Interviewee profile■ Career highlights

Minoru Kobayashi received his B.E. and M.S.degrees from Keio University, Yokohama in 1988 and1990, respectively. He joined NTT Human InterfaceLaboratories in 1990. Since then, he has conductedresearch on computer-supported cooperative work(CSCW) and video-based groupware design. He pro-posed the concept of ClearBoard and implemented thevideo-based prototype ClearBoard-1 and the comput-er-based system ClearBoard-2. He spent 1994 to 1996at Massachusetts Institute of Technology, where heworked on a spatial human interface for audio data andreceived his second M.S. degree. He received hisPh.D. degree in instrumentation engineering fromKeio University in 2000. His research interests includehuman-computer interaction, computer-supportedcooperative work, and video-based groupware design.He is a member of the Association for ComputingMachinery, Computer Society of IEEE, InformationProcessing Society of Japan, and the Institute of Elec-tronics, Information and Communication Engineers ofJapan.