Special Issue

NEC Journal of Advanced Technology, Vol. 1, No. 3 221

Advanced Technologies and Solutions toward Ubiquitous Network Society

ABSTRACT

Ubiquitous Information InterfacesBy Keiji YAMADA,* Shuji SENDA,† Toshiyuki ASAHI,† Toshihiko HIROAKI,†Hiroaki HATTORI* and Akitoshi OKUMURA*

This paper describes ubiquitous information interface, which consists of an input function, anoutput function, and an interactive function. These functions have to be designed under the

criteria such as real time, adaptability, and personalization in addition to the basic user interface designrules. As examples of ubiquitous information interface, it describes a camera-typing interface, a speechinterface, a semantic web browser, a fast document image browser, and an automatic spoken languagetranslator. These examples demonstrate that the ubiquitous information interfaces can overcome the problemswhich may occur in the ubiquitous society and make ubiquitous systems be easy to use and efficient.

KEYWORDS Ubiquitous, Interface, Image processing, Internet, Browser, Speech processing, Translator

*Media and Information Research Laboratories†Internet Systems Research Laboratories

1. INTRODUCTION

At the end of the 1980’s, Mark Weiser proposed theconcept of ubiquitous computing[1] and Norman putforward the idea of invisible computing[2]. They pre-dicted that computers would be miniaturized and em-bedded in various objects, and that in future, comput-ers would be used very differently from the way theywere currently used.

Until now, computers have usually consisted of abox with a keyboard, mouse, and screen. To use acomputer, the user goes to the computer and uses thekeyboard (Fig. 1). With ubiquitous computing, how-ever, users do not have to visit a computer to use it.They can access information services every time theyneed them wherever they are. This is not merely“anywhere and anytime” computing, but the ability touse information services naturally without any spe-cial effort.

As a step towards the ubiquitous information soci-ety, a number of computers can be connected to oneanother via a ubiquitous network that combines sev-eral network systems. These sorts of systems provideaccess to almost unlimited information. To utilizethem efficiently, however, requires the impracticaltask of mastering numerous types of terminal deviceswith different kinds of access networks. ConventionalGUIs (Graphical User Interfaces) are incapable ofsolving this problem. Instead, we need a new kind ofuser interface, which we call a ubiquitous informa-

tion interface. This paper describes the concept of aubiquitous information interface and gives some ex-amples along with their advantages.

2. PROBLEMS OF UBIQUITOUS SYSTEMS

Ubiquitous systems include the following threetypes (see Fig. 1).

(1) Access NetworksSeveral computer systems may be integrated via a

ubiquitous network, which is a seamless combinationof numerous networks such as a Gigabit Ethernet, W-CDMA network, W-LAN, and UWB (Ultra WideBand) network.

(2) Terminal DevicesDifferent types of terminal devices, including mo-

bile phones, PDAs, POS (Point-Of-Sale) terminals,

Fig. 1 From conventional interfaces to ubiq-uitous information interfaces.

NEC J. of Adv. Tech., Summer 2004

Special Issue

222

next-generation IC cards, and terminals attached tovending machines, are connected to one another andexchange information such as users’ attributes andlogs through a ubiquitous network.

(3) Content PresentationInformation may be presented in various multime-

dia formats such as text, images, video, speech, three-dimensional images, or some sort of physical force.

It might seem that these various systems provideusers with access to information services wheneverthey want them. However, they do not offer users thebenefits of a ubiquitous system; rather, they mayactually prevent users from experiencing these ben-efits. Every time users want to access an Internetservice or content, they have to select the optimalaccess network and terminal device depending on thecircumstances. They therefore need to know how touse all sorts of different terminal devices so they cansuccessfully use the one in front of them. This chaoticsituation makes it much more difficult to use ubiqui-tous systems.

To address this problem and to provide a trulyaccessible ubiquitous system, we need a ubiquitousinformation interface that enables everybody to eas-ily access information services using any kind of ter-minal device via any kind of network.

3. CONCEPT OF A UBIQUITOUS INFORMA-TION INTERFACE

An information interface is defined as an interfacebetween the real world and a digital informationsource.

In the real world, information is available in theform of printed documents, notebooks, speech, signs,and objects. Information is also available about oursurroundings, such as the temperature, humidity,and location of acquaintances.

This information is observed by sensors and isinput as digital information. In addition, digital infor-mation is produced by computers. Its examples arecomputer graphics.

An information interface has three functions (seeFig. 2), input, output, and interactive functions. Eachof these needs to be developed to overcome the prob-lems described in the previous section.

(1) Input FunctionThe input function observes information on a real

object to generate digital information with minimaleffort on the part of the user. The digital information

produced is not only stored in a computer, but alsoused as a hyperlink from the real-world object toother digital information related to that object. Forexample, if users want to know the meaning of a wordprinted on the page of a book, all they have to do ispoint their finger to it.

(2) Output FunctionThe output function presents digital information to

the real world via display devices such as screens,printers, and speakers. This function not only outputspieces of digital information to display devices, it alsotransforms them into a suitable form for the displaydevice in front of the user. For example, when imagesand videos are presented on a small display device,essential information such as a headline is extracted.These parts are displayed at a higher resolution andother parts are shrunk and presented in a reducedform. Thus, users can acquire important informationat a glance.

(3) Interactive FunctionThe interactive function enables dialogue between

machines and users or facilitates communication be-tween users. For example, an electronic commercesystem discovers users’ requirements through dia-logue with them and then provides information aboutproducts that best meet their preferences. The systemthen guides them to complete a purchase according totheir individual levels of computer literacy.

Another example is man-machine-man communi-cation. An interactive function inputs a message froman individual and transforms it so that a partner caneasily understand the message. If two people are us-ing different terminal devices, transformation tech-niques, such as video transcoders, web-page-layoutchanges, or text-to-speech synthesis, can be used to

Fig. 2 Ubiquitous information interface.

NEC Journal of Advanced Technology, Vol. 1, No. 3

Ubiquitous Information Interfaces

223

change the media format. In addition, spoken Japa-nese text can be translated into other languages suchas English and Chinese, facilitating internationalcommunication.

4. DESIGN OF A UBIQUITOUS INFORMATIONINTERFACE

A ubiquitous information interface effectively uti-lizes both real-world and digital types of information,making ubiquitous systems easier to use. Each ofthese information types has advantages and disad-vantages, as shown in Table I.

1) Intuitive usability: The most important feature ofreal-world information is that pieces of informationare presented via familiar media such as newspapersand sign boards and the information is intuitivelyrecognized at a glance. There is no need to learn howto use it.

2) Retrieval: Specific digital information can be foundusing information retrieval systems, which is a veryhelpful feature.

3) Amount of information: Unfortunately, not everypiece of real-world information is digitized. Some ef-fort is therefore required to digitize pieces of realworld information into a useful form.

4) Transformation: Even if a piece of digital informa-tion is not represented in the required form, it can betransformed into that form.

A ubiquitous information interface is like a natu-ral extension of current user interface techniques andshould be designed according to Shneiderman’s eightgolden rules for the design of user interfaces[3]. As aconsequence of the previous considerations, the fol-lowing three criteria should be added to these rules(Fig. 3).

(1) Real TimeInput and interactive functions should be designed

to respond quickly in situations where there is a lim-ited amount of digital information and to enhance theintuitive usability of real-world information. Even ifthe input function makes an error when a speechrecognition technique is being used, the operator willtry again as long as the system produces a quickresponse. But if the response is both slow and incor-rect, the operator will give up.

(2) AdaptabilityThe output function should be capable of trans-

forming image content depending on the size of thedisplay and network bandwidth. For improved usabil-ity, not only changes in image size, but also semantictransformation should be implemented based on im-age analysis, speech recognition, and automatictranslation technologies.

(3) PersonalizationTo enable anybody to use ubiquitous information

interfaces easily, they should be adaptable to indi-vidual skills and preferences. That is, an individualshould be able to use different terminal devices in thesame way.

Disadvan-tages

Difficult toretrieveTransforma-tion notpossible

Limitedamount ofinformation

Type

Real worldinformation

Digitalinformation

Examples

Books,newspapers,sign boards,speech,music, facialexpressions,behavior

Digitaldocuments,documentimages,pictures,music,Web pages

Advantages

Intuitiveusability

RetrievableEasilytransformed

Table I Advantages and disadvantages ofreal world information and digital infor-mation.

Fig. 3 Design criteria for ubiquitous infor-mation interfaces.

NEC J. of Adv. Tech., Summer 2004

Special Issue

224

If ubiquitous information interfaces are designedaccording to the eleven criteria described above, theywill overcome the chaotic state produced by the threevarious systems that exist at present and enable gen-eral use of ubiquitous systems. In the following sec-tion, five examples are described.

5. EXAMPLES OF UBIQUITOUS INFORMATIONINTERFACES

5.1 Text Input Interface Using a Camera with a Mo-bile Phone

In Japan, more than 90% of the mobile phonesshipped in the fourth quarter of the 2003 fiscal yearincluded cameras. These cameras are mostly used forvisual communication, but can also provide an inputfunction for ubiquitous information interfaces.

We have developed a new technology called a‘camera-typing interface’ that provides a way of in-putting characters printed on magazines or businesscards, such as URLs, using a low-resolution cam-era[4]. For example, if someone finds a URL withinan interesting magazine article, they can immedi-ately access the web page by shooting it using thecamera on their mobile phone. Similarly, if they seean interesting sign board, they can access digital in-formation relating to it by pushing a few buttons ontheir mobile phone.

Our technology has two main advantages in addi-tion to the capability of recognizing low-quality char-acters with small computational resources; one is au-tomatic concatenation of sequential shots, and theother is automatic error correction by re-shooting. Forexample, to input a long piece of text, the user shootsit bit by bit. The system concatenates the recognitionresults for the partial images by detecting overlapsbetween them (Fig. 4). If users discover recognitionerrors, they can retake the image around themisrecognized character. Then, the system changesthe relevant recognition result to repair the errorautomatically (Fig. 5).

In tests, after automatic correction was performedonce, the recognition rate increased significantly from97.1% to 99.0%. After repeated corrections, the recog-nition rate increased to 99.6%.

5.2 Text Input Interface Based on Speech Recogni-tion

A speech interface is an effective input function forubiquitous information interfaces.

To demonstrate the effectiveness of speech inter-faces, we developed an on-line manual retrieval sys-tem (Fig. 6) for cellular phones with Web-browsing

capability[5]. The system recognizes user’s naturallyspoken queries via a telephone speech recognizer andsearches an on-line manual with a retrieval moduleon a server. The results are then displayed on thescreen via a Web interface. Figure 7 also shows atraffic guide system with a speech interface based onVoIP (Voice over Internet Protocol), which transmitshigh-quality (wide band) speech, thus improving rec-ognition. These systems are based on integratingspeech interfaces and Web access technology andhave been shown to be effective in evaluation experi-ments.

We plan to improve the usability and performanceof the systems through field trials, and to exploretechnologies that will enable speech understandingand synthesis technologies to be applied to ubiquitousinformation interfaces.

5.3 Semantic Zoom Browser and JPEG2000 ImageBrowser for Adaptive Output Function

Currently, web pages are designed for specific ter-minal devices, mostly for PCs. When users access a

Fig. 4 Concatenation of recognition results.

Fig. 5 Correcting misrecognized results byreshooting.

NEC Journal of Advanced Technology, Vol. 1, No. 3

Ubiquitous Information Interfaces

225

web page via a mobile phone, they cannot actuallyview it. One reason for this is that the descriptionlanguage is unsuitable for the terminal. This problemcan be solved by translating description languagessuch as HTML and C-HTML. Another major reasonis that the assumed page size is different from theterminal display size. In this case, users cannot findthe information they need because they can only see avery small area of the whole page. Even more annoy-ing, although the display shows one small part of thepage, it takes a long time to download a large page.We developed two technologies to address these prob-lems: (1) a Semantic Zoom browser for Internetbrowsing using different types of terminals, and (2) afast document image browser based on JPEG2000image formatting.

(1) Semantic Zoom BrowserTo solve the former problem, we developed a Se-

mantic Zoom function that controls the degree of

reduction/enlargement locally as well as the charac-teristics of pointer movement based on the semanticattributes of the content displayed on the screen[6].This function is used in addition to the regular zoomfunction, which simply reduces and enlarges the im-ages on the screen.

Using the regular zoom function, it would be pos-sible to display the entire content of a page on a singlescreen, but the characters and other informationwould be impossible to read. Using the extended zoomfunction, however, the entire screen can be reduced,with titles and other important information main-tained at a larger, more readable size (Fig. 8). Theuser can thus view the content as a whole, even onTVs and other screens with low resolution, and at thesame time can understand the details by reading theheadlines. This function uses XML descriptions ofWeb content layout information (inclusive relations,etc.) to judge the importance of titles and other dis-play elements, enabling control of the degree of zoom

Fig. 6 A manual retrieval system.

Fig. 7 Traffic guide service on VoIP.

NEC J. of Adv. Tech., Summer 2004

Special Issue

226

for each individual display element.In the past, if users wanted to view a segment of

the content that fell outside of the main screen, theyhad to move the pointer to a small area on the screenand scroll to control the display position. This opera-tion was difficult, however, when using a device withlimited input capabilities, such as a remote control.With this newly developed function, the display posi-tion can be controlled by moving the pointer on thescreen directly to the desired position. At the sametime, by adjusting the pointer movement units instages using “semantic units,” for example, large, me-dium or small titles, the user can display the desiredsegment quickly and easily. This function uses thelayout information described above to control theunits of pointer movement.

(2) Fast Document Image BrowserA full-color A4 document image of 600 dpi (dots per

inch) consists of about 100 megabytes of data. Whenusers receive an image like this, it takes about tenseconds to see the page even using a 100Mbps net-work. This means that they cannot leaf through thepages of a magazine. We therefore developed a fastdocument image browser using the JPEG2000 imageformat, which provides hierarchical image descrip-tions using seven levels of resolution from low tohigh[7] (Fig. 9). Users at a terminal with a QVGA-size display do not receive an image with 2,000 ×3,000 dots; instead, the system sends the image to theterminal at the lowest resolution. Then it sends im-ages at progressively higher resolutions. Users can go

to the next page as soon as they want to skip thecurrent page. If they want to magnify an image, thesystem will send that part at a higher resolution sothat it fits the display. For users at terminals withdifferent sizes of display, the system operatesadaptively to the display size. This system enablesusers to view 10 full-color pages of a magazine persecond via 384kbps communication channels.

We also developed image analysis technology thatrecognizes the content of document images. It ex-tracts information about document layout, such astext lines, headlines, figures, and photos and alsoestimates the reading order of the text. The systemsends images of text lines according to the readingorder, enabling users to read them as quickly as pos-sible. It can also send only photos to reduce the trans-mission time if users are looking for photos.

5.4 Automatic Speech Translation for InteractiveFunctions

We developed an automatic speech translation sys-tem for PDAs (Personal Digital Assistants) to assistoral communication between Japanese and Englishspeakers in various situations while traveling (Fig.10)[8]. This compact system requires only a smallmemory to immediately translate speech input inJapanese into English or vice versa and to read aloudthe result.

As shown in Fig. 11, the system consists of speechrecognition, translation and speech synthesis mod-ules. The speech recognition module has a large-scalevocabulary of tens of thousands of words and

Fig. 8 Semantic zoom browser.

NEC Journal of Advanced Technology, Vol. 1, No. 3

Ubiquitous Information Interfaces

227

recognizes a variety of conversational expressionsspoken by travelers. The translation module has alarge-scale grammar dictionary and translates vari-ous travel-related expressions including colloquialidioms and polite expressions.

The performance of the new system has been aidedby the development of a number of functions includ-ing a high-speed large-vocabulary continuous speechrecognition engine that adjusts its operation in accor-dance with the processor and memory capacity used.

We are planning to continue developing automaticspeech translation technology with improved perfor-mance and enhanced functions to provide users with

the ideal translation support solution. We also plan toattempt to increase the number of languages that canbe translated.

6. CONCLUSION

In this paper, we described the design of a ubiqui-tous information interface with properties of realtime, adaptability, and personalization. To date, wehave developed component technologies suitable forpractical application. For input functions, we havedeveloped interfaces using character and speech rec-ognition, which provide hyperlinks between the realworld and digital information. Users can easily accessrich information relating to their interests. For out-put functions, adaptability is essential because differ-ent types of terminals have various-sized displays.

Fig. 9 Progressive transmission of document images.

Fig. 10 Speech translation system on a PDA.

Fig. 11 Overview of speech translation sys-tem.

NEC J. of Adv. Tech., Summer 2004

Special Issue

228

* * * * * * * * * * * * * * *

Received July 26, 2004

* * * * * * * * * * * * * * *

Keiji YAMADA received his B.E., M.E. andDr.E. degrees in Information Science fromKyoto University in 1982, 1984 and 1987, re-spectively. He joined NEC Corporation in 1987and is currently the general Manager of Mediaand Information Research Laboratories.

Dr. Yamada is a member of the IEEE, the IEICE (Instituteof Electronics, Information and Communication Engineer),and IPSJ (Information Processing Society of Japan). He re-ceived Sakai memorial Awards from IPSJ and some otherawards.

Shuji SENDA received his B.E., M.E. and D.E.degrees in information science from KyotoUniversity in 1991, 1993 and 1996, respec-tively. He joined NEC Corporation in 1996,and is now Assistant Manager of UbiquitousSystem Technology Group, Internet Systems

Research Laboratories. He is engaged in the research anddevelopment of character recognition and its application.

Dr. Senda is a member of the Institute of Electronics, Infor-mation and Communication Engineers, and the InformationProcessing Society of Japan.

Toshiyuki ASAHI is a principal researcher atInternet Systems Research Laboratories, NECCorporation. His research interests include us-ability engineering, information visualizationand adaptive user interface. He received theM.E. degree from Osaka University in 1984

and the Ph.D. degree from Nara Institute of Science and Tech-nology in 1997.

Dr. Asahi is a member of the Human Interface Society,Information Processing Society of Japan, and Usability Profes-sionals Association.

Toshihiko HIROAKI received his M.S. degreein material engineering from Tsukuba Univer-sity in 1990. He joined NEC Corporation in1990, and is now Principal Researcher of theInternet Systems Research Laboratories. He isengaged in the research and development of

ubiquitous multimedia information systems and user inter-face.

Mr. Hiroaki is a member of the IPSJ.

Hiroaki HATTORI received his B.E. and M.E.degrees from Hokkaido University, in 1983and 1985, respectively. He joined NEC Corpo-ration in 1985, and is now senior manager ofSpeech Interface Technology Group, Mediaand Information Research Laboratories. He

received Dr. Eng. degree from Hokkaido University in 1994.Dr. Hattori is a member of the Institute of Electronics,

Information and Communications Engineers, and the Acousti-cal Society of Japan.

Akitoshi OKUMURA received his B.E. andM.E. degrees from Kyoto University, Kyoto,Japan, in 1984 and 1986, respectively. Hejoined NEC Corporation in 1986, and is cur-rently researching human language technol-ogy at the Media and Information Research

Laboratories as a Senior Manager. He received his Ph.D. fromDepartment of Computer Science, Tokyo Institute of Technol-ogy in 1999. His current research interests include speechtranslation, translingual information management, informa-tion extraction and summarization.

The system adjusts the content to suit the terminal.Media transcoding such as text-to-speech and text-to-animation will become more important. We also de-scribed an automatic interpreter which provides aninteractive function. In future, we will develop a uni-versal interpreter and a robot to engage in naturaldialogue with people.

REFERENCES

[1] M. Weiser, “Some Computer Science Problems in Ubiqui-tous Computing,” Communications of the ACM, July 1993.

[2] D. A. Norman, “Invisible Computer -Why good productscan ail, the personal computer is so complex, and informa-tion appliances are the solution,” The MIT Press, 1998.

[3] B. Shneiderman, “Designing the User Interface,” AddisonWesley, Third Edition, 1998.

[4] S. Senda, et al., “Camera-Typing Interface for UbiquitousInformation Services,” Proc. of IEEE PerCom 2004,pp.366-370, 2004.

[5] S. Ishikawa, et al., “Speech-Activated Text Retrieval Sys-tem For Multimodal Cellular Phones,” Proc. of ICASSP2004, 1, pp.453-456.

[6] Y. Tatsumi, H. Noda and T. Asahi, “Pointer Zooming:Pointing Method under Limited Input and Output Envi-ronment,” The 10th Workshop on Interactive Systems andSoftware (WISS2002), pp.67-72, 2002.

[7] S. Wang, et al., “Adaptive Data Transmission on Browsingof Scanned Documents Using JPEG2000,” Proc. of IEEEInternational Workshop on Knowledge Media Networking(KMN’02), 2002.

[8] R. Isotani, et al., “An Automatic Speech Translation Sys-tem on PDAs for Travel Conversation,” Proc. ICMI’02,pp.211-216, Oct. 2002.