Bridging the Paper-Digital Divide: The iPaper Interactive Paper Framework

USI Seminar, May 30, 2007

Bridging the Paper-Digital Divide:

The iPaper Interactive Paper Framework

Beat Signer

Institute for Information Systems, ETH Zurich

8092 Zurich, Switzerland

http://vub.academia.edu/BeatSigner

USI Seminar, May 30, 2007 Beat Signer, [email protected] 2

The Paperless Office (1975)


The "Paperless Office" (2003)


The Myth of the Paperless Office

For decades, people have predicted the

office of the future as a paperless office

documents generated, published and

distributed electronically

documents read electronically

What has happened to this

imminent revolution?


Affordances of Paper

The physical properties of an object determine

how people use that object

Properties of paper

light, flexible, robust, porous, opaque, transparent, …

Human actions

grasping, folding, tearing, carrying, writing on, …

Paper supports forms of collaboration and

interaction difficult to mimic in the digital world

USI Seminar, May 30, 2007 Beat Signer, [email protected]

E-Paper versus Digitally Augmented Paper

Electronic Paper (E-Paper)

Digitally Augmented Paper


Outline

Motivation

Interactive paper framework (iPaper)

Cross-media link server (iServer)

Interactive paper applications

functionality

authoring

Future work


Augmented Paper Projects

Digital Desk, Wellner, 1991 Listen Reader, Back et al., 2001

Audio Notebook,

Stifelman et al., 2001

Books with Voices

Klemmer et al., 2003 ButterflyNet, Yeh et al., 2005


Digital Pen and Paper

Developed by Anoto

Different pen manufacturers

Logitech, Maxel, Nokia, ...

Pattern space

60 million km2


Problems of Existing Applications

Many projects focus on the input device, paper,

printing and other hardware technologies rather

than on the data integration and information

management aspects isolated solutions

The linking of paper tends to be based on

physical rather than information-centric concepts

difficult to integrate new input / output devices

10


Our Approach

General interactive framework

different forms of paper-based interactions (enhanced

reading, enhanced writing, …)

rapid prototyping as well as large scale publishing

integration of new types of resources and services

integration of new input devices

The key to a highly integrated interactive paper

solution lies in the introduction of a platform for

general cross-media information management

11


Active Areas and Virtual Page Layers

image

video clip


iPaper Architecture


iServer Metamodel and Resource Plug-ins

iPaper

Shape Page

Application Database

DB ObjectData/Metadata

Query

Movie Clips

Time Span /

Spatial InfoMovie Clip

HTML Documents

XPointer XHTML

Document

Selectors

Entities

Resources

link


iServer

layer

|Layers|

layer

Active

Layers

resource

Containers

activeComponent

Active Com-

ponents

medium

Media

medium

Movies

medium

Images

medium

Webpages

OnLayer

Contains

(1,1)

(1,1) (0,*) (0,*)

document

Documents(0,*)

shape

Shapes

partition (2,*)

(0,*)

rectangle

Rectangles

ellipse

Ellipses

polygon

Polygons

circle

Circles

complexShape

Complex

Shapes

HasShapes

page

PagesOnPage

Contains

Pages

(0,*)

(1,1)

(1,1)

(0,*)

Resources

parameter

Preferences

link

Links

user

Users

group

Groups

individual

Individuals

HasMembers

HasTarget

(0,*)

(0,*)(0,*)

(0,*)

(0,*)

(0,*) (0,*)

(1,*)

(0,*)

(0,*)

(1,1)

(0,*)

partitionpartition

(1,*)

CreatedBy

RefersTo

Has

Preferences

parameter

Properties

selector

Selectors

resource

Resources

HasSource

(0,*)

AccessibleTo

InaccessibleTo

(0,*)

(0,*)

entity

EntitiesHasProperties

Users

Plug-ins

Links

(0,*)

Layers

iPAPER

iSERVER


Active Content

Execute code (services) on the client and server side

Active Component

A

Active Component

B

Active Component

C

iS

erv

er

AC Manager

Server

R

eq

ue

st S

en

de

rActive Component

Stub B

AC Names

ACS Manager

C

lien

t D

evic

e

AC Names

Client

HT

TP


Active Components

Button

Slider

Slider2D

DateSelector

Browser

MapSelector

Rating

Capture

CaptureAndSpeak

Image

Sound

Movie

SQLClient

XCMRequest

ClientCtrlRequest

PaperPointControl

PaperPointAnnot

...


Active Components …

<?xml version="1.0" encoding="UTF-8" ?>

<iserver>

<activeComponent id="s1a" creator="beat">

<name>Show slide 1</name>

<properties>

<parameter>

<key>org.ximtec.iserver.ac:command</key>

<value>showSlide</value>

</parameter>

<parameter>

<key>org.ximtec.iserver.ac:slide</key>

<value>1</value>

</parameter>

</properties>

<identifier>PAPERPOINT_CONTROL</identifier>

</activeComponent>

...

</iserver>

PAPERPOINT_CONTROL

Show slide 1

command showSlide

slide 1


PaperPoint Presentation Tool


XML-Based Authoring

...

<document id="handout" creator="beat">

<name>Slide Handout</name>

<id>h1</id>

<size>

<width>210</width>

<height>297</height>

</size>

<content>handout.pdf</content>

</document>

<page id="p1" creator="beat" document="h1">

<name>Page 1, Slide Controller</name>

<number>1</number>

</page>

...

<rectangle id="fs" creator="beat"

layer="default" resource="p1">

<name>First Slide</name>

<upperLeft>

<point><x>24</x><y>12</y></point>

</upperLeft>

<size>

<width>25</width><height>8</height>

</size>

</rectangle>

...

...

<activeComponent id="fsa" creator="beat">

<name>First slide action</name>

<properties>

<parameter>

<key>org.ximtec.iserver.ac:command</key>

<value>firstSlide</value>

</parameter>

</properties>

<identifier>PAPERPOINT_CONTROL</identifier>

</activeComponent>

...

<link id="fsl" creator="beat" sources="fs"

targets="fsa">

<name>Show the first slide</name>

</link>

...


Digital Link Authoring Tool


PaperPoint Architecture


Edinburgh Fringe Festival

World's largest

international arts festival

4 weeks

~1700 events

~27000 performances


EdFest Components


EdFest Booklet


Content Publishing Framework

XCMcontent publishing

XML data

of festival

venues

and events

iServercross-media link server

• paper link data

• active areas on paper

• layers

• digital resources

• user modelimport

publish

PDF

XML link

definition

data

iPublishpublishing framework

Structure Style

XSL CSS


Print-n-Link

Citations build the basis

for a scientific web of

publications

Use digital services to

search / retrieve cited

publications

Automatic authoring


Print-n-Link Architecture

Display

Printer

Driver

Document DB

iPAPER

iDOC

Publish

Document

Pattern

Semantic Mapper

Citation

Analyser

Citation

Retrieval

Document

Lookup

Citation DB

Print-n-Link Engine

iSERVER

Active Content

Link DB

iPaper Plug-in

Publish

Positions and

Active Content

Definition

Publish

Citations

Desktop

Print

(x,y)

Coordinates

Text-to-

Speech

Web Search

Services

iPaper

Client

Print-n-Link: Weaving the Paper Web

Moira C. NorrieDept. of Computer Science

ETH Zurich8092 Zurich, Switzerland

[email protected]

Beat SignerDept. of Computer Science


[email protected]

Nadir WeibelDept. of Computer Science


[email protected]

ABSTRACT

Citations form the basis for a web of scientific publications.Search engines, embedded hyperlinks and digital librariesall simplify the task of finding publications of interest onthe web and navigating to cited publications or web sites.

However the actual reading of publications often takes placeon paper and frequently on the move. We present a systemPrint-n-Link that uses technologies for interactive paper toenhance the reading process by enabling users to access dig-ital information and/or searches for cited documents from a

printed version of a publication using a digital pen for inter-action. A special virtual printer driver automatically gener-ates links from paper to digital services during the printingprocess based on an analysis of PDF documents. Depend-ing on the user setting and interaction gesture, the systemmay retrieve metadata about the citation and inform the

user through an audio channel or directly display the citeddocument on the user’s screen.

Categories and Subject Descriptors

D.2.11 [ Software Engineering ]: Software Architectures;H.4.m [ Information Systems Applications ]: Miscella-neous; H.5.m [ Information Interfaces and Presenta-tion ]: Miscellaneous

General Terms

Design, Algorithms, Experimentation

Keywords

Interactive paper, document integration, digital library, ci-tation management

1. INTRODUCTIONNowadays we are all familiar with the notion of hyperlinks

as a means of creating interconnected document spaces thatcan be easily searched and navigated with the help of search

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.DocEng’06, October 10–13, 2006, Amsterdam, The Netherlands.Copyright 2006 ACM 1-59593-515-0/06/0010 ...$5.00.

engines and web browsers. The ease with which we cannot only find documents on the web, but also follow linksto related documents is considered to be one of the mainadvantages of working with digital documents as opposedto paper equivalents. Yet many studies have shown thatpaper is still the preferred med ium when it comes to reading

documents for reasons of comfort, mobility and also ease ofannotation.

In this paper, we show how emerging technologies for in-teractive paper enable a bridge to be built between paperand digital document spaces that allows users to travel backand forth between the two using the medium most appropri-

ate to the current task and sit uation. Users can print doc-uments for reading and, by activating hyperlinks on paperwith a digital pen, they can not only access digital informa-tion about related documents, but also initiate web searchesto retrieve copies of those documents. The retrieved docu-

ments may in turn be printed for reading with automaticallygenerated embedded hyperlinks on the paper. Mobility issupported by providing disconnected users on the move withaudio information about related documents and the optionof adding them to a request list to be processed when recon-nected.

As an application, we take the everyday academic task ofreading scientific publications. Whether the reading activ-ity is for the purpose of knowledge discovery or reviewing,readers regularly look up citations to other publications byflicking to the reference list at the back of the paper. In somecases, a search for the cited publication may follow and, if

found, often it is printed and added to a pile of documents tobe read. Citations are the established meansof providing thelinks that form the basis of the scientific web of publicationsthat spans the digital and paper worlds. Just as citationswithin digital documents can be used to automatically gen-erate hyperlinks to cited documents, we can automatically

generate links within paper documents through an analysisof the PDF file when the document is printed.

Section 2 presents a more detailed look at the activityof reading scientific publications and a discussion of relatedwork. Section 3 describes the digital pen and paper tech-

nologies and how these can be used to support interactionbetween paper and digital services. We then present thegeneral functionality of our enhanced reading system in Sec-tion 4 and the overall architecture in Section 5. Details ofthe virtual printer component used to analyse PDF docu-ments for automatic link generation as well as printing are

given in Section 6. Concluding remarks follow in Section 7.

Interactive Paper Architecture

PrintingSemantic Mapper

Document Retrieval


Interactive Paper Applications

Enhanced Reading

interactive paper maps

interactive festival brochures

educational materials

auction brochures

scientific publications

…

Enhanced Writing

photo album

researcher's notebook

mammography

search system

…

Paper-Based Interfaces

PaperPoint

digital libraries

video analysis tool

educational games

…

Artists' Installations

Lost Cosmonaut

Generosa Enterprise

Interactive TableTops


Current Projects

PaperWorks (FP6-516895)

Interweaving Paper and Digital Documents

King's College London, Acreo AB, Anoto, ETH Zurich,

Arjowiggins SAS, Brunel University, Malmo

University, Pearson Education

DELOS

Network of Excellence on Digital Libraries

Paper-based user interfaces for image retrieval

(collaboration with Heiko Schuldt, University of Basel)


Future Work

Design guidelines for interactive paper interfaces

functionalty (active components) and design

Cross-media authoring and publishing

analysis of existing publishing processes (Pearson)

general document model for mapping between printed

and digital document instances

Extensions of iServer

collaborative cross-media information sharing

new resource plug-ins


Mapping between Printed and Digital

Instancesstatic physical rendering

dynamic logical document


Conclusions

Interactive paper framework (iPaper)

information-centric approach supports different

technologies, functionalities and applications

rapid prototyping (reusable active components)

Extensible platform for cross-media information

management (iServer)

Cross-media authoring / publishing tools

manual, content-driven and automatic link authoring

Technology

Bridging the Paper-Digital Divide: The iPaper Interactive Paper Framework