Reinventing Printed DocumentKeeping digital as digital with uPen

Reinventing Printed DocumentKeeping digital as digital with uPen

Jian WangSenior ResearcherMicrosoft Research AsiaJuly 19, 2005

Jian WangSenior ResearcherMicrosoft Research AsiaJuly 19, 2005

OutlineOutline

Challenges and Approach

TechnologiesEmbedded Interaction Code(EIC) system

Computer vision system

HW/SW architectureHardware prototype design

EIC SDK: system design and APIs

Demos

Challenges and Approach

TechnologiesEmbedded Interaction Code(EIC) system

Computer vision system

HW/SW architectureHardware prototype design

EIC SDK: system design and APIs

Demos

ChallengesChallenges

``non-digital way of doing things'' like paper and pencils

PCs still create non-digital documentSuch as printed document

Keyboard/mouse confine people to desktop

Mobile interaction device

``non-digital way of doing things'' like paper and pencils

PCs still create non-digital documentSuch as printed document

Keyboard/mouse confine people to desktop

Mobile interaction device

GoalGoal

Digital original

Keep digital as digital

Makes every printed document digital

Brings digital computing to places where things are still being done in non-digital ways

Digital original

Keep digital as digital

Makes every printed document digital

Brings digital computing to places where things are still being done in non-digital ways

Approach: EIC SystemApproach: EIC System

High data-embedding capacity to provide a gigantic address space up to 2192 bits.

A family of EIC visual patterns to adapt to various hardware implementations, applications and surfaces, and to meet the needs for usability and readability.

Multiple data-embedding channels to embed x and y coordinates, and both global (such as document ID) and local metadata in a digital document.

High data-embedding capacity to provide a gigantic address space up to 2192 bits.

A family of EIC visual patterns to adapt to various hardware implementations, applications and surfaces, and to meet the needs for usability and readability.

Multiple data-embedding channels to embed x and y coordinates, and both global (such as document ID) and local metadata in a digital document.

10

0 1 0 1 1 0

0 0 1 1 1 0

0 1 0 0 1 1

0 1 0 0 1 0

1 0 1 1 1 0

EIC Pattern

EIC: Absolute X, Y Coordinates

A pseudo-random 1D array of 0’s and 1’s with “window” property

An example with primitive polynomial of order 6:

000001000011000101001111010001110010010110111011001101010111111

h(x)=x6+x+1, code length = 26 - 1=63

The window property ensures the unique decodability

Window Property of m-sequence

0

m1 = 9

m2=7

Construction of m-array

0

0

1

0

1

1

1

1

0

0

1

0

1

0

0

1

1

0

0

1

1

1

0

1

0

0

1

1

1

1

1

0

1

0

0

0

0

1

1

0

1

0

0

1

0

1

1

0

0

0

0

0

0

1

0

1

1

1

1

0

1

1

000001000011000101001111010001110010010110111011001101010111111

Create a 2D array with size of m1*m2, where gcd(m1,m2) =1

Folding the sequence

The 2D array retains the window property. If L is the bit location in m-sequence, then its location in m-array:

x= L mod m1

y= L mod m2

Create a 2D array with size of m1*m2, where gcd(m1,m2) =1

Folding the sequence

The 2D array retains the window property. If L is the bit location in m-sequence, then its location in m-array:

x= L mod m1

y= L mod m2

Data-embeddingData-embedding

X, Y Position

67%

Encoded X, Y Position

Three Channels of Embedded-Data

Encoded Global Metadata

25%

Global Metadata(Document ID)

Encoded Local Metadata

Local Metadata

8%

EIC pattern

Decodable areaEIC + Encoded

content

Non-decodable area

Bounding box

Action

Biding EIC with DocumentBiding EIC with Document

Hardware ArchitectureHardware Architecture

MechanicalSize: 14x17x160mm

Ink cartage

Pressure sensor

OpticalCamera

Lens

IR LED

ElectricalProcessor/Memory

Bluetooth

Battery

MechanicalSize: 14x17x160mm

Ink cartage

Pressure sensor

OpticalCamera

Lens

IR LED

ElectricalProcessor/Memory

Bluetooth

Battery

Computer Vision SystemComputer Vision System

To obtain high resolution (> 400 dpi), accurate x and y positions and metadata on printed digital documents (papers).

To obtain actual x and y position of the pen tip regardless of pen posture (tilting and rotating). Calibration error < 0.15mm.

To obtain sufficient spatial and temporal resolution (> 400 dpi x 120 Hz) of uPen digital ink (so that it’s compatible with the resolution of Tablet PC ink).

To obtain high resolution (> 400 dpi), accurate x and y positions and metadata on printed digital documents (papers).

To obtain actual x and y position of the pen tip regardless of pen posture (tilting and rotating). Calibration error < 0.15mm.

To obtain sufficient spatial and temporal resolution (> 400 dpi x 120 Hz) of uPen digital ink (so that it’s compatible with the resolution of Tablet PC ink).

Document/EIC pattern segmentation

.32,,1,

else. ,

);,(),( and ),(),( if ,

);,(),( if ,

),(

ji

Z

jijiPjijiPA

jijiPT

jiLabel .32,,1,

else. ,

);,(),( and ),(),( if ,

);,(),( if ,

),(

ji

Z

jijiPjijiPA

jijiPT

jiLabel T: document content

A: EIC pattern

Z: background area

),( ji adaptive threshold of EIC pattern at (i, j)

),( ji adaptive threshold of document content at (i, j)

31 1

,,13

21 1

1,,12

11 1

1,,1

01 1

,,0

/)||(

/)||(

/)||(

/)||(

NYXQ

NYXQ

NYXQ

NYXQ

i j

i j

i j

i j

n

i

n

jjiji

n

i

n

jjiji

n

i

n

jjiji

n

i

n

jjiji

nYXSW

nYXSB

kl

i

il

j

jk

klji

kl

i

il

j

jk

klji

/)),max((

/)),min((

,

1

1

1

1

,,

,

1

1

1

1

,,

EIC Pattern Processing

kkkkk

ijj

n

ikkkiik

YXXXb

bxyjSibxyJk

k

k

kk

'1'

1

2 }|||{,)(1

}),(,310,31,,1,0

,|),{(

)(

min

,,

,

,,

4,..,4

,,

TyxLabelyx

tktbxayyxP

N

PGray

t

iiiski

ki

Pski

ki

ski

}),(,310,31,,1,0

,|),{(

)(

min

,,

,

,,

4,..,4

,,

TyxLabelyx

tktbxayyxP

N

PGray

t

iiiski

ki

Pski

ki

ski

Determine EIC pattern grid

Determine orientation of pattern

Determine the value of each bits

Efficient Image MatchingEfficient Image Matching

}),mod(,),mod(

],,[],,[|{

),(max

,,

0000,

,

,

yoffsetbarxoffsetbar

nm

jidocumentcamera

SP

ILnILm

dyydyyndxxdxxmPS

IICorrPji

}),mod(,),mod(

],,[],,[|{

),(max

,,

0000,

,

,

yoffsetbarxoffsetbar

nm

jidocumentcamera

SP

ILnILm

dyydyyndxxdxxmPS

IICorrPji

:search radius.dydx,

:Maximum grid alignment error.

CameraCamera IFI F : transformation matrix from camera coordinate to document image coordinate

EIC SDK ArchitectureEIC SDK Architecture

uPen prototype/third-party hardwareuPen prototype/third-party hardware ApplicationsApplications

EIC Embedded SDKEIC Embedded SDK

EIC SDK EIC SDK

uPen FirmwareuPen Firmware

EICPattern

EICPattern MetadataMetadata

EICDocument

EICDocument.Net Compact Framework.Net Compact Framework

DemosDemos

Thank youThank you