1

Optimal Halftoning for Tactile ImagingAmit Nayak,Student Member, IEEEand Kenneth E. Barner,Senior Member,

IEEE

Department of Electrical and Computer EngineeringUniversity of Delaware, Newark, Delaware 19716, USA

nayak@ece.udel.edu, barner@ece.udel.edu

Abstract

Reading of text and understanding images by touch is an important alternative and additional source of information

when sight is absent or lost. Tactile graphics and models such as edge maps, binary output etc. are the solution for

simple access to images for blind persons. This paper introduces an approach to model the human tactile system based

on the responses produced by stimuli on micro-capsule paper. This system is utilized for the purpose of generating

optimum halftone patterns on micro-capsule paper that can be utilized for the effective generation of tactile graphics.

I. INTRODUCTION

Vision plays a pivotal role in information gathering for human beings. For instance, much of

the sciences involve the visualization of materials and concepts. Valuable information in text and

images is thus gathered by humans through the visual sense. This reliance on vision is hampered,

or lost, in individuals with visual impairments. In such cases, it is necessary for the information

display to include alternative modalities.

The main system of written communication utilized by blind people throughout the world is the

Braille system. Invented by Louise Braille in the early nineteenth century, this system proved to

be better than the earlier, difficult and cumbersome, method of embossed letter reading. The use

of Braille method for reading and writing, has been researched at depth for its utility by various

researchers [1]. This method, however, allows tactile access to only text based information.

Tactile imaging is the process of converting a visual image into a touchable binary raised ver-

sion. Applications of tactile graphics have been researched on in several places. Tactile pattern

generation received a boost with the invention of Optacon and the subsequent modification [2],

2

[3]. Loomis [4] and Craig [5] initiated a discussion on the tactile pattern perception and the fac-

tors which could affect this perception. One of the earliest groups to look in to the sensory aspects

of skin in the human finger, Johnsonet al. [6], [7], [8] worked on the the discrimination and

detection of points, gaps, gratings and letters by the human finger. Martinet al. [9] have worked

in the presentation of business graphics for blind people. Pie charts, bar charts and other form

of representation of information in image format have been successfully converted into relevant

form. The data on which the graphical information is based is extracted from the image and then

the tabulated version of this data is conveyed to the user either by speech, sound and/or tactile

form. Speech-synthesizer, using acoustically different frequency tones are some ways and in the

case of bar charts the most favored way is using a 2D-Braille-Display, wherein raised dots are used

to depict the length of the actual bar in the y-axis and the x-axis is coupled as a Braille output.

Klatzky et al. [10], in 1985, discussed the identification of objects by touch and have subse-

quently compared the amount of information garnered by the visual and the haptic interface [11]

in 1987. Similarly the same time investigators had attempted to quantify the information channel

capacity of the human sensory organs and Kokjer [12] suggested an order-of-magnitude upper

limit on this property for the vibrotactile stimulation of human fingertip as 100 bit/s. This, when

compared with corresponding limits in the human ear and eye, a progression of102:104:106 re-

spectively, was found. Thus the capacity of visual sense to receive and perceive information is the

highest, as compared to hearing and touch.

Of the various studies and research done in the field of generation of tactile graphics, of particu-

lar interest are that of [13], [14], which successfully developed software algorithms for automatic

generation of tactile graphics. Meaningful information is identified and extracted from original

image data incorporating a multi-step procedure. These include edge and boundary information

extraction from the original image and representing them in tactile format. Edge and boundary

information refers to identification of regions in an image where a rapid change of intensity takes

place. Also important in the procedures are the successful segmentation of the original image.

These image processing methods were further developed by Hernandezet al. [15], [16], [17] who

3

used the edge information to generate graphics that rendered image boundary in tactile form.

Edge information on its own is, however, not enough to present the complete effect. A tactile

image should contain different textures [18] to help distinguish and discriminate between various

shapes and colors present in an image. We, therefore, propose optimal tactile imaging textures.

The underlying idea is that the various regions in segmented images can now be easily identified

by touch if they are given a separate textures. To accomplish this, we use digital halftoning methods

to create patterns for the images which can then be suitably enhanced to create the feel of various

textures.

Digital Halftoning is the process of generating a binary pixel patterns that create the illusion of

a continuous-tone image. This technique is important for display of gray-scale images in printing

process, where the direct rendition of the gray tones is not possible. In recent years a number

of comprehensive methods have been designed to carry out the halftoning process. The primary

methods are reviewed in [19], [20]. Optimum algorithms [21], [22] were found to be those that

exploit the dot-printer models and take into account the low pass nature of the human vision.

We extend this idea of model-based halftoning to the domain of the human tactile system. As

used extensively in the human visual system (HVS) modeled halftoning system, a transfer function

for the sensory system is required along with an understanding of the printing process. In an anal-

ogous fashion, we develop a model for the human tactile system and the tactile printing process.

In the case considered, we utilize the Tactile Image Enhancer, Fig. 1, operating on microcapsule

paper [13], [14], [23] as the tactile output. The output characteristics of the TIE were examined.

We develop a model for the surface generated from the TIE. We also devise a model of the human

tactile perception based on the spatial frequency response of the human tactile sense. These models

are integrated into the process of generating halftones best suited for a given gray scale.

The remainder of this paper is organized as follows. The working of the TIE and the measure-

ments of the output characteristics and resulting enhancement model are presented in Section 2. In

Section 3, we discuss the work carried out earlier on the characterization and specification of the

human finger pad. New experiments are carried out to find how sensitive the human finger is to

4

Fig. 1. Tactile Image Enhancer (TIE is manufactured by Repro-tronics Inc.).

the surface of the tactile paper, the results of which are reported in Section 4. Optimal halftoning

methods are discussed in Section 5. Finally, the conclusions and further research are presented in

Section 6.

II. I MAGE ENHANCEMENT PROCESS

Tactile images can be formed by enhancing, or raising, the surface of a special paper called

tactile or micro-capsule paper through use of the TIE, Fig. 1. The TIE is manufactured by the

Repro-tronics Inc. The first prototype was introduced in the market in 1993 and have been in use

since then. As of now there is a smaller, more portable version of TIE available in the market. The

TIE works on a very simple principle. It has a motor-driven roller that passes the tactile paper, face

up, underneath a tubular light bulb, Fig. 2. The light bulb acts as a heat source and the rollers help

in guiding the paper through the direct contact of the bulb.

The type of tactile paper which we intend to focus on is manufactured by the Matsumoto Kosan

Company Ltd. in Japan. This is also known as the micro-capsule paper. As the name suggests it

is a white paper coated with microscopic polystyrene capsules. The Micro-capsule coating on the

paper is heat reactive. Black lines or images printed on the paper, when exposed to a heat source,

absorb more heat than the surrounding areas. This causes the underlying capsules to grow. The

result is the generation of raised lines, areas, and symbols,i.e., tactile graphics. A temperature

range of 120-125�C (248-257�F) is appropriate for the heating of the papers. In the present study

5

Image (printed toner particles)

Polystyrene Microcapsules Polyethylene medium

(a)Heating element

Paper path

Transport rollers

(b)Expanded Capsules

(c)

Fig. 2. (a) Micro-capsule paper with image toner particles photocopied on it.(b) Inside the TIE.(c) Raised micro-capsules after the paper is passed through the TIE

we have preferred to use 124�C(255.2�F).

In order to understand and model the changes occurring in the enhancement process, a simple

experiment was carried out. An array of test samples of dots were used. The dot were square

pixels printed on a white paper at a resolution of 100 dots per inch (dpi). The size of the dots was

uniformly increased from a1 � 1 to a20 � 20 dot cluster. The resulting outputs were examined

for their change in dimension. Images were prepared, comprising of black square dots on a white

background and black dots with square gaps simulating white square dots on black background,

Fig. 3. These images were photocopied onto the tactile paper. The tactile paper was then passed

through the TIE.

6

(a) (b)

Fig. 3. (a) Black Dots on white background. (b) White dots on black background.

The structures such generated through this test have profiles that range between -0.5 and 0.5

mm in height. A Scanning Electron Microscope(SEM) was used to view the vertical growth of the

samples. The profiles of the samples were captured and recorded. The profile of one of the black

dot on white background, with a dot size of8 � 8, is shown in Fig 4. Simple image processing

techniques were used to extract the edge information and these were superimposed to see the

growth of the dot clusters as a function of cluster size.

Fig. 5 shows one-dimensional profiles for squares of sizes 1� 1, 2� 2,... etc which are printed at

a resolution of 100 dpi. The inverse case, which we can observe as white dots in black background,

follow a complementary pattern. The growth in both directions were recorded. The average height

or depth of the dot profiles were found to follow the curves shown in Fig. 6.

The clusters of dots are characterized by the way the height/depth and width of the dots behave

after passing through the TIE. The curves of the profiles are modeled as follows:

Fu = Au +Buln(i) (1)

Fd = Ad(e�(i�Bd)

2=Cd � 1) (2)

whereFu andFd denote the height and depth parameters respectively, and the constants were

determined experimentally to beAu=0.026,Bu = 0:16 andAd = 0:5,Bd = 0:75, Cd = 1:596 and

i denotes the size of the cluster of dots. Thus, the height and depth parameter of the dot clusters

7

Fig. 4. The Test pattern corresponding to the 8 x 8 pattern as seen through the SEM.

−0.08 −0.06 −0.04 −0.02 0 0.02 0.04 0.06

0

0.005

0.01

0.015

0.02

0.025

Width of clusters of dots (inches)

Hei

ght (

inch

es)

(a)

−0.04 −0.02 0 0.02 0.04−0.025

−0.02

−0.015

−0.01

−0.005

0

0.005

Width of clusters of dots (inches)

De

pth

(in

che

s)

(b)

Fig. 5. Edge variation for the (a) black dots on a white background, (b) white dots on a black background.

have been modeled either as exponential or logarithmic curves.

8

0 1 2 3 4 50

0.1

0.2

0.3

0.4

0.5

Width of dots (mm)

Hei

ght o

f dot

s (m

m)

Experimental dataLogarithmic curve fit

(a)

1 2 3 4 5

−0.5

−0.4

−0.3

−0.2

−0.1

0

Width of dots (mm)

De

pth

of

do

ts (

mm

)

Curve fitExperimental data

(b)

Fig. 6. (a) Height variation of black dots on white background. (b) Depth variation of white dots on black background.

Consider next the profiles of the dot clusters. On a simple suitable observation, we find that the

profile of a single pixel is approximated by a Gaussian surface as shown in the Fig. 7:

G1;1 = Ae�x2=B (3)

where the constants were determined experimentally to beA = 0:00134, B = 0:02752 andx

represents the distance along the width. Larger clusters are modeled as linear combinations of

dots. Thus an� n cluster is modeled through an amplitude adjusted convolution model:

Fn;n = n;n(G1;1 In;n) (4)

9

−0.015 −0.01 −0.005 0 0.005 0.01 0.0150

5

10

15x 10

−4

Width of dot (inches)

He

igh

t (in

che

s)

Fig. 7. Profile and the model for a unit impulse comprising of dot of size 1x1.

−0.06 −0.04 −0.02 0 0.02 0.04 0.06

0

5

10

15x 10

−3

Width of dot−cluster (inches)

Heig

ht (i

nche

s)

Experimental dataModel fit

Fig. 8. Model fitted curves for the black dots on white background

whereFn;n is the model suitable for an enhanced profile of ann� n cluster of dots andn;n is the

average attainable height/depth of the profiles for ann � n cluster dot, as shown in Fig. 5(a) and

Fig. 5(b).In;n is the collection of impulses representing a cluster of dot of sizen� n. These have

been plotted in Fig. 6. The resulting modeled curves are found to follow the profiles as predicted.

The modeled curves superimposed with the measured curves are seen in Fig. 8. This kind of

modeling can also be achieved for the white dots on black background.

10

III. TACTILE RESOLUTION

Images that are enhanced using the micro-capsule paper processed by the TIE are ready to be

used as tactile graphics. It is necessary, however, to understand how human subjects respond

to the structures on the micro-capsule paper. A group of experiments were carried out to arrive

at a mathematical model for the human tactile system and its response to the structures on the

micro-capsule paper. The spatial resolution of the dots and the ability to produce a dot profile of

substantial distinction as well as the frequency response of the tactile system is investigated in the

experiments.

A. Review of previous work

The discussion on understanding the tactile pattern perception and the factors affecting this

perception was started by Loomis [4] and Craig [5]. Johnsonet al. [6], [7], [8] began the

important exploration of the sensory aspects of the human skin. More recently, investigation was

carried out by Pawluket al. [24], [25] to develop an engineering model describing the response

of the human peripheral tactile system when an object(a flat indentor) dynamically contacts the

finger-pad. Here the system was measured at both mechanically, at the surface of the skin, and

neurophysiologically, as the resulting nerve signal heads towards the brain.

The investigation made into the spatial neural mechanisms underlying the tactile sensation by

Johnsonet al. [6], [7], [8] is divided into three parts. In the first part, specifications of the dis-

crimination behavior that depends strictly on spatial neural mechanism is discussed. Experiments

are carried out in this regard to define which aspects of the spatial discrimination are based on spa-

tial information in the neural discharge patterns. It was found that human subjects could reliably

discriminate between one and two 0.5 mm (0.019 in.) diameter points even when there was no

separation between the two points. A high level of spatial resolution was thus demonstrated.

Another set of experiments were carried out in which the subjects were required to discriminate

between stimuli with and without gaps. A success criteria of75% was used and it was found on this

basis that gap size at the threshold of edge detection was 0.87 mm(0.034 in.). Subjects were also

11

required to feel square-wave gratings that were oriented along or across the axis of the finger. It

was found that grating orientation was difficult to decipher until the gaps exceeded 0.5 mm(0.019

in.). The gap size, which confirmed with the success criteria of75% accuracy, was found to be

0.84 mm(0.033 in.).

In the second part of the experiments a similar study was carried out to find the neural responses

in the monkey finger pad skin. In the third series of experiments a mechanistic model of skin is

developed to predict the stress and strain at mechanoreceptor terminals within the skin.

A parallel approach made by Lamotte and Whitehouse [26], discusses the capabilities of humans

to detect the presence of a single raised dot of550�m(0.021 in.) diameter on a smooth plate and

to judge the magnitude of evoked sensation. The corresponding stimuli on three different kind of

mechanoreceptive peripheral nerve fibers of the monkey finger-pad (namely the slowly adapting

(SA), the rapidly adapting(RA)and the pacinian (PC) mechanoreceptive peripheral nerve fibers )

was studied for dots of different heights.

The mean detection threshold was found to be about 2.1�0.3 �m (9.4�10�5 in). Also it has

been found that a height of 1�m (3.937�10�5 in ) could be detected if the diameter of dot was

600�m (0.0236 in) or wider. In similar fashion a height of 6�m (2.36�10�4 in) was required

for detection if the dot diameter was reduced to 40�m (1.5748�10�3 in). In another experiment,

stroke velocities of about 10 mm/s (0.393 in/s) were used and the dot detection thresholds were

found out to be between 1(3.937�10�5 in) and 3�m (1.181�10�4 in) for all human observers.

This again was different for different mechanoreceptive nerve fibers.

It was reported that the magnitude of sensation evoked in humans increased with an increase

in dot height above the dot detection threshold as defined in the preceding discussion. It was

found that the impulses evoked in monkey RAs increased with dot height as did the widths of RA

receptive fields. Also, it was hypothesized that the mechanical event responsible for the activating

the RA was the lateral deformation of elevated regions of skin. This was found to be true because

the number of impulses evoked in RAs by a dot was greater when the dot was stroked across, as

opposed to along, the papillary ridges. It is conclusively proved that the responses of RAs alone

12

account for the sensory capacity to detect dot of minimal height on a smooth surface with the finger

pad.

In all these experiments described above the object to be felt by the finger-pad is an ideal system

with flat surfaces and definite structures. The enhanced micro-capsule paper does not suit an ideal

system features and hence the results cannot be directly used in its case.

B. Discrimination of dots

This experiment simulated the work carried out by and Whitehouse [26] on the micro-capsule

paper. Test patterns of individual dots of sizes ranging between a single dot and a cluster of20�20

dots were created as shown in Fig. 3. The human subjects were required to feel the patterns and

respond on the detection of the presence or absence of any structures. Sufficient time was provided

to each subject to feel the structures and then define the response. The micro-capsule paper and the

enhanced structures on it were shielded from the subject by blindfolds. The tactile paper sample

containing these test material were made accessible to the human subjects in random order.

The data were derived from experiments conducted on 8 subjects aged between 22 and 30 years

of age. The discrimination of dots experiment tested the ability of subjects to distinguish the

presence or absence of stimuli on the surface of the micro-capsule paper. Both black on white and

white on black dot patterns were used.

Psychophysics is extensively used to answer questions regarding the way human beings react

to external stimuli, such as variation in specified characteristics of environmental stimulation. Ex-

periments usually reveal typical S-shaped curves [27]. The results for the conducted experiments

show that the detection curves follow typical S-shaped curves, Fig. 9.

The threshold value is typically taken as the stimulus value corresponding to the detection re-

sponse of50%. The threshold for the black dot detection in this study was found to be 0.4 mm. The

detection of white dots in black background was found to be around 1.9 mm. Hence there exists

a difference in the detection threshold of the dots in different background. These are attributed to

characteristics and resolution of the tactile paper. The gaps in black background, which are seen

as the white dots, do not produce noticeable depth for small dots. This is because the black dots

13

0 0.5 1 1.5 2 2.5 30

20

40

60

80

100

Stimulus dimension (mm)

Pro

ba

bili

ty o

f fe

elin

g d

ots

(%

)

Black dotsWhite dots

Fig. 9. Detection curves for dots

cause an enhancement that raises the material inside the square block, thereby negating any effect

of depth. However for bigger clusters of white dots, the problem is not as pronounced.

C. Frequency Response

The frequency response of the human tactile system is another way of understanding the way

human subjects react to the raised structures on the tactile paper. There has been substantial re-

search in the designing and conducting of psychophysical experiments [28], [29] to analyze the

human visual system. The basis of the study was to understand what happens to the spatial infor-

mation1 after they are received by the eye. Since the goal is to understand the processing of data

flow within the tactile system, the visual experiments provide us with a good model to carry out

the tactile system experiments.

Visual perception is studied by recording and analyzing the response to sine wave gratings. Spa-

tial sine wave gratings (whose luminance varies sinusoidally across space) are simple building1Spatial information refers to the brightness and constant attributes across the space

14

(a) (b)

Fig. 10. Psychophysical experiments using gratings. (a)Gratings for human visual system tests; (b)Gratings for thehuman tactile system tests

blocks for complex stimuli and they are influenced by frequency, contrast, phase and orientation.

To determine a psychophysical measurement of the sensitivity of humans to the sine wave grat-

ings, human subjects are presented with a gratings of a given spatial frequency, Fig. 10(a). The

frequency range is between 3 and 48 cycles per degree of visual angle [29]. Initially the subject

cannot see a grating as the image of the grating is below threshold. The contrast of the grating is

increased till the subject reports the sighting of the grating. The sensitivity for the particular spatial

frequency is the inverse of the contrast threshold. This procedure is repeated for a large number of

frequencies. The resulting curve plotting contrast sensitivity as a function of spatial frequency is

known as the contrast sensitivity function (CSF).

A number of models have been discussed [30] to model the CSF. The dependence of the contrast

sensitivityShvs on the radial spatial frequency,fhvs was developed by Campbellet al. [31], [32].

It can be written as follows:

Shvs= �[e�2��fhvs � e�2��fhvs] (5)

where�, � and� are constants. The constant� is proportional to the average illumination. The

constants� and� are set depending on the type of model chosen during the course of the experi-

ment.

A similar approach was carried out in the current study to determine the tactile sensitivity func-

15

0 10 20 30 40 50 60 70 80 900

10

20

30

40

50

60

70

80

90

100

Spatial Frequency (cycles/paper)

Ta

ctile

Se

nsi

tivity

Experimental dataModel fit

Fig. 11. Tactile Sensitivity Curve for human beings.

tion (TSF). Gratings such as that shown in Fig. 10(b) were utilized. Non-blind subjects were

blindfolded and presented with square-wave gratings, which were sufficiently raised after passing

through the TIE. These gratings were at a fixed spatial frequency. The spatial frequency used were

in the range between 1 to 30 cycles per unit of space. Noise was introduced in the square gratings

to simulate the change in signal to noise ratio or the contrast of the gratings. The distribution of

the noise varies exponentially with space. At one edge the noise power is very high, so that the

subjects could not feel the gratings. The noise level was reduced until a point is reached where the

subject reports feeling the gratings. The corresponding noise level was recorded and treated as the

sensitivity value for the particular spatial frequency. This experiment was repeated for a number of

different spatial frequencies. The resulting spatial sensitivity values are plotted against the width

of the gratings, as shown in Fig. 11. The measured human TSF is a band-pass function, as it shows

a peak and decreasing sensitivity on either side of this peak.

A model for the tactile system utilizing the TSF data generated from the experiments is given

16

0

5

10

15

20 05

1015

20

40

45

50

55

60

65

70

75

Tac

tile

Sen

sitiv

ity

Fig. 12. Human Tactile Sensitivity model. The x-axis is the horizontal spatial frequency (cycles/mm) and the y-axisis the vertical spatial frequency (cycles/mm)

by:

Shts= (qfhts=C)� (e�fhts=D) (6)

where the constantsC andD are calculated from the empirical data to be4:65 and43:245, respec-

tively. Also, fhts is the radial spatial frequency in cycles per unit of space. The resulting model is

shown in Fig. 12.

IV. TACTILE HALFTONE STRUCTURES

In printing devices where the direct reproduction of all the gray tones is impossible, patterns of

binary pixels or halftones are substituted for the original image, in the process creating a visual

illusion of similarity. This is possible due to the low-pass nature of the human eye, which averages

black dots and white space to perceive a particular gray level. A similar band-pass response of the

human touch sense needs to be taken into account when we are trying to devise a tactile imaging

scheme based on halftones. In conventional halftoning using model-based techniques, the CSF of

the human visual system is utilized to compare the halftoned image and the original continuous

tone image. Our intention is to effectively map this idea to the tactile domain.

17

Image

Process Model

Halftoned Image

TactileTIE

TIE

Error

Human Tactile

System

TactileHuman

System

Halftone

Image

Fig. 13. Block Diagram for the optimal tactile imaging process.

Figure 13 shows the implementation of the proposed model based halftoning process for tactile

imaging. This model is similar to the model-based halftoning method reported in [21], [22], [20].

We have substituted the human tactile system and the TIE for the human visual system and the

printer, respectively.

The model details the algorithm to produce the halftone patterns that are optimum for a par-

ticular halftone process. The original image is passed through an initial halftoning process. The

model based halftoning requires a quantitative comparison between the tactile-perceived original

image and the tactile-perceived halftoned image. The tactile-perceived original image is obtained

by filtering the original image with a band-pass filter modeled according to the tactile sensitivity

function, Equation ( 6) in Section III-C. The halftoned original image is passed through a model

simulating the TIE. This transforms the distribution of dots to the simulated output of the TIE by

taking into consideration the Gaussian surface model (Eq. 4) of the TIE. At this stage, the mini-

mum criteria for recognition of the black and the white dots, as discussed in Section III-B, is used

to screen the halftoned image. The halftoned original image is converted to raised halftone struc-

tures with the necessary constraints on dots characteristics. Finally, the tactile-perceived halftoned

image is the result of output after filtering the raised halftone structures with the tactile sensitivity

function band-pass filter model. The comparison of the two perceived images produces an error

that is then superimposed on the halftoned image to arrive at an optimal dot arrangement.

18

The successful implementation of the model-based halftoning and the production of the opti-

mum halftone structures requires a choice on the type of halftoning method. The following section

discuss the advantages and shortfalls of the common halftoning techniques available. These meth-

ods are then compared with each other through a set of experiments.

A. Halftoning methods

Over the years a number of effective methods have been proposed in the field of halftoning.

Ordered dither, or amplitude modulated (AM) halftoning [19], is one of the simplest methods.

This method of creating an illusion of a continuous tone image is produced by varying the size

of the dots that are printed along a regular lattice. The number of dots, or frequency of the dots,

remains fixed. The original continuous-tone image is divided into blocks of smaller sizes that are

then compared to a threshold in a pixel by pixel comparison. The dither array is composed of

consecutive thresholds clustered together.

Floyd and Steinberg [33] developed a halftoning technique, known as error diffusion, that dis-

tributes the error generated at each pixel. This became the foundation of the present day frequency

modulated (FM) halftoning. In FM halftoning, the required illusion is achieved by varying the

distance between printed dots while the dots are held at a constant size. This process is an adaptive

algorithm that quantizes each pixel. The quantization error produced in each of the single pixel

operations is distributed amongst the neighborhood of yet to be processed pixels.

The pattern generated by this process was referred by Robert Ulichney as the blue-noise halftones

[19]. This name results from the fact that the dither patterns is composed exclusively of high fre-

quency spectral components. For a given gray levelg, the minority pixels are separated by an

average distance of�b, which is the principle blue noise wavelength. The relation between the two

parameters is given by:

�b =

8<: D=(

q(g)) for 0 < g � 1=2

D=(q(1� g)) for 1=2 < g � 1

(7)

where D is the minimum distance between addressable points on the display. The error diffusion

algorithm devised by Floyd and Steinberg, shown in Fig. 14, can be mathematically represented

19

X[n]

X[n]

Y[n]

Y[n]e e

+

+

+

−

B

Fig. 14. Error diffusion (Floyd and Steinberg, 1976)

as follows:

Y [n] =

(1 if (X[n]�BTY e[n]) � 00 else

(8)

whereB = [b1; b2; :::; bN ]T ,PN

i=0 bi = 1, the outputY [n] = [y[n � 1]; y[n � 2]; :::; y[n � N ]]T

andY e[n] = [ye[n � 1]; ye[n � 2]; :::; ye[n � N ]]T , which follows from the equationY e[n] =

Y [n]� (X[n]� (BTY e[n])). The input pixel under consideration in this expression isX[n].

An ideal printer outputs patterns that are composed of perfect square black dots. In high quality

printing situations, where this effect is true to certain extent, blue-noise halftoning is considered the

“optimum technique for minimizing visibility [34] and maximizing the apparent spatial resolution

[35]”, as reported in [20]. In practical considerations involving digital printing, printer distortions

are relevant. They are summarily categorized as dot-gain and dot-loss [20]. Dot-gain is the

increase in size of the printed dot relative to its intended size, and is the main distortion in ink

jet printing. The quality of paper also has a very crucial effect, with denser papers causing ink

to spread instead of absorbing it completely. Dot loss appears in higher resolution printers where

instead of creating a bigger dot, there is a difficulty in printing an isolated black dot. Model

based halftoning and clustering of dots are techniques that have been developed to address these

problems.

The halftoning process that are of particular interest in the present context of clustering are the

AM-FM hybrid stochastic halftoning techniques. These create minority pixel clusters that vary,

according to tone, in both their size and spacing. The spectral characteristics of such patterns have

20

X[n]

X[n]

X[n]

Y[n]

Y[n]e

h

e

+

+

++

+

-

H A

B

Fig. 15. Error diffusion with output-dependent feedback: a weighted sum of the previous output pixels is used to varythe threshold. H-hysteresis term, A-hysteresis filter, B-error filter, X[n]-input image, Y[n]-halftoned image.

mid-frequency components. Hence they are collectively known as green-noise halftone patterns.

In green-noise, the minority pixel clusters are distributed homogeneously. The average separation

(center-to-center) between clusters is termed as�g. Its square is inversely proportional to the

average number of minority pixel clusters per unit area. The green-noise principle wavelength

[20] is given by the following equation.

�g =

8<: D=(

q(g)= �M) for 0 < g � 1=2

D=(q(1� g)= �M) for 1=2 < g � 1

(9)

where D is the minimum distance between addressable points on the display,g is a particular gray

level and �M is the average number of minority pixels per cluster in the binary dither pattern.

The green noise halftone is generated by a error diffusion technique proposed by Levien [36].

He referred to it as the error diffusion with output dependent feedback, Fig. 15. In this algorithm a

weighted sum of the previous output pixels is used to vary the threshold. This makes the minority

pixels more likely to occur in clusters. The amount of clustering is controlled through the hysteresis

constantH. Large values of H can cause large clustering and smaller values lead to smaller clusters.

Levien’s algorithm is precisely defined as follows:

Y [n] =

(1 if (X[n] +BTY e[n] +HATY [n]) � 00 else

(10)

21

whereA = [a1; a2; :::; aN ]T , B = [b1; b2; :::; bN ]

T ,PN

i=0 ai = 1,PN

i=0 bi = 1, the outputY [n] =

[y[n�1]; y[n�2]; :::; y[n�N ]]T andY e[n] = [ye[n�1]; ye[n�2]; :::; ye[n�N ]]T , which follows

from the equationY e[n] = Y [n] � (X[n] + (BTY e[n])). The error filter and hysteresis filter can

take on a wide range of values, including special cases such as Floyd-Steinberg [33], Jarvis [37]

and Stucki [38] filter coefficients. Perturbation, in the form of noise, can also be added to these

filter weights to create desirable results.

B. Results and Evaluation

Psychophysical experiments can be used to determine the optimum tactile halftone technique.

The measure of goodness is defined as the level of effectiveness in feeling and understanding

information through the manual exploration of the structures. As detailed in Fig. 13, the halftoning

systems defined by the model transfer functions for AM, FM, and AM-FM hybrid methods are

considered.

As a first examination, a gray scale ramp test image is used as an original image and the corre-

sponding halftones are generated by each method. These are shown in Fig. 16. A sample image

having four gray levels is also processed by the three halftone techniques, Fig. 17. Upon observa-

tion of the generated halftone structures, we find certain differences that are particularly important

in the case of tactile halftoning.

First note that, as expected, the halftoned gray scale ramp produced by the AM halftoning

method has a uniform distribution of dots, but varying dot size. Due to the regular spacing of

dots, the (micro-capsule) growth of the dots is inhibited. That is, dots begin to merge once they

are of a certain size. Thus no discrimination can be made once the dots begin to merge. Also, due

to the asymmetry of the micro-capsule enhancement process, black dots and white dots merge at

different thresholds. Accordingly, the tactile printing process places significant limitations on the

use of AM halftoning.

In the case of the blue noise, or FM, halftoning technique, the dots have similar size but are

distributed at varying distance depending on the gray level. Once again, the minimum dot size

imposed by the tactile printing process imposes a constraint. Since the minimum dot size for black

22

(a)

(b)

(c)

(d)

Fig. 16. Test Image: (a) Original Ramp of gray scale; Test image halftoned using the HTS by (b)Amplitude Modula-tion (AM) halftoning (c) Frequency Modulation (FM) halftoning and (d) AM-FM hybrid halftoning

23

(a) (b)

(c) (d)

Fig. 17. Example using a test image (a) Original image (b) AM halftoned (c)FM halftoned (d)AM-FM halftoned

and white are different, both constraints cannot be met simultaneously under the FM structure.

This results in merging of white dots and, as in the AM case, leads to a degradation in the quality

of the resulting halftone.

The green noise, or AM-FM hybrid, halftoning structure allows the minimum black and white

dot size to co-exist, and the resulting halftone thus consists of dots within the experimentally

defined limits. Also, since both spacing and dot distribution are stochastic processes, this method

of generating tactile halftones is more robust to errors in either dot location or size. This results

in halftones where the dots are not merged by the printing (micro-capsule paper enhancement)

process. Thus, the produced halftones are tactilely distinct and more easily discriminated than

either the AM or FM generated halftones.

To test the validity of these visual observations, a set of experiments was conducted. The

halftone methods discussed in previous sections were utilized to create halftones for the simple

24

(a) (b)

(c) (d)

(e)

Fig. 18. Test images halftoned by the AM-FM hybrid method for the edge and region identification experiments: (a)Hexagon (b) Triangle (c) Rectangle (d) Circle (e) image of rock with two gray levels.

geometric figures of hexagon, triangle, rectangle, circle and an image of rock. The structures, with

halftoned patterns generated by the AM, FM, and AM-FM methods, constituted the set of stimuli

for the experiments. Figure 18 shows the structures generated by the AM-FM hybrid method.

25

1 2 3 4 50

1

2

3

4

5

6

7

8

9

10

Geometric Figures

Tim

e(s

ecs

)

AM halftoningFM halftoningAM−FM hybrid

Fig. 19. Timed identification of the geometric figures. The figures numbered 1 to 5 are Hexagon, Triangle, Rectangle,Circle, and an image of rock with two gray levels, respectively

The test patterns were enhanced as detailed earlier with the help of the TIE and were presented to

blindfolded human subjects.

The data were derived from experiments conducted on 8 subjects aged between 22 and 30 years

of age, as before. The micro-capsule paper and the enhanced structures on it were shielded from

the subject by blindfolds.

In the first experiment, the halftones of the geometric figures produced by the three methods

were numbered for our identification and presented to the subjects in random order. The subjects

were asked to recognize the shape of the geometric figures by manual exploration of the figures on

the tactile paper. The time taken by each subject to identify the figure, through manual exploration,

was noted and recorded.

This experiment was undertaken to understand how quickly the human subjects were able to

recognize shapes based on each of the halftoning methods. As each halftoning method results

in a different distribution of dots and subsequently different height/depth for the edge dots, this

26

1 2 3 4 50

10

20

30

40

50

60

70

Geometric Figures

Re

spo

nse

to

str

uct

ure

s(%

)

AM halftoningFM halftoningAM−FM hybrid

Fig. 20. Texture Discrimination.The figures numbered 1 to 5 are Hexagon, Triangle, Rectangle, Circle, and an imageof rock with two gray levels, respectively

recording helps in defining which halftoning method produces better recognition. The average

identification times for each method are shown in Fig. 19. The compiled results showed that, in

all cases, the AM-FM hybrid halftoning method leads to the shortest identification time. Thus

it can be noted that the more distinct structures produced by the AM-FM hybrid methods aid in

identification.

The second experiment tested how well a human subject is able to recognize the details in

the textures of the structures. This was aimed at the texture discrimination rather than the shape

identification, which was the goal in the first experiment. The human subjects were presented with

the three halftones of the same geometrical figure and were asked to make a comparison of the

quality of the structures on their surface.

As before, the subjects were asked to manually explore the figures and report which of the three

halftones presented to them, again in random order, was most distincti.e. easily identifiable by

touch. The experiment recorded which halftone method would give the most tactile perceivable

27

patterns on a surface for a given gray level. Figure 20 reports the responses recorded from this ex-

periment. It was found that, in most cases, the AM halftoning is too weak to produce the necessary

discrimination, most likely due to merging of dots. The FM halftoning, on the other hand, enables

greater discrimination than the AM halftoning. The AM-FM hybrid halftoning method however,

yields the greater discrimination.

The psychophysical experiment results are thus in alignment with the analysis of the algorithms.

The hybrid AM-FM halftoning, or green noise halftoning, is optimal for the generation of tactile

output among the methods available in the literature. The performance of the hybrid method is

likely due to its flexibility, in that it can be jointly optimized according to human tactile perception

and the characteristics of the printing (microcapsule enhancement) process. Both the AM and FM

methods produce suboptimal results, which can be attributed to identifiable shortcomings such as

dot merging.

Considerable work must still be undertaken to fully optimized and evaluate halftoning methods

for tactile representations. For instance, it is not currently known what an appropriate number of

tactile gray level representations is. It is unlikely that a user would be able to discriminate amongst,

say, the 256 gray levels in an 8–bit image, but discrimination amongst a smaller number of halfton-

ing patterns should be possible. Further studies are needed to clarify this issue. Additionally, the

methods must be evaluated on other output medium and displays, such as the TIGER embossing

printer. These issues are amongst those that will be the subject of further investigations.

V. CONCLUSIONS

This paper introduced the concept of digital halftoning for tactile imaging. To develop appro-

priate tactile halftoning methods, the characteristics of the Tactile Image Enhancer (TIE) were

studied in detail. This study led to an accurate model of the microcapsule paper enhancement

process. Additionally, experiments were conducted to determine human tactile sensitivity and fre-

quency response for the case of manual exploration of microcapsule paper. The TIE model and

human tactile response were integrated into a general tactile halftoning procedure. This procedure

includes as special cases the widely used AM and FM halftoning, as well as the more flexible

28

AM–FM hybrid green noise technique. Each of the methods were analyzed for their applicabil-

ity in tactile halftoning and sample outputs were generated. Psychophysical experiments on timed

identification and discrimination were conducted. The algorithm analysis and experiments indicate

that the AM–FM hybrid method produces the most tactilely appropriate halftoning patterns. Fur-

ther investigations will be conducted to more fully evaluate the developed methods to determine,

for instance, the number of halftoning patterns that can be discriminated by users, and additional

research will be conducted to optimize the procedures for other tactile display media.

VI. A CKNOWLEDGMENT

This work was supported by the National Science Foundation under the grants 9800175 and

9875658.

REFERENCES

[1] S. Millar, Reading by Touch. Routledge, New York, USA, 1997.[2] J. Bliss, “A relatively high-reslution reading aid for the blind,”IEEE Transactions on Man-Machine Systems, vol. 10, pp. 1–9,

1969.[3] J.G.Linvill and J. Bliss, “A direct translation reading aid for the blind,” inProceedings of the IEEE, vol. 54, pp. 40–51, 1966.[4] J. Loomis, “Tactile pattern perception,”Perception, vol. 10, 1981.[5] J. Craig and C. Sherrick, “Dynamic tactile displays,” inTactual Perception: A Sourcebook(W. Schiff and E. Foulke, eds.),

Cambridge University Press, 1982.[6] J. Phillips and K. Johnson, “Tactile spatial resolution (i): Two-point discrimination, gap detection, grating resolution, and

letter recognition,”Journal of Neurophysiology, vol. 46, pp. 1177–1191, 1981.[7] J. Phillips and K. Johnson, “Tactile spatial resolution (ii): Neural representation of bars, edges and gratings in monkey primary

afferents,”Journal of Neurophysiology, vol. 46, pp. 1192–1203, December 1981.[8] J. Phillips and K. Johnson, “Tactile spatial resolution (iii):a continuum mechanics model of skin predicting mechanorecptor

responses to bars, edges and gratings,”Journal of Neurophysiology, vol. 46, pp. 1204–1225, December 1981.[9] M. Kurze, L. Reichert, and T. Strothotte, “Access to business graphics for blind people,”Proceedings of the RESNA ’94

Annual Conference, June 17-22 1994.[10] R. Klatzky, S. Lederman, and V. Metzger, “Identifying objects by touch: An “expert system”,”Perception and Psychophysics,

vol. 37, no. 4, 1985.[11] R. Klatzky, S. Lederman, and C. Reed, “There’s more to touch than meets the eye: The salience of object attributes for haptics

with and without vision,”Journal of Experimental Psychology: General, vol. 116, 1987.[12] K. Kokjer, “The information capacity of the human fingertip,”IEEE Transactions on Systems, Man, and Cybernetics,

vol. SMC-17, no. 1, 1987.[13] T. P. Way and K. E. Barner, “Automatic visual to tactile translation, part I: Human factors, access methods and image manip-

ulation,” IEEE Transactions on Rehabilitation Engineering, vol. 5, pp. 81–94, Mar. 1997.[14] T. P. Way and K. E. Barner, “Automatic visual to tactile translation, part II: Evaluation of the tactile image creation system,”

IEEE Transactions on Rehabilitation Engineering, vol. 5, pp. 95–105, Mar. 1997.[15] S. Hernandez and K. Barner, “Joint region merging criteria for watershed-based image segm entation,” in2000 International

Conference on Image Processing, (Vancouver, BC, Canada), September 10-13 2000.[16] S. Hernandez and K. E. Barner, “Color image segmentation using watersheds and joint region merging criteria,”IEEE Trans-

actions on Image Processing, Aug. 2000. Submitted for publication.[17] S. Hernandez and K. Barner, “Tactile imaging using watershed-based image segmentation,” inASSETS 2000, (Arlington,

Virginia, USA), November 13-15 2000.[18] W. Schiff and E. Foulke,Tactual perception: a sourcebook. Cambridge University Press, 1982.[19] R. A. Ulichney,Digital Halftoning. Cambridge, MA: MIT Press, 1987.[20] D. L. Lau,Modern Digital Halftoning. PhD thesis, University of Delaware, Newark, Delaware, USA, May 1999.[21] T. N. Pappas and D. L. Neuhoff, “Model-based halftoning,” inProceedings of SPIE, Human Vision, Vision Processing and

Digital Display II (B. E. Rogowitz, M. H. Brill, and J. P. Allebach, eds.), vol. 1453, pp. 244–255, June 1991.

29

[22] T. N. Pappas, “Model-based techniques for digital halftoning,” inProceedings, International Conference on Image Processing(B. Werner, ed.), vol. II, pp. 26–30, IEEE, 1994.

[23] J. Gardner, “Tactile graphics: An overview and resource guide,” Available fromhttp://www.rit.edu/ easi/itd/itdv03n4/article2.html, 1999.

[24] D. Pawluk,A Viscoelastic Model of the Human Fingerpad and a Holistic Model of the Human Touch. PhD thesis, HarvardUniversity, Cambridge, Massachusetts, USA, May 1997.

[25] D. Pawluk and R. Howe, “A holistic model of human touch,” in5th Annual C.N.S. Meeting, 1996.[26] R. LaMotte and J. Whitehouse, “Tactile detection of a dot on a smooth surface: Peripheral neural events,”Journal of Neuro-

physiology, vol. 56, no. 4, 1986.[27] H. Schiffman,Sensation and Perception: An Itegrated Approach. John Wiley and Sons, New York, 1976.[28] T. Cornsweet,Visual Perception. New York: Academic press, 1971.[29] F. W. Campbell, “The transmission of spatial information through the visual system,”The Neurosciences Third Study Program,

pp. 95–103, 1974.[30] H. R. Kang,Digital Color Halftoning. SPIE Optical Engineering Press and IEEE Press,USA, 1999.[31] F. W. Campbell, R. H. Carpenter, and J. Levinson, “Visibility of aperiodic patterns compared with that of sinusoidal gratings,”

The Journal of Physiology, vol. 190, pp. 283–298, 1969.[32] F. W. Campbell, “The human eye as an optical filter,”Proceedings of the IEEE, vol. 56, no. 6, pp. 1009–1014, 1968.[33] R. W. Floyd and L. Steinberg, “An adaptive algorithm for spatial gray-scale,”Proceedings Society Information Display,

vol. 17, pp. 75–78, Feb. 1976.[34] J. Sullivan, L. Ray, and R. Miller, “Design of minimum visual modulation halftone patterns,”IEEE Transactions on Systems,

Man, and Cybernetics, vol. 21, pp. 33–38, Jan. 1991.[35] M. Rodriguez, “Graphic arts perspective on digital halftoning,” inProceedings of SPIE, Human Vision, Visual Processing,

and Digital Display V(B. E. Rogowitz and J. P. Allebach, eds.), vol. 2179, pp. 144–149, Feb. 1994.[36] R. Levien, “Output dependant feedback in error diffusion halftoning,” inIS&T’s Eighth International Congress on Advances

in Non-Impact Printing Technologies, (Williamsburg, Virginia, USA), pp. 280–282, October 25-30 1992.[37] J. F. Jarvis, C. N. Judice, and W. H. Ninke, “A survey of techniques for the display of continuous-tone pictures on bilevel

displays,”Computer Graphics and Image Processing, vol. 5, pp. 13–40, 1976.[38] P. Stucki, “Mecca-a multiple-error correcting computation algorithm for bilevel image hardcopy reproduction,” Tech. Rep.

RZ1060, IBM Research Laboratory, Zurich, Switzerland, 1981.