Upload
jemshidkk
View
223
Download
0
Embed Size (px)
Citation preview
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 1/48
CHAPTER-1
INTRODUCTION
A virtual keyboard is where a full-size image of a QWERTY keyboard is projected onto any
surface. Touching the image of a key generates a unique electronic signal corresponding to a
key's image. Using a virtual keyboard eliminates the chance of breakage and infection
transfer. Additionally virtual keyboards require no cleaning and they have no wires, buttons,
or switches. Virtual keyboards are also compatible with many Smart phones and PDAs.
As the demand for computing environments evolves, new human-computer interfaces have
been implemented to provide multiform interactions between users and machines. Also, the
basis for most human-to-computer interactions remains the binomial keyboard/mouse.
Ordinary keyboards however, to be comfortable and effective, must be reasonably sized.
Thus they are cumbersome to carry and often require wiring. To overcome these problems, a
smaller and more mobile touch-typing device has been proposed which does not have
physical support. This device is known as virtual keyboard.
The depth information supplied by the range camera allows developing simpler and more
efficient computer-vision algorithms to estimate the position of fingertips and to locate the
corresponding stricken key. Simplicity and efficiency are key elements to enable real-time or
even portable applications
.
However, there are still some challenges associated with the range camera utilized in this
project. A number of problems, such as light scattering, and “close target” artifacts impact
achievable depth accuracy. Moreover, the image resolution of the camera is relatively low,
thus restricting its use to applications with large view window. As a result the working area
of the keyboard is limited today to a sub-optimal size. The current power consumption and
size of the range camera are also impediments to its use in truly portable applications
.
On-screen keyboards have a limited area due to the limitations of the screen, but they can
have a dynamic key layout and they can be used in darkness because of the back light. Full
1
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 2/48
text keyboards are too large to fit into a compact mobile device without the keys being
uncomfortably small. However, in this case the user gets a known keyboard that functions
exactly like he is used to.
In this project the main focus will be on implementing a traditional full text keyboard
using Augmented Reality, but future work should consider alternative text input methods.
Projection keyboard
A projection keyboard is a virtual keyboard that can be projected and touched on any surface.
The keyboard watches finger movements and translates them into keystrokes in the device.
Most systems can also function as a virtual mouse or even as a virtual piano. A proposed
system called the P-ISM will combine the technology with a small video projector to create a
portable computer the size of a fountain pen.
How a projection keyboard generally works:
1. A laser or beamer projects visible virtual keyboard onto level surface
2. A sensor or camera in the projector picks up finger movements[3]
3. detected co-ordinates determine actions or characters to be generated
Some devices use a second (invisible infrared) beam:
1. An invisible infrared beam is projected above the virtual keyboard
2. Finger makes keystroke on virtual keyboard. This breaks infrared beam and infrared
light is reflected back to projector
3. Reflected infrared beam passes through infrared filter to camera
4. Camera photographs angle of incoming infrared light
5. Sensor chip determines where infrared beam was broken
6. detected co-ordinates determine actions or characters to be generated
An optical virtual keyboard has been invented and patented by IBM engineers in 1992. It
optically detects and analyses human hand and finger motions and interprets them as operations
on a physically non-existent input device like a surface
2
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 3/48
The virtual keyboard consists of webcam (as a media for sensing the key strings),
Prototype keyboard and MATLAB (as a tool for interfacing). In that way it allows to emulate
unlimited types of manually operated input devices (mouse, keyboard, etc.). All mechanical
input units can be replaced by such virtual devices, optimized for the current application and
for the user's physiology maintaining speed, simplicity and unambiguity of manual data
input.
In this project the main focus will be on implementing a traditional full text keyboard
using Augmented Reality, but future work should consider alternative text input methods.
Why we selected the virtual keyboard
Virtual Keyboard is just another example of today’s computer trend of ‘smaller and faster’.
Computing is now not limited to desktops and laptops, it has found its way into mobile
devices like palm tops and even cell phones. But what has not changed for the last 50 or so
odd years is the input device, the good old QWERTY keyboard. The virtual keyboard
technology is the latest development. The virtual keyboard technology uses sensor
technology and artificial intelligence to let users work on any flat surface as if it were a
keyboard. Virtual Keyboards lets you easily create multilingual text content on almost any
existing platform and output it directly to PDAs or even web pages. Virtual Keyboard, being
a small, handy, well-designed and easy to use application, turns into a perfect solution for
cross platform text input. The main features are: platform-independent multilingual support
for keyboard text input, built-in language layouts and settings, copy/paste etc. operations
support just as in a regular text editor, no change in already existing system language settings,
easy and user-friendly interface and design, and small file size.
Problems of QUERTY keyboard
It is now recognized that it is important to be correctly seated while using a computer. A
comfortable working position will help with concentration, quality of work, and reduce the
risk of long-term problems. This is important for all who use computers, and especially so for
those with disabilities.
The increased repetitive motions and awkward postures attributed to the use of computer
keyboards have resulted in a rise in cumulative trauma disorders (CTDs) that are generally
3
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 4/48
considered to be the most costly and severe disorders occurring in the office. Lawsuits for
arm, wrist, and hand injuries have been filed against keyboard manufacturers allege that
keyboarding equipment is defectively designed and manufacturers fail to provide adequate
warnings about proper use to avoid injury.
As early as1926, Klockenberg described how the keyboard layout required the typist to
assume body postures that were unnatural, uncomfortable and fatiguing. For example,
standard keyboard design forces operators to place their hands in a flat, palm down position
called forearm pronation. The compact, linear key arrangement also causes some typists to
place their wrist in a position that is skewed towards the little fingers, called ulnar deviation.
These awkward postures result in static muscle loading, increased muscular energy
expenditure, reduced muscular waste removal, and eventual discomfort or injury. Researchers
also noted that typing on the QWERTY keyboard is poorly distributed between the hands and
fingers, causing the weaker ring and little fingers to be overwork
VIRTUAL KEYBOARD
Introduction
Virtual Keyboard is just another example of today’s computer trend of "smaller and faster".
Computing is now not limited to desktops and laptops, it has found its way into mobile
devices like palm tops and even cell phones. But what has not changed for the last 50 or so
odd years is the input device, the good old QWERTY keyboard. Alternatives came in the
form of handwriting recognition, speech recognition, abcd input (for SMS in cell phones) etc.
But they all lack the accuracy and convenience of a full-blown keyboard. Speech input has an
added issue of privacy. Even folded keyboards for PDAs are yet to catch on. Thus a new
generation of virtual input devices is now being paraded, which could drastically change the
way we type. Virtual Keyboard uses sensor technology and artificial intelligence to let users
work on any surface as if it were a keyboard. Virtual Devices have developed a flashlight-
size gadget that projects an image of a keyboard on any surface and let’s people input data by
typing on the image. The device detects movement when fingers are pressed down. Those
movements are measured and the device accurately determines the intended keystrokes and
translates them into text. The Virtual Keyboard uses light to project a full-sized computer
keyboard onto almost any surface, and disappears when not in use. The translation process
also uses artificial intelligence. Once the keystroke has been decoded, it is sent to the portable
device either by cable or via wireless.
4
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 5/48
The Virtual Keyboard uses light to project a full-sized computer keyboard onto almost any
surface, and disappears when not in use. Used with Smart Phones and PDAs, it provides a
practical way to do email, word processing and spreadsheet tasks, allowing the user to leave
the laptop computer at home. The technology has many applications in various high-tech and
industrial Sectors. These include data entry and control panel applications in hazardous and
harsh environments and medical markets.
Projection key boards or virtual key boards claim to provide the convenience of compactness
with the advantages of a full-blown QWERTY keyboard. An interesting use of such
keyboards would be in sterile environments where silence or low noise is essential like
operation theaters. The advantage of such a system is that you do not need a surface for
typing, you can even type in plain air. The company's Virtual Keyboard is designed for
anyone who's become frustrated with trying to put information into a handheld but doesn't
want to carry a notebook computer around. There is also the provision for a pause function to
avoid translating extraneous hand movements function, so that users can stop to eat, drink etc
It is also a superior desktop computer keyboard featuring dramatically easier to learn touch-
typing and leaving one hand free for mouse or phone. Combination key presses ("chords") of five main and two extra control keys allow users to type at 25-60 words per minute, with
possibly greater speeds achieved through the use of abbreviation expansion software. Most
users, however, will find memorizing the chords easy and fun, with the included typing
tutorial. The scanner can keep up with the fastest typist, scanning the projected area over 50
times a second. The keyboard doesn't demand a lot of force, easing strain on wrists and digits.
Virtual keyboards solve the problem of sore thumbs that can be caused by typing on the tiny
keyboards of various gadgets like PDAs and cell phones. They are meant to meet the needs of
mobile computer users struggling with cumbersome, tiny, or nonexistent keyboards. It might
help to prevent RSI injuries.
The Virtual Keyboard uses an extremely durable material which is extremely easy to clean.
The Virtual Keyboard is not restricted to the QWERTY touch-typing paradigm; adjustments
can be done to the software to fit other touch-typing paradigms as well, such as the
DVORAK keyboard. It will work with all types of Bluetooth enabled devices such as PDAs
and smart phones, as well as wearable computers. Applications include computer/PDA input,
5
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 6/48
gaming control, TV remote control, and musical applications. Thus virtual keyboards will
make typing easier, faster, and almost a pleasure.
CHAPTER-2
WORKING
The virtual keyboard consists of webcam (as a media for sensing the key strings), Prototype
keyboard and MATLAB (as a tool for interfacing). The program continuously monitor for a
pressed key. The pictures/video taken by the webcam will process by the MATLAB for any
pressed key. The key press is detected by comparing the areas with a predetermined
6
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 7/48
threshold. Change in value indicates a key press. The corresponding key will be detected and
printed on a MATLAB window.
As we started working on to our project with a still camera taking photos manually, and
then transferring the digital image obtained from camera on to the working tool MATLAB
and then processing the obtained image we were able to determine the keys.
Fig1: image taken by the webcam
CHAPTER-3
PROJECT SETUP
7
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 8/48
Fig2: Project set up
Keyboard should be placed on a correct background and the light falling on the keyboard
should be uniform throughout. But in real cases this will not happen mainly due to scattering.
To reduce this scattering effect makes the surface of the keyboard flat.
The camera should focus on the keyboard exactly vertical in order to avoid resizing
problems. The unwanted resizing will ultimately affect or reduce the system performance.
The pictures taken from the camera are moved to the MATLB directory by using Bluetooth
or data cable.8
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 9/48
The CMOS camera continuously captures images of the region where the printed
keyboard is supposed to be placed and checks these images for red color data above a
specified threshold. The threshold concept works in this case because the laser shining on a
typical human finger generates saturating values of red color data, which is very easily
distinguishable from its surroundings.
CHAPTER-4
PROJECT OVERVIEW
In this paper, a complete system is presented which mimics a QWERTY keyboard on an
arbitrary (paper) surface. The system consists of a webcam and a WINDOWS real time
MATLAB program for detecting the typing events. We exploit snapshot feature of the
camera and detect the key pressed using a MATLAB. Fingertips are found by analyzing the
hand picture and compare with a reference keyboard. To detect a key press, we analyze the
hand movements through the snapshots and when we press a key the snap before the event
and after the event will be same and map it back to the reference keyboard to find which key
9
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 10/48
was pressed. These steps are fully automated by MATLAB programming and do not require
human intervention. The resulting key is getting printed on to the MATLAB window and
getting stored in a file.
The steps involved
1. Webcam interfacing and video processing
2. image acquisition, frames acquisition
3. image segmentation
4. keystroke detection
5. key detection
CHAPTER-5
WEBCAM INTERFACING AND VIDEO PROCESSING
10
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 11/48
Fig3: webcam Fig4: virtual keyboard set up
We acquire the images from camera depends on lots of software. Creating the inter phase
between computer and the camera can be done using lower level language like C or C++
will give lots of flexibility, but it will also invoke lots of work and background knowledge.
MATLAB’s image acquisition toolbox has a variety of simple functions
For Mat lab to recognize the video camera you have to create it as an object using the
command obj=videoinput(‘winvideo’). Mat lab will automatically find the webcam
connected to your computer. Once it is an object in your workspace you can edit its settings,such as the number of frames per second, to optimize it for your project. preview () and get
frame() are two useful functions for determining if the camera has been positioned properly.
The first allows you to see what the camera sees, without collecting any data from it, and the
second acquires a single snapshot and stores it as in image.
Create an image acquisition object -- This example creates a video input object for a
Webcam image acquisition device. To run this example on your system, use the imaqhwinfo
function to get the object constructor for your image acquisition device and substitute that
syntax for the following code.
vid = videoinput('winvideo',1); ie converting to static state.
preview
Display preview of live video data
Syntaxes :
11
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 12/48
preview(obj)
preview(obj,himage)
himage = preview(...)
Description
preview(obj) creates a Video Preview window that displays live video data for video input
object obj. The window also displays the timestamp and video resolution of each frame, and
the current status of obj. The Video Preview window displays the video data at 100%
magnification (one screen pixel represents one image pixel).
getsnapshot
It immediately return a single image frame
Syntax
frame = getsnapshot(obj)
Description
frame = getsnapshot(obj) immediately returns one single image frame, frame, from the video
input object obj. The frame of data returned is independent of the video input object
FramesPerTrigger property and has no effect on the value of the FramesAvailable or
FramesAcquired property.
The object obj must be a 1-by-1 video input object.
frame is returned as an H-by-W-by-B matrix where
HImage height, as specified in the ROIPosition propertyWImage width, as specified in the
ROIPosition propertyBNumber of bands associated with obj, as specified in the
NumberOfBands property
frame is returned to the MATLAB workspace in its native data type using the color space
specified by the ReturnedColorSpace property.
12
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 13/48
CHAPTER-6
IMAGE ACQUISITION AND FRAME ACQUISITION
For the image acquisition we use the preview() and getsnapshot(). Using this two functions
we get a coloured images of the keyboard and keyboard with hand motions. The each image
is subjected for conversion to black and white in order to reduce the complexity.
Resize the image to enhance the speed. By resizing the picture the number of pixels
decreases so the processor needs to consider only lesser number of pixels. Rotate the image
to properly align the keyboard.
13
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 14/48
Fig5: captured image
Fig6: Resize and conversion to black and white
14
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 15/48
CHAPTER-7
IMAGE SEGMENTATION
Segmentation means dividing the entire area into small section decided by number of
rows and columns.
The total number of segments will be equal to number of rows multiplies number of columns.
To increasing or decreasing the number of keys in keyboard we only need to change the rows
or columns. The resized image size is stored in a matrix variable.
We can use the loop statement given below to address different areas in the segmented
keyboard.
s = size(kb1); %storing the resized value into new variable s
kalfa = 1; %selecting the first selected area and putting it into loop
15
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 17/48
CHAPTER-8
KEYPRESS & KEYSTROKE DETECTION
The main feature of our keyboard is software enabled key-press detection. In ordinary
keyboards special voltages generation or change in circuit parameter values ( eg: resistors)
are used to detect a key-press. Whenever the parameter values corresponding to key is
happen this event calls a subroutine which will return the key that is pressed.
Our method of key detection is entirely different one. It is based on counting the pixel bits.
The input to this section is the black and white image from the section image acquisition and
frame acquisition. The white bit corresponds to 1 and black corresponds to 0. We are
comparing two consecutive images taken by the webcam in MATLAB. If the two
consecutive images are the same or the difference is less than a threshold value. A key-press
is detected. And this calls the key detection section. That outputs the typed key. The
advantage of this method is that if we are keep on moving our hand then the images taken
from the webcam is not processed by the key detection( So a false key will not be detected
when our hand is moving). But the problem with this method is that we should move our
hand at a faster rate than the capture rate of webcam in between two consecutive key presses.
Key stroke detection
Keystroke detection means detecting the depth of key press. That means the time of repeated key press by without taking hand from a particular key. Our programs
17
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 18/48
adjust/increase the key-press detection rate within the two consecutive press of the same key.
Since the program needs to process the same image (it needs lesser time).
while 1 % function to check whether two snaps are the same
kb1 = getsnapshot(obj);
kb1 = filter_kb(kb1);
if diff_checker(kb1) < 18
diff = sum(sum(abs(kb2-kb1)));
if diff < 500 %if the difference between the two snaps is below 500 detect a key press
dc = diff_checker(kb1);
chara = alfa_det(kb1);
end
end
kb2 = kb1;
end
function [a] = diff_checker(kb)
a = bwarea(kb)/ 1000;
return
18
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 19/48
Fig8: Keystroke detection process
CHAPTER-9
KEY DETECTION
Key detection means pressing a key and detecting the corresponding key pressed.
Many methods are there like potential difference method, latching digital keys etc. in our
method when ever we press a virtual key it’s image will taken and corresponding key press is
detected by comparing the area enclosed by our hand. If our hand cover an area more then
the threshold area the corresponding key is detected and printed on the MATLAB command
window.
Fig9: The segmented key image with hand pressed
Key ‘B’ detected which is having a lesser white area
19
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 20/48
Fig10: The black and white image of the pressed key
1 2 3
Fig11(a): the connected elements
20
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 21/48
Fig11(b): the largest connected element
From the above figure to cut the biggest white area-3 from the three areas area-1, area-2
and area-3 we are using our function filter_big().
function [y] = filter_big(bwsg)
sg_labeled = bwlabel(bwsg);
noce = max(max(sg_labeled)); % number of connected elements
if noce == 0
y = ones(100,200);
% disp(noce)
return
end
for k = 1 : noce
sg_tmp = sg_labeled==k;
t = size(find(sg_tmp==1));
s_sg_tmp(k) = t(1);
end
% s_sg_tmp
kl = find(s_sg_tmp==max(s_sg_tmp));
y = sg_labeled==kl;
return
21
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 22/48
To align the keyboard in a horizontal plane we use the size_calc_rect() function. This
function calculate the alignment of the white are(the largest white area enclosing the
keyboard) in different positions and finding the alignment where the are enclosing the white
area is minimum. This is obtained only at horizontal alignment or in vertical alignment.
function [size_sg, sg_raw] = size_calc_rect(sg_raw)
sg_raw = clip(~sg_raw);
% gonna rotate image to make it fit in the smallest matrix, so that
% allighing orthogonally
k = 0;
rot_ang = 45;
eps_rot = 0.1; % epsilon a small number to represent acceptable tolerance
while (rot_ang > eps_rot)
k = k + 1;
size_sg_raw = size(sg_raw);
size_enc_rect = size_sg_raw(1) * size_sg_raw(2);
size_sg_act = size(find(sg_raw==1));
sg_tmp1 = imrotate(sg_raw, rot_ang);
sg_tmp1 = clip(~sg_tmp1);
size_sg_tmp1 = size(sg_tmp1);
size_enc_rect_tmp1 = size_sg_tmp1(1) * size_sg_tmp1(2);
size_sg_act_tmp1 = size(find(sg_tmp1==1));
22
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 23/48
if size_enc_rect_tmp1 <= size_enc_rect
rot_ang = rot_ang / 2;
sg_raw = sg_tmp1;
else
sg_tmp1 = imrotate(sg_raw, -rot_ang);
sg_tmp1 = clip(~sg_tmp1);
size_sg_tmp1 = size(sg_tmp1);
size_enc_rect_tmp1 = size_sg_tmp1(1) * size_sg_tmp1(2);
size_sg_act_tmp1 = size(find(sg_tmp1==1));
if size_enc_rect_tmp1 <= size_enc_rect
rot_ang = rot_ang / 2;
sg_raw = sg_tmp1;
else
rot_ang = rot_ang / 2;
end
end
end
size_sg = size(sg_raw);
return
23
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 24/48
function imgn=clip(imagen)
%Clip a black letter in word.
%Example:
% imagen=imread('metal.bmp');
% imgn=clip(imagen);
% subplot(2,1,1);imshow(imagen);title('INPUT IMAGE')
% subplot(2,1,2);imshow(~imgn);title('OUTPUT IMAGE')
if ~islogical(imagen)
imagen=im2bw(imagen,0.99);
end
a=~imagen;
[f c]=find(a);
lmaxc=max(c);lminc=min(c);
lmaxf=max(f);lminf=min(f);
imgn=a(lminf:lmaxf,lminc:lmaxc);%Clip image
Fig12 : Initial key board position
24
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 25/48
Fig13(a): horizontal alignment after some rotation
Fig13(b): horizontal alignment after some rotation
25
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 26/48
The above picture is horizontally aligned with an angle of just 0.01 degree. At this position
the area to enclose the keyboard is lesser than in all other cases( see the area of the black
square enclosing the keyboard). After obtaining this picture the image is subjected to image
segmentation
IMAGE SEGMENTATION
Segmentation means dividing the entire area into small section decided by number of
rows and columns.
The total number of segments will be equal to number of rows multiplies number of columns.
To increasing or decreasing the number of keys in keyboard we only need to change the rows
or columns. The resized image size is stored in a matrix variable.
We can use the loop statement given below to address different areas in the segmented
keyboard.
From the previous figure
Fig14: the detected key from fig 7
26
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 27/48
When a key Press is detected the image is scanned from left to write in a row and whenever a
region with minimal white area is detected the key corresponding to that will be get printed
on to a notepad using MATLAB file extension function and also on to the MATLAB
command window. For that we use the alfa_dit()
function [alfa] = alfa_det(kb1)
s = size(kb1); %storing the resized value into new variable s
kalfa = 1; %selecting the first selected area and putting it into loop
for m = 1 : 2
for n = 1 : 4
kbl(:,:,kalfa) = kb1( (((m-1) * floor(s(1) / 2)) + 1 ) : ( (m * floor(s(1) / 2))) , (((n-1) *
floor(s(2) / 4)) + 1 ) : ( (n * floor(s(2) / 4))) );
kba(kalfa) = bwarea(kbl(:,:,kalfa));
kalfa = kalfa + 1;
end
end
kba(1:4) = (kba(1:4) / sum(kba(1:4)) ) * 4;
kba(5:8) = (kba(5:8) / sum(kba(5:8))) * 4;
key = 10;
for k = 1 : 8
if kba(k) < 0.9
key = k;
break;
end
end
27
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 28/48
switch key
case 1
alfa = 'a';
case 2
alfa = 'b';
case 3
alfa = 'c';
case 4
alfa = 'd';
case 5
alfa = 'e';
case 6
alfa = 'f';
case 7
alfa = 'g';
case 8
alfa = 'h';
otherwise
alfa = 'no alfa';
end
fprintf(1, '%c' , alfa); % to print the alphabets in one line
28
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 29/48
fprintf(fid, '%c' ,alfa); %to input the characters into the file data.txt
end
29
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 30/48
CHAPTER-10
BLOCK DIAGRAM
30
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 31/48
Fig15 : Block diagram
31
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 32/48
CHAPTER-11
ALGORITHM
Image capturing and preprocessing
• Reading the image.
• Resizing- to increase the processing speed(since less size means less power)
• Fixing the image to a particular colour.
• Adjust a threshold value for white/black.
• The image is converted to black and white. The level above threshold will be
converted to black/white and vice versa.• Adjust the contrast of white.
Image Conversion
1. Choose the largest white spot from the picture.
2. For this purpose we use filtering.
3. Properly align the largest white spot by a rotating algorithm.
4. Crop the picture to a desired offset(Offset level at bottom, above and sides)
5. The picture is converted to different areas according to pixel size.
Keystroke detection
1. the captured image before and after the key press.(ie, we should keep our hand on the
pressed key to take two picture)
2. if the difference in image between two consecutive images will monitor continuously.
3. if the difference between two frames is less than a predetermined threshold the image
is checked for the key which is pressed.( that means the key press is identified)
Detection
1. Scan the entire area for finding out largest area of black (that is smallest white area).
2. Scanning is row vice.
3. Detection of any large black area above a threshold value will detect the key press and
the remaining check is stopped.
Output
1. Assign alphanumeric character in Chronological order with respect to area.
32
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 33/48
2. The character will be printed on the command window.
CHAPTER-12
ADVANTAGES
33
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 35/48
• It cannot allow multiple key presses at a time, like Ctrl or Alt key functions.
• The webcam needs to be in exactly vertical and perpendicular to the keyboard
• Our keyboard prints the keys only in the MATLAB command window since we were
not developed the system software for this keyboard
• The working surface should be flat.
• Lesser performance due to the webcam
• Necessitates the use of black
Covering on the hand
• Light scattering problems
CHAPTER-14
FUTURE DEVELOPMENT
• The speed of typing can be increase by using faster camera this can also remove slant
error problem by mathematical modeling or by using two cameras
• The multiple key identification(alt ctrl) can be possible by using 3D image modeling
• Light scattering problem can be eliminate by using laser cameras
•
Camera can be place any angle by using high power cameras
• System implementation
35
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 36/48
CHAPTER-15
APPLICATIONS
1. Used with Smart phones, PDAs, email, word processing and spreadsheet
tasks.2. As computer/PDA input.
3. Gaming and TV remote control
36
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 37/48
CHAPTER-16
CONCLUSION
Virtual Keyboard uses sensor technology and artificial intelligence to let users
work on any surface as if it were a keyboard. Projection key boards or virtual key boards
claim to provide the convenience of compactness with the advantages of a full-blown
QWERTY keyboard. The company's Virtual Keyboard is designed for anyone who's become
frustrated with trying to put information into a handheld but doesn't want to carry a notebook
computer around.
Canesta appears to be the most advanced in this class of technology. Different types
of virtual keyboards suit different typing styles. Thus virtual keyboards will make typing
easier, faster, and almost a pleasure.
37
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 38/48
CHAPTER-17
IMPORTANT MATLAB FUNCTIONS CREATED
The following are the important MATLAB functions that we created to make the virtual
keyboard with 8 keys. To work with more number of keys or to change the keyboard design
we only wanted to change the function alfa_det.m
1. cam_adjust.m
clear all
close all
obj = videoinput('winvideo', 2);
preview(obj)
2. cam_init.m
% script to initialise device
% initialising the device
clear all
38
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 39/48
close all
obj = videoinput('winvideo', 2);
preview(obj)
figure(2)
subplot(2,1,1)
kb1 = getsnapshot(obj);
imshow(kb1)
kb1 = filter_kb(kb1);
subplot(2,1,2)
imshow(kb1)
3. kbread2.m
% function [alfa] = kbread
fid = fopen('data1.txt', 'wt');% to open a file data.txt for writing using a variable fid
preview(obj) % to open the hardware of the webcam to fasten the snapshot
kb2 = kb1;
while 1 % function to check whether two snaps are the same
kb1 = getsnapshot(obj);
kb1 = filter_kb(kb1);
39
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 40/48
if diff_checker(kb1) < 18
diff = sum(sum(abs(kb2-kb1)));
if diff < 500 %if the difference between the two snaps is below 500 detect a key
press
dc = diff_checker(kb1);
chara = alfa_det(kb1);
fprintf(1, '%c' , chara); % to print the alphabets in one line
fprintf(fid, '%c' , chara); %to input the characters into the file data.txt
end
end
kb2 = kb1;
end
4. filter_kb.m
% fuction for filtering biggest element and return back
function [kb1] = filter_kb(kb1)
s = size(kb1); %Finding the size of main image(Original image)
kb1 = imresize(kb1, 200/s(1)); %Resizing the image to enhance the processing speed
% kb1 = kb1(:,:,2); % third layer offers highest contrast with human skin, i.e with the blue
layer in std format for RGB
40
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 41/48
kb1 = im2bw(kb1); % we might have to write a function for setting level value, i.e setting the
threshold
kb1 = imrotate(kb1, 180);
kb1 = ~kb1;
kb1 = filter_big(~kb1); %to filter in the biggest white area as keyboard
[skb1, kb1] = size_calc_rect(kb1); %to horizontally align the max white keyboard area so as
to eliminate the unwanted areas
s = size(kb1); %to assign the new size and put it into variable kb1
kb1 = imresize(kb1, [100 200]); %still we are scaling the figure based on column
5. filter_big.m
function [y] = filter_big(bwsg)
sg_labeled = bwlabel(bwsg);
noce = max(max(sg_labeled)); % number of connected elements
if noce == 0
y = ones(100,200);
return
end
41
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 42/48
for k = 1 : noce
sg_tmp = sg_labeled==k;
t = size(find(sg_tmp==1));
s_sg_tmp(k) = t(1);
end
kl = find(s_sg_tmp==max(s_sg_tmp));
y = sg_labeled==kl;
return
6. size_calc_rect.m
% fuction to calculate size of a rectangular piece
function [size_sg, sg_raw] = size_calc_rect(sg_raw)
% double(sg_raw);
sg_raw = clip(~sg_raw);
% sg_bkup = sg_raw; % for test insertion, delete soon this line
% gonna rotate image to make it fit in the smallest matrix, so that
% allighing orthogonally
k = 0;
42
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 44/48
size_sg_tmp1 = size(sg_tmp1);
size_enc_rect_tmp1 = size_sg_tmp1(1) * size_sg_tmp1(2);
size_sg_act_tmp1 = size(find(sg_tmp1==1));
if size_enc_rect_tmp1 <= size_enc_rect
rot_ang = rot_ang / 2;
sg_raw = sg_tmp1;
else
rot_ang = rot_ang / 2;
end
end
end
size_sg = size(sg_raw);
return
7. clip.m
Afunction imgn=clip(imagen)
%Clip a black letter in word.
if ~islogical(imagen)
imagen=im2bw(imagen,0.99);
44
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 45/48
end
a=~imagen;
[f c]=find(a);
maxc=max(c);lminc=min(c);
lmaxf=max(f);lminf=min(f);
imgn=a(lminf:lmaxf,lminc:lmaxc);%Clip image
8. diff_checker.m
% function to keep reading keyboard, and calculate the area
function [a] = diff_checker(kb)
a = bwarea(kb)/ 1000;
return
9. alpha_dit.m
function [alfa] = alfa_det(kb1)
s = size(kb1); %storing the resized value into new variable s
kalfa = 1; %selecting the first selected area and putting it into loop
for m = 1 : 2
for n = 1 : 4
kbl(:,:,kalfa) = kb1( (((m-1) * floor(s(1) / 2)) + 1 ) : ( (m * floor(s(1) / 2))) , (((n-1) *
floor(s(2) / 4)) + 1 ) : ( (n * floor(s(2) / 4))) );
45
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 46/48
kba(kalfa) = bwarea(kbl(:,:,kalfa));
kalfa = kalfa + 1;
end
end
kba(1:4) = (kba(1:4) / sum(kba(1:4)) ) * 4;
kba(5:8) = (kba(5:8) / sum(kba(5:8))) * 4;
key = 10;
for k = 1 : 8
if kba(k) < 0.9
key = k;
break;
end
end
switch key
case 1
alfa = 'a';
case 2
alfa = 'b';
46
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 47/48
case 3
alfa = 'c';
case 4
alfa = 'd';
case 5
alfa = 'e';
case 6
alfa = 'f';
case 7
alfa = 'g';
case 8
alfa = 'h';
otherwise
alfa = 'no alfa';
end
47
8/6/2019 Project Final 23-06-099
http://slidepdf.com/reader/full/project-final-23-06-099 48/48
CHAPTER-18
REFERENCES
1. A three state model of graphical input by William A.S Buxton
www.billbuxton.com/3state.html
2. Kölsch, M. and Turk, M. Keyboards without Keyboards: A Survey of Virtual
Keyboards, Workshop on Sensing and Input for Media-centric Systems, Santa Barbara, CA,
June 20-21, 2002.
3. Alessandro Valli, “ Natural interaction”, Touch Panel, Vol. 2, No. 5, July 2007
4. Discover a powerful new virtual on screen keyboard with unlimited users
http://www.virtual-keyboard.com/special_needs/special_needs.html
5. VKB Bluetooth virtual keyboard- Reviewed by Bonnie Cha
http://reviews.cnet.com/pda-accessories/vkb-bluetooth-virtual-keyboard/4852-6460_7-
31293682.html