41
Customizable Virtual Keyboard Introduction High Level Design Software Implementation Hardware Implementation Testing Conclusion Appendix Please send any comments or questions to: Naweed Paya ([email protected]) Venkat Ganesh ([email protected]) Introduction

Final Report on Virtual Keyboard

Embed Size (px)

Citation preview

Page 1: Final Report on Virtual Keyboard

Customizable Virtual Keyboard

Introduction High Level Design Software Implementation Hardware Implementation Testing Conclusion Appendix

Please send any comments or questions to:

Naweed Paya ([email protected]) Venkat Ganesh ([email protected])

Introduction

It is becoming increasingly difficult for users to interact with the slew of portable gadgets they carry, especially in the area of text entry. Although miniature displays and keyboards make some portable devices, such as cell phones and PDAs, amazingly small, users’ hands do not shrink accordingly.

Page 2: Final Report on Virtual Keyboard

To solve this problem, we proposed a Virtual Keyboard. This device will replace a physical keypad with a customizable keyboard printed on a standard A3 size paper whose “keystrokes” are read and translated to real input. This virtual keyboard can be placed on any flat surface, such as desktops, airplane tray tables, kitchen counters, etc. and can theoretically be interfaced with any computing device that requires text entry. This would eliminate the need to carry anything around and also prevent any chance of mechanical damage to the keypad in harsh environments if a simple lamination is used to protect the paper. In addition, buttons on this device can be reconfigured on-the-fly to give a new keyboard layout using a GUI we built in Java and then transferring that data to the device using a computer’s serial port.

High Level Design

Figure 1: Finger shining bright red when passing through the laser plane

The Virtual Keyboard has three main components: the laser, camera, and printed keyboard. The laser beam is simply a conventional off-the-shelf red laser with a line-generating diffractive optical element attached to it. This assembly generates an invisible plane of red light hovering a few millimeters above the typing surface. When a finger passes through this plane, it shines bright red in that region.

Figure 2: C3038 image sensor module mounted on a custom PCB

The CMOS camera continuously captures images of the region where the printed keyboard is supposed to be placed and checks these images for red color data above a specified threshold. The threshold concept works in this case because the laser shining on a typical human finger generates saturating values of red color data, which is very easily distinguishable from its surroundings.

Page 3: Final Report on Virtual Keyboard

Figure 3: Comparing an actual keyboard with a printed keyboard

Lastly, the printed keyboard is simply a standard A3 size paper that contains a custom keyboard layout. After rigorous testing, we decided to use a black background and blue letters for the printed keyboard because our device doesn’t use its own light source. Therefore, proper contrast is necessary to distinguish the typing finger from the surrounding area in various lighting conditions. The actual programming of the printed keyboard layout into the device can be done using a serial port and a GUI we developed in Java. This GUI basically gives the user a blank grid of buttons and the user can choose to assign any button to any letter or number he/she desires.

Software Implementation

The software component was split into 5 main components:

Implementing the I2C protocol to read and write registers from camera Reading values from camera to obtain 6 frames every second Processing the images to obtain a pressed key Converting the pressed key into a scan code which is then transmitted using the PS/2

protocol Sending serial data from a java application to update the array of scan codes in the

Mega32

Main Operation

At first we initialize PORTA on the Mega32 to take UV input from the camera and PORTC to communicate with the camera over the I2C interface. The baud rate is set to 19,200bps for serial communication. We then run the calibrate function on the camera, which looks at a black keyboard to determine a distinguishable value for red color threshold. Then we call a function called "init_cam" which performs a soft reset on the camera before writing the required values to corresponding camera registers. These registers change the frame size to 176x144, turn on auto white balance, set the frame rate to 6 fps, and set the output format to 16-bit on the Y/UV mode

Page 4: Final Report on Virtual Keyboard

with Y=G G G G and UV = B R B R. The code then enters an infinite loop which checks for the status of the PS2 transmitting queue and tries to process the next captured frame if the queue is empty. If not, the queue is updated and the PS2 transmission is allowed to continue.

Image Processing

The getRedIndex function captures rows of data from the camera and processes each of them. We first wait until a neg edge on the VSYNC, which indicates the arrival of new frame data on the UV and Y lines. We then wait for one HREF to go by since the first row of data is invalid. At this point, we can clock in 176 pixels of data for a given vertical line in the Bayer format.

Figure 4: Bayer color pattern

In the mode where the UV line receives BR data, the output is given by: B11 R22 B13 R24 and so on. Since we only needed red data, we stored an array of 88 values in which we captured the data on the UV line every 2 PCLKS. The OV6630 also repeats the same set of pixels for consecutive rows and thus 2 vertical lines processed would have data about the same pixels. We considered optimizing this by completely dropping data about the even rows, but this was not going to save us anything since all our processing could be done between one neg edge and a pos edge (when data becomes valid again) of HREF.

Since we don’t have enough memory to store entire frames of data to process, we do the processing after each vertical line. After each vertical line of valid data, HREF stays negative for about 0.8ms and the camera data becomes invalid; this gives us ample time to process one line worth of data. After each vertical line was captured, we looped through each pixel to check if it exceeded the red threshold found during calibration. For every pixel that met this threshold, we then checked if the pixel was part of a contiguous line of red pixels, which would indicate the presence of a key press. If such a pixel was found, we then mapped this pixel to a scan code by binary searching through an array of x, y values. If this scan code was found to be valid, we debounced the key by checking for 4 continuous presses, and then added the detected key to the queue of keys to send to the PC.

I2C Communication

A very big part of our challenge was to figure out the correct configuration to use to capture and process the images from the camera. The communication protocol was not easy to work with and there were a total of about 92 registers we could use to set up the camera. At first we considered using the TWI interface provided by CodeVision to communicate with the camera, but we were

Page 5: Final Report on Virtual Keyboard

unable to do so. Thus, we decided to modify and use a version developed by Ruibing Wang, which uses a lot of the TWI settings provided on the Mega32. The protocol uses a 2-wire communication scheme, which is activated by a 10kOhm pull-up resistor. The clock signal to the camera is provided by the SCL line, and the frequency is given by: 16MHz / (16 + 2 x (TWBR)(4TWPS)). We decided the optimal solution would be to satisfy the minimum requirement which was to set the bit rate register (TWBR) to 72 and the status register (TWSR) to 0. The rest of the code just followed the standard protocol defined by Philips. The camera registers could be written by writing a start bit, followed by a target register address and then the target data. We had no need to read from the camera registers except in the initial phase when we had to make sure we had the protocol working properly.

Camera Settings

We decided to use a resolution of 176x144 since that was the minimum required to detect an entire A3 size paper on which the keyboard would be printed. At this resolution, we could capture at most 6 frames of color images per second. The camera output format was set to capture 16 bit UV/Y data, where UV had BRBR data and Y had GGGG data. The Y data was completely ignored.

Figure 5: Screenshot of Java GUI to create custom keyboard layouts on-the-fly

Programming the EEPROM

Since we wanted to be able to change the key assignments on the fly, we stored the array of scan codes corresponding to each key in EEPROM and turned on the RS-232 receive interrupt. We also wrote a java applet that was a simple GUI where the user can enter scan codes of the keys they desire and transmit it to the microcontroller through a standard COM port on the PC.

Keyboard Output (PS/2)

The code was structured using two timer compare interrupts where timer1 compare was used to start transmissions of each data byte and timer2 compare was used to reset the waiting. Since the protocol allows a range of frequencies that a computer would understand, we decided to use a clock time of 250 and wait time of 700. When the timer1 interrupt is fired, it transmits the bits in the following order when the clock is set to high: start bit(0), data bits, parity bit(xor of all bits), and a stop bit(1). If not, the clock state is updated. The rest of the code simply maintains a queue which would hold the elements to transmit as characters. The queue has a get and put method that updates the 2 pointers in an array.

Page 6: Final Report on Virtual Keyboard

Hardware Implementation

The three main components of our hardware design are as follows:

Laser module Camera and its associated circuitry Outer casing for the entire device

Laser

Figure 6: Red laser module with a line-generating DOE attached

Our original plan, at the time of the project proposal, was to use an infrared laser to detect button presses using the CMOS camera, but we realized that user safety would be a major issue in that case. The user would never know even if he/she is staring directly at the laser and, therefore, there would be no way to prevent eye damage. In addition, we also realized that the CMOS camera we’re using (OV6630) is not very effective at detecting infrared light. Hence, we decided to use a Class II 635nm red laser instead.

The laser module we bought came with a built-in driver; therefore, we didn’t have to worry about biasing the laser properly to make it operational. All we had to do was to connect the laser to a 3V power source, which we obtained using a simple 3V voltage regulator.

Figure 7: Laser line generation calculation

Page 7: Final Report on Virtual Keyboard

The laser module also came with a line-generating diffractive optical element attached to it. However, since we didn’t know the fan-angle for this DOE, we had to experiment with various distances in order to obtain a line length of at least 8.5”, which was required to cover the entire width of our printed keypad. In the end, we had to place the laser at a distance of approximately 12.5” to obtain good results.

Camera

Figure 8: C3038-4928IR 1/4” Color Sensor Module

For this project we decided to use the C3038 1/4” color sensor module with digital output, which uses OmniVision’s CMOS image sensor OV6630. The two primary reasons why we chose this specific camera module were low cost and the fact that it is capable of outputting image color data in progressive scan mode. Progressive scanning was an important consideration for us since we don’t have enough computational power available on the 16Mhz Mega32 microcontroller to process entire frames at once; however, we can certainly process images line-by-line as they come in. After rigorous testing and a lot of research, we realized that we could work with only the red channel data from the camera and still be able to identify keystrokes accurately. Hence, we connected the 8-bit red channel output from the camera (UV[7:0]) to PORTA[7:0] on the Mega32.

Casing

The hardware assembly for our device is designed to hold the camera at a fixed position such that it looks over the appropriate region of the printed keyboard. In addition, it also holds the laser module at a fixed position such that the plane of red light completely covers the area above the printed keyboard. In order to ensure that new custom-printed keyboards can be swapped in-and-out of the device while maintaining proper distances, we permanently attached a piece of black poster board of the right length to the assembly and mounted 4 photo-corners on it.

Page 8: Final Report on Virtual Keyboard

Figure 9: Hardware casing for Virtual Keyboard device

Testing

Keystroke Accuracy:

As a result of the limited viewing angle of the camera and positioning of the laser, we had to design and calibrate with various keypad layouts to make sure we could detect all of the buttons with reasonable accuracy. Our final design for the generic keypad and testing results (percentage accuracy) for this layout are given in Figure 3. For the testing, we tried 100 keystrokes per key and set the acceptance threshold at 70% for side areas and 80% for the central area. This means that if we can recognize a certain key accurately at least 70 or 80 times, respectively, out of the 100 times that it’s pressed, that key passes the test.

Figure 10: Testing results for keystroke detection accuracy

Conclusion

Page 9: Final Report on Virtual Keyboard

Although the final project was very satisfying, our results did not completely meet our expectations. The keyboard worked as we predicted but typing speed was minimal (about 60 characters per minute) due to limited processing capabilities of the Mega32 microcontroller.

If we had more time, we would have liked to increase the theoretical maximum typing speed by possibly using another microcontroller in parallel or maybe even an external FPGA to do extra image processing. In addition, we would also like to include sound effects for keystrokes and a dynamic calibration algorithm which can be used to orient the custom-printed keyboard in any direction. This sort of functionality would require performing 2D image transforms on-the-fly, which is not feasible with the existing microcontroller. Last but not least, we could certainly try to improve our current keystroke detection algorithm to improve typing accuracy.

Appendix

Standards

I2C-Bus Specification Version 2.1:

Two wires in an I2C bus, serial data (SDA) and serial clock (SCL), carry information between the devices connected to the bus. Each device is recognized by a unique address and can operate as either a transmitter or receiver, depending on the function of the device. In addition to transmitters and receivers, devices can also be considered as masters or slaves when performing data transfers. A master is the device which initiates a data transfer on the bus and generates the clock signals to permit that transfer. At that time, any device addressed is considered a slave.

PS/2 Keyboard Protocol:

The PS/2 keyboard interface typically uses a bidirectional synchronous serial protocol, but for our implementation we do not need the computer (host) to communicate with the microcontroller (device). Therefore, for our purposes, the device always generates the clock signal and all data is transmitted one byte at a time. Each byte is sent in a frame consisting of 11 bits, in the following order:

1 start bit. This is always 0. 8 data bits, least significant bit first. 1 parity bit (odd parity). 1 stop bit. This is always 1.

The parity bit is set to 1 if there is an even number of 1's in the data bits and set to 0 if there is an odd number of 1's in the data bits. The number of 1's in the data bits plus the parity bit always add up to an odd number (odd parity.) This is used for error detection. Data sent from the device to the host is read on the falling edge of the clock signal; the clock frequency must be in the range 10 - 16.7 kHz. This means clock must be high for 30-50 µs and low for 30-50 µs.

Page 10: Final Report on Virtual Keyboard

Ethical and Legal Considerations

Throughout the final project, we committed ourselves to the highest ethical and professional conduct and closely adhered to the IEEE Code of Ethics. We placed an extra emphasis on the following points mentioned in the Code of Ethics:

1. To accept responsibility in making decisions consistent with the safety, health and welfare of the public, and to disclose promptly factors that might endanger the public or the environment.

2. To avoid real or perceived conflicts of interest whenever possible, and to disclose them to affected parties when they do exist.

3. To be honest and realistic in stating claims or estimates based on available data. 4. To improve the understanding of technology, its appropriate application, and potential

consequences. 5. To maintain and improve our technical competence and to undertake technological tasks

for others only if qualified by training or experience, or after full disclosure of pertinent limitations.

6. To seek, accept, and offer honest criticism of technical work, to acknowledge and correct errors, and to credit properly the contributions of others.

7. To avoid injuring others, their property, reputation, or employment by false or malicious action.

8. To assist colleagues and co-workers in their professional development and to support them in following this code of ethics.

Since we were using a Class II laser device, we always made sure to keep the laser targeted away from other individuals in the Lab. In addition, we designed an enclosure for our device such that the laser would not be visible to the user.

We did not have any legal considerations since we did not use code or algorithms from other sources, did not use parts regulated by federal agencies, and did not infringe upon any existing patents. Although a commercial product similar to our device, called the “Virtual Laser Keyboard,” is currently manufactured by a company known as I-Tech, we believe that we have significantly distinguished our product such that we will not encounter any copyright issues. The commercial product is not dynamically reconfigurable, uses a red laser to project a standard QWERTY keyboard pattern onto a surface, and uses an infrared laser for keystroke detection. Our product, on the other hand, uses a customizable printed keyboard, a red laser for keystroke detection, and custom keystroke detection algorithms.

Safety

Page 11: Final Report on Virtual Keyboard

Figure 11: EM radiation absorption characteristics of the human eye

The human body is vulnerable to the output of certain lasers, and under certain circumstances, exposure can result in damage to the eye and skin. Research relating to injury thresholds of the eye and skin has been carried out in order to understand the biological hazards of laser radiation. It is now widely accepted that the human eye is almost always more vulnerable to injury than human skin. The cornea (the clear, outer front surface of the eye’s optics), unlike the skin, does not have an external layer of dead cells to protect it from the environment. Hence, the cornea absorbs the laser energy and may be damaged. The figure below illustrates the absorption characteristics of the eye for different laser wavelength regions. Since we only used a Class II laser in this project and provided a proper enclosure for the device such that it isn’t directly visible to the user, special protection is not required for normal users. People with sensitive eyesight or other severe vision problems, however, might want to take some precautionary measures and should not use the device for extended periods of time. In addition, users are strongly advised not to look directly into the laser beam at any time.

Budget

Our total budget for this project was $75.00, and we easily managed to keep our costs less than that.

PART COST SOURCE

Total $48.41 --

RS-232 Serial Port $1.00 ECE 476 Digital Lab

MAX233CPP -- Sampled

Red LED -- ECE 476 Digital Lab

Jumpers -- ECE 476 Digital Lab

Surface mount capacitors, resistors -- ECE 476 Digital Lab

16 MHz Crystal Oscillator -- ECE 476 Digital Lab

LM340T5 Voltage Regulator -- Sampled

Slide Switch -- ECE 476 Digital Lab

Atmel ATMega32 $8.00 ECE 476 Digital Lab

40 pin DIP Socket $2.00 ECE 476 Digital Lab

8 pin DIP Socket $0.40 ECE 476 Digital Lab

635nm Laser $8.00 Ebay

9V Power supply -- ECE 476 Digital Lab

Page 12: Final Report on Virtual Keyboard

PART COST SOURCE

OV6630 CMOS Camera $25.03 Electronics123

PS/2 cable and USB adapter -- Previously owned

Poster board $3.99 Walmart

Wood -- Scrap

Task Distribution

Naweed Paya:

Solder prototype board and C3088 camera module Camera implementation Design and assemble hardware casing PS/2 communication Testing and debugging Final report

Venkat Ganesh:

Laser and camera implementation Java applet I2C communication Testing and debugging Final report

Acknowledgements

We would like to thank Prof. Bruce Land and the ECE 476 staff for their continual support, insightful comments, and suggestions which altogether made this project possible. We would also like to thank Ruibing Wang, an ECE M. Eng student at Cornell University, for his assistance in getting the I2C protocol to work. This protocol was necessary to communicate with the camera, and was therefore an integral component of our project.

In addition, we would like to reference two past ECE 476 projects. To get an initial idea of the camera settings required for proper operation of the OV6630 camera using a Mega32 microcontroller, we looked at the camera implementation in the project titled “Autonomous SearchBot” by John and Diego. In addition, we used the “Wireless Keyboard” project by Luke Hejnar and Sean Leventhal as an example to implement the PS/2 keyboard protocol using a Mega32 microcontroller.

Page 13: Final Report on Virtual Keyboard

Code Listing

Custom keyboard design GUI Serial communication from PC to Virtual Keyboard Camera operation and Image processing PS/2 keyboard protocol I2C communication

Schematic

Download in PDF format Download in SCH format

Datasheets

Mega32 microcontroller C3038 camera module OV6630 optical sensor ADP3300ART-32 voltage regulator

Code References

AVRCam Bayer color pattern

Vendor Sites

Digikey Electronics123

Background Sites

ECE 476 AVR serial communication PS/2 Keyboard Protocol

©2008 Cornell University

Page 14: Final Report on Virtual Keyboard

Custom keyboard design gui code.

import java.awt.BorderLayout;import java.awt.Container;import java.awt.FlowLayout;import java.awt.GridLayout;import java.awt.event.ActionEvent;import java.awt.event.ActionListener;

import javax.swing.JButton;import javax.swing.JFrame;import javax.swing.JTextArea;import javax.swing.JTextField;import javax.swing.SwingUtilities;

public class keyboardGUI extends JFrame implements ActionListener{private JButton program;private int NUM_COLUMNS = 10;private int NUM_ROWS =4;private JTextField input[] = new JTextField[NUM_ROWS*NUM_COLUMNS];private Container inputPane, infoPane;private JTextField port;private JTextArea msgs;private String[] orig = {"15", "1D", "24", "2D", "2C", "35", "3C",

"43", "44", "4D","1C", "1B", "23", "2B", "34", "33", "3B", "42", "4B",

"66","1A", "22", "21", "2A", "32", "31", "3A", "41", "49",

"5A","ff", "ff", "58", "29", "29", "29", "29", "29", "ff",

"ff"};public static void main(String[] args) {

SwingUtilities.invokeLater(new Runnable(){public void run(){

createAndShowGUI();}

});}public static void createAndShowGUI(){

JFrame keyboardFrame = new keyboardGUI("ECE 476 Virtual Keyboard Programmer");

JFrame.setDefaultLookAndFeelDecorated(true);keyboardFrame.setDefaultCloseOperation(EXIT_ON_CLOSE);keyboardFrame.pack();keyboardFrame.setVisible(true);

}

public keyboardGUI(String title){super(title);setLayout(new BorderLayout());program = new JButton("Program!");program.setToolTipText("Program current configuration!");infoPane = new Container();infoPane.setLayout(new FlowLayout(FlowLayout.CENTER));infoPane.add(program);

Page 15: Final Report on Virtual Keyboard

port = new JTextField("COM6");infoPane.add(port);msgs = new JTextArea();infoPane.add(msgs);msgs.setEditable(false);add(infoPane, BorderLayout.SOUTH);inputPane = new Container();inputPane.setLayout(new GridLayout(NUM_ROWS, NUM_COLUMNS));for (int i=0; i<NUM_ROWS*NUM_COLUMNS; i++){

input[i] = new JTextField(2);inputPane.add(input[i]);input[i].setText("" + orig[i]);

}add(inputPane, BorderLayout.NORTH);program.addActionListener(this);

}

private char[] computeByteStream(){char[] toReturn = new char[NUM_ROWS*NUM_COLUMNS];for (int i=0; i<NUM_COLUMNS*NUM_ROWS; i++){

if (input[i].getText()!=null && input[i].getText().trim()!=""){

toReturn[i] = (char)hex2decimal(input[i].getText());

} else toReturn[i] = 255;}System.out.println(toReturn);return toReturn;

}@Overridepublic void actionPerformed(ActionEvent e) {

if (e.getSource().equals(program)){// send serial datamsgs.setText(SerialComm.tranmit(computeByteStream(),

port.getText()));}

}

public static int hex2decimal(String s) { String digits = "0123456789ABCDEF"; s = s.toUpperCase(); int val = 0; for (int i = 0; i < s.length(); i++) { char c = s.charAt(i); int d = digits.indexOf(c); val = 16*val + d; } return val; }

}

Page 16: Final Report on Virtual Keyboard

Serial communication from pc to virtuall keyboard code.

import java.io.OutputStream;import java.io.PrintWriter;

import javax.comm.CommPortIdentifier;import javax.comm.SerialPort;

public class SerialComm {

static CommPortIdentifier portId1;

OutputStream outputStream; SerialPort serialPort; public static String tranmit(char[] data, String port) { try { portId1 = CommPortIdentifier.getPortIdentifier(port); new SerialComm(data); return "Programming successful!"; } catch (Exception e) { e.printStackTrace(); return "Programming failed due to " + e.getLocalizedMessage();

} };

public SerialComm(char[] data) throws Exception { serialPort = (SerialPort) portId1.open("ComControl", 2000); serialPort.setSerialPortParams(19200, SerialPort.DATABITS_8, SerialPort.STOPBITS_1, SerialPort.PARITY_NONE); serialPort.setDTR(false); serialPort.setRTS(false); outputStream = serialPort.getOutputStream(); for (int i=0; i<data.length; i++) { outputStream.write(data[i]); Thread.sleep(100); } outputStream.close(); }}

Page 17: Final Report on Virtual Keyboard

Image processing code.

/* Standard Includes */#include <Mega32.h>#include <delay.h>#include "debugger.c"

/* I2C communication */#include "twi_master.h"

/* PS/2 communication */#include "keyboard.c"

/* System Constants */#define CPU_FREQUENCY_HZ 16000000

/* TWI Constants */#define TWI_PULLUP_RESISTOR_ENABLE TRUE#define TWI_CAMERA_WRITE_ADDRESS 0xC0#define TWI_CAMERA_READ_ADDRESS 0xC1#define TWI_DATA_BUFFER_LENGTH 1

/* Camera Pin Assignments */#define HREF PINC.4#define VSYN PINC.3#define PCLK PINC.2

/* Image constants */#define LENGTH 144#define WIDTH 88#define CAPTURE_IMG FALSE#define MAX_VERT 115#define MIN_VERT 45

/* Keyboard constants */#define YSIZE 4#define XSIZE 10#define null 255

/* Global Variables */

// I2C variables for camera read and writeunsigned char twi_data_buffer[TWI_DATA_BUFFER_LENGTH];unsigned char horiz,vert, startw, write, startr, read;

// red threshold value above which we detect a key presschar rThres;

// serial port recieve buffer - used to update the eeprom to swap scanCodeschar r_char

// debouncing variablesunsigned char activeChar, activeCount;

Page 18: Final Report on Virtual Keyboard

// y-coordinates of each vertical line (determined by calibration)flash unsigned char YgridArray[YSIZE+1] = {46, 62, 74, 92, 102};

// x-coordinates of each key (determined by calibration)flash unsigned char XgridArray[YSIZE][XSIZE+1] ={1, 7, 13, 18, 27, 36, 43, 50, 59, 63, 72,2, 9, 17, 22, 29, 37, 43, 50, 57, 65, 72,5, 11, 17, 24, 30, 37, 43, 50, 57, 64, 71,7, 12, 19, 25, 32, 38, 43, 50, 57, 63, 70};

// scanCodes associated with each keyeeprom unsigned char scanCodes[YSIZE][XSIZE] = {0x15, 0x1D, 0x24, 0x2D, 0x2C, 0x35, 0x3C, 0x43, 0x44, 0x4D 0x1C, 0x1B, 0x23, 0x2B, 0x34, 0x33, 0x3B, 0x42, 0x4B, 0x66, 0x1A, 0x22, 0x21, 0x2A, 0x32, 0x31, 0x3A, 0x41, 0x49, 0x5A, 0xff, 0xff, 0x58, 0x29, 0x29, 0x29, 0x29, 0x29, 0xff, 0xff};

// used when updating the scanCodes arrayunsigned char scan_index;unsigned char searchArray(unsigned char, unsigned char);

//UART character-ready ISR: updates the scanCodes array on each byteinterrupt [USART_RXC] void uart_rec() {

r_char=UDR; scanCodes[scan_index/XSIZE][scan_index%XSIZE] = r_char; scan_index++;}

// read datafrom camera at register regNumunsigned char camera_read(unsigned char regNum) { startw = twi_master_start((0xC0>>1), 0); // start write write = twi_write_byte(regNum); // write byte twi_stop(); // stop startr = twi_master_start((0xC0>>1), 1); // fresh start read = twi_read_byte(twi_data_buffer); // read into buffer twi_stop(); // stop return twi_data_buffer[0];}

// write data to camera at regNumvoid camera_write(unsigned char regNum, unsigned char data) { startw = twi_master_start((0xC0>>1), 0); // start write write = twi_write_byte(regNum); // write register number write = twi_write_byte(data); // write data twi_stop(); // stop}

// looks for spots above the calibrated red threshold. If found, updates the keyboard queue to transmit charactervoid getRedIndex(unsigned char max) {

unsigned char red; unsigned char RData[WIDTH]; unsigned char redArrX[50];

while(!VSYN); // wait till new frame is ready while(VSYN); // start read on neg edge

Page 19: Final Report on Virtual Keyboard

while(!HREF); // discard first row of data sent by camera while(HREF);

// we are only concerned about data till max rows for(vert = 0; vert < max; vert++) { unsigned char redCountInside = 0; unsigned char key, refreshed; while(!HREF); for(horiz = 0; horiz < WIDTH; horiz++) { while(!PCLK); // wait for pos edge of PCLK to read data while(PCLK); // discard the Blue data while(!PCLK); // wait for another edge and then record Red data RData[horiz] = PINA; while(PCLK); } // post processing on each row for(horiz = 0; horiz < WIDTH; horiz++) { if (RData[horiz]>rThres) { // if we find some point above threshold, we look for contiguous points (2 points not more than 2 pixels apart) if (redCountInside!=0 && horiz-redArrX[redCountInside-1]<3) { redArrX[redCountInside++] = horiz; } else { redArrX[0] = horiz; redCountInside=1; } // we have found a point within the keyboard limits to consider if (vert>MIN_VERT && redCountInside>2) { // find the key pressed key = searchArray(horiz-1, vert); // debounce by counting how many frames have caught the same key if (key == activeChar && activeCount>0) activeCount++; else { activeChar = key; activeCount = 1; } // if it's been captured for 6 frames, we output that key if (activeCount==6 && key!=255) { activeCount = 0; queuePut(key); queuePut(0xf0); queuePut(key); return;

Page 20: Final Report on Virtual Keyboard

} } } } while(HREF); } }

// print data - to capture an image for debug purposes if (CAPTURE_IMG) { for(horiz = 0; horiz < WIDTH; horiz++) { red= RData[horiz]; reportVhex(red); } }}

// binary searches the x, y arrays to find the scanCode of the key at button (x, y)unsigned char searchArray(unsigned char x, unsigned char y) { unsigned char i, j, temp, yVal; // y out of bounds if (y > YgridArray[YSIZE] || y < YgridArray[0]) return null; i = 0; j = YSIZE; // update the bounds depending on whether the value is bigger or smaller than current value while (i<j) { temp = (i + j); temp = (temp>>1) + (1&temp); if (YgridArray[temp]>=y && YgridArray[temp-1]<=y) break; else if (YgridArray[temp] < y) i = temp; else j = temp; } i = 0; j = XSIZE; // store y position yVal = temp-1; // x out of bounds if (x > XgridArray[yVal][XSIZE] || x < XgridArray[yVal][0]) return null; // update the bounds depending on whether the value is bigger or smaller than current value while (i<j) { temp = (i + j); temp = (temp>>1) + (1&temp); if (XgridArray[yVal][temp]>x && XgridArray[yVal][temp-1]<x) break;

Page 21: Final Report on Virtual Keyboard

else if (XgridArray[yVal][temp] < x) i = temp; else j = temp; } // get the appropriate scanCode return scanCodes[yVal][temp-1];}

// perform on a blank keyboard - captures the highest red value and determines an appropriate thresholdunsigned char calibrate(){ unsigned char RData[WIDTH]; unsigned char thres = 0; while(!VSYN); while(VSYN); while(!HREF); while(HREF); // loop through all the pixels to capture maximum red value for(vert = 0; vert < LENGTH; vert++) { while(!HREF); for(horiz = 0; horiz < WIDTH; horiz++) { while(!PCLK); while(PCLK); while(!PCLK); RData[horiz] = PINA; while(PCLK); } if (vert>MIN_VERT && vert<MAX_VERT) { for(horiz = 0; horiz < WIDTH; horiz++) { if (RData[horiz]>thres) thres = RData[horiz]; } } while(HREF); } // if background red content is already too high, keyboard can't be used; attempt to capture points beyond 215 if (thres>215) return 215; // otherwise just look for values 10 above threshold return thres + 10;}

void initCam(){ /* define registers of the camera with following data: * Frame size = 176*144 * Auto white balance on * Frame rate = 6 * Output Format = 16 bit Y/UV line with Y = G G G G and UV = R B R B */ delay_ms(10); camera_write(0x12, 0x80); // initiate a soft reset to make sure that all the regsiters are at default values

Page 22: Final Report on Virtual Keyboard

delay_ms(2); camera_write(0x14, 0x20); // reduce frame size delay_ms(2); camera_write(0x12, 0x2C); // AGC enable, set RGB mode, with AWB delay_ms(2); camera_write(0x28, 0x04); // set color sequencer delay_ms(2); camera_write(0x13, 0x01); // un-tri-state the Y/UV lines, select 16 bit format delay_ms(2); camera_write(0x11, 0x08); // reduce the fps delay_ms(2); camera_write(0x0D, 0x00); // white balance red ratio adjust delay_ms(2); camera_write(0x02, 0x81); // red color gain delay_ms(2); camera_write(0x01, 0x81); // blue color gain delay_ms(2); camera_write(0x03, 0xF8); // change saturation value delay_ms(2); camera_write(0x05, 0x48); // adjust contrast delay_ms(2); camera_write(0x07, 0xc6); // adjust sharpness delay_ms(2); camera_write(0x0F, 0x15); //select luminance level in AWB (auto white balance) control delay_ms(2);}

void main(void) { unsigned char i; char myState; // checks current keyboard state char temp = 0xff; init_twi_master(CPU_FREQUENCY_HZ, TWI_PULLUP_RESISTOR_ENABLE); // initialize camera initKeyboard(); //initialize keyboard

DDRA = 0x00; // UV data - configure to capture blue, red data out of which only the red is processed DDRD.2 = 1; // set up LED as output UBRRL = 51; // set baud rate to 19200 UCSRB = 0b10011000; // enable recieve interrupt // initialize keyboard debouncing variables activeCount = 0; activeChar = 255; // initialize scan_index every time system restarts - so programming of scanCodes can be done scan_index = 0; // camera calibration delay delay_ms(500);

// define VSYN and PCLK as inputs PORTC.3 = 0; PORTC.2 = 0;

Page 23: Final Report on Virtual Keyboard

initCam(); // compute the red threshold value PORTD.2 = 0; rThres calibrate(); PORTD.2 = 1;

while(1) { myState=clockStateNow(); if (queueEmpty) { // read from camera if there is nothing more to transmit PORTD.2=0; getRedIndex(MAX_VERT); PORTD.2=1; } else if(myState==IDLE) { // send only when line is idle delay_ms(4); PS2byte = queueGet(); // get next character in queue if(PS2byte!=0) { transmitting = 1; position = 0; clockState = HIGH; PS2parity = parity(PS2byte); // compute the parity } } } }

Page 24: Final Report on Virtual Keyboard

PS/2 KEYBOARD PROTOCOL CODE.

/* * PS2 Keyboard Transmission * Ports Used: * D.7 - Output Clock * D.6 - Output Data */#include <Mega32.h>#include <delay.h>

#define CLK_TIME 250#define RESPOND_WAIT 700

//define some useful maps to bits#define IDLE 0#define BUSY 1

//define ps/2 data line states#define CLK_OUT PORTD.7#define DATA_OUT PORTD.6

#define breakCode 0xF0

//state of the data linechar transmitting, waiting;

//variables for writing to PS/2 portchar position, PS2byte, PS2parity;char clockState;

//define clock states#define HIGH 0#define FALLING 1#define LOW 2#define RISING 3

//define queue values#define QUEUELEN 100

char queue[QUEUELEN]; //queue of data sent by keyboard. (first in, first out)char queueFull; //indicates if queue is fullchar queueEmpty; //indicates if queue is emptychar queueIn; //indicates where to put data into queuechar queueOut; //indicates where to take data out of queue

char queuePut(char d); //put data into queuechar queueGet(void); //get data from queue interrupt [TIM1_COMPB] void t1_cmpB(void) {

waiting=0;TCNT1=0;

}

Page 25: Final Report on Virtual Keyboard

//this interrupt occurs four times per clock cycle on the ps/2 lineinterrupt [TIM1_COMPA] void t1_cmpA(void) {

TCNT1=0;if(transmitting) { if(clockState==HIGH) { //write data out here

if(position==0) { //start bit DATA_OUT = 0; } else if(position<9) { //data bits DATA_OUT = PS2byte&0x01; PS2byte = PS2byte>>1; } else if(position==9) { //parity bit DATA_OUT = PS2parity; } else if(position == 10) { //stop bit DATA_OUT = 1; } else if(position==11) { //end transmission, set up delay transmitting = 0; waiting = 1; TCNT1 = CLK_TIME+1; } position++;

clockState = FALLING; } else if(clockState==FALLING) {

CLK_OUT = 0;clockState = LOW;

} else if(clockState==LOW) {

clockState = RISING; } else {

CLK_OUT = 1;clockState = HIGH;

}}

}

/*calculate the parity of a character*/char parity(char x) {

char temp, i;temp=1;for(i=0;i<8;i++) {

temp=temp^(x&1);x>>=1;

}return temp;

}

Page 26: Final Report on Virtual Keyboard

//insert data into queue. Return 1 if queue full, or 0 if inserted data sucessfullychar queuePut(char d) {

if (queueFull==TRUE) //check if queue is fullreturn(TRUE);

queue[queueIn]=d; //insert d into queuequeueIn++; //increment where to stick in the next d

valuequeueEmpty=FALSE; //indicate queue isnt' empty anymore

if (queueIn==QUEUELEN) //if reached the end of the queuequeueIn=0; //wrap around to the beginning

if (queueIn==queueOut) //if queueIn caught up to queueOutqueueFull=TRUE; //indicate queue is full

return(0);}

//get data out of queue. Return 0 if queue empty or the actual data if not emptychar queueGet(void) {

char d;if (queueEmpty==TRUE) //check if queue is empty

return(0);d=queue[queueOut]; //get data out of queuequeueOut++; //increment location where to get next

d valuequeueFull=FALSE; //indicate queue isn't full anymore

if (queueOut==QUEUELEN) //if reached the end of the queuequeueOut=0; //wrap around to the beginning

if (queueOut==queueIn) //if queueOut caught up to queueInqueueEmpty=TRUE; //indicate queue is empty

return(d); //return the data from queue}

//determine the state of the data linechar clockStateNow(void) {

if(transmitting || waiting)return(BUSY);

return(IDLE);}

// initialize the keyboardvoid initKeyboard(void) {

OCR1A = CLK_TIME; //compare match to drive the clock to PCOCR1B = RESPOND_WAIT;TCCR1B = 0x01; //Run counter at full speedTCCR1A = 0x00; //nothing needed here, no outputTIMSK = 0x18; //Set the compare A interrupt and

compare B interrupt

queueFull = FALSE; //queue is not fullqueueEmpty = TRUE; //queue is empty

Page 27: Final Report on Virtual Keyboard

queueIn = 0; //where to insert into queuequeueOut = 0; //where to take out of queue

DDRD = 0xC0; //set data direction (see initial wiriing comments)

PORTD = 0xC0; //initialize outputs to highDDRD.2 = 1;PORTD.2 = 0;

#asmsei#endasm

}

Page 28: Final Report on Virtual Keyboard

I2/C COMMUNICATION CODE

/* Global Constants */#define TRUE 1#define FALSE 0

/* General Constants for TWI */#define TWI_FREQUENCY_HZ 100000#define TWI_READ 1#define TWI_WRITE 0#define TWI_GENERAL_ADDRESS 0#define TWSR_MASK 0xF8

/* Constants for TWCR Register */#define TWINT 0b10000000#define TWEA 0b01000000#define TWSTA 0b00100000#define TWSTO 0b00010000#define TWWC 0b00001000#define TWEN 0b00000100#define TWIE 0b00000001

/* General TWI Master staus codes */ #define TWI_START 0x08 // START has been transmitted #define TWI_REP_START 0x10 // Repeated START has been transmitted#define TWI_ARB_LOST 0x38 // Arbitration lost

/* TWI Master Transmitter staus codes */ #define TWI_MT_ADR_ACK 0x18 // SLA+W has been tramsmitted and ACK received#define TWI_MT_ADR_NACK 0x20 // SLA+W has been tramsmitted and NACK received #define TWI_MT_DATA_ACK 0x28 // Data byte has been tramsmitted and ACK received#define TWI_MT_DATA_NACK 0x30 // Data byte has been tramsmitted and NACK received

/* TWI Master Receiver staus codes */ #define TWI_MR_ADR_ACK 0x40 // SLA+R has been tramsmitted and ACK received#define TWI_MR_ADR_NACK 0x48 // SLA+R has been tramsmitted and NACK received#define TWI_MR_DATA_ACK 0x50 // Data byte has been received and ACK tramsmitted#define TWI_MR_DATA_NACK 0x58 // Data byte has been received and NACK tramsmitted

/* TWI Slave Transmitter staus codes */#define TWI_ST_ADR_ACK 0xA8 // Own SLA+R has been received; ACK has been returned#define TWI_ST_ADR_ACK_M_ARB_LOST 0xB0 // Arbitration lost in SLA+R/W as Master; own SLA+R has been received; ACK has been returned#define TWI_ST_DATA_ACK 0xB8 // Data byte in TWDR has been transmitted; ACK has been received

Page 29: Final Report on Virtual Keyboard

#define TWI_ST_DATA_NACK 0xC0 // Data byte in TWDR has been transmitted; NOT ACK has been received#define TWI_ST_DATA_ACK_LAST_BYTE 0xC8 // Last data byte in TWDR has been transmitted (TWEA = “0”); ACK has been received

/* TWI Slave Receiver staus codes */#define TWI_SR_ADR_ACK 0x60 // Own SLA+W has been received ACK has been returned#define TWI_SR_ADR_ACK_M_ARB_LOST 0x68 // Arbitration lost in SLA+R/W as Master; own SLA+W has been received; ACK has been returned#define TWI_SR_GEN_ACK 0x70 // General call address has been received; ACK has been returned#define TWI_SR_GEN_ACK_M_ARB_LOST 0x78 // Arbitration lost in SLA+R/W as Master; General call address has been received; ACK has been returned#define TWI_SR_ADR_DATA_ACK 0x80 // Previously addressed with own SLA+W; data has been received; ACK has been returned#define TWI_SR_ADR_DATA_NACK 0x88 // Previously addressed with own SLA+W; data has been received; NOT ACK has been returned#define TWI_SR_GEN_DATA_ACK 0x90 // Previously addressed with general call; data has been received; ACK has been returned#define TWI_SR_GEN_DATA_NACK 0x98 // Previously addressed with general call; data has been received; NOT ACK has been returned#define TWI_SR_STOP_RESTART 0xA0 // A STOP condition or repeated START condition has been received while still addressed as Slave

/* TWI Miscellaneous status codes */#define TWI_NO_STATE 0xF8 // No relevant state information available; TWINT = “0”#define TWI_BUS_ERROR 0x00 // Bus error due to an illegal START or STOP condition

/* Constants for TWI Functions' Return Status */#define TWI_SUCCESS 0#define TWI_INVALID_STATUS_ERROR 1#define TWI_NO_ACK_ERROR 2#define TWI_INVALID_SLA_ERROR 3#define TWI_ARB_LOST_ERROR 4#define TWI_NO_DATA_ERROR 5

/* Global Variables for TWI */char currStatus;

/* Prototype Functions for TWI */void init_twi_master(unsigned long cpu_freq, unsigned char pullup_resistor);unsigned char twi_master_start(unsigned char sla, unsigned char mode);unsigned char twi_write_byte(unsigned char data);unsigned char twi_write(unsigned char *data, unsigned char length);unsigned char twi_read_byte(unsigned char *data);unsigned char twi_read(unsigned char *data, unsigned char length);void twi_stop(void);

void init_twi_master(unsigned long cpu_freq, unsigned char pullup_resistor) {float temp;

if (pullup_resistor) {DDRC.0 = 0;

Page 30: Final Report on Virtual Keyboard

DDRC.1 = 0;PORTC.0 = 1;PORTC.1 = 1;

}

temp = (float) cpu_freq / (float) TWI_FREQUENCY_HZ; TWBR = (temp - 16) / 2 ; // TWI Bit Rate Register

// I2C Freq = CPU Freq / (16 + 2 * TWBR * 4 ^ TWPS)

TWSR = 0x00; // TWI Status Register// 7:3 - TWI Status// 2 - Reserved// 1:0 - TWI Prescaler Bits// 00 - 1// 01 - 4// 10 - 16// 11 - 64

}

unsigned char twi_master_start(unsigned char sla, unsigned char mode) {if (0x80 & sla) // Check SLA is 7 bits

return TWI_INVALID_SLA_ERROR;

TWCR = TWINT | TWSTA | TWEN; // Send START condition

while(!(TWCR & TWINT)); // Wait for transmission currStatus = TWSR & TWSR_MASK; reportVhex(currStatus);

switch (TWSR & TWSR_MASK) { // Check statuscase TWI_REP_START: // OKcase TWI_START: // OK

break;case TWI_ARB_LOST: // Lost arbitration, can

try againreturn TWI_ARB_LOST_ERROR;

default: // Not in START condition, do not send STOP condition

return TWI_INVALID_STATUS_ERROR;}

TWDR = (sla << 1) | mode; // Send SLA+R/WTWCR = TWINT | TWEN; // Clear interrupt to start

transmission

while(!(TWCR & TWINT)); // Wait for transmissioncurrStatus = TWSR & TWSR_MASK;

reportVhex(currStatus);if (mode == TWI_WRITE) { // For SLA+W

switch (TWSR & TWSR_MASK) { // Check statuscase TWI_MT_ADR_ACK: // OK

break;case TWI_MT_ADR_NACK: // Device busy during write, can

try againreturn TWI_NO_ACK_ERROR;

Page 31: Final Report on Virtual Keyboard

case TWI_ARB_LOST: // Lost arbitration, can try again

return TWI_ARB_LOST_ERROR;default: // Error, must

send STOP condition return TWI_INVALID_STATUS_ERROR;

}}else { // For SLA+R

switch (TWSR & TWSR_MASK) { // Check statuscase TWI_MR_ADR_ACK: // OK

break;case TWI_MR_ADR_NACK: // Device busy during write

return TWI_NO_ACK_ERROR;case TWI_ARB_LOST: // Lost arbitration, can try

againreturn TWI_ARB_LOST_ERROR;

default: // Error, must send STOP condition

return TWI_INVALID_STATUS_ERROR;}

}

return TWI_SUCCESS;}

unsigned char twi_write_byte(unsigned char data) {TWDR = data;TWCR = TWINT | TWEN; // Clear interrupt to start

transmissionwhile (!(TWCR & TWINT)); // Wait for transmissionswitch (TWSR & TWSR_MASK) { // Check status

case TWI_MT_DATA_ACK: // OKreturn TWI_SUCCESS;

case TWI_MT_DATA_NACK: // Device write protectedreturn TWI_NO_ACK_ERROR;

case TWI_ARB_LOST: // Lost arbitrationreturn TWI_ARB_LOST_ERROR;

default: // Error, must send STOP condition

return TWI_INVALID_STATUS_ERROR;}

}

unsigned char twi_write(unsigned char *data, unsigned char length) {unsigned char i;

for (i = 0; i < length; i++) {TWDR = *data++;TWCR = TWINT | TWEN; // Clear interrupt to start

transmissionwhile (!(TWCR & TWINT)); // Wait for transmissionswitch (TWSR & TWSR_MASK) { // Check status

case TWI_MT_DATA_ACK: // OKbreak;

case TWI_MT_DATA_NACK: // Device write protectedreturn TWI_MT_DATA_NACK;

Page 32: Final Report on Virtual Keyboard

case TWI_ARB_LOST: // Lost arbitration, re-arbitrate (unlimited)

return TWI_ARB_LOST_ERROR;default: // Error, must send stop

conditionreturn TWI_INVALID_STATUS_ERROR;

}}

return TWI_SUCCESS;}

unsigned char twi_read_byte(unsigned char *data) {TWCR = TWINT | TWEN;

while(!(TWCR & TWINT));switch (TWSR & TWSR_MASK) {

case TWI_MR_DATA_NACK:case TWI_MR_DATA_ACK:

break;default:

return TWI_INVALID_STATUS_ERROR;}*data = TWDR;return TWI_SUCCESS;

}

unsigned char twi_read(unsigned char *data, unsigned char length) {unsigned int i;

for (i = 0; i < length; i++) {if (i == (length-1))

TWCR = TWINT | TWEN;else

TWCR = TWINT | TWEN | TWEA;while (!(TWCR & TWINT));switch (TWSR & TWSR_MASK) {

case TWI_MR_DATA_NACK:case TWI_MR_DATA_ACK:

*data++ = TWDR;break;

default:return TWI_INVALID_STATUS_ERROR;

}}return TWI_SUCCESS;

}

void twi_stop(void) {TWCR = TWINT | TWEN | TWSTO; // Send STOP condition

while(TWCR & TWSTO); // Wait for transmission }