DEVELOPlVlENT OF AN IPAQ-BASEO ROBOTIC VEHICLE 'VITA\VIRELESS CONTROL AND VIDEO FEEDBACK

\Vayne T. Padgett Adam 1. ThomasElectrical and Computer Engineering Dept.

Rose-Hulman Institute of TechnologyTan: Haute, IN 47803

ABSTRACT

The iPAQTM computer makes an excellenthardware platform for the development of arobotic vehicle with advanced video andwireless Iletwork features. Because thehardware and accessories are off-the-shelf massmarket prouucts, a reliable vehicle can becreated \vith very little hardware development,and the cost is minimized. This paper describesthe development of the vehicle hardware andsoftware by four undergraduate students atRose-Hulman as part of an independent-studyproject course. Potential uses as part of nrobotics / image understanding course arediscussed.

fNTRODUCTION

This project began as a prototype design for aground vehicle for the International AerialRobotics Competition [I J. The current rules 0 fthe competition require a small vehicle that canmove and return video to a controlling computerseveral kilometers away. Because reliability isdi fficult to achieve in custom bui It hardware andbecause custom software tools are usuallyrudimentary, this experiment was designed totest the idea of using commercial hardware.Since the iPAQ is capabk of supporting awireless network card and a video-capabkdigital camera [2] it became the target CPU forthe robotic vehicle. This project was only aproof-of-concept so a simple toy firetruck waschosen as the vehicle to be controlled with theiPAQ system. The firetruck has a number ofaccessories to control just as the competitionvehicle would have. A picture of the tiretruck isshown in Figure I.

COI\IPUTERS I~ EOLJCATION JOt! RI~AL

Although there remain a few software issues toresolve, the resulting prototype allows the userto control the vehicle from a remote computerover the \virdess network, and watch video ofthe scene ahead of the vehlcle.

TEAM

Four Rose- Hul man undergraduate studentsworked on this project. Ryan Bechtloff, a seniorcomputer science major, and Adam Thomas, afreshman computer science student, were bothin charge of liguring out the NexiCam™ API,developing the server/client software for thesystem, and creating code that displays picturesfrom the NexiCam. Thomas Penne, a computerscience freshman, and Chris Dupin, an electricalengineering senior. headed lip the efforts tocontrol the lire truck with a PIC microcontrollerand to get the iPAQ to talk to the new controlcircuit via RS-232 connection. In total, the teamworked about 420 hours over the course of a tenweek term.

HARDWARE

The iPAQ H3470, which requires a wirelessnetwork card accessory, was obtained through agrant from Hev\ lett-Packard to encourageeducational use of the handheld cumputer.Current iPAQ models such as the H4155include built-in wireless LAN and are in theS500 range. The iPAQ has a "sleeve" interf~lce

for supporting various accessories, such as extrahnttcries, extra card slots, and other types ofinterfaces. One Stich accessory is the NexiCam[2] camera sleeve which costs about S 120. TheNexiCam allm,\s the iPAQ to operate as adigital

23

, :..

" _:"":.'~~:~::-;~~{'::~~(~'-~

.,>·~.:;:,Y~(~~~ ,:/~~~i~Figure 1. The iPAQ controlled firetruck. Thecustom cradle on top holds the iPAQ so that theNexiCam camera can view the forward path.The black cable is a serial link to an internalpre processor that does the motor switching.The iPAQ is not installed because it was used totake the picture.

camera, and the camera lens is on a rotating annso that it can be used for videoconferencing aswell. The rotating ann allows the iPAQ to bemounted in a variety of positions and still havethe camera aimed toward the vehicle's forwardpath. The NexiCam also includes a CompactFlash slot, primarily for image storage, but inthis project it is used for the D-Link WirelessLAN card which cost about $\ 00. Gettingdigital I/O from the iPAQ is a challenge. Inkeeping with the '·off·the-shel f' approach, theoriginal plan was to use a USB-based 1/0 box.but the iPAQ has a slave USB interface and sodoes the 1/0 box, so there was 110 bus master.The serial port was used instead. which isslower. but simpler to work with, both inhardware and software.

A rrc 16F874 was lIsed to reCt:lve the serialcontrol commands and control the L298 H­bridgl:S and lRF540 MOSFETs that switch themotors and lights on the tiretruck. The tiretruckcan even pump water under iPAQ control. Theentire vehIcle is controlled (rom a PC andaccessed through the internet, using the Rose­Hulman wireless network. Figure 2 shows ablock diagram of the hardware contiguration.

COMPUTERS IN EDUCATION JOURNAL

Figure 2. The Hardware. The iPAQ has aNexiCam camera sleeve, which includes aCompact Flash slot for the wireless card. Themotor and light control is done through theserial port.

The vehicle hardware design was simplified bychoosing a firetruck that used a 6V supply.which was easy to regulate down to 5V for thePIC. and the accessories did not require levelconversion for operation with the PIC's logiclevels. Serial communication with the iPAQ didrequire level shifting to +/~ 12V, \vhich wasaccomplished using a MAX232 chip.

COST

The cost of the entire project was well under$1000. Because the iPAQ was donated by HP.the out of pocket expense for this prototype wasunder $350. Fortunately, the cost of iPAQhardware is dropping, and the iPAQ cal\ also beused for other projects. since it is not modified.Table 1 shows a breakdown of project costs.

Clearly, the main costs are the iPAQ and theNexiCam. The costs of the controlling PC andthe wireless hub are not included here becausethey are part of the infrastnlcture at Rose­Hulman, and many other schools. The cost ofmanufacturing a small PC board can varywidely. At Rose-Hulman, we have: a millingmachine for manufacnlring circuit board

24

ApproximateP;.lIt Cost

iPAQ (with wirdess) $500NexiCam $120Remote Control Vehicle(inc [uding battery and charger) $45

PIC 16F874 $10iPAQ Serial Cable $25PCB fabrication(RHIT prototype milling process) $30Circuit Components $10

Totals I $740

be controlled by placing a delay between imagesand running the vehicle for only a few secondsat a time. Finally a compromise was reachedwhere two separate client/server programs wereset up, one to control the vehicle and one to takepictures, as shown in Figure 3. ThePictureServer/Client software was designed sothat if the server (PC side) locked up, it could beshut down, restarted, and then the client (iPAQside) would automatically timeout its previousconnection and attempt to establish a new one.

, I

PC Sid< iPA!) S;o<

Table I. Itemized Expenses. The hardwarecost is quite low. The costs shown aboveassume the purchase of a newer iPAQ whichwould include a built-in wireless card, andminimal PCB fabrication costs for the PICboard.

prototypes at minimal cost, however, numerousservices such as ExpressPCB [3] can quicklymanufacture small boards for as little as 3boards for $60. It is also quite practical to builda PIC system on a prototyping board (hand­wired) or even on a solderless breadboard. Notethat the solderless breadboards are not veryreliable, but worked well enough that after twoattempts at building a circuit board werethwarted by hole sizing errors, the student teamleft their PIC implementation on a breadboardand moved on.

SOFTWARE

The purpose of the software was to allow aperson to control the vehicle from a remotecomputer via the iPAQ and be able to drive thevehicle around using reaJ-time streaming imagessent over the wireless network from the iPAQ.This requires two programs, a server running onthe PC side as a user interface, and a client,running on the iPAQ side to control the vehicleand the NexiCam. All of these features havebeen successfully implemented, however thestreaming images were not reliable because theycaused the server program to lock upperiodically thus preventing control of thevehicle in real-time. The vehicle could however

COMPUTERS IN EDUCATION JOURNAL

r=------1'-idClit'tll

---_., RI.."l:t'ivts Commands; f,.rJm t:scrl p,,,<s I"e", 10 PIC

-~~~L.__-=-I P1C j. .-\~~CJl1blv Prt.lgr.1m, .

Ii ('Ol)trol~; 1\ h'lOrs

:md Lighls~---_..._.---~

Figure 3. The Software. The client and serverprograms were divided into two parts so thatproblems with image display would not severcontrol of the vehicle, and so that thePictureServer could be reset. Note there are notwo way communication paths, and vehiclesensors would require a NetClient return path.

USING THE IPAQ TO TAKE PICTURES

The tirst step in implementation \vas to learnhow to program f()r the Windows CE basediPAQ. Embedded Visual C++ was used as thecompiler and. after a lengthy trial and crrorcompi ling can figuration process, the standard"Hello World!" program sucCl:ssfully ran on theiPAQ. The next step was to attempt to take apicture. Navicom (Nexian is Navicom's U.S.distributor) graciously provided the NaiCamAPT and SDK manual, but it was not Lip to dateso the camera interface took extra work.Eventually the included writcjpcg program was

25

modified to take a picture by carefully pickingthrough the camera.h file for commands thatseemed likely to work. Since the task of takingpictures was completed early in the quarter, itwas decided that the project should be expandedso it would take the full ten weeks. The studentteam was divided into two major focus areas:the first consisted of Adam and Ryan whoworked on making connections over thewireless network and eventually sendingpictures and commands; the second consisted ofChris and Tom who worked on using a serialcable connected to the iPAQ to provide input toa PIC microcontroller which controlled thevehicle.

CREATING A WIRELESS CONNECTION

Creating a wireless connection to send databetween the iPAQ and a host computer was thesecond step after learning how to write basicprograms for the iPAQ. After reading severalarticles on the web, the decision was made touse WinSockets to establish networkconnections and send data, because they werethe most easily ported to the iPAQ. First,MyNet, a simple server/client program waswritten, that could be used between twocomputers running Windows. MyNet allowedtwo computers that were connected to theinternet to send text messages between them.Next, MyNet was modified and converted overto the iPAQ which runs Windows CEo Thisproved to be more difficult that originallyanticipated. First of all Windows CE does notsupport all of the functions that Windows doesand furthermore the iPAQ itself only supportedcertain Winsock 2.0 functions so an olderversion had to be used, Winsock 1.1. Even withWinsock 1.1, a few functions would not onwork the iPAQ such as GetHostByNameO.This prevented setting the iPAQ up as a serverthat any computer could connect to because theport number and IP address had to be entered inbefore a connection is initiated. So the iPAQsoftware had to be implemented as a client thatwould connect to a server.

COMPUTERS IN EDUCATION JOU&~AL

The second stage was to take the pictures onthe iPAQ and send them over the network to theserver. This was harder than sending textbecause first the send and recv functionsWinsock used only accepted the char data typeso everything had to be type cast char beforesending and then type cast back to BYTE after itwas received so it could be written to a file.There was a problem with the recv functiononly receiving a limited amount of data before itreturned so only half or a quarter of a picturewould arrive at the display PC. This problemwas resolved by sending the size of the picturefirst and then looping the recv function until ithad received the full file. There was also a stackoverflow problem that required reading thereceived picture into a buffer on the heapinstead of the stack. The code was latermodified to pass this buffer directly to theLoadPictureBuffer function instead of writing itto a file.

The final stage of software implementationwas to control the PIC microcontroller throughthe serial port from the (iPAQ) client which wastaking commands from the (PC) server program.Chris and Tom developed a program to sendcommands from the iPAQ over the serial port.A series of modifications of existing coderesulted in a system to send text from the PCcontrol server to the iPAQ which then convertedthem to PIC commands and relayed to them tothe PIC.

SERIAL COMMUNICATIONS AND PICMICROCONTROLLERS

Serial communication begins with the iPAQsending an ASCII character out its serial port.A cable runs from the iPAQ to a Max RS232chip, which changes the RS-232 signal to aTTL/CMOS level, and then to the USART inputon a PIC16F874. Once the built-in USART inthe PIC has received the 8 bit ASCII characterand stored it, an interrupt vector is executed thatprocesses each serial input and changes theoutput state of the corresponding ports on thePIC. PORT B has four pins that control the fire

26

trlli.:k's driving motions. PORT A has four pinsthat control the tower's motion, and PORT Dhas two pins to control the water pump and theheadlights. The four pins for the driving motorsand the four pins for the tower's motors eachhave an L298N H-bridge to control the directionof the motion and allow for a higher currentthan the PIC can output.

DISPLAYING PICTURES

Displaying a picture turned out to be relativelyeasy, since a jpeg image display program wasavailable from Microsoft's website. TheMicrosoft example code had to be modifiedslightly, but it displayed the images as needed.The Microsoft example required the incomingimage to be read from a file, so the imagescoming from the iPAQ had to be written to atile, and then read back into memory to bedisplayed on the screen. Displaying multipleimages in rapid succession was more difficult.The display functions had no error checking soerror reporting had to be added to find out whatthe problems were. In order to increase thespeed of the display, the example code wasmodified to read straight from a memory bufferinstead of writing to a file first. The imageserver often locked up entirely if it tried to paintthe screen and load a new image at the sametime, so a simple mutual exclusion lock wasadded to prevent that from occurring. The causeof an E_FAIL error which is specifically anunspecified error associated with theOLELoadPichlreO function, remains a mystery.It was possible to identify that error and makethe program continue each time it occurs, butnot prevent the error from occurring. Thepicture server program locked up occasionallydue to an error which was not identified duringthe project. The problem was later determinedto be caused by incomplete jpeg files being sentto the OLELoadPictureO function, whichcaused it to lock up instead of throw anexception. The problem does not occur when aSleep(2000), which stalls the program for 2seconds, is added to the loop, so there is may bea problem in our early version of the NexiCam.

COMPUTERS IN EDUCATION .JOURNAL

The most attractive compromise solution was todesign the client program so that the user couldshut the picture server program down and restartit if it locked up and the picture client wouldautomatica Ily reconnect.

POSSIBLE IMPROVEMENTS

NEXICAM

The NexiCam had some disadvantages for thisproject. First, the APl was in an early stage ofdevelopment. Hopefully, a more mature versionwill be available for future work. Second, theNexiCam API is designed around a still-cameraconcept, which is not ideal for streaming video.Finally, the direct attachment of the NexiCam tothe iPAQ requires the iPAQ to be mounted atthe front of the vehicle wWere it is most easilydamaged. A separate video camera that couldbe mounted away from the iPAQ, or even set upwith multiple cameras (for switching views)would be nicer, but no such products seem to beavailable.

PROGRAMMING LANGUAGE

Visual C++ was the obvious choice for thisproject because that's v"hat the Nex iCam ;\ PIsupported for the iPAQ clients. Since it seemedimpractical to write a translation of Winsocketsfor another language, Visual (t +- was also theobvious choice for the PC servers. Furtherresearch on Winsockets seems to show that theserver could have heen programmed in adifferent language such as Java or Visual BaSICwhich is more graphically oriented and thus,may have prevented the mystlTiolis prohlemthat causes the server program to lock lip.

PROGRAM DESIGN

During the project it became apparent that It

was necessary to use threads with the send andrecv functions hecause they needed to I()opsimultaneously while waiting to s\.:nd (bra rromthe user or camera and receive it on thl' uthcrend. The students had very limited exrl'fICnCe

27

with threaded programming, so the problemcausing the server to lock up could also be aresult of poor multithreaded design.

The function used to load the pictureOleLoadPictureO was not the fastest optionavailable. Many articles on the web suggestedusing Intel's Integrated Perfonnance Primitiveslibrary (4] which cuts the time in half of loadingpictures in a Windows program. Thisinfonnation became apparent during the courseof the project, but it was too late to converteverything over. Pricing for this library seemsquite affordable, especially with academicdiscounts.

PCB

During the last two weeks of the 10 weekproject, the students tried to make a PCB for thePIC circuit. However, the first PCB had someholes that were too small and it had to bescrapped. The second PCB had a short whichthe students were unable to find, rendering ituseless as well. Since the soiderless breadboardprototype stilI worked, it became the "fmal"implementation. It would have been desirableto have more .time to complete a functioningPCB.

SENSORS

It would add significant new ftmctionality ifIR sensors and a heat sensor for the fire truckcould have been added. The IR sensors wouldbe for wall-following, so the truck could driveinto a room, and then automatically follow awall until it found a target on the camera orfrom the heat sensor. The heat sensor wouldallow simulated firefighting, and possibly videofeedback on the water stream aiming point.Another possible sensor addition would be anoptical shaft encoder for the wheels so thataccurate measurements of travel distance couldbe reported to the controller.

COMPUTERS IN EDUCATION JOURNAL

COMPETITIVE DEVICE

A Google search for robot vision will tum upanother "off the shelf' solution forexperimentation: Evolution Robotics' ER-l (5].This is a simple robot system designed to carrya laptop as its Cpu. It includes a frame, steppermotors, a web cam, and a USB stepper motorcontroller. Clearly, the concept is similar to theiPAQ based robot described above. The maindifferences are in size and weight, since thelaptop is a larger device than the iPAQ. TheER-l does have the advantage of using a webcam as mentioned above, and having the sameoperating system at the controller and on thevehicle.

APPLICAnONS

Although this project was conceived as a proofof concept prototype for the Aerial RoboticsCompetition, it has become clear that theresulting system is an affordable platfonn forautonomous vehicle experimentation andcomputer vision research. Because both thecontrol and images are available to thestationary PC, it is simple to use the images forpath planning and obstacle avoidancealgorithms, and test the methods on real data.The vehicle can be driven as slowly as desired,and as much computational power as you wishcan be applied at the stationary PC end of thesystem.

Because the moving vehicle takes multipleimages of roughly the same scene from differentpositions, algorithms similar to stereo vision canbe applied. The image analysis can takeadvantage of substantial infonnation about thevehicle's motion, since it controls the motion.A great deal of research has been done on the"structure from motion" problem [6], and this isan excellent topic for an upper level course.The ability to test student solutions in a realexperiment should be highly motivational forthe typical student, just as it was for the studentswho worked on this project.

28

CONCLUSIONS

It is both possible and practical to build anaffordable, small, reliable robot for research andexperimentation. The majority of thecomponents can be obtained from commercial"off the shelf' sources, and the requiredsoftware is within the reach of undergraduatestudents. The exercise of building the softwareand hardware for such a robot is an excellentlearning experience, and the resulting robotmakes an excellent platform for further researchand experimentation. With a remote computer incontrol of the robot, it is simple to experimentwith autonomous vehicle behavior in a realenvironment for testing path planning, imageunderstanding, obstacle avoidance, and structurefrom motion algorithms. Many schools couldbenefit from the opportunity to have hands onlab work in these advanced topics, with a verysmall budget.

REFERENCES

I. Michelson, Robert. (2004, Feb. 19)Association for Unmanned Vehicle Systems,International's Aerial Robotics Competition.[Online]. Available: http://avdil.gtri.gatech.Edu/AUVS/IA RCLaunchPoint.htrnl

2. Nexian, Inc. (2004, Feb. 19) N'exiCamDigital Camera for the HP IPAQ PocketPc. [Online]. Available:http://www.nexian.com/product/nexicam.asp

3. Engineering Express. (2004, Feb. 19)Introduction to ExpressPCB. [Online].Avai lab Ie: http://www.expresspcb.coml

4. Intel Corp. (2004, Feb. 19) Intel IntegratedPerformance Primitives. [Online].Available: http://www.in!el.com/software/products/ipp

5. Evolution Robotics, Inc. (2004, Feb. 19)ER I. Personal Robot System. [Online].Availabk http://www.evolution.comJerIl

COMPUTERS iN EDUCATION .JOURNAL

6. 1. OIiensis, (2004, Feb. 19) "A Critique ofStructure from Motion Algorithms," NECITR 1997. [Online]. Available:http://citeseer.nj.nec.coml01 iensisOOcritique.html

BIOGRAPHICAL lNFORMATION

Wayne T. Padgett was born in Syracuse, NY.He received the B.E.E. Degree from AuburnUniversity, Auburn, AL, in 1989. He receivedthe M.S. and Ph.D. degree from GeorgiaInstitute of Technology, Atlanta, GA, in 1990and 1994, respectively. He is currently anAssociate Professor of Electrical and ComputerEngineering at Rose-Hulman Institute ofTechnology. He has consulted with a variety ofcompanies on digital signal and imageprocessing. Dr. Padgett is a member of IEEEEand ASEE. He received the CoED Division'sWoody Everett Award for best paper in 1999.His research interests include robotics, fixedpoint algorithm design, and microphone arrays.

Adam J. Thomas was born in Terre Haute, IN.He attended Northview High School in Brazil,IN. Currently he is a junior attending Rose­Hulman Institute of Technology and majoring inComputer Science with a minor in AppliedBiology. His course concentrations includeartificial intelligence and genetics. He plans topursue a future career in biotechnology.

29