The Minimum You Need to Know About Qt and Databases

About Qt and Databases

By Roland Hughes

Logikal Solutions

The Minimum You Need to Know

Copyright © 2010 by Roland HughesAll rights reserved

ISBN13 9780982358054

This book was published by Logikal Solutions for the author. Neither Logikal Solutionsnor the author shall be held responsible for any damage, claim or expense incurred by auser of this book as a result of its use or reliance upon.

These trademarks belong to the following companies:Ask.com IAC (InterActiveCorp)Borland Borland Software CorporationDataBoss Kedwell SoftwareDatatrieve HewlettPackard Development Company, L.P.DEC HewlettPackard Development Company, L.P.Firebird General Motors Corporation when talking about carsFirebird Firebird Foundation Incorporated when speaking of database software Pontiac General Motors CorporationInterbase Embarcadero Technologies.Linux Linus TorvaldsLotus Approach International Business Machines Corp.MySQL MySQL ABOpenVMS HewlettPackard Development Company, L.P.PostgreSQL PostgreSQL Global Development GroupQuicken Intuit Inc.Qt Nokia CorporationSourceForge Geeknet, Inc.Ubuntu Canonical Ltd.Unix The Open GroupVista Microsoft CorporationWindows Microsoft Corporation

All other trademarks inadvertently missing from this list are trademarks of their respective owners. A best effort wasmade to appropriately capitalize all trademarks that were known at the time of this writing. Neither the publisher nor theauthor can attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting thevalidity of any trademark or service mark.

Acknowledgments

I'd like to thank my loyal base of readers and the incredible core team of Qt developers slaving away on an Open Source library that stands full torso above the commercial C++ cross platform libraries of old.

Source Code License

Any person owning a copy of this book may use the source code from this bookfreely when developing software for their personal use, their company's use, or theirclient's use. Such persons may include the source code either modified or unmodifiedprovided that the source delivered makes reference to the original author and is part of afully functional application. It is expressly forbidden for anyone to post this sourcecode on any bulletin board system, Internet Web site, or other electronic distributionmedium other than SourceForge.net without the express written permission of theauthor. It is also expressly forbidden to sell this source as part of a library or sharewaredistribution of source. All files for all projects presented in this book will be availablefrom http://www.theminimumyouneedtoknow.com

Users of the source code contained within this book agree to hold harmless both theauthor and the publisher for any errors, omissions, losses, or other financialconsequences resulting from the use of said source. The source code is supplied “as is”with no warranty of any kind expressed or implied.

http://www.theminimumyouneedtoknow.com/

Table of ContentsIntroduction......................................................................................................................1

1.1 Where We've Been and Where We Are Going.....................................................11.2 Not a GUI Tutorial................................................................................................51.3 ClientServer Full Circle.......................................................................................61.4 Today's Cross Platform Challenge........................................................................81.5 Why Ubuntu?........................................................................................................91.6 Why Qt?..............................................................................................................12

Table 1: Qt Database Drivers...............................................................................141.7 A Tale of Two Market Segments........................................................................171.8 The MutantNinja Data Model............................................................................201.9 Exercises.............................................................................................................23

Cautionary Information..................................................................................................252.1 Scope of This Chapter.........................................................................................252.2 Development Restrictions...................................................................................252.3 Building a Qt Program........................................................................................262.4 The Database Username Restriction...................................................................272.5 Garbage Collection Restrictions..........................................................................312.6 More Database Issues..........................................................................................332.7 Pitfalls of Mandatory Passwords.........................................................................352.8 Database Frailty..................................................................................................372.9 Main.cpp for What?............................................................................................402.10 Transaction Integrity.........................................................................................422.11 SQL != SQL......................................................................................................492.12 Lassie Might Not Come Home..........................................................................522.13 Consistently Inconsistent...................................................................................542.14 Just Because You Found It, That Doesn't Mean It is There...............................572.15 To tr() Or Not to tr()?........................................................................................592.16 You Might be Optically Isolated.......................................................................642.17 What? You Want to Create a Database?...........................................................662.18Exercises............................................................................................................67

PostgreSQL and Qt........................................................................................................693.1 Scope of This Chapter.........................................................................................69

3.2 Installing and Configuring PostgreSQL..............................................................783.3 Designer..............................................................................................................813.4 Inserting Into a PostgreSQL Table....................................................................1093.5 Our Flow and Why............................................................................................1143.6 qDebug or QMessageBox.................................................................................1193.7 Error Handling for Remote Support..................................................................1243.8 Plugin vs. Compile In Database Interfaces........................................................1313.9 Logging On.......................................................................................................1353.10 Hidden Signal and Slot Connections...............................................................1513.11 Reading a Text File.........................................................................................1523.12 Our Main Window..........................................................................................1553.13 Reporting........................................................................................................1773.14 The Boolean Issue Again................................................................................1903.15 Browsing Database Records............................................................................1923.16 Follow Up.......................................................................................................1993.17 Backup Via CSV.............................................................................................2003.18 Assignment One..............................................................................................2073.19 Assignment Two.............................................................................................2073.20 Assignment Three...........................................................................................2083.21 Assignment Four.............................................................................................2093.22 Assignment Five.............................................................................................2093.23 Assignment Six...............................................................................................2103.24 Assignment Seven...........................................................................................2103.25 Assignment Eight............................................................................................2103.26 Assignment Nine.............................................................................................2103.27 Exercises.........................................................................................................211

Firebird and Qt.............................................................................................................2154.1 Why Firebird?...................................................................................................2154.2 Differences........................................................................................................2184.3 Obtaining and Installing Firebird......................................................................2194.4 Quirks Between Flavors....................................................................................2244.5 Creating Users...................................................................................................2254.6 Creating Databases............................................................................................2264.7 Aliases...............................................................................................................2304.8 No Boolean.......................................................................................................234

4.9 QIBASE Not Provided by Default....................................................................2354.10 CSV Import and Export Part 1........................................................................2424.11 Console Applications and IDEs.......................................................................2474.12 #!/bin/sh..........................................................................................................2584.13 Some Serious SQL Differences.......................................................................2604.14 CSV Import and Export Part 2........................................................................2664.15 xpnsfb..............................................................................................................3004.16 Programming Assignment 1............................................................................3284.17 Programming Assignment 2............................................................................3284.18 Programming Assignment 3............................................................................3284.19 Programming Assignment 4............................................................................3294.20 Programming Assignment 5............................................................................3294.21 Programming Assignment 6............................................................................3294.22 Firebird Server Version Followup.................................................................3294.23 Deploying This Contraption............................................................................3314.24 Firebird Summary...........................................................................................3344.25 Exercises.........................................................................................................335

SQLite and Qt..............................................................................................................3395.1 Pros and Cons...................................................................................................3395.2 Which SQLite?..................................................................................................3465.3 Monkey Studio..................................................................................................3475.4 Programming Assignment 1..............................................................................3785.5 Programming Assignment 2..............................................................................3785.6 Programming Assignment 3..............................................................................3785.7 Programming Assignment 4..............................................................................3795.8 Programming Assignment 5..............................................................................3795.9 Our Application with Eclipse............................................................................3795.10 Programming Assignment 4............................................................................4155.11 Programming Assignment 5............................................................................4155.12 Programming Assignment 6............................................................................4165.13 Building Static vs. Plugin ...............................................................................4165.14 SQLite Followup...........................................................................................4175.15 Exercises.........................................................................................................419

Ruminations and Observations.....................................................................................4216.1 CTRLY, CTRLL, and ANSI...........................................................................421

6.2 Plugin vs. Static Compile..................................................................................4246.3 What Database is Right for Your Application?.................................................4276.4 Unstart...............................................................................................................4316.5 Outages and Support.........................................................................................4336.6 Ubuntu's Original Sin........................................................................................4376.7 Qt's Achilles Heel..............................................................................................4406.8 Package Maintainer vs. Marketing Fraud..........................................................4416.9 Code What You Know......................................................................................4436.10 Thursdays When It Rains................................................................................4456.11 Bitflags............................................................................................................448

Chapter 1

Introduction

1.1 Where We've Been and Where We Are Going

Hopefully you've been reading along in this book series and are quickly progressingas a developer or at least gaining a significant appreciation for what developers do.While you do not need to have read the rest of the series, it might help. The source codefor the PostgreSQL expense tracking example in this book has been uploaded toSourceForge.net and can be found in the xpnsqt project. All source files will be zippedup and placed on http://www.theminimumyouneedtoknow.com for download as well.Unlike most books in this series, there will not be an accompanying CDROM.

While I did not write the books in exactly this order, the correct order of readingwould have been:

The Minimum You Need to Know About Logic to Work in IT

ISBN13 9780977086627

The Minimum You Need to Know to Be an OpenVMS Application Developer

ISBN13 9780977086634

The Minimum You Need to Know About Java on OpenVMS

ISBN13 9780977086610

The Minimum You Need to Know About Service Oriented Architecture

ISBN13 9780977086672

The Minimum You Need to Know About Java and xBaseJ

ISBN13 9780982358030

Infinite Exposure

ISBN13 9780977086696

http://www.theminimumyouneedtoknow.com/

2 Chapter 1 Introduction

Originally I wrote the OpenVMS Application Developer book to fill a need. Lots ofplaces run OpenVMS and new installations are happening all of the time, yet there is nosource for new OpenVMS developers. It wasn't until after that book was published thatI started getting feedback about recent college graduates not understanding logic. Atthat point I decided to make this a series and began writing the logic book.

With the logic and traditional application development books out in the field therewas a natural progression. First we covered Java on OpenVMS using the sameapplication we had developed in COBOL, BASIC, FORTRAN, C and C++ in theapplication book. It was necessary to cover accessing of RMS files and RDB databasesvia Java along with using FMS as your screen management tool. Seasoned developerswanted to learn some of the new technology and recent college grads only understoodnew technology without even the faintest clue about how to access the data whichactually ran the company.

Next we stepped forward with Java on the Ubuntu platform accessing servicespresented by OpenVMS on an internal company network. Some of you might beshocked to learn that “The Minimum You Need to Know About Service OrientedArchitecture” won a 2008 Best Book Award in the category Business:Computers/Technology/Internet from USA Book News and was a 2009 Eric HofferFinalist. There are a lot of books about Service Oriented Architecture on the market,but damned few actually cover exposing the heritage data silos. They usually try to sellyou a chunk of middleware and another never ending support contract along with shinynew security holes.

The next book in the series didn't follow a logical progression per se, it came aboutbased upon need. I was in the process of designing and writing a fuel surchargeapplication which would eventually be released as a Open Source project and should beusable by someone who isn't any smarter than a Microsoft product. That lastrequirement meant the application had to use its own indexed file system rather thanmandate a user install, configure, and become a DBA for some database product.

Chapter 1 – Introduction 3

The XBASE family of file formats are the most widely used throughout the historyof PC computing, but the Java language doesn't directly include anything to support it,nor does it include any other usable indexed file system. (Yes, there is now a massivedevelopment effort called Java DB, but not at the time I needed it.) That meant I had tolearn a library. I decided that if I had to take the time to learn a new library with scarcedocumentation that I would write a tutorial and make life easier for the next guy.

Unlike the previous books in this series, the xBaseJ book is available in electronicform only. You can find the PDF with the files for the project on SourceForge. Youcan also find it listed here:

http://www.freeebooks.net/ebook/TheMinimumYouNeedtoKnowAboutJavaandxBaseJ

From an educational perspective, I need to provide you with a book which uses apopular cross platform C++ GUI tool to do some classic standalone developmentagainst a database server. Many in the world of IT will call this “ClientServer”development even though both the client and the server are on the same machine.Historically, this type of development occurred prior to anyone even thinking aboutinventing the Internet, but modern education wise, it is best to cover this topic aftercovering Service Oriented Architecture.

Why?

Kids today know very little about application development, but they know a lotabout surfing the Web and buying things online. It is easier for them to grasp a processwhich involves some kind of Web page sending data to some back end system (ServiceOriented Architecture.) Only a few of them will have had to work with Quicken orsome other personal finance/accounting application hosted on their own computer. Astudent who has actually used an application on a desktop computer which stores dataon a remote database without going through the Internet will be a rare find indeed.

“Infinite Exposure” is quite simply a novel which was written in response to aninterview question I got for the OpenVMS Application book. While it has gotten somegreat reviews, it is not part of my technical book writing.

http://www.free-ebooks.net/ebook/The-Minimum-You-Need-to-Know-About-Java-and-xBaseJ

http://www.free-ebooks.net/ebook/The-Minimum-You-Need-to-Know-About-Java-and-xBaseJ


Some of the previous books in this series used the Mega Zillionare applicationdeveloped over and over again so you could learn the differences and tradeoffsassociated with each tool choice. I could bore the living Hell out of you doing thatagain, but we don't have different tools, only different databases. I'm going to have todeviate from the norm and develop a new application for multiple databases to showyou the main issues. Yes, Qt does quite a good job of hiding the database if you happento be using a relational database, but in most cases, you cannot simply change thedriver namerecompileandgo. There are quite a few differences between supporteddatabases.

Don't worry, I know that most of you will be GUI only developers at best andcannot begin to function without the continual hand holding of an IDE (IntegratedDevelopment Environment.) While I developed the PostgreSQL version of ourapplication with nothing more than a text editor and the Qt Designer for form layout,our later examples will include using multiple IDEs and provide quite a few projectcreation screen shots. You will be exposed to QtCreator, QDevelop, Monkey Studio,and Eclipse with the Qt plugin.

Our application will be an expense tracking system which initially uses thePostgreSQL database. We will then redevelop the application using Firebird (BorlandInterbase) and SQLite. I will spend one chapter covering the use of stored proceduresin PostgreSQL and Firebird. Our final technical chapter will cover problems with longqueries and threading. It is too bad Oracle hasn't created a Qt plugin to access RDB or Icould show you how to use a modern desktop with the world's highest availabilitydatabase.

We won't cover MySQL. I know the database is popular. I know many of you willwant to use it, but it's quite the quagmire right now. By default, it is not a relationaldatabase. MyISAM is the default storage engine and is now the only storage engine forthe embedded version. InnoDB was purchased by Oracle some time ago and Oracle hasstarted to squeeze that orange.

http://digitizor.com/2010/11/05/innodbdroppedfromoraclemysqlclassicedition/

http://digitizor.com/2010/11/05/innodb-dropped-from-oracle-mysql-classic-edition/


While InnoDB is still available in the “community” edition, the fact it has beendropped from the Classic edition and InnoDB now has its own embedded/embeddableversion seems to indicate the “community” version of MySQL is going to be trappedwith a stagnant InnoDB release.

MyISAM creates a lot of data storage and application design problems. It doesn'tcurrently have a boolean type completely implemented and there is no support forforeign key constraints. While the Boolean problem could be considered a nuisance, thelack of foreign key constraints really torpedoes this project. What good are referencetables containing categories and expenses if the database isn't going to enforcereferential integrity? I went through this exact same problem eons ago when I usedXBASE files for each table. Without an engine, I had to enforce the integrity from theapplication level. Not such a good thing.

1.2 Not a GUI Tutorial

This book will not be a GUI tutorial nor will it spend much time covering areas ofthe Qt library which do not pertain to our sample application's database portions. Whileit is true that many developers could use this as a Qt tutorial along with the manualsTrolltech provides online, that is not the goal of this book.

If you need additional information on the graphical portions of the library alongwith the philosophy behind it all I can recommend is a pair of excellent books.

“Foundations of Qt Development” ISBN13: 9781590598313

“C++ GUI Programming with Qt 4 Second Edition” ISBN13: 9780132354165

In truth, this book is meant to cover the one area where those books really felldown. It's sad because it is the one area most developers actually need to understand.Very few of us get paid to write painting and drawing applications. The bulk ofbusiness development revolves around getting data into a database and getting reportsout of the database.


1.3 ClientServer Full Circle.

The Web doesn't live up to its hype for most applications. In many cases you wantcontrol, security, and limited deployment. In some cases you simply want a standaloneapplication or an application that you control access via controlling where it getsinstalled. You don't want information about your new supersecret billion dollar R&Deffort some place it could easily be poached. You want that application secured on itsown internal network or machine without any connection to the Internet at all.

Standalone programs on standalone personal computers make sharing data a bittough. Without a network and exposing some portion of your disk storage to outsiders,the options for sharing data are limited to physical media transfers along with dialanddeliver options.

Most of your successful standalone development happened with midrange andmainframe computers where hundreds to thousands of terminals were scattered about acompany, but they all connected into a single computer. This made exposing the data assimple as granting a user access to the files or databases.

During the late 1980s through the mid1990s ClientServer was both the rage, andthe bane of computing. Nobody was ever able to get it right, but everybody kept trying.The real problem with clientserver was application deployment. We didn't have anyautomated methods of “pushing” updates out to users and most users didn't have thetechnical ability to perform an installation on their own.

When corporate desktops all ran some form of nonGUI DOS, file servers seemedto be the ultimate solution. Products like Netware provided security, the ability toaddress much more disk storage than most versions of DOS, and even provided anindexed file system which handled multiuser access. As long as the desktop computertrying to run the application didn't have too much of that precious 640K alreadyconsumed, they could run the executables a user could access.


Users wanted more and more sophisticated applications. Developers wanted to usedifferent tools which did more and more of the programming work for them. Thesetools had various licensing systems which invariably involved installing some kind ofencrypted license key file along with a runtime library (RTL) of some kind. While thefile server could still be a great place to host the installation files and the actual datacreated by the application, the days of having a single executable file which everybodycould run were pretty much over.

Another drastic thing happened near the end of the 1990s: corporate desktopsstarted to diversify. While a great many people will be ignorant enough to say thatMicrosoft rules the corporate desktop, they would be wrong. Apple started makingsome serious inroads due to things the platform could do which Microsoft simply couldnot. After the disastrous release of Windows Vista, many corporations started loadingvarious flavors of Linux on their company desktops as well as their servers.

Currently, the PC desktop industry is in a state of turmoil. This turmoil will existfor roughly another decade, but it will be good for the industry in general. All but themost hardened supporters of Microsoft have realized the company has a legacy platformwhich is in the mode of continually shrinking market share. Apple has a knack when itcomes to getting people to pay a premium for their stuff, and recently, per the stockmarket, became bigger than Microsoft on a total_shares * share_price basis. Anotherplayer gaining a lot of momentum and respect are the various Linux distributions. AfterWindows Vista, anything looked stable, so companies gave it a try. Then the economytanked worldwide. Suddenly company heads started looking at the annual licensingand support costs of Microsoft products and running the numbers against the Linux andOpen Source alternatives. When the savings started being $20$120 per employee peryear the 10,000+ employee companies started taking note and issuing marching orders.

The turmoil is what lead to the SOA (Service Oriented Architecture) stampede.Some companies needed to get custom applications working on two to three versions ofWindows as well as the current Apple OS and three to four different Linuxdistributions. Rather than fight the cross platform battle they lunged forward at the ideaof providing Webbased applications which used Java and other Open Source tools that


ran in/on the major browsers for each operating system.

A great deal of effort was put forth by the W3C and many others to make SOAwork. A method of deploying services for discovery via WSDL (Web ServicesDescription Language) was created so any application anywhere could find what itneeded and simply pluginto the service. The problem was much of this design centeredaround openness, not privacy and security.

It never ceases to amaze me when young MBAs refuse to write a new applicationon OpenVMS or even an IBM mainframe. PCs are all they have ever seen, therefore,that must be all there is to use. They are so dead set against creating a “green screen”application which would be able to restrict access to only those who should really see,that they bring us full circle, back to the mid1980s and cross platform development.

1.4 Today's Cross Platform Challenge.

In today's world, cross platform has a slightly different definition. It used to justmean different operating systems on the desktop, but now it extends to handheld deviceslike netbooks, PDAs (Personal Digital Assistant), and smart phones. In truth, thedistinction between cellular phone, PDA, and netbook is becoming blurred by thecontinual addition of features to smart phones and the emergence of the iPad.Companies are making wide use of VPNs (Virtual Private Networks) on handhelddevices so personnel in the field can run applications which access databases securedbehind the corporate firewall. They have a need to maintain one application yet have itrun on every desktop, netbook, and PDA deployed by the company. Let us not forgetyou now have to deal with virtual keyboards and mice on most handheld devices.

I'm no stranger to cross platform development. I was there when the first majorplunge happened. Some of you might be old enough to remember the “Zinc It!” bookseries. It covered the Zinc Application Framework on DOS, OS/2, Windows, and tosome extent MAC. The reason I know about that series is I was the author. ZincSoftware was a poorly managed company. The product was purchased by Wind RiverSystems and rolled into various products they offer. Recently, perhaps because of thistrend, I have found Zinc is once again being sold as a product by a company. I haven'tdug too deeply into it, but they claim to support desktop and embedded platforms.


There is also talk of an Open Source version as well.

Let's be honest. While some commercial product vendors will use the crossplatform capabilities of the library, many inhouse developers will simply use thelibrary on a single platform with an eye toward being able to migrate quickly if whimschange. A lot of companies which had been staunchly (and stupidly) in the Windowsdesktop category are now starting to use Qt for Windows development. They are in theprocess of migrating everything to Qt so they can deploy Ubuntu, OpenSuSE, or someother Linux distribution on the desktop as soon as the bean counters realize just howmuch it costs to keep Microsoft around. Microsoft has been a victim of its ownmarketing now. When you actually total up all of the costs, it ends up being the mostexpensive platform per user of any computing platform, including IBM mainframes andOpenVMS midrange systems. A lot of companies have learned that the OpenVMS boxwhich is still housing their data and some core applications is/was the cheapest per usersolution they had even if they didn't like green screen applications.

1.5 Why Ubuntu?

If you have already read “The Minimum You Need to Know About ServiceOriented Architecture”, then you aren't asking the above question. For those of youwho skipped that book in the series, Ubuntu is quickly becoming the desktop Linuxdistribution of choice for most IT professionals and even quite a few corporations. Dellhas even started preloading Ubuntu on machines for corporate customers. Yes, Lenovois preloading OpenSuSE, but most other vendors are embracing Ubuntu. OpenSuSEhas been tainted by a business deal between Novell and Microsoft which infuriated mostof the various Unix/Linux factions and spawned a rewrite of the GPL (GNU PublicLicense) which put Novell in a world of hurt.

We won't get too deep into arguments about which is technically better. I have usedboth in the past. I used to actually pay for the professional edition of SuSE before it gotthe silent “Open” added to the front of it. All code in this book will be written andtested on version 10.10 AMD 64bit KUbuntu. You should be able to adapt things foryour own environment. I can warn you that many versions of OpenSuSE made it nearlyimpossible to get PostgreSQL running. It would eventually run, but you had to scour


the Internet looking for hints, snippets, and advice. Ubuntu made it an easy install andnearly automatic configuration. If you are a developer that doesn't want to be a SysAdmin and you don't like hacking kernel startup scripts, Ubuntu is the simpler choice.

Developers who want lots of development tools included with the initial installationtend to prefer using OpenSuSE. Corporations and their support staff tend to preferUbuntu. In either case, you don't notice a lot of difference unless you are willing toopen up a terminal window and start keying in commands, or you need to installsomething different. Ubuntu uses the Synaptic Package Manager for installations whileOpenSuSE uses YAST (Yet Another Setup Tool) for everything, including addingusers, changing the system time, etc. You can load KDE or Gnome front ends undereither distribution.

Corporations are preferring Ubuntu desktop distributions because the defaultdesktop interface is based on Gnome. This is a clean, minimalist interface. Users whodon't understand the inner workings of Linux will find themselves somewhat limited tosurfing the Web, creating documents, checking email, and whatever application linkswere preloaded by the corporation. To get both the KDE and Gnome sessioninterfaces, you need to navigate your way to the package manager after installation andinstall the KDE interface (KUBUNTU). KDE is a much more functional interface thanGnome. I will admit there are certain specific things which are much easier in Gnomethan KDE, but most common things are far easier in KDE than Gnome. If you haveboth interfaces installed, you can simply log out and log back in changing your sessionto which ever interface is easier for what you need to do.

If you install both interfaces on Ubuntu though be prepared for a lot of things toeither not work or work spastically. The Ubuntu crowd tests only one installedinterface with each release. I have tried many times to make both KUbuntu and Ubuntuplay nice together, and finally gave up. Every time you try to log a support issue youget told to wipe everything and install just Ubuntu.

Corporations are more than happy to stick their users with the Gnome interfacebecause they want to restrict what people do on their desktop. How many billions havebeen lost to people installing stuff on their Windows desktop only to find out it had avirus or some other nasty security hole? A restricted interface keeps users from poking


around and installing stuff. The user account can also be set up to not allowinstallations.

The other major selling point for Ubuntu is stability. Yes, many flavors of Linuxand other PCbased operating systems claim stability, but when you ask them how theyachieve it, you get a lot of buzzwords along with a song and dance. Ubuntu is generallya year behind the bleeding edge found in the source trunks. While they will pull insome patches and fixes which are newer, they are generally at least six months behindthe last stable release of any package which tends to be four to six months behind thebleeding edge of the development trunk. The core developers for Ubuntu will tell youthey release every six months so they are only six months behind. This is true only ifyou are measuring from the last “stable” release each package developer turned out.

OpenSuSE releases once every eight months. They don't put out an LTS release,but they don't have a spastic numbering system like Ubuntu. OpenSuSE 11 has aroadmap spanning several years and multiple .x releases. Ubuntu releases during thefourth and tenth month of every year. Their versions are number YEAR.MONTH.

In short, we are using Ubuntu because it plays the game the best and doesn't have ataint from the MicrosoftNovell deal and doesn't have YAST. That said, Ubuntu hasabout a five year window to fix what is wrong with it before it disappears from themarket place like Corel Linux. For more information on that read, “Ubuntu's OriginalSin” in the Ruminations chapter.


1.6 Why Qt?

To start with, there is an Open Source version which is regularly updated. It is theGUI library used for all KDE project development. Linux is gaining a lot of popularityand KDE is the major desktop. While Gnome gained a lot of corporate support becauseit is a bare interface which doesn't let the experimental end user get into too muchtrouble, for most developers and seasoned computer users, Gnome is much like rubbingsticks together to make fire.

While it can be correctly stated that Open Source projects let in far too many juniorprogrammers with buggy code, that statement would be incorrect when it comes to theQt library. There are commercial developers sifting the wheat from the chaff. This isnot to say the library doesn't have bugs. What I'm saying is that there are literallythousands of developers working on various portions of KDE (and now the new UbuntuGnomelike interface based on Qt instead of Gtk+.) These developers report bugs asthey find them and in many cases send along source changes to be incorporated into themain product trunk. Few commercial products can afford thousands of professionaldevelopers. By going Open Source with much of the library, Trolltech (now part ofNokia) has affectively added thousands of developers to their staff at little to no cost.The cross platform libraries from days of old didn't go this route. They tried to get bywith a limited number of inhouse developers funded by astronomical license fees.

Let us not forget that Nokia has chosen Qt as its standard (basically by purchasingthe product) and there is a lot of development going on for handheld applications in thatcompany. More and more embedded systems developers are choosing to use Qt ontheir embedded project as a result of this. Ubuntu even announced that the maindevelopment library for its Gnomelike front end will now be Qt as well. While thereare many Gtk+ diehards out there, the writing is on the wall. Qt managed to achieve aconsensus of what should be provided on all platforms and the “benefits” of proprietarytoolkits no longer exceed the “costs” of using them.


There is a lot of online documentation on Qt. Trolltech and Nokia have gone togreat lengths to make all of Qt's documentation available online. Many other sites alsohost copies of it in case the primary site goes down. In addition, the documentation isincluded in the free QtAssistant document browser which is installed with mostdevelopment packages. Much of the documentation is “expert friendly.” It works wellif you are already a seasoned veteran and just need to look something up. It doesn'twork well from the learning aspect.

I needed to develop the xpnsqt application to replace something I had written inLotus Approach many years ago. I kept waiting and waiting for Lotus/IBM to have adivine visitation and port the Lotus SmartSuite to Linux. I guess all of the major deitieshave been busy for the past decade. Now that OpenOffice has made great strides theword processor and spreadsheet market simply wouldn't be profitable. Approach wasone of those tools nobody used much. It did things well so you simply didn't rememberwhat tool you used for development. The xpnsqt application took me 45 minutes todevelop with Lotus Approach well over a decade ago without ever having usedApproach before. The replacement weighed in around four days. Two and a half ofthose days were spent trying to use a new Qt database class one of the books was gagaabout. That class simply wasn't ready to run with the big dogs.

You have quite a few database choices when developing with Qt. The databaseclasses are actually wrapper classes for various database plugins. Many of the pluginscan be compiled into the library so they are statically linked with your application.Static linking provides an immense amount of data and application security over pluginuse. For more on that topic see “plugin vs. Static Compile” in the Ruminations chapter.At the time of this writing Qt supports the following database plugins:


Driver Database

QDB2 IBM DB2 version 7.1 and later

QIBASE Borland InterBase

QMYSQL MySQL

QOCI ORACLE (Oracle Call Interface)

QODBC ODBC (includes Microsoft SQL Server)

QPSQL PostgreSQL 7.3 and later

QSQLITE SQLite version 3

QSQLITE2 SQLite version 2

QTDS Sybase Adaptive Server

Table 1: Qt Database Drivers

Not all drivers are shipped with all versions of Qt. Some drivers only work onsome platforms because the database only runs on some platforms. Other than anODBC driver, I don't know how you would access Oracle RDB from within Qt.(Remember that Oracle and Oracle RDB are two completely different databaseproducts.) Interestingly enough, the OpenVMS community has been porting Qt to theOpenVMS platform for quite some time. Trolltech doesn't seem willing to adopt theseports, so the community has been working diligently at them. You can find someinteresting information in this message thread:

http://groups.google.com/group/comp.os.vms/browse_thread/thread/4e1bacfc0f33584c/00c4ad8634cf9f0e?hl=en&lnk=gst&q=Qt#00c4ad8634cf9f0e




Qt comes (on most platforms) with its own version of SQLite. This is like thedefault MySQL for many Linux distributions, without the additional administrationissues. It is an SQL interface wrapped around an indexed file system. (MySQLdefaults to an indexed file system on most platforms. You have to actually forceInnoDB at time of database creation to get a relational engine.) Since SQLite is muchlighter than MySQL, it will let you create your indexed files pretty much where everyou happen to be. MySQL requires databases to be created where its startup script orsystem logical points (at least for the server version.) It is popular with PCmindeddevelopers because it allows them to gain the benefits of an SQL interface withouthaving to learn how to actually design and develop around a database. (Those of youlooking for more information about logicals should read ISBN13 9780977086634.)

In theory, the wrapper classes protect you from the quirks found in each database. Isay in theory because the higher level wrappers don't work cleanly yet, so you end upwriting quite a bit of SQL and using query functions. Each target database has someminor differences in their query languages and connection methods. Why hasn't thisbecome a major issue? Quite simply, very few applications ever get ported betweendatabase products. The primary development effort with Qt has been to use it writingKDE (the K Desktop Environment for Unix/Linux), many text editors, and a few IDEs.A major secondary market has been to develop applications for Nokia phones and otherhandheld devices. Very few applications are ported between the two environments, butthat is starting to change. Most of those which are ported can get by with the limitedcapabilities of SQLite.

These days, many companies will start development with a cross platform toolknowing they will have to port to another platform in the future, even if they don't wantto port. How is that possible? Take a look at the DOS market. There were a rash ofDOS specific GUI libraries during DOS's heyday. Then came Windows (GUI DOS),Windows for Work Groups (even more GUI DOS), Windows 3.x (still more GUIDOS), Windows 95 (GUIer DOS), Windows 98 (DOS locked in the dungeon still doingthe bulk of the work for the GUI), Windows NT, Windows XP, and most recentlyWindows Vista. (Yes, I skipped OS/2, a better DOS than DOS and a better Windowsthan Windows.) Every three to ten years, the underlying environment changed. A crossplatform library can minimize the impact to your application. In a best case scenario


you simply have to recompile and create a new installation procedure.

When Zinc came out, it straddled the DOS, OS/2, Mac, and Windows markets. Theunderlying problems with the product all stemmed from not having an Open Sourceversion. The vendor was left trying to develop and support a rash of moving targets,and was always hampered by the limited address space of the DOS environmentcombined with near zero free DOS extenders. By the time some of the better qualityDOS extender tools became “free” or cheaply licensed, the market had moved. Duringthe first round of cross platform libraries we also had a large number of compilervendors combined with evolving standards for both C and C++. Today there are veryfew compiler vendors and the standards have settled. While there still are commercialC++ compilers out there, most developers are migrating to the Open Source compilersof either GNU or Watcom (which oddly enough used to be the highest qualitycommercial compiler back in the day. OpenWatcom now has a Linux version besidesall of the other versions it used to have.http://www.openwatcom.org/index.php/Main_Page )

How many of you have been around long enough to remember the pain of portingfrom Windows 3.x to Windows 95 to Windows 98 and the later ports? If you didn'thave a wrapper library things were ugly. Because Vista took so long to get into themarket, people forgot and started using a lot of native tools when developing on XP.The release of Vista taught another generation of developers the sad lesson of usingplatform and vendor specific tool sets. The much (and rightfully) maligned Vista alsoopened the door for Linux distributions on the desktop. People decided they weren'tgoing to go through this yet again simply to stay current.

The other benefit of using a tool set which runs on many different operating systems(notice I avoided saying cross platform) is that your pool of potential developers getsmuch wider. Yes, many are contributing things to the Qt library itself, but whathappens when you need to hire new developers? Most companies have learned the painof using proprietary systems like Windows and Microsoft specific tool sets. Collegesquit teaching things that aren't free when they are strapped for cash. Your pool ofpossible employees shrinks every semester once that trend starts. Vendors keepincreasing the pricing of their tools so the only way you can get new talent is to steal

http://www.openwatcom.org/index.php/Main_Page


someone else's old talent. A lone wolf simply isn't going to plunk down thousands ofdollars to obtain all of the tools in the “hope” they will land a good paying contract.

I have seen job openings posted for months listing six to twelve vendor specifictools. When the posting stays open in excess of a year they claim there is an ITshortage. There is no shortage. They were simply posting a job opening looking to pay$60,000/year for a skill set commanding $120/hour. Had they been willing to pay theactual market rate, they would have gotten the talent they wanted. Had they beenwilling to hire a programmer who had none of those skills, but was willing and able tolearn, they would have gotten somebody for the money they offered.

A lot of companies are moving to Qt to avoid being trapped by a proprietaryplatform like Windows and its proprietary tool sets. If a developer already knows howto develop with Qt, they only have to learn a few of the quirks and the editors on thenew platform. This is, of course, making the assumption you have managed to migrateall of your data into relational databases which have plugins/compileins available.

1.7 A Tale of Two Market Segments

What is really the fundamental difference between the Mac market and the PCmarket? Apple has completely tossed out prior architecture on more than one occasion,yet Apple is now the leading college campus computer manufacturer. Vista requiredsuch a massive hardware upgrade for most existing Windows users that the industryrefused to upgrade. After officially dropping support for Windows XP, Microsoftbegan selling a “Business Edition” Vista license. I set up one of these computers forsomeone. Do you want to know what the “Business Edition” really is? It comes withbootable media which will wipe Vista off your machine and install Windows XP Pro.Microsoft counts this as a Vista license sale, but they really just sold another copy ofWindows XP Pro. Oh, let's not forget about when the netbooks came out! They didn'thave enough resources for Vista, so Microsoft began selling XP on them again. Somuch for the sunsetting of XP.


How is it that the Apple customers are willing to endure this, but the Microsoftcustomers are not? When Apple throws out the prior technology, they throw it out lockstockandbarrel. A major improvement in both the operating system and the hardwareshipping with it occurs. There is a compelling reason to ditch the old stuff for the newbecause it always feels like you are upgrading from a Model T to a fully loaded CapriceClassic. Apple also targets a niche. Apple does not market itself as a general officesupply like a stapler or a handheld calculator.

Vista was not a major improvement. Microsoft doesn't control the computerhardware like Apple does. Anyone who has seen any of the “I'm a Mac and I'm a PC”commercials where the PC user was rejoicing about being error free for nearly a weekcan get a good understanding of just how bad Vista has been. So bad, Microsoft hascome up with a new campaign where they won't even use the name Vista anymore. Toobad for them the eBook version of this book will remain for sale on servers long afterMicrosoft has ceased to exist.

Over the course of its existence, Microsoft had managed to compel people andcompanies to upgrade and stay current with the latest toys. That worked until the PCbecame a standard office supply. Companies look to spend as little as possible when itcomes to standard office supplies. If any of you have tried to get a name brand surgeprotector because of some delicate equipment through your local office supply group,you know what I mean. What they ship you is a $2 generic that may or may notsuppress anything.

Windows XP Pro became successful because it remained virtually unchanged forsix years. Companies developed a lot of software for it, and most employees alreadyhad some idea of how to check email and surf the Web on it when they were hired. It isa heritage system. The very thing Microsoft (and most everybody trying to sellsomething new into IT) rails against, claiming it is holding people and companies backfrom reaching their full potential. What the marketing types don't understand is whyheritage systems exist:


1. They are a sunk cost.

2. They do their job.

3. Most companies don't make money changing software or hardware platforms.

Apple can get away with a complete redesign of their platform because the officedesks they are found on need/want all of the improvements the new platform provides.They occupy creative niches at companies and really are still personal machines. ThePC targeted the much wider “business tool” market. What Microsoft shareholders andleaders haven't realized yet is that once a bean counter is in charge of buying hammersfor the carpenters, the carpenters will all be carrying the cheapest hammer out there.When Apple moved to Intelbased chips it opened the door for going head to head withMicrosoft and some companies warmed to the product line. It should also be noted thatApple recently exceeded Microsoft in market cap size per the SEC formula ofshare_price * authorized_shares. This isn't because Apple is taking over the businessdesktop, rather it is because Microsoft is losing the business desktop to Linux, thecheapest hammer out there.

Vista made Apple look like a better deal in the eyes of many bean counters as well.Vista required significant hardware upgrades and was deservedly maligned in theindustry trade press. When the bean counters started adding up the upgrade costs andtotaling in the cost of Vista problem resolution along with lost business productivity thehigher priced Apple products started looking cheaper by the minute; of course Linux,with its free office software suite which could run on the existing desktops without anyhardware upgrades looked the best of all, but that required a bit more selling.


1.8 The MutantNinja Data Model

Many of you might be long enough in the tooth to remember when you couldn'tpick up a weekly geek magazine without seeing a bunch of ads and reading morearticles about how object databases were going to completely replace all other databaseengines. Well, that didn't happen, but the industry did get hungry for some capabilitieswhich only object databases could deliver. Likewise, hierarchical databases fell out offavor, but never disappeared. We are now beginning to see what some geeks will callthe MutantNinja database model, but others will call the objectrelationalembeddedhierarchical model. Some even try to leave out the fact there is an embeddedhierarchical database by simply calling it the objectrelational model. I prefer to call itthe MutantNinja Data Model.

There aren't really any of these for us to play with in the Open Source world. Forsome strange reason the only people working on them are the commercial databaseproviders. Given the nature of the Open Source world it would be a lot smarter to comeup with a design in the commercial world, then Open Source it to shake out theproblems.

For a while we had some database engines using relational as the outer engine,accessing blobs via an object model of some kind, and still providing hashed index fileservices. DBAs, to a large extent, hand pick each storage type, database model, andlocation. We are less than ten years away from that changing on a permanent basis.

In a not too distant future, a DBA will run an interactive tool when they want tocreate a database. It will ask them some questions about initial row counts, columnnames and data types, foreign keys, and key/index columns needed. The DBA willchoose existing tablespace, or be allowed to create shiny new tablespace, then, when allquestions have been answered, the tool will create a database in the format it chooseskeeping hidden all knowledge of its real organization. We will no longer use a databasemodel, but a Data Storage System that will determine all of the “how” yet promise uswe can view the information as we wish as long as we access it via the URL it provides.


There is nothing which can be done about this. It is the only logical progression.Each database model has a class of problem it is uniquely perfect at solving. XML isused nearly universally for external communications and companies are seekingmethods of storing this beast. An XML document is, by definition, a hierarchicaldocument. When you have a limited number of rows in a reference table, nothing isfaster than a sorted sequential file and a binary search, unless the data is really short,then linear search. The “almost small” row count data set lends itself to a hash table,but “almost small” tends to be defined by the quality of the available hash. When youhave a mediumsized data set with less than four segmented keys, VSAM (RMSIndexed) rules. When you have real time data coming in that requires embeddedtriggers to automatically fire once certain methods return true, object makes it a piece ofcake. When you have tables with massive amounts of data requiring serious amounts ofreferential integrity, relational is currently the answer.

We have spent the last 20 years migrating perfectly good business systems torelational database systems for no other reason than to get SQL access.

While it is true that some of those systems had their lifespan greatly extendedbecause the relational model didn't have the same file size limitations their previousstorage models had, it is also true that we had 4GL tools the industry chose to shit upon.One of the most impressive 4GL tools I ever worked with was Cognos PowerHouse. Itran on many platforms and you could link a native indexed file to an RDB database toan Oracle database to a Sybase database to N other databases all in the same report,batch update module, or data entry screen. The tool is still available today, but Cognosis now owned by IBM. You really should try it before your IT career ends.

What happened was the industry went through a period where proprietary thingswere considered bad (yet for some odd reason proprietary database systems like Oracleand DB2 were considered good.) The only reason for it at the time was the fact theGartner Group was paid to market “Open” systems. The definition of “Open” was“whatever they were paid to sell this week” because somehow Microsoft, one of themost proprietary operating systems on the face of the planet having a single vendor anddamned little accurate kernel documentation, was also branded as “Open.”


The end result of this disaster was an MBA mindset that you couldn't pay a supportcontract for a cross platform 4GL, but you could pay one for a proprietary relationaldatabase because SQL was an “open” standard, never mind that every database had itsown dialect of SQL and that very little production code could ever migrate. Since theMBAs wanted to be able to access all information the company had via an “oooh look atthe colors” dashboard application, all data had to be in the relational database thatdashboard used.

Now you find many SQL front end tools and libraries for old data storage formats.It wasn't until recently that upper management learned the SQL wrapper provided byMySQL really worked with an indexed file system rather than a relational database bydefault on most platforms. Most didn't know that until Oracle bought InnoDB, whichwas the relational engine shipping with MySQL and a public brewhaha ensued in thetrade press.

Long hours combined with massive quantities of caffeine and nicotine are nowhappening behind closed doors at all of the major database vendors, and probably someof the Open Source projects. Of course they will have to come up with a sexier namethan DSS (didn't IBM call some ancient form of DASD DSS?) and they cannot useDataWarehouse because that's already been used for other things, but the solution,however named, will be the same. The DBA will use a tool which lets them specifyhow the columns will be viewed by the outside world, but the database engine willchoose how to store them. More importantly, it will dynamically switch from oneformat to another when it senses a limit being reached and automatically rebuild indexeswhen it notices performance degradation. We will no longer have any low levelrecovery options. We will use the backup and recovery tools provided by the systemand the people who created SOX will be happy because now we have only one tool fordata storage and their audits will be easy.

Of course, some of you might think this is sounding all too familiar. Well, it is, ifyou happen to ever work with the PICK operating system.

http://en.wikipedia.org/wiki/Pick_operating_system

http://en.wikipedia.org/wiki/Pick_operating_system


We will have a different database engine, but we will be headed back there shortly.What do you want to bet we end up with a mishmash of BASIC and SQL as theprimary languages again?

1.9 Exercises

1. Why do heritage systems exist (3 reasons)?

2. What does WSDL stand for?

3. Without a network and without the ability to expose part of your disk farm tooutsiders, how do you have to share data?

4. What are the three main reasons which would force a clientserver solutioninstead of a Web based solution?

5. What was the really big problem with clientserver computing during the 1980sthrough the mid1990s.

6. What does YAST stand for?


Page left blank intentionally.

Chapter 2

Cautionary Information

2.1 Scope of This Chapter

In this chapter we will cover some of the major issues encountered when using Qtand/or databases. These will be the kind of things which “shouldn't” happen. Whenthey do happen you chase your tail for hours and sometimes days tracking the problemdown.

2.2 Development Restrictions

If you come from a development environment on a robust platform which wascentered around a code management library, developing with the Qt tool set is going toseem like an act of extreme sacrilege. The Unix, and by extension Linux, platforms donot have any concepts of logical names or distributed architectures. Here, you have atbest, a PC mindset. That mindset, being small, simple, and not looking further aheadthan the weekend, developed a tool set which matched its thought patterns. There isonly one guiding principle.

Everything in a single directory tree.

You will find this philosophy wrapped up in every Open Source project youdownload or use. Back in the days when we only had indexed files we could muddlethrough. When you start actually running software as an installed image and needmultiple versions, this philosophy starts to break down. It goes completely out thewindow when you need to run D,T,S, and P versions of databases. (Development,integration Testing, production Support, and Production, all on the same machine orcluster. Generally running Production on your development machine(s) is part of adisaster recovery plan.) I covered the bulk of the DTSP issues in ISBN13: 9780977086603, please refer to it for further information.

26 Chapter 2 – Cautionary Information

2.3 Building a Qt Program

Once you have all of your source files in a single directory, you need to create a Qtproject file. If Qt has been correctly installed on your machine, you can do this bysimply typing:1)2) qmake -project3)

A file using the current directory name with a .pro extension will be generated.The name of the executable file which comes out of the entire build process will also bethe same as the directory. (I think you are now starting to see why I harped on the“Everything in a single directory tree” philosophy.) It was never a good idea onmidrange platforms to have a directory provide the name for an executable containedwithin the directory, and it was an even worse idea on the mainframe platform, but hey,the Open Source community is all gaga over using a convention which has failedmiserably on every other platform.

When dealing with a relational database via the Qt libraries, you need to manuallyedit the resulting PRO file to add the following highlighted line prior to generating theMakefile.

4) ######################################################################5) # Automatically generated by qmake (2.01a) Sun Jun 6 19:10:04 20106) ######################################################################7)8) TEMPLATE = app9) TARGET = 10) DEPENDPATH += .11) INCLUDEPATH += .12) QT += sql13)14) # Input

I don't know why the qmake tool isn't smart enough to do a quick scan inside of thefiles and determine an application uses the SQL classes. Once you've added the line andsaved the file you can run qmake one more time without any parameters to generate aMakefile. Your final act of building an executable is to type make and watch it run.Assuming you don't have any typos or other compilation errors, you will get anexecutable with the same name as your directory.

Chapter 2 – Cautionary Information 27

Because this section of the book is about development restrictions, take a look at theother lines above our highlighted line. The “.” stands for the current directory.Everything is assumed to be in your current directory. If you want to change theresulting executable name you must provide a value to the TARGET line. When youare using third party libraries and need to force recompilation due to a header filechanging, you need to add the directory tree holding the header files to DEPENDPATH.To ensure the compiler looks in that third party directory for header files, you need toalso add it to the INCLUDEPATH. Use a space or a “;” to separate entries in the pathlist. There is a long list of variables which can be defined in this file. Use your favoriteWeb search engine (hopefully not Google) and search for “Qt includepath” without thequotes. You should see at least one link to a Nokia/Trolltech documentation page.Hopefully the documentation page has a version number close to the version you areusing of Qt so everything works the way it is described.

2.4 The Database Username Restriction

One of the most frustrating aspects you will encounter is obtaining the usernameand password for database access. It sounds simple. Using a procedural tool, youwould pop up a window to prompt for these, then delete that window and display yourmain form or forms. Even with our Java applet in “The Minimum You Need to KnowAbout Service Oriented Architecture” we handled this cleanly. Not so when it comes todesigning your Qtbased applications. Handling this one task dramatically impacts yourapplication design. Most examples you will find for Qt concentrate only on the GUIaspect since it was originally a GUI library. Each will have a main.cpp file which looksmuch like the following.1) #include <QApplication>2)3) #include "mydialog.h"4)5) int main(int argc, char *argv[])6) {7) QApplication app(argc, argv);8) MyDialog *dialog = new MyDialog;9) dialog->show();10) return app.exec();11) }


When the initial GUI object is a dialog you typically allocate it by calling new andlet the Qt environment delete the attached object when the application exits. What youhave to remember is that Qt programs aren't just GUI OOP programs, they are eventdriven. Everything occurs in the event loop handled by the QApplication object methodexec(). I've been through this many times with other products. I'm very glad I am notcovering device IO in this book like I did in my first Zinc book. When you end uphandling IO from 32+ port serial cards and controlling relays via digital IO and Opto22components you end up having to code your own event loop to handle such things.

Adding a database to this mix opens nearly the same can of worms, not because youhave to hack your own event loop, but because in the Qt world the database object isglobal data. It really needs to be owned by the main.cpp file so it never goes out ofscope and isn't garbage collected on you. Keep in mind what I told you about C++ in“The Minimum You Need to Know to Be an OpenVMS Application Developer”: Oncean object goes out of scope it is eligible for garbage collection. Some compilersgarbage collect immediately and others postpone the collection until idle time. Javadevelopers will have an easier time understanding this given that Java almost alwayswaits until idle time to perform garbage collection.

Your second major problem with adding a database to the mix is the methoddifferent databases use to validate a user. Some relational engines are configured sothat any user on the local machine has automatic validation, other relational databasesprovide their own user authentication. Some databases use both methods. The first forlocal users and the second for users accessing via TCP/IP services.

Many of you are probably having trouble understanding what I'm complainingabout, so let me give you a small example. Here is the main.cpp from the xpnsqtproject.


1) /****************************************************************************2) * Originally created by Roland Hughes at Logikal Solutions.3) * Project is being released under GPL2 license.4) *5) * At some point in the future either Logikal Solutions or Roland Hughes6) * may elect to publish a book with this original source code in it. If7) * the book is written it will be part of "The Minimum You Need to Know"8) * book series.9)

****************************************************************************/10) #include <QtGui>11) #include <QtSql>12) #include <QtDebug>13)14) #include "XpnsLogonDialog.h"15)16) int main(int argc, char *argv[])17) {18) int i_x;19) QString qtDbName;20) QTime theTime = QTime::currentTime();21)22)23) QApplication app(argc, argv);24)25) {26) QString driverName = "QPSQL";27)28) //29) // Set up information for driver and see if driver is available30) //31) qtDbName = "xpns" + theTime.toString("hhmmsszzz");32) QSqlDatabase db = QSqlDatabase::addDatabase(driverName, qtDbName);33) if (!QSqlDatabase::isDriverAvailable( driverName)) {34) QMessageBox::critical( 0, "Missing PostgreSQL Driver",35) "The Qt driver for PostgreSQL "36) + driverName37) + " is not installed. You must install it before running

this program",38) QMessageBox::Ok);39) db = QSqlDatabase(); // reset to avoid warning40) return 0;41) }42)43) XpnsLogonDialog *xpnsLogonDialog = new XpnsLogonDialog( 0, qtDbName);44) xpnsLogonDialog->show();45) i_x = app.exec();46) qDebug() << "About to delete dialog";47) delete xpnsLogonDialog; 48) qDebug() << "Dialog deleted, now clearing database"; 49)50) db = QSqlDatabase::database(); // clear connection to avoid error51) } // end scope of logon dialog52)53) QSqlDatabase::removeDatabase(qtDbName);


54)55) return i_x;}

Pay special attention to the highlighted {} surrounding the inner code in this module(listing lines 25 through 51.) This is required to ensure the db object goes out of scopeand is garbage collected. Just prior to exiting that block I have to assign the db object tothe default database. You can only remove a database from the global storage area onceall references to it have been removed.

The global QSqlDatabase object creates a “default” database object whenever youinclude SQL support in your program. Many developers will only use the defaultdatabase. The difference between a default database and other databases is the defaultdatabase doesn't have a name. Notice the highlighted line calling the methodaddDataBase(). The first parameter is a string containing the driver name. The secondparameter is a name you give this database object. Any other module in your programcan use any database object you add to the global database object. In affect it is a list ofdatabases your program has access to and is stored globally. You cannot remove adatabase from this list as long as there is still an object referencing it. Oddly enough,the reference count isn't reduced when the db object goes out of scope and is garbagecollected. I assume they will fix this bug eventually. For now, you need to remove thatreference by reassigning the object to the default database.

If you fail to remove named database objects, or fail to remove all references tothem, you will receive warning messages in the default output window when yourapplication closes. Many developers simply ignore these messages and set thereapplications up to be launched from screen icons rather than run inside of a terminalwindow. If there is no default output device (like a terminal window) you simply don'tsee these messages.

Notice how I had to launch a login dialog as the initial screen/form for thisapplication. When we get to the chapter covering this application you will see that thelogin dialog had to launch the actual main screen/form once it completed the loginprocess. This isn't the pinnacle of OOP benefits.


Why did I have to launch the login dialog as the initial form? Application design.In order to have combo boxes populated from various tables in your database, you haveto actually have a connection to the database and those tables. In order to get thatconnection, you have to ensure you have a username and password. You cannot assumethe database will allow free access to anyone who wants it. In fact, the current wrapperprovided by Qt doesn't allow you to connect without a username and password, evenwhen PostgreSQL allows this as we have seen. Now my login dialog is tightly coupledwith the application.

If you choose to not have any combo boxes or database populated pick lists on yourmain form, you can write a generic login dialog which can be popped up after yourmain form has been created and displayed. This generic dialog will be loosely coupledwith your application and infinitely reusable. The problem is that a developer rarelygets to make the choices about what is and isn't on a form. Users want it all and theydon't care how you have to warp the architecture to produce it.

2.5 Garbage Collection Restrictions

Java rots your brain. I really admire developers who can easily slide between Javain the morning and C++ in the afternoon without ever creating a memory leak on anysystem and without using tools to grind their application checking for memory leaks.My personal guess is that there are about five such people in the software developmentuniverse. I have no idea who they are, but I firmly believe there cannot be more thanfive of them. I especially doubt that number is increasing.

When you are programming in Java, you have to allocate everything except thescalar types with new. The JVM keeps track of your references and does garbagecollection whenever it deems garbage collection is appropriate. C++ automaticallycleans up stack allocated variables when they go out of scope, but variables allocatedvia new must be physically killed off by a call to delete.


Qt tries to aid in this by telling you it deletes all objects which are derived fromQObject. I have seen this statement in a lot of documentation, and I'm not exactlybuying it. Actually, it is mostly true, but telling people this is causing more problemsthan it is solving. The main problem occurs with QSqlDatabase. You have to clean upthe references yourself. Earlier I showed you some code where I had to reassign ourpointer to the default instance in order to free the reference, then I was able to removethe database from the list of databases.

Why do we find ourselves in this bind? Location, location, location. QSqlDatabaseis global. The instances you declare of it tend to be on the stack rather than on thedynamic heap.

QSqlDatabase db = QSqlDatabase::addDatabase(driverName, qtDbName);

Only dynamically allocated objects can be removed by the delete operator in C++.Objects allocated on the stack are supposed to be cleaned up with the stack clean upprior to return, but you cannot remove them. Because you cannot physically deletestack allocated objects yourself, you cannot get rid of the references to global objectscleanly.

But when the object went out of scope it should have been garbage collected, right?No. It was made available to garbage collection, but each platform implements garbagecollection at a time and place of its own choosing. The only way to clear the referencewas to reassign my stack object to the default database connection prior to exiting theenclosing block.

db = QSqlDatabase::database(); // clear connection to avoid error


2.6 More Database Issues

Almost all of the examples I find floating around the Web show how to use thedefault database connection, not a named connection. Why should you avoid using thedefault database connection? Come on, think about it, you know the answer. Modulereusability. If you always pass in a string with the name of the database as Qt has itstored, you can call your modules/classes from any application you develop. It doesn'tmatter if that connection is the first or the 50th connection.

You will find the xpnsqt application is coded quite differently from themega_zillionare application used in other books in this series. The main.cpp for xpnsqtappears earlier in this chapter. It is the one we have been discussing with respect todatabase issues. I didn't notice it when working on Ubuntu 8.10, but when working onSuSE 11.x I was seeing error messages like the following:roland@roland-desktop:~/megaqt$ ./megaqt QSqlDatabasePrivate::database: unable to open database: Access denied for user

'roland'@'localhost' (using password: NO) QMYSQL: Unable to connect Successfully connected to database

The instruction which tossed this error wasn't the open statement. It was the call tomake a copy of the database instance.

QSqlDatabase db = QSqlDatabase::database( m_qtDbName);

I must say I was really taken aback by this. All of the information you could findonline said you needed the main.cpp file to declare your first instance of QSqlDatabaseso it would be owned by the last module to complete execution. Most examples I couldfind did it pretty much the way I was doing. It appears that when you attempt to get acopy or instance of an existing database connection, the method returning that instanceautomatically tries to open it for you. Quite the little helper isn't it. “The road to Hellis paved with good intentions.” A lot of truth in that statement.

There was a slightly larger shock waiting for me when I ran the xpxnqt applicationand saw the following in my terminal window:


roland@lgkl-destop:~/xpnsqt2$ ./xpnsqt2QInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: No such file

or directoryQFileSystemWatcher: failed to add paths: /home/roland/.config/ibus/busBus::open: Can not get ibus-daemon's address. IBusInputContext::createInputContext: no connection to ibus-daemon QSqlDatabasePrivate::database: unable to open database: "FATAL: database

"roland" does not existQPSQL: Unable to connect" About to delete dialog Dialog deleted, now clearing database

When you look through the various support forums and email lists on the subject ofQt programming, you will find quite a few hacks, suggestions, and work arounds.Everybody seems to be happy simply getting one to three lines of code which makes theerror go away. They don't seem to be interested in actually fixing the problem. In thiscase, the problem was caused by someone trying to be helpful. It only became aproblem because a lot of other people tried to be helpful telling people they needed toget an instance of the database object in their main.cpp file to keep it from going out ofscope during the life of their program.

What we really need to make things work cleanly is our main.cpp to declare a driverfor the default database connection and get a copy of the instance. We need a driverwhich will not throw an error trying to connect when a method tries to get a copy of apartially configured entry.

I guess it shouldn't be any surprise that Qt comes with its own version of SQLite onmost platforms. I could not find an example of this anywhere I searched, but this isbasically what you need to do. We will see this done in the megaqt (mega_zillionare)example, not the xpnsqt example.

If you make a typo in your driver name, and you happen to have launched yourapplication from a terminal window, you will see a helpful error message along the linesof the following:roland@roland-desktop:~/megaqt$ ./megaqt QSqlDatabase: QMYSQL driver not loaded QSqlDatabase: available drivers: QSQLITE QMYSQL3 QMYSQL QPSQL7 QPSQL roland@roland-desktop:~/megaqt$


Of course you might have the driver typed correctly and even see the name of it inthe list of available drivers. In that case, you are using a static Qt library. In the sourcefile which is actually making reference to the driver you need to type the followingmacro:Q_IMPORT_plugin(qsqlmysql)

and in the .pro file you need to add:QTPLUGIN +=qmysql

Here is another beautiful error message which will have your head slammingagainst the keyboard:QSqlDatabasePrivate::removeDatabase: connection 'blah' is still in use, all

queries will cease to work.

I loved seeing this message. It took a long time to figure it out. This is the messageyou get when you forgot to remove your last reference to a database before callingremoveDatabase(). I've already shown you the fix for it. Simply assign your databasevariable to the default database before calling remove. That will clear the reference inmost cases. You get this error even if the database is closed because the referencecounter is greater than zero.

2.7 Pitfalls of Mandatory Passwords

You have already seen a few examples from the world of PostgreSQL where it willlet a properly configured user connect to a database without forcing a password prompt.The database itself has to be told to trust local system security before such things areallowed. roland@lgkl-destop:~$ psql -d postgrespsql (8.4.4)Type "help" for help.

postgres=#


Most of your security type geeks make it a point to turn this feature off on anynewly created system. The last thing they want is someone with access to a sensitivedatabase (like one full of credit card info) walking away from their terminal withoutlocking it. Sometimes the user community wins and sometimes it doesn't.

I'm pointing this out in the cautionary section because it can really screw yourdesign. Actually, it can make entire applications impossible to develop. In the nextchapter you will encounter our XPNSQT application. While it does not try todownload, install, and configure PostgreSQL automatically for the user, it does try tocreate a missing database.

You should not be surprised to learn that creating a year specific expense databasewill require user input. It should surprise you even less to learn that the applicationbuilds a SHELL command on the fly using the tax year and other information to createthe database. cmd = "createdb " + db.databaseName() + " 'Expense Information';"; int x = system( cmd.toAscii());

The system() command in C/C++ does not create a shiny new terminal windowwhere a user could view and respond to prompts that might be encountered when thecommand runs. If someone needs or wants that capability, they will have to modify theapplication to use QProcess provided by Qt.

I need to point out that “createdb” is a helper program provided with thePostgreSQL database package. It is expected to reside in a directory which is ineveryone's path. There are actually quite a few of these helper programs. Where theyget installed depends upon your platform and database version.

roland@lgkl-destop:/$ dir /usr/lib/postgresql/8.4/binclusterdb createlang dropdb dropuser oid2name pg_controldata pg_dump pg_resetxlog pg_standby postmaster reindexdb vacuumlocreatedb createuser droplang initdb pgbench pg_ctl pg_dumpall pg_restore postgres psql vacuumdbroland@lgkl-destop:/$


2.8 Database Frailty

If you work in IT long enough, you will find out there are reasons for mosteverything. Many times you will find that none of those reasons were good, but acourse of action was chosen anyway. This is especially true of database products.

On the OpenVMS platform, we have a wonderful database called RDB. It wasoriginally developed by DEC (Digital Equipment Corporation) and is now owned byOracle. No, it isn't the Oracle database you now see running on useless Windowsboxes, I'm talking about a good database. RDB is completely integrated with theoperating system. It uses the ACLs (Access Control Lists) and rights identifiers fromthe same UAF (User Authorization File) as the operating system. In short, it doesn'thave to provide some hokey rollyourown user administration and rights identifiersystem like almost every other database on the face of the planet.

OpenVMS is the only operating system which truly clusters, many claim it, but onlyOpenVMS actually delivers it. The OS kernel understands the concept of a recordalong with the concepts of ACLs, rights identifiers, logicals, and user privileges. Thisenables the kernel to support and utilize a lock manager which has complete recoverybuilt into the system boot process. This enables the cluster to have a distributed lockmanager that works in unison with every lock manager on every node in the cluster.The lock managers provide the foundation for the distributed transaction managerwhich allows a transaction to span multiple databases, indexed files, journaled flat files,and message queues with full rollback/commit capabilities. It may not be sexy, but it isdata Nirvana.

RDB allows you some choices with database creation. You can create a single filedatabase, or you can spread your database across multiple drives and directories. Singlefile databases aren't speed demons, but they are pretty cool. You can lose the systemdisk which had RDB running, create a shiny new system disk with RDB on it, boot, andtell RDB to open your database. All of the information needed is stored in the singlefile database and its directory. You have complete recovery. It's a bit trickier with themultifiledisk version.


I'm bringing this up to you now to let you know that none of the databases we willbe covering have that capability. Most PC users think about the cheapness or ease ofuse when choosing a database, not about what happens when things go bad. I guess thatis kind of understandable since a lot of those people are using a computer with a singlehard drive they rarely back up and never take the time to create a full image backup.When the spindle spits a platter onto the floor they are well and truly pooched.

Professionals, on the other hand, have made a living designing systems on midrangeand mainframe computers. We've had to build in fault tolerance to levels which wouldmake the average PC user faint if they tried to wrap their heads around it.

I'm on my soapbox now so that you will understand just how shaky things are.Every database we will be covering has the exact same flaw:

If you lose the disk containing the database engine directory and have toreinstall on a new drive, all of your data is toast. It doesn't matter how manydrives you spread the data across or how religiously you backed it up withsome third party tool.

The flaw is caused by the database engine keeping all of the schema, user, access,space, and proprietary information in the directory with the engine, or somesubdirectory thereof. When you lose that information, you lose touch with the data, nomatter where it is stored. MySQL, as well as most of the others, tries to hide this factfrom users by making the default location for database creation the same directory treeas the database engine which puts it on the same spindle as the operating system. Thisway you lose it all together and don't realize the flaw. Of course, creating a productiondatabase on the operating system disk drive is a systems management nono of biblicalproportions.

I found this out about PostgreSQL the hard way.

No binary backup made with any backup tool is of any use if it cannot restore yourdatabase engine directory to exactly what it needs to be so database information isn't


lost. The “backup method” used by all of these databases is for you to write a script (oruse a provided utility) which dumps the data into a text only CSV (Comma SeparatedValue) file. Once you recreate your database from the script files (you did use a scriptinstead of just keying the definitions in at the interactive prompt didn't you?) the CSVfiles are to be imported into their corresponding tables.

If you aren't an IT person by trade, that pitfall probably doesn't seem so bad. I aman IT person by trade. The first two versions of my expense tracking software usedstandalone indexed files which I kept in different directories for different years on aseparate disk drive. I also backed that drive up religiously. I could easily recover fromany situation prior to moving to an Open Source relational database. Now, in order tobe able to recover, I have to quintuple my storage requirements. Once all of the data isentered and verified, I must export each table to a CSV and backup that expandedversion along with the scripts to create and restore the CSV.

What I'm showing you will be good enough for your own personal development. Itmay even be good enough for a small company as long as you are religious about CSVexport and backup. Some companies have been burned pretty bad by this frailty. I haveseen some companies map the system boot disk (which contains the database engine andproprietary information) to a RAID 6 volume on their SAN so they could be protectedfrom two spindles burning out at the same time.

Only you can decide what your data is worth. Don't make that decision after you'vedeveloped the application.

It's difficult for a regular PC user or even the average Open Source developer tograsp just how critical this issue is in a production environment. If your database iscurrently approaching 1TB in size and is filled with binary data (mostly numbers, notmuch text) you will need 5TB of contiguous free disk space to create your CSV file.Right now the biggest SATA drive you can buy for your PC is 2TB so unless you linkthree of them together with a RAID controller, you cannot backup.


2.9 Main.cpp for What?

One of the larger issues I have with the “accepted” method of Qt development andits “one directory tree” philosophy is the concept that every application should have amain.cpp file. I find this in all of the books I look at and most of the online examples.It appears to be the will of the Qt programming community.

Let me be the first to tell you, this is a disaster waiting to happen!

Yes, I really wanted one sentence all by itself like that. You need to pay attention tothis issue. If every project has to have a main.cpp and they are all custom for thatproject, you have a ticking time bomb on your hands. Let me tell you a story.

Back in the days of my yoot, I worked at a DEC VAR (Value Added Reseller.)Besides selling DEC equipment, OpenVMS, and its layered products, we sold acustomized ERP system. It was like many of the ERP systems of its day. All of theorder entry modules were named OEnnna so the main order entry source file would besomething like OE710 and the supporting source files would be named either OE710A,OE710B, OE710C or OE711, OE712, OE713, depending upon what they where andhow the developer split things. (There were rules, but they aren't relevant to thisdiscussion.)

We had no source code control system. Each customer would have a temporarydirectory tree set up for them by the System Analyst assigned to the project. We wouldall take source from that tree and work on it in our own little development accounts.When we had completed our coding and testing the Systems Analyst would be giveninformation on where to pull the source from our account and what files to pull. If itpassed testing, they put the new source into the client source tree.

If business was going good, we never had enough disk space to go around. Once aclient had been installed and a few days of “shake out” time had passed, their treeincluding source, data, etc. was backed up to a removable platter and/or tape, thendeleted from the system. Many times there were races between a tape restore processand getting the developers to delete enough items for the restore to completesuccessfully. Some of our clients actually purchased enough disk space that we couldstore a copy of the source files which had been modified for them out on their systems


as well. A token few even dropped the $1200 it cost for that 300 baud modem so wecould dial in to support their systems.

So now, scattered across three disk drives and literally thousands of subdirectories,were various copies of every source file in the system. You can guess what happenedwhen a new programmer started. They would be given a list of custom changes to onemodule and would invariably pluck that module from a directory tree other than the onethe System Analyst (SA) had created. Since they had no idea as to what changes hadalready been made for that client, they had no idea that some things were missing fromthis version. The SA would find that out the hard way when they got around tointegration testing.

Oh, there was one other thing which would happen with new people. They wouldn'tcreate a different subdirectory for each project they were working on and leave it untilthe client had been removed from the system. Given all of our time was billable, wedidn't just get to sit around waiting for bug reports to come back from the SA. Wehanded off our stuff and got a new assignment at that time. If you had just gotten donemodifying IN110 for client A, the next SA may give you the exact same module tomodify for Client B since you were already familiar with it. Half way through yourmodifications the first SA would come back with bugs they needed fixed right away dueto an installation date coming up. You guessed it. They would pull that module intotheir working directory and walk on everything they had done for Client B withoutknowing it.

I'm pointing this out because it is a massive pet peeve of mine. It is physicallyimpossible to say enough bad and degrading things about code management systemswhich allow directories to be created inside of them or which enforce some kind ofdirectory structure on a project. These are not code management tools, they are codedisasters in the making. Naturally CVS and the other tools which support this happen tobe the free stuff used by many, if not all, Open Source projects. These are bad bad badbad bad bad things.


A quality code management package will allow a source file name to exist oncewithin a code library. If you want to fork off development, you have to create acompletely different code library which is optically isolated from the first code library.A developer (or build procedure) can pull a specific generation of any element at anytime without having to know what directory the thing was squirreled away in. Whenyou issue a command like “SHOW HISTORY OE710.BAS” you get the completehistory for the element and each generation is either a linear or flat progression. (Linear= put back with changes from prior version, Flat = put back without any changes,sometimes programmer error, other times result of migration from Development to Testto Production to Support.)

For a better explanation on how a quality code management system works read,“The Minimum You Need to Know to Be an OpenVMS Application Developer” ISBN13 9780977086603. In particular you want to focus on the chapter covering CMS.

Readers looking for general information on code management systems can starthere: http://en.wikipedia.org/wiki/Revision_control though the author(s) don't appear tounderstand the problems of allowing branches and forks within the same repository.What they do talk quite a bit about is how CVS can, in many cases, merge the codefrom one fork/branch with another. That's a scary thing Maynard.

2.10 Transaction Integrity

This topic is so important that I broke it out into its own section. While the SQLinterface provided by Qt can do a great many things, one thing it is not designed to do isprovide transaction integrity. The database engine, or the driver providing access to thedatabase, has been made completely responsible for transaction integrity. This meansyou are personally responsible.

Far too many developers expect any SQL interface to provide them completetransaction integrity. I have seen many a catastrophe caused by stupid developers whobreak the cardinal rule of database application development.

Keep your transactions short.

http://en.wikipedia.org/wiki/Revision_control


That's it. That's the one rule you have to remember and it is the one rule most ofyou ignore. I've even been guilty of it. Actually I've had business units at client sitesforce upon me application design rules which meant I either created a massivetransaction or I wrote an awful lot of recovery code which may or may not work in theevent of a system or hardware failure.

Please consider the applications being presented in this book. While they “could”be multiuser, they are really designed to be standalone. (If two different users with theXPNSQT application try to create a new tax year database at the exact same instant, oneof them is going to have a less than happy experience with the application.)

When we design standalone applications, we seldom worry about multiuser issuesor the size of our transactions. One client site used to have a rule which was enforced incode reviews that no database write transaction could contain more than 300 recordsbefore a commit had to occur. While hardware improved, they kept this rule in placeand currently have applications in production which still adhere to this rule. While itmight sound stupid today, that rule existed for two reasons:

1. Database lock timeouts

2. Snapshot file size

If you have 400 different users all attempting to do some bulk insert of 20300records on the same table at the same time, lock timeout can be an issue. It certainlywas an issue back in the early 1990s when disk access was comparatively slow so eachof those users need one to three seconds of lock on the same table. You don't have to bea math major to figure out that at two seconds each: 300 * 2 = 600 seconds /60 = 10minutes. Basically every user past the 2 minute mark was going to go into a rollbackstate due to time out which, at the time, was even less efficient than a commit.

Today we have similar issues with the freely available databases on our PCs. Whilethese issues aren't multiuser per se, they amount to the same thing, especially if yourapplication is being accessed via a Web site with each connection having its own thread.Imagine how things would fail at eBay with 100,000+ bidders all trying to update thebid history table simultaneously if they each needed a one second lock on the entire


table? They don't have to be bidding on the same item, just bidding on AN item to forcethe need for an entry.

• Have you ever been a Project Manager?

• Have you ever been a Business Analyst?

• Have you ever read SDLC (Systems Development Life Cycle)documentation for a shiny new project which involves a database?

• Have you ever sat in a meeting with the IT team and the MBAs where theywere arguing about what new things needed to be purchased to make thisproject work?

What is the one thing which gets questioned on every project?

Oh come on. We have all sat in college database classes and been told the reasonrelational databases can provide a rollback capability is because all of the pendingchanges are kept off in a snapshot file or area until the COMMIT occurs. We have allsat there at our terminal and typed something like:

DELETE * FROM PERSONNEL;xxxx rows deletedROLLBACK;SELECT * FROM PERSONNEL;

And simply been amazed to see all of the data on screen. Then we repeated thesame exercise with a COMMIT instead of a ROLLBACK and saw zero rows make it tothe report at the end. No matter what we did, that data was gone for good and had to bereloaded from the source file. Some of the older database systems would even let us dothe SELECT after the DELETE without either a COMMIT or ROLLBACK so we couldsee the impact of the DELETE prior to deciding whether to COMMIT or ROLLBACK.

I really feel sorry for DBAs when we are sitting in those meetings. It doesn't matterhow many times I point this database example out, or how many times they write theformula from the database vendor on the board, MBAs simply can't understand why weneed to spend so much money on disk space that will never be allocated to actualdatabase storage.


Perhaps it is time to rehash another lesson here since many of you will be PC usersand have no idea what disk storage for real computers costs. Yes, you can find 1TBSATA hard drives for under $100 through many major retailers these days. The priceseems to drop on them every couple of months. These drives, however, don't reallywork for real computers. Real computer systems obtain much of their storage from aSAN (Storage Area Network) device. This device may be colocated in the computerroom, or somewhere else in the country.

Many of the 1TB SAN units require a forklift to be moved and their own 200AMPservice to operate. These are high availability devices. Every drive, except the hotswap in cabinet spares, stays spinning while the unit has power for as long as it haspower. When the unit senses a failure, it can recreate the failing/failed drive on one ofthe internal spares while the operator replaces the failing/failed drive with a new one offthe shelf. When the unit senses a “quiet” time it will recreate the original drive in itsoriginal place returning the in cabinet spare to the spare pool. (Okay, some require asystem manager to issue a few commands, others will only recreate in place, but someare this way too. Let's not chase a red herring and lose sight of the lesson.)

The SAN allows the systems managers to configure any number of spindles as alogical disk drive available to one or more computer systems. Those spindles can beconfigured as raw storage or as various RAID levels (which is what allows the hot swapwithout data loss.) Many of the most robust SAN systems used only SCSI disk drivesbecause they were built far tougher than their IDE counterparts. It was not unusual tosee a $300+ price difference between SCSI and IDE for the same size drives from thesame manufacturer. Later, some sleazy drive manufacturers tried simply putting theirSCSI logic cards on top of their IDE manufactured drives as a way of increasing profits.They were quickly weeded out of the marketplace when replacement drives from themstarted lasting days instead of years inside of the SAN cabinet.

You see, the IDE drive (which is now morphed into SATA and PATA driveinterfaces over the same platter set) achieves its five year MTB by spending much of itslife powered off. Of course, marketing has been able to play this up as “Green”technology and many have even gotten Energy Star ratings. Once again, spending


much of your life powered off makes this possible.

Life in the SAN is different. The SAN must provide high availability. The reasonSANs were invented was to provide high availability and make disk management easier.Just how many users do you think will sit around waiting to place a bid if they have towait 30 seconds for one of the spindles the database is spread across to spin up and forthe database to verify its own structural integrity?

Some vendors are now trying to blend high availability with cheap technology innewer SAN devices. This requires two or more sets of standby racks since each standbyneeds to match the device type it will dynamically replace.

Other vendors are simply relying on cheap individual spindle prices and massivereplication. While the implementations vary, most simply require one to six additionalcold mirror spindles for every hot spindle. From the system manager perspective theysimply assigned a primary 1TB spindle and a mirror spindle, but internally the SANallocated twelve additional spindles to cold mirror those. The software in the SANattempts to regulate the amount of hours each spindle has to be hot. When it gets closeto time it spins up one of the cold mirrors and begins the update process directing mostnew traffic to the new spindle. Once the update is complete, the original hot spindleshuts down and cools while awaiting its next active time. Eventually someone will getthis technology to work really well, but I've been hearing mixed things about it for awhile now.

One of the big drawbacks with SAN in general is the bandwidth required to flashcopy 1TB between hot and cold spindles. By the time that issue gets resolved 2TBdrives will probably cost around $50 and the problems will start all over again. A majorreason a lot of SAN technology stuck with 36GB SCSI drives was the fact they couldcopy that much in just a few seconds and do all of the work on the SCSI controllerwithout involving the SAN CPU. Part of the “high availability” requirement means thata database request won't time out while you are in the middle of a hot swap.

How many of you have actually sat through a full format of your 1TB SATA drive?Not one of those “quick formats” which just writes out the file allocation table andupdates the MBR, but one which actually writes a pattern to each drive sector? Youaren't even doing IO at that point for most of these new drives. The drive has a


command which tells it to do that and it still seems to take forever. Imagine just howwidespread SAN technology would be if that was how long it took to “hot swap” aspindle?

Enter our project DBA. He or she has to sit in the project meeting and say that ifyou plan to have 2TB of data in your database, or one of the tables could have in excessof half a billion records, that they need 1TB of snapshot space to ensure all users access.They have to sell that to an MBA with a half a million dollar price tag for the SAN plusthe cost of getting yet another 200AMP service installed in the computer room and theadditional air conditioning capacity. In some cases it means having to put the SAN in adifferent data center or having to build an additional data center because you are alreadydrawing the maximum amount of power your municipality will allow an office businessto draw.

Laugh all you want at not being able to get the electricity to run the device. I haveseen that happen more than once. One client built an office building in a shiny newoffice park where the buildings were limited to 1000 AMPS each. It didn't sound baduntil upper management started putting more people in the building than originallyplanned. Before everybody got moved in, the data center didn't have enough power torun the single SAN it was supposed to run. Adding insult to injury, the BusinessRecovery Plan on file stated that the data center in the new office building would runthe company in the event they lost the primary data center … and I never saw it getupdated.

Many people erroneously believe that it is the responsibility of the DBA to ensureintegrity in the database. Nothing could be further from the truth. It is theresponsibility of the DBA to ensure all of the constraints are in place to ensurereferential integrity exists within the database, but it is the application developer andBusiness Analyst's responsibility to ensure that referential and domain integrity matchesthe business requirements. It is the responsibility of the DBA group to put in place abackup procedure which ensures recoverable archives of the database get created, but itis the responsibility of the system managers to ensure those procedures get executed andto coordinate the handling of backup media. It is the responsibility of the BusinessAnalyst to ensure the backup and recovery process occur within an outage window the


business unit is willing to live with and to obtain the funding necessary to get backupand recovery to occur within the outage window the business is willing to live with.Lastly, it is the responsibility of the application developer to ensure their softwareprovides transactional integrity.

If you, the application developer, are not a partner and friend to the DBA on yourproject, you are going to have a miserable project experience. If you start a transactionat the same time you initially connect to the database, but don't actually have data orcomplete the transaction until a user provides some response, you are going to be calledon the carpet when your program has resources locked for hours. If you try to selectevery row in the database for update without first obtaining a count to see if that isreasonable, you are going to have a very bad day.

Perhaps the biggest problem you will ever encounter in your career is the differentdefinitions “transaction” has throughout any given business.

• To the DBA, a transaction is a write or delete and commit to a single table.You might get them to agree to multiple tables within a single database, but notsomething spanning multiple databases or data sources.

• Your business people view an entire business process as a single transaction. Ifyou are working in a wholesale business supplying retail/dealer locations, thebusiness is going to consider a stock order coming in from the dealer all theway through the order acknowledgment with instock and back order indicatorsalong with pricing going back to the dealer as a single business transaction.This “transaction” not only encompasses checking the inventory at allwarehouses, but also checking available dealer credit and available shippingmethods. It shouldn't take a rocket scientist to figure out that much of thisinformation will be provided by either different databases or interfaces providedby completely different systems.

• Many portions of IT will refer to the business transaction as a multisegmentedtransaction. That definition is important, because once you define the segmentsyou can bust up the overall transaction into restartable units of work. Each unitof work will be a transaction a DBA can like but your logical transaction willrequire the successful commit of many physical transactions before it can


commit or be considered successful.

If you are lucky enough to be working on OpenVMS, it has a distributed transactionmanager (DECdtm) which allows you to commit and rollback transactions spanningmultiple data sources without having nasty database locking issues. When you areworking on lesser platforms, you have to provide your own recovery logic forsegmented transactions. What you cannot do is keep a different transaction open andpending in every database you are using until you know you are completely done.

Many developers are absolute geniuses when it comes to coding in C/C++, Boost,STL, or a host of other tools and languages. Many of them have only coded OpenSource or standalone applications on a PC and while they may be coding savants, theystill get fired when they try to work for big companies using real computers. Thereason they are canned is that they don't understand transactional integrity.

2.11 SQL != SQL

I've said this before and I really catch a lot of grief for it, but as a professionaldeveloper you have to believe it.

There is no SQL standard.

Yes, I know the entire ANSI membership began hunting me with pitchforks andtorches the instance I typed that line, but it's really the truth. Yes, there were somestandards published and yes, every SQL engine for every database will claim to supportANSI standard SQL, but less than half of the commands you type will work on morethan one database product, even if the database tables are declared exactly the same.Every database you encounter supporting some flavor of SQL will exploit“implementation defined” holes in the “standard”. They will also “enhance” thestandard at every opportunity.

You have been reading an excerpt. Please visit http://www.theminimumyouneedtoknow.com to learn more about thisbook series. You can also find out how to purchase works by this author.

Documents

The Minimum You Need to Know About Qt and Databases