24

The InterBase and Firebird Developer Magazine, Issue 1, 2005

Embed Size (px)

DESCRIPTION

The first issue of "The InterBase and Firebird Developer Magazine". Main topics: "Inside Savepoints of Firebird and InterBase", "How to avoid 10054 errors", "10 InterBase Myths", "Working with temporary tables in InterBase 7.5" and materials dedicated for Unicode and Hyperthreading issues with databases. Authors are Vlad Khorsun (Firebird core developer), Dmitry Kuzmenko (CEO IBSurgeon), Dmitry Kovalenko (IBProvider).

Citation preview

Page 1: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Cover story

2

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

ContentsCover storyby Dmitri YemanovInside Savepoints 3

by IBSurgeon Research Labs researchib-aidcomType conversion throughCOMPUTED BY 10

Database maintenanceby Dmitri KouzmenkoEmbedded User Authenticationin InterBase 75 11

by Vasiliy OvchinnikovUsing KEEPALIVE-sockets to detect and release hung InterBaseand Firebird client connectionsor how to avoid 10054104 errors 15

Developers areaby Dmitri KouzmenkoWorking with temporary tables in InterBase 75 18

by Dmitri KovalenkoWorking with UNICODE in InterBaseFirebird 21

by Dmitri KouzmenkoHyperthreading SMP and InterBase Firebird Yaffil 23

Dear colleagues

I am happy to present to you thefirst issue of ldquoThe InterBase andFirebird Developer Magazinerdquo Ithas been a long time since the ini-tial idea for such a magazine wasborn and now you are readingthe first issue

Why ldquoInterBase and Firebirdrdquo

I think the ldquoeditorrsquos noterdquo is the rightplace to answer some obvious ques-tions about a newborn magazine Firstof all why does its title contain bothldquoInterBaserdquo and ldquoFirebirdrdquo Somenewbies and gurus of InterBase andFirebird insist that they are differentproducts ndash and I completely agreeThey are different but people who usethem are the same Developmentapproaches problems and techniquesare almost always the same

Of course InterBase and Firebird aredifferent but it is not the same kind ofdifference as between Oracle and MSSQL ndash they are different like brothers

Who need our magazineYou sometimes hear the opinion that

ldquoInterBase is not a serious DBMSrdquo andldquoFirebird Whatrsquos thisrdquo And it is ahardly known fact that the size of ourcommunity is close to the communitysize of such monsters as Oracle andMS SQL Despite the fact that Oracle

and Microsoft command huge PR-budgets we are very well positioned inthis competition And I am sure thatthere is huge hidden potential in ourcommunity

Now the situation is changing Wehave got at least one great book aboutFirebird we have annual conferenceevents and now we have got our ownmagazine ndash just like other big players

I hope that our magazine will be usefuland interesting for all people who workwith InterBase and Firebird ndash as DBAsas developers or even as project man-agers We are concentrating on ldquoalltimes greatest hitsrdquo problems and willtry to make this magazine not a one-offreading but a long-play interestingnovel that makes for some good reainga number of times Also in future issueswe will pay attention to communityevents like BorCon and the FirebirdConference

In this issue

Ok what do we have in this issue Thefirst and largest article ldquoInside Save-pointsrdquo by Dmitri Yemanov is devote tothe internal details of server save-points Explicit savepoints are a ratherrecent improvement in both InterBaseand Firebird and I think it will be veryinteresting for all readers to know nitty-gritty details of their functioning

Another hot article is ldquoUsing Embed-ded User Authentication in InterBase75rdquo by Dmitri Kouzmenko We have

all been waiting for this feature for along time and I suppose we wont bedisappointed

ldquoWorking with temporary tables inInterBase 75rdquo is also very interestingTo my mind adding user temporarytables is the most important recentimprovement of InterBase so everydeveloper should know its details

And I think many InterBase and Fire-bird developers came across the mostannoying error ndash error 10054 Thearticle of Vasily Ovchinnikov is devotedto practical ways to avoid 10054errors

Of course this is not all of it in this issuendash please feel invited to read on

We need your feedbackThis is the first issue of ldquoThe Interbaseand Firebird Developer MagazinerdquoWe have got plenty of ideas for the fol-lowing issues and will try to make ldquoTheInterbase and Firebird DeveloperMagazinerdquo the best database devel-oper magazine around And the mostimportant thing we would like to knowis your opinion and your thoughtsabout our work Please do not hesitateto contact us

readersibdevelopercom

Sincerely yours Alexey Kovyazin

Chief Editor

Such a nice newbornEditorrsquos note

CreditsAlexey Kovyazin Chief EditorDmitri Kouzmenko EditorLev Tashchilin DesignerVolker Rehn Subeditor

Subscribe now To receive future issues notifications send email tosubscribeibdevelopercom

rollbacks the server also uses them forexception handling Each SQL andorPSQL operator is enclosed in a save-point frame which allows to rollbackthis particular operator keeping theprevious ones unchanged This guaran-tees either successful execution of theoperator or automatic cancellation ofall changes made and a correspondingerror will be initiated

For exception handling in PSQL eachBEGINhellipEND block is also enclosed ina savepoint frame which allows you tocancel all changes made by this blockLets consider some details of how save-points work

Savepoints in actionA savepoint is a data structure which islocated in the servers dynamic storage(transaction pool) and has a uniquenumerical ID A list of activities madewithin the savepoint context is associat-ed with this savepoint Such a list iscalled an undo log Savepoints form astack within a transaction and that isthe reason why only sequential rollbackof savepoints is possible

Undo log fragments are distributedacross savepoints that store the historyA savepoint which is active when a

General informationA savepoint is an internal mechanism ofthe database which binds any changesin the database to a specific point oftime during a transaction and in case ofnecessity allows a user to cancel allchanges which were made after settingthis particular savepoint This process isalso known as rolling back to savepoint

Also server uses savepoint mechanismto implement transaction handling Thismechanism helps either to commit or tocancel all changes made during a trans-action For those purposes the serveruses the global savepoint

Such a savepoint is set automaticallywhen a transaction starts and it is thefirst savepoint in the transaction contextWhen transaction rollback is initiatedall changes made within its context arecancelled using the transaction globalsavepoint After that the transaction iscommited () within the TransactionInventory Page (TIP) This is necessaryin order to avoid housekeeping opera-tions in the future

However if the number of changes intransaction context becomes too big(approx10000 - 1000000 records)then the storing of rollback listsbecomes expensive and the serverdeletes the transaction global save-point switching to the standard TIPmechanism to mark the transaction asdead

In addition to the use of savepoints for

Cover story

3

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Inside Savepoints

record is being changed is called cur-rent Information about record changesis stored in the current savepoint undolog If a rollback to the savepoint is per-formed the undo log is unwound andrecords are reconstructed As a resultthe record becomes as it was at themoment this savepoint was set

TipIf you expect that during transaction many changes are to be made thenit makes sense to specify the isc_tpb_no_auto_undo transaction parame-ter which disables usage of the global savepoint for rollback In somecases it allows to increase servers performance during batch operations

Author Dmitri Yemanovdimitruserssourceforgenet

In case there are no exception handlers available the records may bereconstructed down to the global savepoint providing completetransaction rollback After reconstruction of all modified records thesavepoint is usually deleted from the transaction context

Releasing savepointsIn addition to the rollback to savepoint operation there is an opera-tion of regular deletion (release) of a savepoint In case a savepoint isdeleted its undo log is merged with the undo log of the previous onein the savepoint stack In this sense each savepoint is nested

It is clear that regular deletion of all savepoints located deeper thanthe global one would lead to the transfer of all changes to the trans-action global savepoint

Figure 1 Savepoints in action

Cover story

4

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Thus the combination of all changeswhich were successfully executed dur-ing transaction is stored in undo logsThat is why when the automatic undolog is enabled the server can cancel allperformed changes and in the TIP thistransaction would be marked as com-mitted and not as dead

When specifying theisc_tpb_no_auto_undo parameter attransaction start a global savepoint is

not created and if regular deletion of the current stack is performed thecombined undo log is simply deleted and transaction is marked dead andall changes are to be cleared (garbage collection)

Savepoints and exception handlingThere are several events which make server create system-defined (ieuser-uncontrollable) savepoints

1 Execution of any user SQL-query As has been said above this done inorder to guarantee atomicity of a query That is to say if any exceptionoccurs during query execution the changes made in the database willalways be canceled After the query is executed the savepoint will be

automatically deleted

2 Execution of the BEGINhellipEND blockin PSQL (stored procedure or a trigger)in case the block contains an error han-dler (WHEN-block) In that case eachBEGIN operator sets a savepoint anda corresponding END operator deletesit This enables to provide error han-dling

in the PSQL-block

3 Execution of an SQL-operator in thecontext of a ldquoBEGINhellipENDrdquo blockwhich contains an error handler(WHEN-block) That is if the block con-tains an error handler any SQL-opera-tor in this block is framed by a save-point frame

Exception handling

Letrsquos consider the errors handlingprocess on the server When an excep-tion occurs automatic rollback to thelast set savepoint is performed As aresult all operations performed by aninvalid SQL-query would be canceled

Then in case of a PSQL-block it ischecked if there is a custom WHEN-handler If it does exist control is trans-ferred to it and after exiting savepointis deleted

Then the process repeats recursivelyuntil embedded handlers end Thus if astored procedure doesnt contain ahandler which would be able to handlethis error the whole procedure will beconsidered as a single SQL-operatorand canceled

Letrsquos make a summary

Firstly if there is no WHEN-handlerany PSQL-block (including stored pro-cedure and trigger) becomes atomic

and would be canceled entirely if anerror occurs

Secondly in the presence of aWHEN-handler the occurrence of anerror leads a rollback of the only oper-ator and after that the process is man-aged by the handler

That is to say there are obvious dif-ferences in the serverrsquos reactions whichdepend on the presence of a WHEN-handler

It is worth considering a knownanomaly which does not fit the schemedescribed above The scheme works asfollows the paragraph 3 (enclosing ofeach SQL-operator inside a PSQL-block by its own savepoint frame) istrue for SQL-operators only (moreovernot for all of them) In other words forexample an assignment operator willnot be framed by a savepoint frame

As a result of an error in assign-ment execution leads to a normal roll-back to the previous savepoint whichishellip exactly ndash the blockrsquos savepoint

That is to say even if there is aWHEN-handler the error may causethe rollback of the whole block beforecontrol to the error handler

I consider this situation as a seriousflaw in the serverrsquos exceptions handlinglogic Below is the full list of operatorsfor which the savepoint frame is creat-ed INSERT UPDATE DELETE EXCEP-TION EXECUTE STATEMENT [INTO]

To understand the reasons of thisanomaly it is necessary to take intoaccount the following 2 facts

bullIf an error occurs rollback to thelast savepoint is performed uncondi-tionally

Figure 2 Releasing a savepoint undo data moved to upward savepoint

TipAs is illustrated below each custom SQL-query is a set of savepoints within a transaction

TipIt is necessary to note that the isc_tpb_no_auto_undo parameter does not disable the savepoints mecha-nism (this is impossible due to the atomicity guarantee for SQL-operators ) It only disables the creationof a transaction undo log as a single whole

Cover story

5

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

bullA blockrsquos savepoint deletion (or roll-back to it) is performed together with theverification of the savepointrsquos identifier

This means that when an error occurs inoperators not included to the above list(ie not enclosed in a savepoint frame)that actually the wrong (blocks) save-point is deleted But the block itself is tol-erant to that fact since it only deletes itsown savepoint (in case it has one) Thisis the reason why the described servererror does not cause fatal conse-quences

Let us illustrate such a situation using thedescription of the [428903] ExceptionHandling Bug error To clarify this weprovide these examples with commentsabout show the server deals with save-points

Example 1A procedure with an error handler anderror generation in the assignmentoperator

In this case since savepoint 1 is thenearest to the erratic operator (save-

InterBase Myths 1When performing a restore outdated versions aredeleted (and therefore are stored in backup) orgbak -g creates a backup file without versions andby default versions are included in the fileNothing of the kind No records versions are stored in a back-up because it is unnecessary Generally the backup process is an ordi-nary snapshot transaction (repeatable read) and it reads only thoserecords versions which were relevant at the beginning of the transac-tion The ldquono_garbage_collectrdquo flag controls collecting garbage versionsin the database itself This flag can also be used during ordinary con-nections accessing the AP (ie in applications when say one needs toaccelerate sampling in some cases)

point 2 was deleted right before assignment execution) INSERT will becanceled Therefore a rollback of the whole BEGINEND block will beperformed before entering the handler

Below is the same procedure with an explicit exception statement enclosedin a block without a WHEN-handler

As you see no savepoint frame near assignment operator wascreated Therefore the result would be similar to the previous one

Below is the same procedure with an explicit exception call in theblock which contains WHEN-handler

CREATE PROCEDURE PROC1AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES

(01)-- end savepoint 2X = 1 0

WHEN ANY DOEXIT

-- end savepoint 1END

CREATE PROCEDURE PROC2AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (23)-- end savepoint 2BEGIN

X = 1 0END

WHEN ANY DOEXIT

-- end savepoint 1END

CREATE PROCEDURE PROC3AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (45)-- end savepoint 2-- start savepoint 3BEGIN

X = 1 0WHEN ANY DO

EXIT-- end savepoint 3

ENDWHEN ANY DO

EXIT-- end savepoint 1END Here we see a created savepoint frame As a result rollback is performed only for thenearest BEGIN level (savepoint 3) and the INSERT operator remained executed

Example 2A procedure with error handler and explicit exception call

CREATE PROCEDURE PROC4AS-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (67)-- end savepoint 2-- start savepoint 3EXCEPTION E-- end savepoint 3

WHEN ANY DOEXIT

-- end savepoint 1END

Cover story

6

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

In this case INSERT will not be canceleddue to the fact that the exception initiationof E results in rollback to savepoint 3 andsubsequent transfer of control to the errorhandler In order to cancel the INSERToperator in this case you should inhibit theexecution of the deletion handler

In addition a few words aboutexception handling adequacyregarding the SQL-standard

The standard allows three types ofhandlers in PSQL CONTINUEEXIT and UNDO With a CON-TINUE-handler the server mustrollback the erroneous operatorexecute the handler code and thencontinue the execution of the blockbeginning with the operator next tothe which one caused the error

An EXIT-handler requires finishingof the execution of the block rightafter exiting the handler code

An UNDO-handler requires a roll-back of all actions of the blockbefore entering the handler

Current versions of the server(InterBase as well as Firebird) donot support the explicit specifica-tion of the handler type and workaccording to the EXIT principle(however there is a possibility ofUNDO-behavior due to the anom-aly described above)

I suppose that in the future it

would be desirable to provide analternative to choose betweenUNDO- and EXIT- behavior of ahandler and repair the describedanomaly

Custom savepointsIn addition to the internalrealization of savepoints attransaction (and opera-torblock) levels the latestversions of servers (Inter-Base 71 Firebird 15 andYaffil 11) provide an SQL-interface developed for

this mechanism

Note savepointsrsquo syntax andsemantics are declared in the SQL-99 standard (see section 4371 ofthe specification)

Custom savepoints (also known asnested transactions) provide a con-venient business logic error han-dling method with no need to roll-back the whole transaction

Note rollback to a savepoint isalso sometimes called ldquopartialtransaction rollbackrdquo

New SQL operator (SAVEPOINT)was added to define a savepoint inthe transaction context to which arollback can be performed lateron

CREATE PROCEDURE PROC5ASBEGIN

INSERT INTO TAB (COL) VALUES (89)EXCEPTION E

END

a transaction If you attempt to createtwo savepoints with similar names thefirst savepoint is deleted and the speci-fied name is given to the second one

For rollback to a savepoint the follow-ing operator is used

SAVEPOINT ltnamegtltnamegt - the string identifier of a save-point As soon as a savepoint is creat-ed you can either continue transactioncommit (or cancel) the whole transac-tion or perform a rollback to a particu-lar savepoint Savepointsrsquo names (iden-tifiers) must be unique in the context of

Note the SAVEPOINT keyword isobligatory in InterBase 71

During execution of this operator thefollowing actions are performed

bullRollback of all changes made afterthe savepoint was set

bullAll savepoints set after this one aredeleted The current savepoint remainsunchanged and thus you can performseveral rollbacks to a savepoint Previ-ous savepoints remain unchanged aswell

Note Performing a rollback to save-point in InterBase 71 deletes the select-ed savepoint

bullAll explicit and implicit write locksoccupied after the savepoint was setare released At that other transac-tions which requested an access to therecords blocked by the transaction afterthe savepoint was set continue waitingfor the current transaction to be fin-ished Transactions which did notrequest access to the records may con-tinue and get access to them

Note This behavior refers to Firebird15 and can be changed in higher ver-sions

Since each savepoint uses certain sys-tem resources and also clogs thenamespace it makes sense to release

(delete) savepoints when they are no longer necessary This can be accom-plished using the following operator

RELEASE SAVEPOINT ltnamegt [ONLY]This command deletes the selected (and all following) savepoints fromtransaction context The ldquoONLYrdquo option is a switch to delete the selectedsavepoint only at that all following savepoints will be saved If a savepoint

was not released explicitly it will be automatically deleted assoon as the transaction is finished

Note The ldquoONLYrdquo option is non-standard extension and isnot supported by InterBase 71

ROLLBACK [WORK] TO [SAVEPOINT] ltnamegt

FastReport 3 - new generation of the reporting tools

Visual report designer with rulers guides andzooming wizards for base type reports exportfilters to html tiff bmp jpg xls pdf Dot matrix

reports support support most popular DB-engines

Full WYSIWYG text rotation 0360 degreesmemo object supports simple html-tags

(font color b i u sub sup) improved stretching(StretchMode ShiftMode properties)

Access to DB fields styles text flow URLs Anchors

httpwwwfast-reportcomenreadersphp5 discount for all our readers

Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers

At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail

First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint

It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set

through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71

A question emerges how would customsavepoints coexist with internal save-points managed by the server

Cover story

7

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title

A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation

Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative

In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO

LOOPTRY

SAVEPOINT DOClthellipgt single document handling commands

EXCEPTROLLBACK TO SAVEPOINT DOC

lthellipgt either log the error or display it

END END

CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name

Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)

START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY

lthellipgt operations on databaseEXCEPT

ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID

ENDCOMMIT

Cover story

8

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation

CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )

ASBEGIN

BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)

VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO

BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID

ENDEND

END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here

BEGININSERT INTO hellip

WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id

END

This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution

Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN

SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip

INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

WHEN ANY DOBEGIN

ROLLBACK TO SAVEPOINTS1

ERROR = REC_IDSUSPEND

ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well

FOR SELECT ID hellip INTOREC_ID hellipBEGIN

BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

ENDWHEN ANY DO

BEGINERROR = REC_IDSUSPEND

ENDEND

In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler

Therefore in virtually any case one

can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks

A bit of practice

Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear

Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as

SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1

work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above

1 savepoints constitute a stack andcan be canceled sequentially only

2 each block of PSQL-code with anexception handler is enclosed in a frame

Thus we arrive at a conclusion that anycode area of the following type

SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1

hellipWHEN

is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base

Thank God the InterBase developersdid not create such cardinal realization

and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in

this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler

However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample

Statement failed SQLCODE = -504Savepoint ltnamegt unknown

Cover story

9

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip

END

This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode

INSERT INTO TABLE1 hellipBEGIN

ROLLBACK TO SAVEPOINT S1WHEN ANY DO

EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon

SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is

ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)

neither available in Firebird nor in Yaffil

Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed

Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions

For this purpose three new API functions are introduced

As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to

Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)

Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)

Create savepoint isc_start_savepoint(status amptrans A)

Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)

Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)

all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)

stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316

Thus we can make up several conclu-sions

bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)

bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead

bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0

bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place

bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results

Tips

10

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Commit coordinated transaction isc_commit_transaction (status amptrans)

Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)

As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed

And that would be that way but forthe manually executed RELEASESAVEPOINT A

At first savepoint rollback was per-formed for the first connection and allchanges were canceled

Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible

The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints

That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process

Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole

Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API

Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means

The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability

It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature

InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used

ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain

[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]

As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source

The following experiment can be performed Create a table with structure as shownbelow

(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)

Type conversion throughCOMPUTED BY

CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))

Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be

AuthorIBSurgeon Research Labs researchib-aidcom

Database maintenance

11

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Embedded User Authenticationin InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version

Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database

In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base

1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server

2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one

3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)

As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted

This scheme is insufficient when used in

bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib

bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase

bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work

Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to

manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)

This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages

bullproduct DATABASE ACCESS is notlicensed

for databases in which EUA is enabledor disabled

bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)

for databases in which EUA was neverenabled

In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)

continued on page 12

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 2: The InterBase and Firebird Developer Magazine, Issue 1, 2005

rollbacks the server also uses them forexception handling Each SQL andorPSQL operator is enclosed in a save-point frame which allows to rollbackthis particular operator keeping theprevious ones unchanged This guaran-tees either successful execution of theoperator or automatic cancellation ofall changes made and a correspondingerror will be initiated

For exception handling in PSQL eachBEGINhellipEND block is also enclosed ina savepoint frame which allows you tocancel all changes made by this blockLets consider some details of how save-points work

Savepoints in actionA savepoint is a data structure which islocated in the servers dynamic storage(transaction pool) and has a uniquenumerical ID A list of activities madewithin the savepoint context is associat-ed with this savepoint Such a list iscalled an undo log Savepoints form astack within a transaction and that isthe reason why only sequential rollbackof savepoints is possible

Undo log fragments are distributedacross savepoints that store the historyA savepoint which is active when a

General informationA savepoint is an internal mechanism ofthe database which binds any changesin the database to a specific point oftime during a transaction and in case ofnecessity allows a user to cancel allchanges which were made after settingthis particular savepoint This process isalso known as rolling back to savepoint

Also server uses savepoint mechanismto implement transaction handling Thismechanism helps either to commit or tocancel all changes made during a trans-action For those purposes the serveruses the global savepoint

Such a savepoint is set automaticallywhen a transaction starts and it is thefirst savepoint in the transaction contextWhen transaction rollback is initiatedall changes made within its context arecancelled using the transaction globalsavepoint After that the transaction iscommited () within the TransactionInventory Page (TIP) This is necessaryin order to avoid housekeeping opera-tions in the future

However if the number of changes intransaction context becomes too big(approx10000 - 1000000 records)then the storing of rollback listsbecomes expensive and the serverdeletes the transaction global save-point switching to the standard TIPmechanism to mark the transaction asdead

In addition to the use of savepoints for

Cover story

3

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Inside Savepoints

record is being changed is called cur-rent Information about record changesis stored in the current savepoint undolog If a rollback to the savepoint is per-formed the undo log is unwound andrecords are reconstructed As a resultthe record becomes as it was at themoment this savepoint was set

TipIf you expect that during transaction many changes are to be made thenit makes sense to specify the isc_tpb_no_auto_undo transaction parame-ter which disables usage of the global savepoint for rollback In somecases it allows to increase servers performance during batch operations

Author Dmitri Yemanovdimitruserssourceforgenet

In case there are no exception handlers available the records may bereconstructed down to the global savepoint providing completetransaction rollback After reconstruction of all modified records thesavepoint is usually deleted from the transaction context

Releasing savepointsIn addition to the rollback to savepoint operation there is an opera-tion of regular deletion (release) of a savepoint In case a savepoint isdeleted its undo log is merged with the undo log of the previous onein the savepoint stack In this sense each savepoint is nested

It is clear that regular deletion of all savepoints located deeper thanthe global one would lead to the transfer of all changes to the trans-action global savepoint

Figure 1 Savepoints in action

Cover story

4

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Thus the combination of all changeswhich were successfully executed dur-ing transaction is stored in undo logsThat is why when the automatic undolog is enabled the server can cancel allperformed changes and in the TIP thistransaction would be marked as com-mitted and not as dead

When specifying theisc_tpb_no_auto_undo parameter attransaction start a global savepoint is

not created and if regular deletion of the current stack is performed thecombined undo log is simply deleted and transaction is marked dead andall changes are to be cleared (garbage collection)

Savepoints and exception handlingThere are several events which make server create system-defined (ieuser-uncontrollable) savepoints

1 Execution of any user SQL-query As has been said above this done inorder to guarantee atomicity of a query That is to say if any exceptionoccurs during query execution the changes made in the database willalways be canceled After the query is executed the savepoint will be

automatically deleted

2 Execution of the BEGINhellipEND blockin PSQL (stored procedure or a trigger)in case the block contains an error han-dler (WHEN-block) In that case eachBEGIN operator sets a savepoint anda corresponding END operator deletesit This enables to provide error han-dling

in the PSQL-block

3 Execution of an SQL-operator in thecontext of a ldquoBEGINhellipENDrdquo blockwhich contains an error handler(WHEN-block) That is if the block con-tains an error handler any SQL-opera-tor in this block is framed by a save-point frame

Exception handling

Letrsquos consider the errors handlingprocess on the server When an excep-tion occurs automatic rollback to thelast set savepoint is performed As aresult all operations performed by aninvalid SQL-query would be canceled

Then in case of a PSQL-block it ischecked if there is a custom WHEN-handler If it does exist control is trans-ferred to it and after exiting savepointis deleted

Then the process repeats recursivelyuntil embedded handlers end Thus if astored procedure doesnt contain ahandler which would be able to handlethis error the whole procedure will beconsidered as a single SQL-operatorand canceled

Letrsquos make a summary

Firstly if there is no WHEN-handlerany PSQL-block (including stored pro-cedure and trigger) becomes atomic

and would be canceled entirely if anerror occurs

Secondly in the presence of aWHEN-handler the occurrence of anerror leads a rollback of the only oper-ator and after that the process is man-aged by the handler

That is to say there are obvious dif-ferences in the serverrsquos reactions whichdepend on the presence of a WHEN-handler

It is worth considering a knownanomaly which does not fit the schemedescribed above The scheme works asfollows the paragraph 3 (enclosing ofeach SQL-operator inside a PSQL-block by its own savepoint frame) istrue for SQL-operators only (moreovernot for all of them) In other words forexample an assignment operator willnot be framed by a savepoint frame

As a result of an error in assign-ment execution leads to a normal roll-back to the previous savepoint whichishellip exactly ndash the blockrsquos savepoint

That is to say even if there is aWHEN-handler the error may causethe rollback of the whole block beforecontrol to the error handler

I consider this situation as a seriousflaw in the serverrsquos exceptions handlinglogic Below is the full list of operatorsfor which the savepoint frame is creat-ed INSERT UPDATE DELETE EXCEP-TION EXECUTE STATEMENT [INTO]

To understand the reasons of thisanomaly it is necessary to take intoaccount the following 2 facts

bullIf an error occurs rollback to thelast savepoint is performed uncondi-tionally

Figure 2 Releasing a savepoint undo data moved to upward savepoint

TipAs is illustrated below each custom SQL-query is a set of savepoints within a transaction

TipIt is necessary to note that the isc_tpb_no_auto_undo parameter does not disable the savepoints mecha-nism (this is impossible due to the atomicity guarantee for SQL-operators ) It only disables the creationof a transaction undo log as a single whole

Cover story

5

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

bullA blockrsquos savepoint deletion (or roll-back to it) is performed together with theverification of the savepointrsquos identifier

This means that when an error occurs inoperators not included to the above list(ie not enclosed in a savepoint frame)that actually the wrong (blocks) save-point is deleted But the block itself is tol-erant to that fact since it only deletes itsown savepoint (in case it has one) Thisis the reason why the described servererror does not cause fatal conse-quences

Let us illustrate such a situation using thedescription of the [428903] ExceptionHandling Bug error To clarify this weprovide these examples with commentsabout show the server deals with save-points

Example 1A procedure with an error handler anderror generation in the assignmentoperator

In this case since savepoint 1 is thenearest to the erratic operator (save-

InterBase Myths 1When performing a restore outdated versions aredeleted (and therefore are stored in backup) orgbak -g creates a backup file without versions andby default versions are included in the fileNothing of the kind No records versions are stored in a back-up because it is unnecessary Generally the backup process is an ordi-nary snapshot transaction (repeatable read) and it reads only thoserecords versions which were relevant at the beginning of the transac-tion The ldquono_garbage_collectrdquo flag controls collecting garbage versionsin the database itself This flag can also be used during ordinary con-nections accessing the AP (ie in applications when say one needs toaccelerate sampling in some cases)

point 2 was deleted right before assignment execution) INSERT will becanceled Therefore a rollback of the whole BEGINEND block will beperformed before entering the handler

Below is the same procedure with an explicit exception statement enclosedin a block without a WHEN-handler

As you see no savepoint frame near assignment operator wascreated Therefore the result would be similar to the previous one

Below is the same procedure with an explicit exception call in theblock which contains WHEN-handler

CREATE PROCEDURE PROC1AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES

(01)-- end savepoint 2X = 1 0

WHEN ANY DOEXIT

-- end savepoint 1END

CREATE PROCEDURE PROC2AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (23)-- end savepoint 2BEGIN

X = 1 0END

WHEN ANY DOEXIT

-- end savepoint 1END

CREATE PROCEDURE PROC3AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (45)-- end savepoint 2-- start savepoint 3BEGIN

X = 1 0WHEN ANY DO

EXIT-- end savepoint 3

ENDWHEN ANY DO

EXIT-- end savepoint 1END Here we see a created savepoint frame As a result rollback is performed only for thenearest BEGIN level (savepoint 3) and the INSERT operator remained executed

Example 2A procedure with error handler and explicit exception call

CREATE PROCEDURE PROC4AS-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (67)-- end savepoint 2-- start savepoint 3EXCEPTION E-- end savepoint 3

WHEN ANY DOEXIT

-- end savepoint 1END

Cover story

6

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

In this case INSERT will not be canceleddue to the fact that the exception initiationof E results in rollback to savepoint 3 andsubsequent transfer of control to the errorhandler In order to cancel the INSERToperator in this case you should inhibit theexecution of the deletion handler

In addition a few words aboutexception handling adequacyregarding the SQL-standard

The standard allows three types ofhandlers in PSQL CONTINUEEXIT and UNDO With a CON-TINUE-handler the server mustrollback the erroneous operatorexecute the handler code and thencontinue the execution of the blockbeginning with the operator next tothe which one caused the error

An EXIT-handler requires finishingof the execution of the block rightafter exiting the handler code

An UNDO-handler requires a roll-back of all actions of the blockbefore entering the handler

Current versions of the server(InterBase as well as Firebird) donot support the explicit specifica-tion of the handler type and workaccording to the EXIT principle(however there is a possibility ofUNDO-behavior due to the anom-aly described above)

I suppose that in the future it

would be desirable to provide analternative to choose betweenUNDO- and EXIT- behavior of ahandler and repair the describedanomaly

Custom savepointsIn addition to the internalrealization of savepoints attransaction (and opera-torblock) levels the latestversions of servers (Inter-Base 71 Firebird 15 andYaffil 11) provide an SQL-interface developed for

this mechanism

Note savepointsrsquo syntax andsemantics are declared in the SQL-99 standard (see section 4371 ofthe specification)

Custom savepoints (also known asnested transactions) provide a con-venient business logic error han-dling method with no need to roll-back the whole transaction

Note rollback to a savepoint isalso sometimes called ldquopartialtransaction rollbackrdquo

New SQL operator (SAVEPOINT)was added to define a savepoint inthe transaction context to which arollback can be performed lateron

CREATE PROCEDURE PROC5ASBEGIN

INSERT INTO TAB (COL) VALUES (89)EXCEPTION E

END

a transaction If you attempt to createtwo savepoints with similar names thefirst savepoint is deleted and the speci-fied name is given to the second one

For rollback to a savepoint the follow-ing operator is used

SAVEPOINT ltnamegtltnamegt - the string identifier of a save-point As soon as a savepoint is creat-ed you can either continue transactioncommit (or cancel) the whole transac-tion or perform a rollback to a particu-lar savepoint Savepointsrsquo names (iden-tifiers) must be unique in the context of

Note the SAVEPOINT keyword isobligatory in InterBase 71

During execution of this operator thefollowing actions are performed

bullRollback of all changes made afterthe savepoint was set

bullAll savepoints set after this one aredeleted The current savepoint remainsunchanged and thus you can performseveral rollbacks to a savepoint Previ-ous savepoints remain unchanged aswell

Note Performing a rollback to save-point in InterBase 71 deletes the select-ed savepoint

bullAll explicit and implicit write locksoccupied after the savepoint was setare released At that other transac-tions which requested an access to therecords blocked by the transaction afterthe savepoint was set continue waitingfor the current transaction to be fin-ished Transactions which did notrequest access to the records may con-tinue and get access to them

Note This behavior refers to Firebird15 and can be changed in higher ver-sions

Since each savepoint uses certain sys-tem resources and also clogs thenamespace it makes sense to release

(delete) savepoints when they are no longer necessary This can be accom-plished using the following operator

RELEASE SAVEPOINT ltnamegt [ONLY]This command deletes the selected (and all following) savepoints fromtransaction context The ldquoONLYrdquo option is a switch to delete the selectedsavepoint only at that all following savepoints will be saved If a savepoint

was not released explicitly it will be automatically deleted assoon as the transaction is finished

Note The ldquoONLYrdquo option is non-standard extension and isnot supported by InterBase 71

ROLLBACK [WORK] TO [SAVEPOINT] ltnamegt

FastReport 3 - new generation of the reporting tools

Visual report designer with rulers guides andzooming wizards for base type reports exportfilters to html tiff bmp jpg xls pdf Dot matrix

reports support support most popular DB-engines

Full WYSIWYG text rotation 0360 degreesmemo object supports simple html-tags

(font color b i u sub sup) improved stretching(StretchMode ShiftMode properties)

Access to DB fields styles text flow URLs Anchors

httpwwwfast-reportcomenreadersphp5 discount for all our readers

Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers

At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail

First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint

It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set

through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71

A question emerges how would customsavepoints coexist with internal save-points managed by the server

Cover story

7

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title

A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation

Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative

In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO

LOOPTRY

SAVEPOINT DOClthellipgt single document handling commands

EXCEPTROLLBACK TO SAVEPOINT DOC

lthellipgt either log the error or display it

END END

CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name

Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)

START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY

lthellipgt operations on databaseEXCEPT

ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID

ENDCOMMIT

Cover story

8

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation

CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )

ASBEGIN

BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)

VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO

BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID

ENDEND

END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here

BEGININSERT INTO hellip

WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id

END

This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution

Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN

SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip

INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

WHEN ANY DOBEGIN

ROLLBACK TO SAVEPOINTS1

ERROR = REC_IDSUSPEND

ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well

FOR SELECT ID hellip INTOREC_ID hellipBEGIN

BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

ENDWHEN ANY DO

BEGINERROR = REC_IDSUSPEND

ENDEND

In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler

Therefore in virtually any case one

can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks

A bit of practice

Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear

Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as

SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1

work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above

1 savepoints constitute a stack andcan be canceled sequentially only

2 each block of PSQL-code with anexception handler is enclosed in a frame

Thus we arrive at a conclusion that anycode area of the following type

SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1

hellipWHEN

is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base

Thank God the InterBase developersdid not create such cardinal realization

and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in

this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler

However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample

Statement failed SQLCODE = -504Savepoint ltnamegt unknown

Cover story

9

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip

END

This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode

INSERT INTO TABLE1 hellipBEGIN

ROLLBACK TO SAVEPOINT S1WHEN ANY DO

EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon

SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is

ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)

neither available in Firebird nor in Yaffil

Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed

Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions

For this purpose three new API functions are introduced

As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to

Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)

Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)

Create savepoint isc_start_savepoint(status amptrans A)

Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)

Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)

all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)

stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316

Thus we can make up several conclu-sions

bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)

bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead

bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0

bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place

bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results

Tips

10

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Commit coordinated transaction isc_commit_transaction (status amptrans)

Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)

As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed

And that would be that way but forthe manually executed RELEASESAVEPOINT A

At first savepoint rollback was per-formed for the first connection and allchanges were canceled

Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible

The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints

That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process

Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole

Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API

Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means

The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability

It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature

InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used

ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain

[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]

As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source

The following experiment can be performed Create a table with structure as shownbelow

(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)

Type conversion throughCOMPUTED BY

CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))

Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be

AuthorIBSurgeon Research Labs researchib-aidcom

Database maintenance

11

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Embedded User Authenticationin InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version

Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database

In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base

1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server

2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one

3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)

As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted

This scheme is insufficient when used in

bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib

bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase

bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work

Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to

manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)

This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages

bullproduct DATABASE ACCESS is notlicensed

for databases in which EUA is enabledor disabled

bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)

for databases in which EUA was neverenabled

In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)

continued on page 12

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 3: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Cover story

4

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Thus the combination of all changeswhich were successfully executed dur-ing transaction is stored in undo logsThat is why when the automatic undolog is enabled the server can cancel allperformed changes and in the TIP thistransaction would be marked as com-mitted and not as dead

When specifying theisc_tpb_no_auto_undo parameter attransaction start a global savepoint is

not created and if regular deletion of the current stack is performed thecombined undo log is simply deleted and transaction is marked dead andall changes are to be cleared (garbage collection)

Savepoints and exception handlingThere are several events which make server create system-defined (ieuser-uncontrollable) savepoints

1 Execution of any user SQL-query As has been said above this done inorder to guarantee atomicity of a query That is to say if any exceptionoccurs during query execution the changes made in the database willalways be canceled After the query is executed the savepoint will be

automatically deleted

2 Execution of the BEGINhellipEND blockin PSQL (stored procedure or a trigger)in case the block contains an error han-dler (WHEN-block) In that case eachBEGIN operator sets a savepoint anda corresponding END operator deletesit This enables to provide error han-dling

in the PSQL-block

3 Execution of an SQL-operator in thecontext of a ldquoBEGINhellipENDrdquo blockwhich contains an error handler(WHEN-block) That is if the block con-tains an error handler any SQL-opera-tor in this block is framed by a save-point frame

Exception handling

Letrsquos consider the errors handlingprocess on the server When an excep-tion occurs automatic rollback to thelast set savepoint is performed As aresult all operations performed by aninvalid SQL-query would be canceled

Then in case of a PSQL-block it ischecked if there is a custom WHEN-handler If it does exist control is trans-ferred to it and after exiting savepointis deleted

Then the process repeats recursivelyuntil embedded handlers end Thus if astored procedure doesnt contain ahandler which would be able to handlethis error the whole procedure will beconsidered as a single SQL-operatorand canceled

Letrsquos make a summary

Firstly if there is no WHEN-handlerany PSQL-block (including stored pro-cedure and trigger) becomes atomic

and would be canceled entirely if anerror occurs

Secondly in the presence of aWHEN-handler the occurrence of anerror leads a rollback of the only oper-ator and after that the process is man-aged by the handler

That is to say there are obvious dif-ferences in the serverrsquos reactions whichdepend on the presence of a WHEN-handler

It is worth considering a knownanomaly which does not fit the schemedescribed above The scheme works asfollows the paragraph 3 (enclosing ofeach SQL-operator inside a PSQL-block by its own savepoint frame) istrue for SQL-operators only (moreovernot for all of them) In other words forexample an assignment operator willnot be framed by a savepoint frame

As a result of an error in assign-ment execution leads to a normal roll-back to the previous savepoint whichishellip exactly ndash the blockrsquos savepoint

That is to say even if there is aWHEN-handler the error may causethe rollback of the whole block beforecontrol to the error handler

I consider this situation as a seriousflaw in the serverrsquos exceptions handlinglogic Below is the full list of operatorsfor which the savepoint frame is creat-ed INSERT UPDATE DELETE EXCEP-TION EXECUTE STATEMENT [INTO]

To understand the reasons of thisanomaly it is necessary to take intoaccount the following 2 facts

bullIf an error occurs rollback to thelast savepoint is performed uncondi-tionally

Figure 2 Releasing a savepoint undo data moved to upward savepoint

TipAs is illustrated below each custom SQL-query is a set of savepoints within a transaction

TipIt is necessary to note that the isc_tpb_no_auto_undo parameter does not disable the savepoints mecha-nism (this is impossible due to the atomicity guarantee for SQL-operators ) It only disables the creationof a transaction undo log as a single whole

Cover story

5

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

bullA blockrsquos savepoint deletion (or roll-back to it) is performed together with theverification of the savepointrsquos identifier

This means that when an error occurs inoperators not included to the above list(ie not enclosed in a savepoint frame)that actually the wrong (blocks) save-point is deleted But the block itself is tol-erant to that fact since it only deletes itsown savepoint (in case it has one) Thisis the reason why the described servererror does not cause fatal conse-quences

Let us illustrate such a situation using thedescription of the [428903] ExceptionHandling Bug error To clarify this weprovide these examples with commentsabout show the server deals with save-points

Example 1A procedure with an error handler anderror generation in the assignmentoperator

In this case since savepoint 1 is thenearest to the erratic operator (save-

InterBase Myths 1When performing a restore outdated versions aredeleted (and therefore are stored in backup) orgbak -g creates a backup file without versions andby default versions are included in the fileNothing of the kind No records versions are stored in a back-up because it is unnecessary Generally the backup process is an ordi-nary snapshot transaction (repeatable read) and it reads only thoserecords versions which were relevant at the beginning of the transac-tion The ldquono_garbage_collectrdquo flag controls collecting garbage versionsin the database itself This flag can also be used during ordinary con-nections accessing the AP (ie in applications when say one needs toaccelerate sampling in some cases)

point 2 was deleted right before assignment execution) INSERT will becanceled Therefore a rollback of the whole BEGINEND block will beperformed before entering the handler

Below is the same procedure with an explicit exception statement enclosedin a block without a WHEN-handler

As you see no savepoint frame near assignment operator wascreated Therefore the result would be similar to the previous one

Below is the same procedure with an explicit exception call in theblock which contains WHEN-handler

CREATE PROCEDURE PROC1AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES

(01)-- end savepoint 2X = 1 0

WHEN ANY DOEXIT

-- end savepoint 1END

CREATE PROCEDURE PROC2AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (23)-- end savepoint 2BEGIN

X = 1 0END

WHEN ANY DOEXIT

-- end savepoint 1END

CREATE PROCEDURE PROC3AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (45)-- end savepoint 2-- start savepoint 3BEGIN

X = 1 0WHEN ANY DO

EXIT-- end savepoint 3

ENDWHEN ANY DO

EXIT-- end savepoint 1END Here we see a created savepoint frame As a result rollback is performed only for thenearest BEGIN level (savepoint 3) and the INSERT operator remained executed

Example 2A procedure with error handler and explicit exception call

CREATE PROCEDURE PROC4AS-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (67)-- end savepoint 2-- start savepoint 3EXCEPTION E-- end savepoint 3

WHEN ANY DOEXIT

-- end savepoint 1END

Cover story

6

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

In this case INSERT will not be canceleddue to the fact that the exception initiationof E results in rollback to savepoint 3 andsubsequent transfer of control to the errorhandler In order to cancel the INSERToperator in this case you should inhibit theexecution of the deletion handler

In addition a few words aboutexception handling adequacyregarding the SQL-standard

The standard allows three types ofhandlers in PSQL CONTINUEEXIT and UNDO With a CON-TINUE-handler the server mustrollback the erroneous operatorexecute the handler code and thencontinue the execution of the blockbeginning with the operator next tothe which one caused the error

An EXIT-handler requires finishingof the execution of the block rightafter exiting the handler code

An UNDO-handler requires a roll-back of all actions of the blockbefore entering the handler

Current versions of the server(InterBase as well as Firebird) donot support the explicit specifica-tion of the handler type and workaccording to the EXIT principle(however there is a possibility ofUNDO-behavior due to the anom-aly described above)

I suppose that in the future it

would be desirable to provide analternative to choose betweenUNDO- and EXIT- behavior of ahandler and repair the describedanomaly

Custom savepointsIn addition to the internalrealization of savepoints attransaction (and opera-torblock) levels the latestversions of servers (Inter-Base 71 Firebird 15 andYaffil 11) provide an SQL-interface developed for

this mechanism

Note savepointsrsquo syntax andsemantics are declared in the SQL-99 standard (see section 4371 ofthe specification)

Custom savepoints (also known asnested transactions) provide a con-venient business logic error han-dling method with no need to roll-back the whole transaction

Note rollback to a savepoint isalso sometimes called ldquopartialtransaction rollbackrdquo

New SQL operator (SAVEPOINT)was added to define a savepoint inthe transaction context to which arollback can be performed lateron

CREATE PROCEDURE PROC5ASBEGIN

INSERT INTO TAB (COL) VALUES (89)EXCEPTION E

END

a transaction If you attempt to createtwo savepoints with similar names thefirst savepoint is deleted and the speci-fied name is given to the second one

For rollback to a savepoint the follow-ing operator is used

SAVEPOINT ltnamegtltnamegt - the string identifier of a save-point As soon as a savepoint is creat-ed you can either continue transactioncommit (or cancel) the whole transac-tion or perform a rollback to a particu-lar savepoint Savepointsrsquo names (iden-tifiers) must be unique in the context of

Note the SAVEPOINT keyword isobligatory in InterBase 71

During execution of this operator thefollowing actions are performed

bullRollback of all changes made afterthe savepoint was set

bullAll savepoints set after this one aredeleted The current savepoint remainsunchanged and thus you can performseveral rollbacks to a savepoint Previ-ous savepoints remain unchanged aswell

Note Performing a rollback to save-point in InterBase 71 deletes the select-ed savepoint

bullAll explicit and implicit write locksoccupied after the savepoint was setare released At that other transac-tions which requested an access to therecords blocked by the transaction afterthe savepoint was set continue waitingfor the current transaction to be fin-ished Transactions which did notrequest access to the records may con-tinue and get access to them

Note This behavior refers to Firebird15 and can be changed in higher ver-sions

Since each savepoint uses certain sys-tem resources and also clogs thenamespace it makes sense to release

(delete) savepoints when they are no longer necessary This can be accom-plished using the following operator

RELEASE SAVEPOINT ltnamegt [ONLY]This command deletes the selected (and all following) savepoints fromtransaction context The ldquoONLYrdquo option is a switch to delete the selectedsavepoint only at that all following savepoints will be saved If a savepoint

was not released explicitly it will be automatically deleted assoon as the transaction is finished

Note The ldquoONLYrdquo option is non-standard extension and isnot supported by InterBase 71

ROLLBACK [WORK] TO [SAVEPOINT] ltnamegt

FastReport 3 - new generation of the reporting tools

Visual report designer with rulers guides andzooming wizards for base type reports exportfilters to html tiff bmp jpg xls pdf Dot matrix

reports support support most popular DB-engines

Full WYSIWYG text rotation 0360 degreesmemo object supports simple html-tags

(font color b i u sub sup) improved stretching(StretchMode ShiftMode properties)

Access to DB fields styles text flow URLs Anchors

httpwwwfast-reportcomenreadersphp5 discount for all our readers

Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers

At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail

First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint

It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set

through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71

A question emerges how would customsavepoints coexist with internal save-points managed by the server

Cover story

7

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title

A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation

Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative

In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO

LOOPTRY

SAVEPOINT DOClthellipgt single document handling commands

EXCEPTROLLBACK TO SAVEPOINT DOC

lthellipgt either log the error or display it

END END

CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name

Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)

START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY

lthellipgt operations on databaseEXCEPT

ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID

ENDCOMMIT

Cover story

8

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation

CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )

ASBEGIN

BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)

VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO

BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID

ENDEND

END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here

BEGININSERT INTO hellip

WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id

END

This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution

Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN

SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip

INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

WHEN ANY DOBEGIN

ROLLBACK TO SAVEPOINTS1

ERROR = REC_IDSUSPEND

ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well

FOR SELECT ID hellip INTOREC_ID hellipBEGIN

BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

ENDWHEN ANY DO

BEGINERROR = REC_IDSUSPEND

ENDEND

In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler

Therefore in virtually any case one

can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks

A bit of practice

Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear

Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as

SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1

work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above

1 savepoints constitute a stack andcan be canceled sequentially only

2 each block of PSQL-code with anexception handler is enclosed in a frame

Thus we arrive at a conclusion that anycode area of the following type

SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1

hellipWHEN

is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base

Thank God the InterBase developersdid not create such cardinal realization

and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in

this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler

However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample

Statement failed SQLCODE = -504Savepoint ltnamegt unknown

Cover story

9

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip

END

This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode

INSERT INTO TABLE1 hellipBEGIN

ROLLBACK TO SAVEPOINT S1WHEN ANY DO

EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon

SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is

ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)

neither available in Firebird nor in Yaffil

Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed

Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions

For this purpose three new API functions are introduced

As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to

Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)

Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)

Create savepoint isc_start_savepoint(status amptrans A)

Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)

Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)

all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)

stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316

Thus we can make up several conclu-sions

bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)

bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead

bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0

bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place

bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results

Tips

10

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Commit coordinated transaction isc_commit_transaction (status amptrans)

Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)

As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed

And that would be that way but forthe manually executed RELEASESAVEPOINT A

At first savepoint rollback was per-formed for the first connection and allchanges were canceled

Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible

The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints

That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process

Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole

Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API

Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means

The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability

It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature

InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used

ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain

[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]

As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source

The following experiment can be performed Create a table with structure as shownbelow

(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)

Type conversion throughCOMPUTED BY

CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))

Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be

AuthorIBSurgeon Research Labs researchib-aidcom

Database maintenance

11

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Embedded User Authenticationin InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version

Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database

In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base

1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server

2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one

3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)

As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted

This scheme is insufficient when used in

bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib

bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase

bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work

Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to

manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)

This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages

bullproduct DATABASE ACCESS is notlicensed

for databases in which EUA is enabledor disabled

bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)

for databases in which EUA was neverenabled

In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)

continued on page 12

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 4: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Cover story

5

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

bullA blockrsquos savepoint deletion (or roll-back to it) is performed together with theverification of the savepointrsquos identifier

This means that when an error occurs inoperators not included to the above list(ie not enclosed in a savepoint frame)that actually the wrong (blocks) save-point is deleted But the block itself is tol-erant to that fact since it only deletes itsown savepoint (in case it has one) Thisis the reason why the described servererror does not cause fatal conse-quences

Let us illustrate such a situation using thedescription of the [428903] ExceptionHandling Bug error To clarify this weprovide these examples with commentsabout show the server deals with save-points

Example 1A procedure with an error handler anderror generation in the assignmentoperator

In this case since savepoint 1 is thenearest to the erratic operator (save-

InterBase Myths 1When performing a restore outdated versions aredeleted (and therefore are stored in backup) orgbak -g creates a backup file without versions andby default versions are included in the fileNothing of the kind No records versions are stored in a back-up because it is unnecessary Generally the backup process is an ordi-nary snapshot transaction (repeatable read) and it reads only thoserecords versions which were relevant at the beginning of the transac-tion The ldquono_garbage_collectrdquo flag controls collecting garbage versionsin the database itself This flag can also be used during ordinary con-nections accessing the AP (ie in applications when say one needs toaccelerate sampling in some cases)

point 2 was deleted right before assignment execution) INSERT will becanceled Therefore a rollback of the whole BEGINEND block will beperformed before entering the handler

Below is the same procedure with an explicit exception statement enclosedin a block without a WHEN-handler

As you see no savepoint frame near assignment operator wascreated Therefore the result would be similar to the previous one

Below is the same procedure with an explicit exception call in theblock which contains WHEN-handler

CREATE PROCEDURE PROC1AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES

(01)-- end savepoint 2X = 1 0

WHEN ANY DOEXIT

-- end savepoint 1END

CREATE PROCEDURE PROC2AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (23)-- end savepoint 2BEGIN

X = 1 0END

WHEN ANY DOEXIT

-- end savepoint 1END

CREATE PROCEDURE PROC3AS

DECLARE VARIABLE X INT-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (45)-- end savepoint 2-- start savepoint 3BEGIN

X = 1 0WHEN ANY DO

EXIT-- end savepoint 3

ENDWHEN ANY DO

EXIT-- end savepoint 1END Here we see a created savepoint frame As a result rollback is performed only for thenearest BEGIN level (savepoint 3) and the INSERT operator remained executed

Example 2A procedure with error handler and explicit exception call

CREATE PROCEDURE PROC4AS-- start savepoint 1BEGIN

-- start savepoint 2INSERT INTO TAB (COL) VALUES (67)-- end savepoint 2-- start savepoint 3EXCEPTION E-- end savepoint 3

WHEN ANY DOEXIT

-- end savepoint 1END

Cover story

6

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

In this case INSERT will not be canceleddue to the fact that the exception initiationof E results in rollback to savepoint 3 andsubsequent transfer of control to the errorhandler In order to cancel the INSERToperator in this case you should inhibit theexecution of the deletion handler

In addition a few words aboutexception handling adequacyregarding the SQL-standard

The standard allows three types ofhandlers in PSQL CONTINUEEXIT and UNDO With a CON-TINUE-handler the server mustrollback the erroneous operatorexecute the handler code and thencontinue the execution of the blockbeginning with the operator next tothe which one caused the error

An EXIT-handler requires finishingof the execution of the block rightafter exiting the handler code

An UNDO-handler requires a roll-back of all actions of the blockbefore entering the handler

Current versions of the server(InterBase as well as Firebird) donot support the explicit specifica-tion of the handler type and workaccording to the EXIT principle(however there is a possibility ofUNDO-behavior due to the anom-aly described above)

I suppose that in the future it

would be desirable to provide analternative to choose betweenUNDO- and EXIT- behavior of ahandler and repair the describedanomaly

Custom savepointsIn addition to the internalrealization of savepoints attransaction (and opera-torblock) levels the latestversions of servers (Inter-Base 71 Firebird 15 andYaffil 11) provide an SQL-interface developed for

this mechanism

Note savepointsrsquo syntax andsemantics are declared in the SQL-99 standard (see section 4371 ofthe specification)

Custom savepoints (also known asnested transactions) provide a con-venient business logic error han-dling method with no need to roll-back the whole transaction

Note rollback to a savepoint isalso sometimes called ldquopartialtransaction rollbackrdquo

New SQL operator (SAVEPOINT)was added to define a savepoint inthe transaction context to which arollback can be performed lateron

CREATE PROCEDURE PROC5ASBEGIN

INSERT INTO TAB (COL) VALUES (89)EXCEPTION E

END

a transaction If you attempt to createtwo savepoints with similar names thefirst savepoint is deleted and the speci-fied name is given to the second one

For rollback to a savepoint the follow-ing operator is used

SAVEPOINT ltnamegtltnamegt - the string identifier of a save-point As soon as a savepoint is creat-ed you can either continue transactioncommit (or cancel) the whole transac-tion or perform a rollback to a particu-lar savepoint Savepointsrsquo names (iden-tifiers) must be unique in the context of

Note the SAVEPOINT keyword isobligatory in InterBase 71

During execution of this operator thefollowing actions are performed

bullRollback of all changes made afterthe savepoint was set

bullAll savepoints set after this one aredeleted The current savepoint remainsunchanged and thus you can performseveral rollbacks to a savepoint Previ-ous savepoints remain unchanged aswell

Note Performing a rollback to save-point in InterBase 71 deletes the select-ed savepoint

bullAll explicit and implicit write locksoccupied after the savepoint was setare released At that other transac-tions which requested an access to therecords blocked by the transaction afterthe savepoint was set continue waitingfor the current transaction to be fin-ished Transactions which did notrequest access to the records may con-tinue and get access to them

Note This behavior refers to Firebird15 and can be changed in higher ver-sions

Since each savepoint uses certain sys-tem resources and also clogs thenamespace it makes sense to release

(delete) savepoints when they are no longer necessary This can be accom-plished using the following operator

RELEASE SAVEPOINT ltnamegt [ONLY]This command deletes the selected (and all following) savepoints fromtransaction context The ldquoONLYrdquo option is a switch to delete the selectedsavepoint only at that all following savepoints will be saved If a savepoint

was not released explicitly it will be automatically deleted assoon as the transaction is finished

Note The ldquoONLYrdquo option is non-standard extension and isnot supported by InterBase 71

ROLLBACK [WORK] TO [SAVEPOINT] ltnamegt

FastReport 3 - new generation of the reporting tools

Visual report designer with rulers guides andzooming wizards for base type reports exportfilters to html tiff bmp jpg xls pdf Dot matrix

reports support support most popular DB-engines

Full WYSIWYG text rotation 0360 degreesmemo object supports simple html-tags

(font color b i u sub sup) improved stretching(StretchMode ShiftMode properties)

Access to DB fields styles text flow URLs Anchors

httpwwwfast-reportcomenreadersphp5 discount for all our readers

Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers

At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail

First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint

It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set

through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71

A question emerges how would customsavepoints coexist with internal save-points managed by the server

Cover story

7

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title

A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation

Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative

In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO

LOOPTRY

SAVEPOINT DOClthellipgt single document handling commands

EXCEPTROLLBACK TO SAVEPOINT DOC

lthellipgt either log the error or display it

END END

CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name

Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)

START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY

lthellipgt operations on databaseEXCEPT

ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID

ENDCOMMIT

Cover story

8

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation

CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )

ASBEGIN

BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)

VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO

BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID

ENDEND

END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here

BEGININSERT INTO hellip

WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id

END

This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution

Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN

SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip

INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

WHEN ANY DOBEGIN

ROLLBACK TO SAVEPOINTS1

ERROR = REC_IDSUSPEND

ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well

FOR SELECT ID hellip INTOREC_ID hellipBEGIN

BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

ENDWHEN ANY DO

BEGINERROR = REC_IDSUSPEND

ENDEND

In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler

Therefore in virtually any case one

can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks

A bit of practice

Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear

Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as

SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1

work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above

1 savepoints constitute a stack andcan be canceled sequentially only

2 each block of PSQL-code with anexception handler is enclosed in a frame

Thus we arrive at a conclusion that anycode area of the following type

SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1

hellipWHEN

is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base

Thank God the InterBase developersdid not create such cardinal realization

and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in

this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler

However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample

Statement failed SQLCODE = -504Savepoint ltnamegt unknown

Cover story

9

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip

END

This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode

INSERT INTO TABLE1 hellipBEGIN

ROLLBACK TO SAVEPOINT S1WHEN ANY DO

EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon

SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is

ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)

neither available in Firebird nor in Yaffil

Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed

Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions

For this purpose three new API functions are introduced

As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to

Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)

Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)

Create savepoint isc_start_savepoint(status amptrans A)

Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)

Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)

all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)

stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316

Thus we can make up several conclu-sions

bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)

bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead

bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0

bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place

bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results

Tips

10

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Commit coordinated transaction isc_commit_transaction (status amptrans)

Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)

As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed

And that would be that way but forthe manually executed RELEASESAVEPOINT A

At first savepoint rollback was per-formed for the first connection and allchanges were canceled

Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible

The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints

That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process

Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole

Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API

Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means

The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability

It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature

InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used

ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain

[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]

As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source

The following experiment can be performed Create a table with structure as shownbelow

(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)

Type conversion throughCOMPUTED BY

CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))

Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be

AuthorIBSurgeon Research Labs researchib-aidcom

Database maintenance

11

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Embedded User Authenticationin InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version

Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database

In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base

1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server

2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one

3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)

As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted

This scheme is insufficient when used in

bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib

bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase

bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work

Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to

manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)

This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages

bullproduct DATABASE ACCESS is notlicensed

for databases in which EUA is enabledor disabled

bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)

for databases in which EUA was neverenabled

In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)

continued on page 12

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 5: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Cover story

6

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

In this case INSERT will not be canceleddue to the fact that the exception initiationof E results in rollback to savepoint 3 andsubsequent transfer of control to the errorhandler In order to cancel the INSERToperator in this case you should inhibit theexecution of the deletion handler

In addition a few words aboutexception handling adequacyregarding the SQL-standard

The standard allows three types ofhandlers in PSQL CONTINUEEXIT and UNDO With a CON-TINUE-handler the server mustrollback the erroneous operatorexecute the handler code and thencontinue the execution of the blockbeginning with the operator next tothe which one caused the error

An EXIT-handler requires finishingof the execution of the block rightafter exiting the handler code

An UNDO-handler requires a roll-back of all actions of the blockbefore entering the handler

Current versions of the server(InterBase as well as Firebird) donot support the explicit specifica-tion of the handler type and workaccording to the EXIT principle(however there is a possibility ofUNDO-behavior due to the anom-aly described above)

I suppose that in the future it

would be desirable to provide analternative to choose betweenUNDO- and EXIT- behavior of ahandler and repair the describedanomaly

Custom savepointsIn addition to the internalrealization of savepoints attransaction (and opera-torblock) levels the latestversions of servers (Inter-Base 71 Firebird 15 andYaffil 11) provide an SQL-interface developed for

this mechanism

Note savepointsrsquo syntax andsemantics are declared in the SQL-99 standard (see section 4371 ofthe specification)

Custom savepoints (also known asnested transactions) provide a con-venient business logic error han-dling method with no need to roll-back the whole transaction

Note rollback to a savepoint isalso sometimes called ldquopartialtransaction rollbackrdquo

New SQL operator (SAVEPOINT)was added to define a savepoint inthe transaction context to which arollback can be performed lateron

CREATE PROCEDURE PROC5ASBEGIN

INSERT INTO TAB (COL) VALUES (89)EXCEPTION E

END

a transaction If you attempt to createtwo savepoints with similar names thefirst savepoint is deleted and the speci-fied name is given to the second one

For rollback to a savepoint the follow-ing operator is used

SAVEPOINT ltnamegtltnamegt - the string identifier of a save-point As soon as a savepoint is creat-ed you can either continue transactioncommit (or cancel) the whole transac-tion or perform a rollback to a particu-lar savepoint Savepointsrsquo names (iden-tifiers) must be unique in the context of

Note the SAVEPOINT keyword isobligatory in InterBase 71

During execution of this operator thefollowing actions are performed

bullRollback of all changes made afterthe savepoint was set

bullAll savepoints set after this one aredeleted The current savepoint remainsunchanged and thus you can performseveral rollbacks to a savepoint Previ-ous savepoints remain unchanged aswell

Note Performing a rollback to save-point in InterBase 71 deletes the select-ed savepoint

bullAll explicit and implicit write locksoccupied after the savepoint was setare released At that other transac-tions which requested an access to therecords blocked by the transaction afterthe savepoint was set continue waitingfor the current transaction to be fin-ished Transactions which did notrequest access to the records may con-tinue and get access to them

Note This behavior refers to Firebird15 and can be changed in higher ver-sions

Since each savepoint uses certain sys-tem resources and also clogs thenamespace it makes sense to release

(delete) savepoints when they are no longer necessary This can be accom-plished using the following operator

RELEASE SAVEPOINT ltnamegt [ONLY]This command deletes the selected (and all following) savepoints fromtransaction context The ldquoONLYrdquo option is a switch to delete the selectedsavepoint only at that all following savepoints will be saved If a savepoint

was not released explicitly it will be automatically deleted assoon as the transaction is finished

Note The ldquoONLYrdquo option is non-standard extension and isnot supported by InterBase 71

ROLLBACK [WORK] TO [SAVEPOINT] ltnamegt

FastReport 3 - new generation of the reporting tools

Visual report designer with rulers guides andzooming wizards for base type reports exportfilters to html tiff bmp jpg xls pdf Dot matrix

reports support support most popular DB-engines

Full WYSIWYG text rotation 0360 degreesmemo object supports simple html-tags

(font color b i u sub sup) improved stretching(StretchMode ShiftMode properties)

Access to DB fields styles text flow URLs Anchors

httpwwwfast-reportcomenreadersphp5 discount for all our readers

Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers

At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail

First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint

It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set

through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71

A question emerges how would customsavepoints coexist with internal save-points managed by the server

Cover story

7

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title

A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation

Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative

In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO

LOOPTRY

SAVEPOINT DOClthellipgt single document handling commands

EXCEPTROLLBACK TO SAVEPOINT DOC

lthellipgt either log the error or display it

END END

CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name

Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)

START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY

lthellipgt operations on databaseEXCEPT

ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID

ENDCOMMIT

Cover story

8

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation

CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )

ASBEGIN

BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)

VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO

BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID

ENDEND

END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here

BEGININSERT INTO hellip

WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id

END

This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution

Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN

SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip

INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

WHEN ANY DOBEGIN

ROLLBACK TO SAVEPOINTS1

ERROR = REC_IDSUSPEND

ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well

FOR SELECT ID hellip INTOREC_ID hellipBEGIN

BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

ENDWHEN ANY DO

BEGINERROR = REC_IDSUSPEND

ENDEND

In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler

Therefore in virtually any case one

can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks

A bit of practice

Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear

Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as

SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1

work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above

1 savepoints constitute a stack andcan be canceled sequentially only

2 each block of PSQL-code with anexception handler is enclosed in a frame

Thus we arrive at a conclusion that anycode area of the following type

SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1

hellipWHEN

is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base

Thank God the InterBase developersdid not create such cardinal realization

and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in

this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler

However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample

Statement failed SQLCODE = -504Savepoint ltnamegt unknown

Cover story

9

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip

END

This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode

INSERT INTO TABLE1 hellipBEGIN

ROLLBACK TO SAVEPOINT S1WHEN ANY DO

EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon

SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is

ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)

neither available in Firebird nor in Yaffil

Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed

Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions

For this purpose three new API functions are introduced

As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to

Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)

Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)

Create savepoint isc_start_savepoint(status amptrans A)

Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)

Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)

all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)

stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316

Thus we can make up several conclu-sions

bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)

bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead

bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0

bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place

bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results

Tips

10

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Commit coordinated transaction isc_commit_transaction (status amptrans)

Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)

As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed

And that would be that way but forthe manually executed RELEASESAVEPOINT A

At first savepoint rollback was per-formed for the first connection and allchanges were canceled

Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible

The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints

That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process

Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole

Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API

Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means

The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability

It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature

InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used

ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain

[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]

As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source

The following experiment can be performed Create a table with structure as shownbelow

(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)

Type conversion throughCOMPUTED BY

CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))

Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be

AuthorIBSurgeon Research Labs researchib-aidcom

Database maintenance

11

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Embedded User Authenticationin InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version

Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database

In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base

1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server

2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one

3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)

As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted

This scheme is insufficient when used in

bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib

bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase

bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work

Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to

manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)

This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages

bullproduct DATABASE ACCESS is notlicensed

for databases in which EUA is enabledor disabled

bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)

for databases in which EUA was neverenabled

In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)

continued on page 12

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 6: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers

At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail

First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint

It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set

through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71

A question emerges how would customsavepoints coexist with internal save-points managed by the server

Cover story

7

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title

A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation

Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative

In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO

LOOPTRY

SAVEPOINT DOClthellipgt single document handling commands

EXCEPTROLLBACK TO SAVEPOINT DOC

lthellipgt either log the error or display it

END END

CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name

Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)

START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY

lthellipgt operations on databaseEXCEPT

ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID

ENDCOMMIT

Cover story

8

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation

CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )

ASBEGIN

BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)

VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO

BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID

ENDEND

END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here

BEGININSERT INTO hellip

WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id

END

This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution

Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN

SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip

INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

WHEN ANY DOBEGIN

ROLLBACK TO SAVEPOINTS1

ERROR = REC_IDSUSPEND

ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well

FOR SELECT ID hellip INTOREC_ID hellipBEGIN

BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

ENDWHEN ANY DO

BEGINERROR = REC_IDSUSPEND

ENDEND

In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler

Therefore in virtually any case one

can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks

A bit of practice

Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear

Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as

SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1

work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above

1 savepoints constitute a stack andcan be canceled sequentially only

2 each block of PSQL-code with anexception handler is enclosed in a frame

Thus we arrive at a conclusion that anycode area of the following type

SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1

hellipWHEN

is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base

Thank God the InterBase developersdid not create such cardinal realization

and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in

this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler

However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample

Statement failed SQLCODE = -504Savepoint ltnamegt unknown

Cover story

9

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip

END

This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode

INSERT INTO TABLE1 hellipBEGIN

ROLLBACK TO SAVEPOINT S1WHEN ANY DO

EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon

SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is

ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)

neither available in Firebird nor in Yaffil

Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed

Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions

For this purpose three new API functions are introduced

As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to

Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)

Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)

Create savepoint isc_start_savepoint(status amptrans A)

Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)

Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)

all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)

stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316

Thus we can make up several conclu-sions

bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)

bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead

bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0

bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place

bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results

Tips

10

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Commit coordinated transaction isc_commit_transaction (status amptrans)

Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)

As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed

And that would be that way but forthe manually executed RELEASESAVEPOINT A

At first savepoint rollback was per-formed for the first connection and allchanges were canceled

Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible

The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints

That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process

Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole

Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API

Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means

The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability

It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature

InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used

ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain

[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]

As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source

The following experiment can be performed Create a table with structure as shownbelow

(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)

Type conversion throughCOMPUTED BY

CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))

Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be

AuthorIBSurgeon Research Labs researchib-aidcom

Database maintenance

11

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Embedded User Authenticationin InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version

Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database

In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base

1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server

2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one

3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)

As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted

This scheme is insufficient when used in

bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib

bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase

bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work

Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to

manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)

This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages

bullproduct DATABASE ACCESS is notlicensed

for databases in which EUA is enabledor disabled

bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)

for databases in which EUA was neverenabled

In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)

continued on page 12

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 7: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Cover story

8

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation

CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )

ASBEGIN

BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)

VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO

BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID

ENDEND

END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here

BEGININSERT INTO hellip

WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id

END

This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution

Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN

SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip

INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

WHEN ANY DOBEGIN

ROLLBACK TO SAVEPOINTS1

ERROR = REC_IDSUSPEND

ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well

FOR SELECT ID hellip INTOREC_ID hellipBEGIN

BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip

ENDWHEN ANY DO

BEGINERROR = REC_IDSUSPEND

ENDEND

In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler

Therefore in virtually any case one

can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks

A bit of practice

Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear

Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as

SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1

work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above

1 savepoints constitute a stack andcan be canceled sequentially only

2 each block of PSQL-code with anexception handler is enclosed in a frame

Thus we arrive at a conclusion that anycode area of the following type

SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1

hellipWHEN

is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base

Thank God the InterBase developersdid not create such cardinal realization

and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in

this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler

However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample

Statement failed SQLCODE = -504Savepoint ltnamegt unknown

Cover story

9

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip

END

This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode

INSERT INTO TABLE1 hellipBEGIN

ROLLBACK TO SAVEPOINT S1WHEN ANY DO

EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon

SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is

ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)

neither available in Firebird nor in Yaffil

Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed

Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions

For this purpose three new API functions are introduced

As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to

Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)

Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)

Create savepoint isc_start_savepoint(status amptrans A)

Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)

Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)

all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)

stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316

Thus we can make up several conclu-sions

bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)

bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead

bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0

bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place

bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results

Tips

10

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Commit coordinated transaction isc_commit_transaction (status amptrans)

Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)

As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed

And that would be that way but forthe manually executed RELEASESAVEPOINT A

At first savepoint rollback was per-formed for the first connection and allchanges were canceled

Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible

The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints

That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process

Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole

Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API

Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means

The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability

It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature

InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used

ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain

[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]

As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source

The following experiment can be performed Create a table with structure as shownbelow

(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)

Type conversion throughCOMPUTED BY

CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))

Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be

AuthorIBSurgeon Research Labs researchib-aidcom

Database maintenance

11

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Embedded User Authenticationin InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version

Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database

In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base

1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server

2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one

3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)

As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted

This scheme is insufficient when used in

bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib

bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase

bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work

Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to

manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)

This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages

bullproduct DATABASE ACCESS is notlicensed

for databases in which EUA is enabledor disabled

bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)

for databases in which EUA was neverenabled

In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)

continued on page 12

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 8: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Cover story

9

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip

END

This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode

INSERT INTO TABLE1 hellipBEGIN

ROLLBACK TO SAVEPOINT S1WHEN ANY DO

EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon

SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is

ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)

ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)

neither available in Firebird nor in Yaffil

Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed

Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions

For this purpose three new API functions are introduced

As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to

Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)

Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)

Create savepoint isc_start_savepoint(status amptrans A)

Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)

Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)

all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)

stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316

Thus we can make up several conclu-sions

bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)

bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead

bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0

bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place

bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results

Tips

10

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Commit coordinated transaction isc_commit_transaction (status amptrans)

Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)

As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed

And that would be that way but forthe manually executed RELEASESAVEPOINT A

At first savepoint rollback was per-formed for the first connection and allchanges were canceled

Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible

The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints

That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process

Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole

Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API

Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means

The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability

It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature

InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used

ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain

[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]

As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source

The following experiment can be performed Create a table with structure as shownbelow

(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)

Type conversion throughCOMPUTED BY

CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))

Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be

AuthorIBSurgeon Research Labs researchib-aidcom

Database maintenance

11

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Embedded User Authenticationin InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version

Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database

In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base

1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server

2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one

3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)

As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted

This scheme is insufficient when used in

bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib

bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase

bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work

Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to

manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)

This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages

bullproduct DATABASE ACCESS is notlicensed

for databases in which EUA is enabledor disabled

bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)

for databases in which EUA was neverenabled

In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)

continued on page 12

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 9: The InterBase and Firebird Developer Magazine, Issue 1, 2005

stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316

Thus we can make up several conclu-sions

bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)

bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead

bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0

bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place

bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results

Tips

10

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Inside Savepoints

Commit coordinated transaction isc_commit_transaction (status amptrans)

Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)

As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed

And that would be that way but forthe manually executed RELEASESAVEPOINT A

At first savepoint rollback was per-formed for the first connection and allchanges were canceled

Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible

The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints

That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process

Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole

Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API

Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means

The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability

It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature

InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used

ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain

[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]

As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source

The following experiment can be performed Create a table with structure as shownbelow

(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)

Type conversion throughCOMPUTED BY

CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))

Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be

AuthorIBSurgeon Research Labs researchib-aidcom

Database maintenance

11

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Embedded User Authenticationin InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version

Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database

In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base

1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server

2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one

3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)

As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted

This scheme is insufficient when used in

bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib

bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase

bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work

Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to

manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)

This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages

bullproduct DATABASE ACCESS is notlicensed

for databases in which EUA is enabledor disabled

bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)

for databases in which EUA was neverenabled

In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)

continued on page 12

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 10: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Database maintenance

11

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Embedded User Authenticationin InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version

Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database

In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base

1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server

2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one

3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)

As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted

This scheme is insufficient when used in

bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib

bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase

bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work

Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to

manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)

This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages

bullproduct DATABASE ACCESS is notlicensed

for databases in which EUA is enabledor disabled

bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)

for databases in which EUA was neverenabled

In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)

continued on page 12

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 11: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib

Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database

bullThe server opens a specific database

1 EUA disabled ndash user authentication is accomplished from adminib

2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database

That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password

Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not

Database maintenance

12

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

Enabling EUAThere are two ways of enabling EUA in a database

1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE

2 For any ODS 112 database enter the following operator

ALTER DATABASE SET ADMIN OPTION INACTIVEand activated

ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)

You can completely delete EUA together with all user records by the command

ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)

User managementIf EUA is enabled you can manage users

CREATE | ALTER USER SET

option PASSWORD [NO] DEFAULT ROLE

[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE

Examples

CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC

As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set

CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC

continued on page 13

ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)

When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified

EUA can be temporarily deactivated by the command

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 12: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Database maintenance

13

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed

User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes

1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants

2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it

3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)

4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-

words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled

5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA

The picture illustrates an example ofhow two different users connect to dif-ferent databases

User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)

User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled

BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)

After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database

The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored

To date (16052005) in Borland it is considered as bug IB 750174

A workaround

disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib

Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled

1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib

2 Enable EUA in the database

ALTER DATABASE ADD ADMIN OPTION

In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1

3 Add a local user USR

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 13: The InterBase and Firebird Developer Magazine, Issue 1, 2005

create user USR set password usr

4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo

5 try to backup from TEST user through adminib

gbak -b dbib dbibk -v -user TEST -pass tttt

does not pass Try a user from EUA

gbak -b dbib dbibk -v -user TEST -pass test

it does pass That is only the user specified in eua can do backup (ie databaseowner)

6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not

gbak -c dbibk 1db -v user TEST -pass test

does not pass as it was expected The TEST user has a different name in adminib

gbak -c dbibk 1db -v user TEST -pass tttt

restore is successfully accomplished

However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)

Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)

ConclusionRegarding all that we can come to several conclusions

bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode

bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords

Database maintenance

14

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Embedded UA in InterBase

bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers

bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 14: The InterBase and Firebird Developer Magazine, Issue 1, 2005

closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL

By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)

Below adjustment of these parametersfor Windows and Linux operating sys-tems is described

Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl

IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture

If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high

For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released

As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records

Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors

Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections

In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article

Note As we know from previous experience stability of the dummy-pack-

The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response

If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection

Thus a failed client connection will be

et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero

KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver

For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow

In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters

KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table

Database maintenance

15

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors

Author Vasiliy Ovchinnikovovatkvcru

Parameter Description

KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start

KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes

KEEPALIVE_PROBES Number of KEEPALIVE-probes

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 15: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Substitute the lt valuegt word with necessary values

If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added

FLAGS=REUSE KEEPALIVE

Adjusting KEEPALIVE in Windows 9598ME Register branch

HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP

Everything about adjustment of TCP can be found here

httpsupportmicrosoftcomdefaultaspxscid=kben-us158474

Parameters

bullKeepAliveTime = milliseconds

Type DWORD

For Windows 98 type STRING

Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)

bullKeepAliveInterval = 32-digit value

Type DWORD

For Windows 98 STRING type

Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified

KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)

bullMaxDataRetries = 32-digit value

Type STRING

Defines maximum number of KEEPALIVE-probes Default value is 5

Setting KEEPALIVE in Windows 2000NTXP Register branch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

Everything about TCP adjustment

2000 NT httpsupportmicrosoftcomkb120642

XP httpsupportmicrosoftcomkb314053

The MaxDataRetries parameter is substituted by

TCPMaxDataRetransmissions

All other parameters have the same names as in Windows 9x

Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters

parameter DisableDHCPMediaSense=1 See a description of this parameterhere

httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14

Example

Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS

bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)

helliphelliphelliphelliphellip

DummyPacketsInterval=0

helliphelliphelliphelliphellip

bullMake sure there is the etcxinetdfirebird configuration file

Database maintenance

16

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

For the first case the following lines should be edited

procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes

For the second case the following commands should be executed

sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value

Time value is expressed in seconds

For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf

netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 16: The InterBase and Firebird Developer Magazine, Issue 1, 2005

bullConnect to any database on the server from any network client

bullCheck traffic on the server using any packet filter

If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on

bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on

Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected

Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value

Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes

General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by

KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic

Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors

Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed

bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes

Database maintenance

17

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Using Keepalive

We kept everything unchanged as it was registered during installation Nothingneeds to be added

bullChange the TCP stack parameters

KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes

InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement

sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5

bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes

KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds

bullClients use any connections downtime is not regulated

KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes

bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes

KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes

We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 17: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Developers area

18

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Working with temporarytables in InterBase 75

Author Dmitri Kouzmenkokdvib-aidcom

In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data

Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird

Letrsquos consider what temporary tables in InterBase 75 really are

MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)

Syntax of creation of temporary tables is as following

CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )

[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which

records are deleted only by delete+commit

GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)

RDB$RELATION_TYPE Description

GLOBAL TEMPORARYDELETE

Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit

GLOBAL TEMPORARYPRESERVE

Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection

It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa

For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo

Letrsquos create such a table and try to work with it

CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT DELETE ROWS

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 18: The InterBase and Firebird Developer Magazine, Issue 1, 2005

and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-

For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection

Letrsquos create a table

Developers area

19

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

Now you may create a procedure which would fill in the table with some data

CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN

I = 0WHILE (I lt 10000) DOBEGIN

INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1

ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)

Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table

TIP

In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table

At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following

InterBase Myths 5

InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)

This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase

CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )

ON COMMIT PRESERVE ROWS

Create a record in this table (it also can be done in IBExpert)

INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error

As soon as you close the current connection and open a new one the entered datawill be lost

Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However

when creating FK one should take into consideration therecord view area in both tables For example you create twotables

Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68

MASTER on commit deleteDETAIL on commit preserve

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 19: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Developers area

20

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with temporary tables

mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)

To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE

This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table

Temporary tables of all types cannot use FK pointing to constant tables

Garbage collecting

As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur

An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions

bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung

Resume

bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database

bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes

approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records

bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records

Resume

bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds

SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit

Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work

Table type When garbage is collected

ON COMMIT DELETE At first exclusive connection to the database

ON COMMIT PRESERVE When canceling the connection created the record

ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time

ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)

Registering for the Conference

Call for papers

Sponsoring the Firebird Conference

httpfirebird-conferencecom

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 20: The InterBase and Firebird Developer Magazine, Issue 1, 2005

to a database specify the codepage which was used during database creation

In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format

Working with databaseExchange with server uses the following data categories

bullGeneral text fields with CHAR and VARCHAR types

bullBLOB text fields

bullArrays

bullSQL query text

UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8

InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage

Developers area

21

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

Working with UNICODEin InterBaseFirebird

Author Dmitri Kovalenko LCPIdimalcpilipetskru

What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for

bullIt provides transport for UNICODE texts based on regular ASCII text

bullData packing The characters with codes less than 128 are as usual representedas one byte

Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable

Bits Hex Min Hex Max UTF-8 Binary Encoding

7 00000000 0000007F 0xxxxxxx

11 00000080 000007FF 110xxxxx 10xxxxxx

16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx

21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Table 1 Converting UCS-2 to UTF-8

Example

CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION

The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it

That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)

Creation of database using UNICODE_FSSNothing out of the way everything is as usual

CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS

Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use

However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 21: The InterBase and Firebird Developer Magazine, Issue 1, 2005

When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes

If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data

However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message

Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared

A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes

The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow

So please consider it if you want to useconnections with several codepages inthe context of a single database

BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB

That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR

The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding

Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss

ArraysUNICODE support in text arrays is simi-lar to the one for text fields

As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated

In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage

SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer

There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query

containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit

Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism

Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding

Developers area

22

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Working with UNICODEin InterBaseFireDird

TIP

By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte

InterBase Myths 6

Compiled procedures store query plans

Not a bit (unless the query plan is explicit)

This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 22: The InterBase and Firebird Developer Magazine, Issue 1, 2005

When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo

Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)

However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers

There are many reasons why Hyper-threading should be turned off on theserver

Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading

1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary

Total charge of several real (as well asvirtual) processors can occur only oncalculating applications

2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15

3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors

[conferences на newsgroups]

4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS

Hardware

23

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Hyperthreading SMP andInterBase Firebird Yaffil

Author Dmitri Kouzmenkokdvib-aidcom

This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000

Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom

Page 23: The InterBase and Firebird Developer Magazine, Issue 1, 2005

Hardware

24

The InterBase and Firebird Developer Magazine 2005 ISSUE 1

wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved

Hyperthreading amp SMP + InterBaseFirebird Yaffil

Server Server type Operatingsystem

2 and moreprocessors

InterBase 60 SuperServer Windows No

Linux No

Solaris-SPARC No

Classic Linux Yes

InterBase 65 SuperServer Windows No

Linux No

Solaris-SPARC No

InterBase 7x SuperServer Windows Yes

Linux Yes

Solaris-SPARC Yes

Firebird 10 SuperServer Windows No

Linux No

Solaris-x86 No

HP-UX No

Classic Linux Yes

Solaris-x86 Yes

Solaris-SPARC Yes

FreeBSD Yes

MacOSX Yes

HP-UX Yes

Firebird 15 SuperServer Windows No

Linux No

Classic Windows Yes

Linux Yes

FreeBSD Yes

MacOSX Yes

Sinixz Yes

Solaris-x86 Yes

Yaffil SuperServer Windows No

Classic Windows Yes

httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp

Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default

5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled

(search Google for bsod+hyperthread-ing)

6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on

two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)

httpwwwintelcomtechnologyhyperthread

SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system

In the 2 and more processors column

No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)

Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor

For clarity sake yes variantsin the table are typed in bold

Subscribe now To receive future issues notifications send email to subscribeibdevelopercom