Upload
doanduong
View
231
Download
6
Embed Size (px)
Citation preview
Cover story
2
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
ContentsCover storyby Dmitri YemanovInside Savepoints 3
by IBSurgeon Research Labs researchib-aidcomType conversion throughCOMPUTED BY 10
Database maintenanceby Dmitri KouzmenkoEmbedded User Authenticationin InterBase 75 11
by Vasiliy OvchinnikovUsing KEEPALIVE-sockets to detect and release hung InterBaseand Firebird client connectionsor how to avoid 10054104 errors 15
Developers areaby Dmitri KouzmenkoWorking with temporary tables in InterBase 75 18
by Dmitri KovalenkoWorking with UNICODE in InterBaseFirebird 21
by Dmitri KouzmenkoHyperthreading SMP and InterBase Firebird Yaffil 23
Dear colleagues
I am happy to present to you thefirst issue of ldquoThe InterBase andFirebird Developer Magazinerdquo Ithas been a long time since the ini-tial idea for such a magazine wasborn and now you are readingthe first issue
Why ldquoInterBase and Firebirdrdquo
I think the ldquoeditorrsquos noterdquo is the rightplace to answer some obvious ques-tions about a newborn magazine Firstof all why does its title contain bothldquoInterBaserdquo and ldquoFirebirdrdquo Somenewbies and gurus of InterBase andFirebird insist that they are differentproducts ndash and I completely agreeThey are different but people who usethem are the same Developmentapproaches problems and techniquesare almost always the same
Of course InterBase and Firebird aredifferent but it is not the same kind ofdifference as between Oracle and MSSQL ndash they are different like brothers
Who need our magazineYou sometimes hear the opinion that
ldquoInterBase is not a serious DBMSrdquo andldquoFirebird Whatrsquos thisrdquo And it is ahardly known fact that the size of ourcommunity is close to the communitysize of such monsters as Oracle andMS SQL Despite the fact that Oracle
and Microsoft command huge PR-budgets we are very well positioned inthis competition And I am sure thatthere is huge hidden potential in ourcommunity
Now the situation is changing Wehave got at least one great book aboutFirebird we have annual conferenceevents and now we have got our ownmagazine ndash just like other big players
I hope that our magazine will be usefuland interesting for all people who workwith InterBase and Firebird ndash as DBAsas developers or even as project man-agers We are concentrating on ldquoalltimes greatest hitsrdquo problems and willtry to make this magazine not a one-offreading but a long-play interestingnovel that makes for some good reainga number of times Also in future issueswe will pay attention to communityevents like BorCon and the FirebirdConference
In this issue
Ok what do we have in this issue Thefirst and largest article ldquoInside Save-pointsrdquo by Dmitri Yemanov is devote tothe internal details of server save-points Explicit savepoints are a ratherrecent improvement in both InterBaseand Firebird and I think it will be veryinteresting for all readers to know nitty-gritty details of their functioning
Another hot article is ldquoUsing Embed-ded User Authentication in InterBase75rdquo by Dmitri Kouzmenko We have
all been waiting for this feature for along time and I suppose we wont bedisappointed
ldquoWorking with temporary tables inInterBase 75rdquo is also very interestingTo my mind adding user temporarytables is the most important recentimprovement of InterBase so everydeveloper should know its details
And I think many InterBase and Fire-bird developers came across the mostannoying error ndash error 10054 Thearticle of Vasily Ovchinnikov is devotedto practical ways to avoid 10054errors
Of course this is not all of it in this issuendash please feel invited to read on
We need your feedbackThis is the first issue of ldquoThe Interbaseand Firebird Developer MagazinerdquoWe have got plenty of ideas for the fol-lowing issues and will try to make ldquoTheInterbase and Firebird DeveloperMagazinerdquo the best database devel-oper magazine around And the mostimportant thing we would like to knowis your opinion and your thoughtsabout our work Please do not hesitateto contact us
readersibdevelopercom
Sincerely yours Alexey Kovyazin
Chief Editor
Such a nice newbornEditorrsquos note
CreditsAlexey Kovyazin Chief EditorDmitri Kouzmenko EditorLev Tashchilin DesignerVolker Rehn Subeditor
Subscribe now To receive future issues notifications send email tosubscribeibdevelopercom
rollbacks the server also uses them forexception handling Each SQL andorPSQL operator is enclosed in a save-point frame which allows to rollbackthis particular operator keeping theprevious ones unchanged This guaran-tees either successful execution of theoperator or automatic cancellation ofall changes made and a correspondingerror will be initiated
For exception handling in PSQL eachBEGINhellipEND block is also enclosed ina savepoint frame which allows you tocancel all changes made by this blockLets consider some details of how save-points work
Savepoints in actionA savepoint is a data structure which islocated in the servers dynamic storage(transaction pool) and has a uniquenumerical ID A list of activities madewithin the savepoint context is associat-ed with this savepoint Such a list iscalled an undo log Savepoints form astack within a transaction and that isthe reason why only sequential rollbackof savepoints is possible
Undo log fragments are distributedacross savepoints that store the historyA savepoint which is active when a
General informationA savepoint is an internal mechanism ofthe database which binds any changesin the database to a specific point oftime during a transaction and in case ofnecessity allows a user to cancel allchanges which were made after settingthis particular savepoint This process isalso known as rolling back to savepoint
Also server uses savepoint mechanismto implement transaction handling Thismechanism helps either to commit or tocancel all changes made during a trans-action For those purposes the serveruses the global savepoint
Such a savepoint is set automaticallywhen a transaction starts and it is thefirst savepoint in the transaction contextWhen transaction rollback is initiatedall changes made within its context arecancelled using the transaction globalsavepoint After that the transaction iscommited () within the TransactionInventory Page (TIP) This is necessaryin order to avoid housekeeping opera-tions in the future
However if the number of changes intransaction context becomes too big(approx10000 - 1000000 records)then the storing of rollback listsbecomes expensive and the serverdeletes the transaction global save-point switching to the standard TIPmechanism to mark the transaction asdead
In addition to the use of savepoints for
Cover story
3
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Inside Savepoints
record is being changed is called cur-rent Information about record changesis stored in the current savepoint undolog If a rollback to the savepoint is per-formed the undo log is unwound andrecords are reconstructed As a resultthe record becomes as it was at themoment this savepoint was set
TipIf you expect that during transaction many changes are to be made thenit makes sense to specify the isc_tpb_no_auto_undo transaction parame-ter which disables usage of the global savepoint for rollback In somecases it allows to increase servers performance during batch operations
Author Dmitri Yemanovdimitruserssourceforgenet
In case there are no exception handlers available the records may bereconstructed down to the global savepoint providing completetransaction rollback After reconstruction of all modified records thesavepoint is usually deleted from the transaction context
Releasing savepointsIn addition to the rollback to savepoint operation there is an opera-tion of regular deletion (release) of a savepoint In case a savepoint isdeleted its undo log is merged with the undo log of the previous onein the savepoint stack In this sense each savepoint is nested
It is clear that regular deletion of all savepoints located deeper thanthe global one would lead to the transfer of all changes to the trans-action global savepoint
Figure 1 Savepoints in action
Cover story
4
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Thus the combination of all changeswhich were successfully executed dur-ing transaction is stored in undo logsThat is why when the automatic undolog is enabled the server can cancel allperformed changes and in the TIP thistransaction would be marked as com-mitted and not as dead
When specifying theisc_tpb_no_auto_undo parameter attransaction start a global savepoint is
not created and if regular deletion of the current stack is performed thecombined undo log is simply deleted and transaction is marked dead andall changes are to be cleared (garbage collection)
Savepoints and exception handlingThere are several events which make server create system-defined (ieuser-uncontrollable) savepoints
1 Execution of any user SQL-query As has been said above this done inorder to guarantee atomicity of a query That is to say if any exceptionoccurs during query execution the changes made in the database willalways be canceled After the query is executed the savepoint will be
automatically deleted
2 Execution of the BEGINhellipEND blockin PSQL (stored procedure or a trigger)in case the block contains an error han-dler (WHEN-block) In that case eachBEGIN operator sets a savepoint anda corresponding END operator deletesit This enables to provide error han-dling
in the PSQL-block
3 Execution of an SQL-operator in thecontext of a ldquoBEGINhellipENDrdquo blockwhich contains an error handler(WHEN-block) That is if the block con-tains an error handler any SQL-opera-tor in this block is framed by a save-point frame
Exception handling
Letrsquos consider the errors handlingprocess on the server When an excep-tion occurs automatic rollback to thelast set savepoint is performed As aresult all operations performed by aninvalid SQL-query would be canceled
Then in case of a PSQL-block it ischecked if there is a custom WHEN-handler If it does exist control is trans-ferred to it and after exiting savepointis deleted
Then the process repeats recursivelyuntil embedded handlers end Thus if astored procedure doesnt contain ahandler which would be able to handlethis error the whole procedure will beconsidered as a single SQL-operatorand canceled
Letrsquos make a summary
Firstly if there is no WHEN-handlerany PSQL-block (including stored pro-cedure and trigger) becomes atomic
and would be canceled entirely if anerror occurs
Secondly in the presence of aWHEN-handler the occurrence of anerror leads a rollback of the only oper-ator and after that the process is man-aged by the handler
That is to say there are obvious dif-ferences in the serverrsquos reactions whichdepend on the presence of a WHEN-handler
It is worth considering a knownanomaly which does not fit the schemedescribed above The scheme works asfollows the paragraph 3 (enclosing ofeach SQL-operator inside a PSQL-block by its own savepoint frame) istrue for SQL-operators only (moreovernot for all of them) In other words forexample an assignment operator willnot be framed by a savepoint frame
As a result of an error in assign-ment execution leads to a normal roll-back to the previous savepoint whichishellip exactly ndash the blockrsquos savepoint
That is to say even if there is aWHEN-handler the error may causethe rollback of the whole block beforecontrol to the error handler
I consider this situation as a seriousflaw in the serverrsquos exceptions handlinglogic Below is the full list of operatorsfor which the savepoint frame is creat-ed INSERT UPDATE DELETE EXCEP-TION EXECUTE STATEMENT [INTO]
To understand the reasons of thisanomaly it is necessary to take intoaccount the following 2 facts
bullIf an error occurs rollback to thelast savepoint is performed uncondi-tionally
Figure 2 Releasing a savepoint undo data moved to upward savepoint
TipAs is illustrated below each custom SQL-query is a set of savepoints within a transaction
TipIt is necessary to note that the isc_tpb_no_auto_undo parameter does not disable the savepoints mecha-nism (this is impossible due to the atomicity guarantee for SQL-operators ) It only disables the creationof a transaction undo log as a single whole
Cover story
5
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
bullA blockrsquos savepoint deletion (or roll-back to it) is performed together with theverification of the savepointrsquos identifier
This means that when an error occurs inoperators not included to the above list(ie not enclosed in a savepoint frame)that actually the wrong (blocks) save-point is deleted But the block itself is tol-erant to that fact since it only deletes itsown savepoint (in case it has one) Thisis the reason why the described servererror does not cause fatal conse-quences
Let us illustrate such a situation using thedescription of the [428903] ExceptionHandling Bug error To clarify this weprovide these examples with commentsabout show the server deals with save-points
Example 1A procedure with an error handler anderror generation in the assignmentoperator
In this case since savepoint 1 is thenearest to the erratic operator (save-
InterBase Myths 1When performing a restore outdated versions aredeleted (and therefore are stored in backup) orgbak -g creates a backup file without versions andby default versions are included in the fileNothing of the kind No records versions are stored in a back-up because it is unnecessary Generally the backup process is an ordi-nary snapshot transaction (repeatable read) and it reads only thoserecords versions which were relevant at the beginning of the transac-tion The ldquono_garbage_collectrdquo flag controls collecting garbage versionsin the database itself This flag can also be used during ordinary con-nections accessing the AP (ie in applications when say one needs toaccelerate sampling in some cases)
point 2 was deleted right before assignment execution) INSERT will becanceled Therefore a rollback of the whole BEGINEND block will beperformed before entering the handler
Below is the same procedure with an explicit exception statement enclosedin a block without a WHEN-handler
As you see no savepoint frame near assignment operator wascreated Therefore the result would be similar to the previous one
Below is the same procedure with an explicit exception call in theblock which contains WHEN-handler
CREATE PROCEDURE PROC1AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES
(01)-- end savepoint 2X = 1 0
WHEN ANY DOEXIT
-- end savepoint 1END
CREATE PROCEDURE PROC2AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (23)-- end savepoint 2BEGIN
X = 1 0END
WHEN ANY DOEXIT
-- end savepoint 1END
CREATE PROCEDURE PROC3AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (45)-- end savepoint 2-- start savepoint 3BEGIN
X = 1 0WHEN ANY DO
EXIT-- end savepoint 3
ENDWHEN ANY DO
EXIT-- end savepoint 1END Here we see a created savepoint frame As a result rollback is performed only for thenearest BEGIN level (savepoint 3) and the INSERT operator remained executed
Example 2A procedure with error handler and explicit exception call
CREATE PROCEDURE PROC4AS-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (67)-- end savepoint 2-- start savepoint 3EXCEPTION E-- end savepoint 3
WHEN ANY DOEXIT
-- end savepoint 1END
Cover story
6
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
In this case INSERT will not be canceleddue to the fact that the exception initiationof E results in rollback to savepoint 3 andsubsequent transfer of control to the errorhandler In order to cancel the INSERToperator in this case you should inhibit theexecution of the deletion handler
In addition a few words aboutexception handling adequacyregarding the SQL-standard
The standard allows three types ofhandlers in PSQL CONTINUEEXIT and UNDO With a CON-TINUE-handler the server mustrollback the erroneous operatorexecute the handler code and thencontinue the execution of the blockbeginning with the operator next tothe which one caused the error
An EXIT-handler requires finishingof the execution of the block rightafter exiting the handler code
An UNDO-handler requires a roll-back of all actions of the blockbefore entering the handler
Current versions of the server(InterBase as well as Firebird) donot support the explicit specifica-tion of the handler type and workaccording to the EXIT principle(however there is a possibility ofUNDO-behavior due to the anom-aly described above)
I suppose that in the future it
would be desirable to provide analternative to choose betweenUNDO- and EXIT- behavior of ahandler and repair the describedanomaly
Custom savepointsIn addition to the internalrealization of savepoints attransaction (and opera-torblock) levels the latestversions of servers (Inter-Base 71 Firebird 15 andYaffil 11) provide an SQL-interface developed for
this mechanism
Note savepointsrsquo syntax andsemantics are declared in the SQL-99 standard (see section 4371 ofthe specification)
Custom savepoints (also known asnested transactions) provide a con-venient business logic error han-dling method with no need to roll-back the whole transaction
Note rollback to a savepoint isalso sometimes called ldquopartialtransaction rollbackrdquo
New SQL operator (SAVEPOINT)was added to define a savepoint inthe transaction context to which arollback can be performed lateron
CREATE PROCEDURE PROC5ASBEGIN
INSERT INTO TAB (COL) VALUES (89)EXCEPTION E
END
a transaction If you attempt to createtwo savepoints with similar names thefirst savepoint is deleted and the speci-fied name is given to the second one
For rollback to a savepoint the follow-ing operator is used
SAVEPOINT ltnamegtltnamegt - the string identifier of a save-point As soon as a savepoint is creat-ed you can either continue transactioncommit (or cancel) the whole transac-tion or perform a rollback to a particu-lar savepoint Savepointsrsquo names (iden-tifiers) must be unique in the context of
Note the SAVEPOINT keyword isobligatory in InterBase 71
During execution of this operator thefollowing actions are performed
bullRollback of all changes made afterthe savepoint was set
bullAll savepoints set after this one aredeleted The current savepoint remainsunchanged and thus you can performseveral rollbacks to a savepoint Previ-ous savepoints remain unchanged aswell
Note Performing a rollback to save-point in InterBase 71 deletes the select-ed savepoint
bullAll explicit and implicit write locksoccupied after the savepoint was setare released At that other transac-tions which requested an access to therecords blocked by the transaction afterthe savepoint was set continue waitingfor the current transaction to be fin-ished Transactions which did notrequest access to the records may con-tinue and get access to them
Note This behavior refers to Firebird15 and can be changed in higher ver-sions
Since each savepoint uses certain sys-tem resources and also clogs thenamespace it makes sense to release
(delete) savepoints when they are no longer necessary This can be accom-plished using the following operator
RELEASE SAVEPOINT ltnamegt [ONLY]This command deletes the selected (and all following) savepoints fromtransaction context The ldquoONLYrdquo option is a switch to delete the selectedsavepoint only at that all following savepoints will be saved If a savepoint
was not released explicitly it will be automatically deleted assoon as the transaction is finished
Note The ldquoONLYrdquo option is non-standard extension and isnot supported by InterBase 71
ROLLBACK [WORK] TO [SAVEPOINT] ltnamegt
FastReport 3 - new generation of the reporting tools
Visual report designer with rulers guides andzooming wizards for base type reports exportfilters to html tiff bmp jpg xls pdf Dot matrix
reports support support most popular DB-engines
Full WYSIWYG text rotation 0360 degreesmemo object supports simple html-tags
(font color b i u sub sup) improved stretching(StretchMode ShiftMode properties)
Access to DB fields styles text flow URLs Anchors
httpwwwfast-reportcomenreadersphp5 discount for all our readers
Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers
At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail
First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint
It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set
through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71
A question emerges how would customsavepoints coexist with internal save-points managed by the server
Cover story
7
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title
A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation
Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative
In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO
LOOPTRY
SAVEPOINT DOClthellipgt single document handling commands
EXCEPTROLLBACK TO SAVEPOINT DOC
lthellipgt either log the error or display it
END END
CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name
Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)
START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY
lthellipgt operations on databaseEXCEPT
ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID
ENDCOMMIT
Cover story
8
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation
CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )
ASBEGIN
BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)
VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO
BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID
ENDEND
END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here
BEGININSERT INTO hellip
WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id
END
This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution
Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN
SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip
INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
WHEN ANY DOBEGIN
ROLLBACK TO SAVEPOINTS1
ERROR = REC_IDSUSPEND
ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well
FOR SELECT ID hellip INTOREC_ID hellipBEGIN
BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
ENDWHEN ANY DO
BEGINERROR = REC_IDSUSPEND
ENDEND
In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler
Therefore in virtually any case one
can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks
A bit of practice
Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear
Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as
SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1
work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above
1 savepoints constitute a stack andcan be canceled sequentially only
2 each block of PSQL-code with anexception handler is enclosed in a frame
Thus we arrive at a conclusion that anycode area of the following type
SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1
hellipWHEN
is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base
Thank God the InterBase developersdid not create such cardinal realization
and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in
this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler
However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample
Statement failed SQLCODE = -504Savepoint ltnamegt unknown
Cover story
9
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip
END
This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode
INSERT INTO TABLE1 hellipBEGIN
ROLLBACK TO SAVEPOINT S1WHEN ANY DO
EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon
SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is
ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)
neither available in Firebird nor in Yaffil
Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed
Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions
For this purpose three new API functions are introduced
As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to
Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)
Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)
Create savepoint isc_start_savepoint(status amptrans A)
Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)
Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)
all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)
stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316
Thus we can make up several conclu-sions
bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)
bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead
bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0
bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place
bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results
Tips
10
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Commit coordinated transaction isc_commit_transaction (status amptrans)
Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)
As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed
And that would be that way but forthe manually executed RELEASESAVEPOINT A
At first savepoint rollback was per-formed for the first connection and allchanges were canceled
Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible
The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints
That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process
Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole
Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API
Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means
The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability
It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature
InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used
ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain
[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]
As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source
The following experiment can be performed Create a table with structure as shownbelow
(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)
Type conversion throughCOMPUTED BY
CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))
Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be
AuthorIBSurgeon Research Labs researchib-aidcom
Database maintenance
11
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Embedded User Authenticationin InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version
Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database
In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base
1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server
2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one
3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)
As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted
This scheme is insufficient when used in
bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib
bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase
bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work
Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to
manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)
This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages
bullproduct DATABASE ACCESS is notlicensed
for databases in which EUA is enabledor disabled
bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)
for databases in which EUA was neverenabled
In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)
continued on page 12
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
rollbacks the server also uses them forexception handling Each SQL andorPSQL operator is enclosed in a save-point frame which allows to rollbackthis particular operator keeping theprevious ones unchanged This guaran-tees either successful execution of theoperator or automatic cancellation ofall changes made and a correspondingerror will be initiated
For exception handling in PSQL eachBEGINhellipEND block is also enclosed ina savepoint frame which allows you tocancel all changes made by this blockLets consider some details of how save-points work
Savepoints in actionA savepoint is a data structure which islocated in the servers dynamic storage(transaction pool) and has a uniquenumerical ID A list of activities madewithin the savepoint context is associat-ed with this savepoint Such a list iscalled an undo log Savepoints form astack within a transaction and that isthe reason why only sequential rollbackof savepoints is possible
Undo log fragments are distributedacross savepoints that store the historyA savepoint which is active when a
General informationA savepoint is an internal mechanism ofthe database which binds any changesin the database to a specific point oftime during a transaction and in case ofnecessity allows a user to cancel allchanges which were made after settingthis particular savepoint This process isalso known as rolling back to savepoint
Also server uses savepoint mechanismto implement transaction handling Thismechanism helps either to commit or tocancel all changes made during a trans-action For those purposes the serveruses the global savepoint
Such a savepoint is set automaticallywhen a transaction starts and it is thefirst savepoint in the transaction contextWhen transaction rollback is initiatedall changes made within its context arecancelled using the transaction globalsavepoint After that the transaction iscommited () within the TransactionInventory Page (TIP) This is necessaryin order to avoid housekeeping opera-tions in the future
However if the number of changes intransaction context becomes too big(approx10000 - 1000000 records)then the storing of rollback listsbecomes expensive and the serverdeletes the transaction global save-point switching to the standard TIPmechanism to mark the transaction asdead
In addition to the use of savepoints for
Cover story
3
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Inside Savepoints
record is being changed is called cur-rent Information about record changesis stored in the current savepoint undolog If a rollback to the savepoint is per-formed the undo log is unwound andrecords are reconstructed As a resultthe record becomes as it was at themoment this savepoint was set
TipIf you expect that during transaction many changes are to be made thenit makes sense to specify the isc_tpb_no_auto_undo transaction parame-ter which disables usage of the global savepoint for rollback In somecases it allows to increase servers performance during batch operations
Author Dmitri Yemanovdimitruserssourceforgenet
In case there are no exception handlers available the records may bereconstructed down to the global savepoint providing completetransaction rollback After reconstruction of all modified records thesavepoint is usually deleted from the transaction context
Releasing savepointsIn addition to the rollback to savepoint operation there is an opera-tion of regular deletion (release) of a savepoint In case a savepoint isdeleted its undo log is merged with the undo log of the previous onein the savepoint stack In this sense each savepoint is nested
It is clear that regular deletion of all savepoints located deeper thanthe global one would lead to the transfer of all changes to the trans-action global savepoint
Figure 1 Savepoints in action
Cover story
4
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Thus the combination of all changeswhich were successfully executed dur-ing transaction is stored in undo logsThat is why when the automatic undolog is enabled the server can cancel allperformed changes and in the TIP thistransaction would be marked as com-mitted and not as dead
When specifying theisc_tpb_no_auto_undo parameter attransaction start a global savepoint is
not created and if regular deletion of the current stack is performed thecombined undo log is simply deleted and transaction is marked dead andall changes are to be cleared (garbage collection)
Savepoints and exception handlingThere are several events which make server create system-defined (ieuser-uncontrollable) savepoints
1 Execution of any user SQL-query As has been said above this done inorder to guarantee atomicity of a query That is to say if any exceptionoccurs during query execution the changes made in the database willalways be canceled After the query is executed the savepoint will be
automatically deleted
2 Execution of the BEGINhellipEND blockin PSQL (stored procedure or a trigger)in case the block contains an error han-dler (WHEN-block) In that case eachBEGIN operator sets a savepoint anda corresponding END operator deletesit This enables to provide error han-dling
in the PSQL-block
3 Execution of an SQL-operator in thecontext of a ldquoBEGINhellipENDrdquo blockwhich contains an error handler(WHEN-block) That is if the block con-tains an error handler any SQL-opera-tor in this block is framed by a save-point frame
Exception handling
Letrsquos consider the errors handlingprocess on the server When an excep-tion occurs automatic rollback to thelast set savepoint is performed As aresult all operations performed by aninvalid SQL-query would be canceled
Then in case of a PSQL-block it ischecked if there is a custom WHEN-handler If it does exist control is trans-ferred to it and after exiting savepointis deleted
Then the process repeats recursivelyuntil embedded handlers end Thus if astored procedure doesnt contain ahandler which would be able to handlethis error the whole procedure will beconsidered as a single SQL-operatorand canceled
Letrsquos make a summary
Firstly if there is no WHEN-handlerany PSQL-block (including stored pro-cedure and trigger) becomes atomic
and would be canceled entirely if anerror occurs
Secondly in the presence of aWHEN-handler the occurrence of anerror leads a rollback of the only oper-ator and after that the process is man-aged by the handler
That is to say there are obvious dif-ferences in the serverrsquos reactions whichdepend on the presence of a WHEN-handler
It is worth considering a knownanomaly which does not fit the schemedescribed above The scheme works asfollows the paragraph 3 (enclosing ofeach SQL-operator inside a PSQL-block by its own savepoint frame) istrue for SQL-operators only (moreovernot for all of them) In other words forexample an assignment operator willnot be framed by a savepoint frame
As a result of an error in assign-ment execution leads to a normal roll-back to the previous savepoint whichishellip exactly ndash the blockrsquos savepoint
That is to say even if there is aWHEN-handler the error may causethe rollback of the whole block beforecontrol to the error handler
I consider this situation as a seriousflaw in the serverrsquos exceptions handlinglogic Below is the full list of operatorsfor which the savepoint frame is creat-ed INSERT UPDATE DELETE EXCEP-TION EXECUTE STATEMENT [INTO]
To understand the reasons of thisanomaly it is necessary to take intoaccount the following 2 facts
bullIf an error occurs rollback to thelast savepoint is performed uncondi-tionally
Figure 2 Releasing a savepoint undo data moved to upward savepoint
TipAs is illustrated below each custom SQL-query is a set of savepoints within a transaction
TipIt is necessary to note that the isc_tpb_no_auto_undo parameter does not disable the savepoints mecha-nism (this is impossible due to the atomicity guarantee for SQL-operators ) It only disables the creationof a transaction undo log as a single whole
Cover story
5
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
bullA blockrsquos savepoint deletion (or roll-back to it) is performed together with theverification of the savepointrsquos identifier
This means that when an error occurs inoperators not included to the above list(ie not enclosed in a savepoint frame)that actually the wrong (blocks) save-point is deleted But the block itself is tol-erant to that fact since it only deletes itsown savepoint (in case it has one) Thisis the reason why the described servererror does not cause fatal conse-quences
Let us illustrate such a situation using thedescription of the [428903] ExceptionHandling Bug error To clarify this weprovide these examples with commentsabout show the server deals with save-points
Example 1A procedure with an error handler anderror generation in the assignmentoperator
In this case since savepoint 1 is thenearest to the erratic operator (save-
InterBase Myths 1When performing a restore outdated versions aredeleted (and therefore are stored in backup) orgbak -g creates a backup file without versions andby default versions are included in the fileNothing of the kind No records versions are stored in a back-up because it is unnecessary Generally the backup process is an ordi-nary snapshot transaction (repeatable read) and it reads only thoserecords versions which were relevant at the beginning of the transac-tion The ldquono_garbage_collectrdquo flag controls collecting garbage versionsin the database itself This flag can also be used during ordinary con-nections accessing the AP (ie in applications when say one needs toaccelerate sampling in some cases)
point 2 was deleted right before assignment execution) INSERT will becanceled Therefore a rollback of the whole BEGINEND block will beperformed before entering the handler
Below is the same procedure with an explicit exception statement enclosedin a block without a WHEN-handler
As you see no savepoint frame near assignment operator wascreated Therefore the result would be similar to the previous one
Below is the same procedure with an explicit exception call in theblock which contains WHEN-handler
CREATE PROCEDURE PROC1AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES
(01)-- end savepoint 2X = 1 0
WHEN ANY DOEXIT
-- end savepoint 1END
CREATE PROCEDURE PROC2AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (23)-- end savepoint 2BEGIN
X = 1 0END
WHEN ANY DOEXIT
-- end savepoint 1END
CREATE PROCEDURE PROC3AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (45)-- end savepoint 2-- start savepoint 3BEGIN
X = 1 0WHEN ANY DO
EXIT-- end savepoint 3
ENDWHEN ANY DO
EXIT-- end savepoint 1END Here we see a created savepoint frame As a result rollback is performed only for thenearest BEGIN level (savepoint 3) and the INSERT operator remained executed
Example 2A procedure with error handler and explicit exception call
CREATE PROCEDURE PROC4AS-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (67)-- end savepoint 2-- start savepoint 3EXCEPTION E-- end savepoint 3
WHEN ANY DOEXIT
-- end savepoint 1END
Cover story
6
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
In this case INSERT will not be canceleddue to the fact that the exception initiationof E results in rollback to savepoint 3 andsubsequent transfer of control to the errorhandler In order to cancel the INSERToperator in this case you should inhibit theexecution of the deletion handler
In addition a few words aboutexception handling adequacyregarding the SQL-standard
The standard allows three types ofhandlers in PSQL CONTINUEEXIT and UNDO With a CON-TINUE-handler the server mustrollback the erroneous operatorexecute the handler code and thencontinue the execution of the blockbeginning with the operator next tothe which one caused the error
An EXIT-handler requires finishingof the execution of the block rightafter exiting the handler code
An UNDO-handler requires a roll-back of all actions of the blockbefore entering the handler
Current versions of the server(InterBase as well as Firebird) donot support the explicit specifica-tion of the handler type and workaccording to the EXIT principle(however there is a possibility ofUNDO-behavior due to the anom-aly described above)
I suppose that in the future it
would be desirable to provide analternative to choose betweenUNDO- and EXIT- behavior of ahandler and repair the describedanomaly
Custom savepointsIn addition to the internalrealization of savepoints attransaction (and opera-torblock) levels the latestversions of servers (Inter-Base 71 Firebird 15 andYaffil 11) provide an SQL-interface developed for
this mechanism
Note savepointsrsquo syntax andsemantics are declared in the SQL-99 standard (see section 4371 ofthe specification)
Custom savepoints (also known asnested transactions) provide a con-venient business logic error han-dling method with no need to roll-back the whole transaction
Note rollback to a savepoint isalso sometimes called ldquopartialtransaction rollbackrdquo
New SQL operator (SAVEPOINT)was added to define a savepoint inthe transaction context to which arollback can be performed lateron
CREATE PROCEDURE PROC5ASBEGIN
INSERT INTO TAB (COL) VALUES (89)EXCEPTION E
END
a transaction If you attempt to createtwo savepoints with similar names thefirst savepoint is deleted and the speci-fied name is given to the second one
For rollback to a savepoint the follow-ing operator is used
SAVEPOINT ltnamegtltnamegt - the string identifier of a save-point As soon as a savepoint is creat-ed you can either continue transactioncommit (or cancel) the whole transac-tion or perform a rollback to a particu-lar savepoint Savepointsrsquo names (iden-tifiers) must be unique in the context of
Note the SAVEPOINT keyword isobligatory in InterBase 71
During execution of this operator thefollowing actions are performed
bullRollback of all changes made afterthe savepoint was set
bullAll savepoints set after this one aredeleted The current savepoint remainsunchanged and thus you can performseveral rollbacks to a savepoint Previ-ous savepoints remain unchanged aswell
Note Performing a rollback to save-point in InterBase 71 deletes the select-ed savepoint
bullAll explicit and implicit write locksoccupied after the savepoint was setare released At that other transac-tions which requested an access to therecords blocked by the transaction afterthe savepoint was set continue waitingfor the current transaction to be fin-ished Transactions which did notrequest access to the records may con-tinue and get access to them
Note This behavior refers to Firebird15 and can be changed in higher ver-sions
Since each savepoint uses certain sys-tem resources and also clogs thenamespace it makes sense to release
(delete) savepoints when they are no longer necessary This can be accom-plished using the following operator
RELEASE SAVEPOINT ltnamegt [ONLY]This command deletes the selected (and all following) savepoints fromtransaction context The ldquoONLYrdquo option is a switch to delete the selectedsavepoint only at that all following savepoints will be saved If a savepoint
was not released explicitly it will be automatically deleted assoon as the transaction is finished
Note The ldquoONLYrdquo option is non-standard extension and isnot supported by InterBase 71
ROLLBACK [WORK] TO [SAVEPOINT] ltnamegt
FastReport 3 - new generation of the reporting tools
Visual report designer with rulers guides andzooming wizards for base type reports exportfilters to html tiff bmp jpg xls pdf Dot matrix
reports support support most popular DB-engines
Full WYSIWYG text rotation 0360 degreesmemo object supports simple html-tags
(font color b i u sub sup) improved stretching(StretchMode ShiftMode properties)
Access to DB fields styles text flow URLs Anchors
httpwwwfast-reportcomenreadersphp5 discount for all our readers
Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers
At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail
First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint
It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set
through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71
A question emerges how would customsavepoints coexist with internal save-points managed by the server
Cover story
7
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title
A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation
Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative
In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO
LOOPTRY
SAVEPOINT DOClthellipgt single document handling commands
EXCEPTROLLBACK TO SAVEPOINT DOC
lthellipgt either log the error or display it
END END
CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name
Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)
START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY
lthellipgt operations on databaseEXCEPT
ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID
ENDCOMMIT
Cover story
8
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation
CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )
ASBEGIN
BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)
VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO
BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID
ENDEND
END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here
BEGININSERT INTO hellip
WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id
END
This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution
Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN
SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip
INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
WHEN ANY DOBEGIN
ROLLBACK TO SAVEPOINTS1
ERROR = REC_IDSUSPEND
ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well
FOR SELECT ID hellip INTOREC_ID hellipBEGIN
BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
ENDWHEN ANY DO
BEGINERROR = REC_IDSUSPEND
ENDEND
In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler
Therefore in virtually any case one
can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks
A bit of practice
Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear
Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as
SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1
work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above
1 savepoints constitute a stack andcan be canceled sequentially only
2 each block of PSQL-code with anexception handler is enclosed in a frame
Thus we arrive at a conclusion that anycode area of the following type
SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1
hellipWHEN
is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base
Thank God the InterBase developersdid not create such cardinal realization
and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in
this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler
However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample
Statement failed SQLCODE = -504Savepoint ltnamegt unknown
Cover story
9
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip
END
This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode
INSERT INTO TABLE1 hellipBEGIN
ROLLBACK TO SAVEPOINT S1WHEN ANY DO
EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon
SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is
ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)
neither available in Firebird nor in Yaffil
Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed
Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions
For this purpose three new API functions are introduced
As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to
Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)
Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)
Create savepoint isc_start_savepoint(status amptrans A)
Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)
Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)
all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)
stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316
Thus we can make up several conclu-sions
bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)
bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead
bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0
bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place
bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results
Tips
10
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Commit coordinated transaction isc_commit_transaction (status amptrans)
Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)
As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed
And that would be that way but forthe manually executed RELEASESAVEPOINT A
At first savepoint rollback was per-formed for the first connection and allchanges were canceled
Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible
The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints
That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process
Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole
Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API
Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means
The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability
It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature
InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used
ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain
[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]
As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source
The following experiment can be performed Create a table with structure as shownbelow
(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)
Type conversion throughCOMPUTED BY
CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))
Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be
AuthorIBSurgeon Research Labs researchib-aidcom
Database maintenance
11
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Embedded User Authenticationin InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version
Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database
In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base
1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server
2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one
3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)
As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted
This scheme is insufficient when used in
bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib
bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase
bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work
Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to
manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)
This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages
bullproduct DATABASE ACCESS is notlicensed
for databases in which EUA is enabledor disabled
bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)
for databases in which EUA was neverenabled
In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)
continued on page 12
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Cover story
4
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Thus the combination of all changeswhich were successfully executed dur-ing transaction is stored in undo logsThat is why when the automatic undolog is enabled the server can cancel allperformed changes and in the TIP thistransaction would be marked as com-mitted and not as dead
When specifying theisc_tpb_no_auto_undo parameter attransaction start a global savepoint is
not created and if regular deletion of the current stack is performed thecombined undo log is simply deleted and transaction is marked dead andall changes are to be cleared (garbage collection)
Savepoints and exception handlingThere are several events which make server create system-defined (ieuser-uncontrollable) savepoints
1 Execution of any user SQL-query As has been said above this done inorder to guarantee atomicity of a query That is to say if any exceptionoccurs during query execution the changes made in the database willalways be canceled After the query is executed the savepoint will be
automatically deleted
2 Execution of the BEGINhellipEND blockin PSQL (stored procedure or a trigger)in case the block contains an error han-dler (WHEN-block) In that case eachBEGIN operator sets a savepoint anda corresponding END operator deletesit This enables to provide error han-dling
in the PSQL-block
3 Execution of an SQL-operator in thecontext of a ldquoBEGINhellipENDrdquo blockwhich contains an error handler(WHEN-block) That is if the block con-tains an error handler any SQL-opera-tor in this block is framed by a save-point frame
Exception handling
Letrsquos consider the errors handlingprocess on the server When an excep-tion occurs automatic rollback to thelast set savepoint is performed As aresult all operations performed by aninvalid SQL-query would be canceled
Then in case of a PSQL-block it ischecked if there is a custom WHEN-handler If it does exist control is trans-ferred to it and after exiting savepointis deleted
Then the process repeats recursivelyuntil embedded handlers end Thus if astored procedure doesnt contain ahandler which would be able to handlethis error the whole procedure will beconsidered as a single SQL-operatorand canceled
Letrsquos make a summary
Firstly if there is no WHEN-handlerany PSQL-block (including stored pro-cedure and trigger) becomes atomic
and would be canceled entirely if anerror occurs
Secondly in the presence of aWHEN-handler the occurrence of anerror leads a rollback of the only oper-ator and after that the process is man-aged by the handler
That is to say there are obvious dif-ferences in the serverrsquos reactions whichdepend on the presence of a WHEN-handler
It is worth considering a knownanomaly which does not fit the schemedescribed above The scheme works asfollows the paragraph 3 (enclosing ofeach SQL-operator inside a PSQL-block by its own savepoint frame) istrue for SQL-operators only (moreovernot for all of them) In other words forexample an assignment operator willnot be framed by a savepoint frame
As a result of an error in assign-ment execution leads to a normal roll-back to the previous savepoint whichishellip exactly ndash the blockrsquos savepoint
That is to say even if there is aWHEN-handler the error may causethe rollback of the whole block beforecontrol to the error handler
I consider this situation as a seriousflaw in the serverrsquos exceptions handlinglogic Below is the full list of operatorsfor which the savepoint frame is creat-ed INSERT UPDATE DELETE EXCEP-TION EXECUTE STATEMENT [INTO]
To understand the reasons of thisanomaly it is necessary to take intoaccount the following 2 facts
bullIf an error occurs rollback to thelast savepoint is performed uncondi-tionally
Figure 2 Releasing a savepoint undo data moved to upward savepoint
TipAs is illustrated below each custom SQL-query is a set of savepoints within a transaction
TipIt is necessary to note that the isc_tpb_no_auto_undo parameter does not disable the savepoints mecha-nism (this is impossible due to the atomicity guarantee for SQL-operators ) It only disables the creationof a transaction undo log as a single whole
Cover story
5
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
bullA blockrsquos savepoint deletion (or roll-back to it) is performed together with theverification of the savepointrsquos identifier
This means that when an error occurs inoperators not included to the above list(ie not enclosed in a savepoint frame)that actually the wrong (blocks) save-point is deleted But the block itself is tol-erant to that fact since it only deletes itsown savepoint (in case it has one) Thisis the reason why the described servererror does not cause fatal conse-quences
Let us illustrate such a situation using thedescription of the [428903] ExceptionHandling Bug error To clarify this weprovide these examples with commentsabout show the server deals with save-points
Example 1A procedure with an error handler anderror generation in the assignmentoperator
In this case since savepoint 1 is thenearest to the erratic operator (save-
InterBase Myths 1When performing a restore outdated versions aredeleted (and therefore are stored in backup) orgbak -g creates a backup file without versions andby default versions are included in the fileNothing of the kind No records versions are stored in a back-up because it is unnecessary Generally the backup process is an ordi-nary snapshot transaction (repeatable read) and it reads only thoserecords versions which were relevant at the beginning of the transac-tion The ldquono_garbage_collectrdquo flag controls collecting garbage versionsin the database itself This flag can also be used during ordinary con-nections accessing the AP (ie in applications when say one needs toaccelerate sampling in some cases)
point 2 was deleted right before assignment execution) INSERT will becanceled Therefore a rollback of the whole BEGINEND block will beperformed before entering the handler
Below is the same procedure with an explicit exception statement enclosedin a block without a WHEN-handler
As you see no savepoint frame near assignment operator wascreated Therefore the result would be similar to the previous one
Below is the same procedure with an explicit exception call in theblock which contains WHEN-handler
CREATE PROCEDURE PROC1AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES
(01)-- end savepoint 2X = 1 0
WHEN ANY DOEXIT
-- end savepoint 1END
CREATE PROCEDURE PROC2AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (23)-- end savepoint 2BEGIN
X = 1 0END
WHEN ANY DOEXIT
-- end savepoint 1END
CREATE PROCEDURE PROC3AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (45)-- end savepoint 2-- start savepoint 3BEGIN
X = 1 0WHEN ANY DO
EXIT-- end savepoint 3
ENDWHEN ANY DO
EXIT-- end savepoint 1END Here we see a created savepoint frame As a result rollback is performed only for thenearest BEGIN level (savepoint 3) and the INSERT operator remained executed
Example 2A procedure with error handler and explicit exception call
CREATE PROCEDURE PROC4AS-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (67)-- end savepoint 2-- start savepoint 3EXCEPTION E-- end savepoint 3
WHEN ANY DOEXIT
-- end savepoint 1END
Cover story
6
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
In this case INSERT will not be canceleddue to the fact that the exception initiationof E results in rollback to savepoint 3 andsubsequent transfer of control to the errorhandler In order to cancel the INSERToperator in this case you should inhibit theexecution of the deletion handler
In addition a few words aboutexception handling adequacyregarding the SQL-standard
The standard allows three types ofhandlers in PSQL CONTINUEEXIT and UNDO With a CON-TINUE-handler the server mustrollback the erroneous operatorexecute the handler code and thencontinue the execution of the blockbeginning with the operator next tothe which one caused the error
An EXIT-handler requires finishingof the execution of the block rightafter exiting the handler code
An UNDO-handler requires a roll-back of all actions of the blockbefore entering the handler
Current versions of the server(InterBase as well as Firebird) donot support the explicit specifica-tion of the handler type and workaccording to the EXIT principle(however there is a possibility ofUNDO-behavior due to the anom-aly described above)
I suppose that in the future it
would be desirable to provide analternative to choose betweenUNDO- and EXIT- behavior of ahandler and repair the describedanomaly
Custom savepointsIn addition to the internalrealization of savepoints attransaction (and opera-torblock) levels the latestversions of servers (Inter-Base 71 Firebird 15 andYaffil 11) provide an SQL-interface developed for
this mechanism
Note savepointsrsquo syntax andsemantics are declared in the SQL-99 standard (see section 4371 ofthe specification)
Custom savepoints (also known asnested transactions) provide a con-venient business logic error han-dling method with no need to roll-back the whole transaction
Note rollback to a savepoint isalso sometimes called ldquopartialtransaction rollbackrdquo
New SQL operator (SAVEPOINT)was added to define a savepoint inthe transaction context to which arollback can be performed lateron
CREATE PROCEDURE PROC5ASBEGIN
INSERT INTO TAB (COL) VALUES (89)EXCEPTION E
END
a transaction If you attempt to createtwo savepoints with similar names thefirst savepoint is deleted and the speci-fied name is given to the second one
For rollback to a savepoint the follow-ing operator is used
SAVEPOINT ltnamegtltnamegt - the string identifier of a save-point As soon as a savepoint is creat-ed you can either continue transactioncommit (or cancel) the whole transac-tion or perform a rollback to a particu-lar savepoint Savepointsrsquo names (iden-tifiers) must be unique in the context of
Note the SAVEPOINT keyword isobligatory in InterBase 71
During execution of this operator thefollowing actions are performed
bullRollback of all changes made afterthe savepoint was set
bullAll savepoints set after this one aredeleted The current savepoint remainsunchanged and thus you can performseveral rollbacks to a savepoint Previ-ous savepoints remain unchanged aswell
Note Performing a rollback to save-point in InterBase 71 deletes the select-ed savepoint
bullAll explicit and implicit write locksoccupied after the savepoint was setare released At that other transac-tions which requested an access to therecords blocked by the transaction afterthe savepoint was set continue waitingfor the current transaction to be fin-ished Transactions which did notrequest access to the records may con-tinue and get access to them
Note This behavior refers to Firebird15 and can be changed in higher ver-sions
Since each savepoint uses certain sys-tem resources and also clogs thenamespace it makes sense to release
(delete) savepoints when they are no longer necessary This can be accom-plished using the following operator
RELEASE SAVEPOINT ltnamegt [ONLY]This command deletes the selected (and all following) savepoints fromtransaction context The ldquoONLYrdquo option is a switch to delete the selectedsavepoint only at that all following savepoints will be saved If a savepoint
was not released explicitly it will be automatically deleted assoon as the transaction is finished
Note The ldquoONLYrdquo option is non-standard extension and isnot supported by InterBase 71
ROLLBACK [WORK] TO [SAVEPOINT] ltnamegt
FastReport 3 - new generation of the reporting tools
Visual report designer with rulers guides andzooming wizards for base type reports exportfilters to html tiff bmp jpg xls pdf Dot matrix
reports support support most popular DB-engines
Full WYSIWYG text rotation 0360 degreesmemo object supports simple html-tags
(font color b i u sub sup) improved stretching(StretchMode ShiftMode properties)
Access to DB fields styles text flow URLs Anchors
httpwwwfast-reportcomenreadersphp5 discount for all our readers
Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers
At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail
First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint
It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set
through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71
A question emerges how would customsavepoints coexist with internal save-points managed by the server
Cover story
7
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title
A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation
Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative
In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO
LOOPTRY
SAVEPOINT DOClthellipgt single document handling commands
EXCEPTROLLBACK TO SAVEPOINT DOC
lthellipgt either log the error or display it
END END
CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name
Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)
START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY
lthellipgt operations on databaseEXCEPT
ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID
ENDCOMMIT
Cover story
8
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation
CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )
ASBEGIN
BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)
VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO
BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID
ENDEND
END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here
BEGININSERT INTO hellip
WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id
END
This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution
Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN
SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip
INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
WHEN ANY DOBEGIN
ROLLBACK TO SAVEPOINTS1
ERROR = REC_IDSUSPEND
ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well
FOR SELECT ID hellip INTOREC_ID hellipBEGIN
BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
ENDWHEN ANY DO
BEGINERROR = REC_IDSUSPEND
ENDEND
In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler
Therefore in virtually any case one
can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks
A bit of practice
Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear
Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as
SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1
work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above
1 savepoints constitute a stack andcan be canceled sequentially only
2 each block of PSQL-code with anexception handler is enclosed in a frame
Thus we arrive at a conclusion that anycode area of the following type
SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1
hellipWHEN
is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base
Thank God the InterBase developersdid not create such cardinal realization
and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in
this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler
However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample
Statement failed SQLCODE = -504Savepoint ltnamegt unknown
Cover story
9
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip
END
This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode
INSERT INTO TABLE1 hellipBEGIN
ROLLBACK TO SAVEPOINT S1WHEN ANY DO
EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon
SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is
ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)
neither available in Firebird nor in Yaffil
Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed
Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions
For this purpose three new API functions are introduced
As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to
Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)
Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)
Create savepoint isc_start_savepoint(status amptrans A)
Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)
Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)
all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)
stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316
Thus we can make up several conclu-sions
bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)
bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead
bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0
bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place
bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results
Tips
10
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Commit coordinated transaction isc_commit_transaction (status amptrans)
Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)
As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed
And that would be that way but forthe manually executed RELEASESAVEPOINT A
At first savepoint rollback was per-formed for the first connection and allchanges were canceled
Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible
The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints
That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process
Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole
Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API
Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means
The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability
It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature
InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used
ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain
[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]
As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source
The following experiment can be performed Create a table with structure as shownbelow
(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)
Type conversion throughCOMPUTED BY
CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))
Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be
AuthorIBSurgeon Research Labs researchib-aidcom
Database maintenance
11
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Embedded User Authenticationin InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version
Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database
In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base
1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server
2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one
3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)
As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted
This scheme is insufficient when used in
bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib
bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase
bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work
Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to
manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)
This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages
bullproduct DATABASE ACCESS is notlicensed
for databases in which EUA is enabledor disabled
bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)
for databases in which EUA was neverenabled
In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)
continued on page 12
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Cover story
5
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
bullA blockrsquos savepoint deletion (or roll-back to it) is performed together with theverification of the savepointrsquos identifier
This means that when an error occurs inoperators not included to the above list(ie not enclosed in a savepoint frame)that actually the wrong (blocks) save-point is deleted But the block itself is tol-erant to that fact since it only deletes itsown savepoint (in case it has one) Thisis the reason why the described servererror does not cause fatal conse-quences
Let us illustrate such a situation using thedescription of the [428903] ExceptionHandling Bug error To clarify this weprovide these examples with commentsabout show the server deals with save-points
Example 1A procedure with an error handler anderror generation in the assignmentoperator
In this case since savepoint 1 is thenearest to the erratic operator (save-
InterBase Myths 1When performing a restore outdated versions aredeleted (and therefore are stored in backup) orgbak -g creates a backup file without versions andby default versions are included in the fileNothing of the kind No records versions are stored in a back-up because it is unnecessary Generally the backup process is an ordi-nary snapshot transaction (repeatable read) and it reads only thoserecords versions which were relevant at the beginning of the transac-tion The ldquono_garbage_collectrdquo flag controls collecting garbage versionsin the database itself This flag can also be used during ordinary con-nections accessing the AP (ie in applications when say one needs toaccelerate sampling in some cases)
point 2 was deleted right before assignment execution) INSERT will becanceled Therefore a rollback of the whole BEGINEND block will beperformed before entering the handler
Below is the same procedure with an explicit exception statement enclosedin a block without a WHEN-handler
As you see no savepoint frame near assignment operator wascreated Therefore the result would be similar to the previous one
Below is the same procedure with an explicit exception call in theblock which contains WHEN-handler
CREATE PROCEDURE PROC1AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES
(01)-- end savepoint 2X = 1 0
WHEN ANY DOEXIT
-- end savepoint 1END
CREATE PROCEDURE PROC2AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (23)-- end savepoint 2BEGIN
X = 1 0END
WHEN ANY DOEXIT
-- end savepoint 1END
CREATE PROCEDURE PROC3AS
DECLARE VARIABLE X INT-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (45)-- end savepoint 2-- start savepoint 3BEGIN
X = 1 0WHEN ANY DO
EXIT-- end savepoint 3
ENDWHEN ANY DO
EXIT-- end savepoint 1END Here we see a created savepoint frame As a result rollback is performed only for thenearest BEGIN level (savepoint 3) and the INSERT operator remained executed
Example 2A procedure with error handler and explicit exception call
CREATE PROCEDURE PROC4AS-- start savepoint 1BEGIN
-- start savepoint 2INSERT INTO TAB (COL) VALUES (67)-- end savepoint 2-- start savepoint 3EXCEPTION E-- end savepoint 3
WHEN ANY DOEXIT
-- end savepoint 1END
Cover story
6
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
In this case INSERT will not be canceleddue to the fact that the exception initiationof E results in rollback to savepoint 3 andsubsequent transfer of control to the errorhandler In order to cancel the INSERToperator in this case you should inhibit theexecution of the deletion handler
In addition a few words aboutexception handling adequacyregarding the SQL-standard
The standard allows three types ofhandlers in PSQL CONTINUEEXIT and UNDO With a CON-TINUE-handler the server mustrollback the erroneous operatorexecute the handler code and thencontinue the execution of the blockbeginning with the operator next tothe which one caused the error
An EXIT-handler requires finishingof the execution of the block rightafter exiting the handler code
An UNDO-handler requires a roll-back of all actions of the blockbefore entering the handler
Current versions of the server(InterBase as well as Firebird) donot support the explicit specifica-tion of the handler type and workaccording to the EXIT principle(however there is a possibility ofUNDO-behavior due to the anom-aly described above)
I suppose that in the future it
would be desirable to provide analternative to choose betweenUNDO- and EXIT- behavior of ahandler and repair the describedanomaly
Custom savepointsIn addition to the internalrealization of savepoints attransaction (and opera-torblock) levels the latestversions of servers (Inter-Base 71 Firebird 15 andYaffil 11) provide an SQL-interface developed for
this mechanism
Note savepointsrsquo syntax andsemantics are declared in the SQL-99 standard (see section 4371 ofthe specification)
Custom savepoints (also known asnested transactions) provide a con-venient business logic error han-dling method with no need to roll-back the whole transaction
Note rollback to a savepoint isalso sometimes called ldquopartialtransaction rollbackrdquo
New SQL operator (SAVEPOINT)was added to define a savepoint inthe transaction context to which arollback can be performed lateron
CREATE PROCEDURE PROC5ASBEGIN
INSERT INTO TAB (COL) VALUES (89)EXCEPTION E
END
a transaction If you attempt to createtwo savepoints with similar names thefirst savepoint is deleted and the speci-fied name is given to the second one
For rollback to a savepoint the follow-ing operator is used
SAVEPOINT ltnamegtltnamegt - the string identifier of a save-point As soon as a savepoint is creat-ed you can either continue transactioncommit (or cancel) the whole transac-tion or perform a rollback to a particu-lar savepoint Savepointsrsquo names (iden-tifiers) must be unique in the context of
Note the SAVEPOINT keyword isobligatory in InterBase 71
During execution of this operator thefollowing actions are performed
bullRollback of all changes made afterthe savepoint was set
bullAll savepoints set after this one aredeleted The current savepoint remainsunchanged and thus you can performseveral rollbacks to a savepoint Previ-ous savepoints remain unchanged aswell
Note Performing a rollback to save-point in InterBase 71 deletes the select-ed savepoint
bullAll explicit and implicit write locksoccupied after the savepoint was setare released At that other transac-tions which requested an access to therecords blocked by the transaction afterthe savepoint was set continue waitingfor the current transaction to be fin-ished Transactions which did notrequest access to the records may con-tinue and get access to them
Note This behavior refers to Firebird15 and can be changed in higher ver-sions
Since each savepoint uses certain sys-tem resources and also clogs thenamespace it makes sense to release
(delete) savepoints when they are no longer necessary This can be accom-plished using the following operator
RELEASE SAVEPOINT ltnamegt [ONLY]This command deletes the selected (and all following) savepoints fromtransaction context The ldquoONLYrdquo option is a switch to delete the selectedsavepoint only at that all following savepoints will be saved If a savepoint
was not released explicitly it will be automatically deleted assoon as the transaction is finished
Note The ldquoONLYrdquo option is non-standard extension and isnot supported by InterBase 71
ROLLBACK [WORK] TO [SAVEPOINT] ltnamegt
FastReport 3 - new generation of the reporting tools
Visual report designer with rulers guides andzooming wizards for base type reports exportfilters to html tiff bmp jpg xls pdf Dot matrix
reports support support most popular DB-engines
Full WYSIWYG text rotation 0360 degreesmemo object supports simple html-tags
(font color b i u sub sup) improved stretching(StretchMode ShiftMode properties)
Access to DB fields styles text flow URLs Anchors
httpwwwfast-reportcomenreadersphp5 discount for all our readers
Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers
At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail
First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint
It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set
through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71
A question emerges how would customsavepoints coexist with internal save-points managed by the server
Cover story
7
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title
A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation
Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative
In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO
LOOPTRY
SAVEPOINT DOClthellipgt single document handling commands
EXCEPTROLLBACK TO SAVEPOINT DOC
lthellipgt either log the error or display it
END END
CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name
Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)
START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY
lthellipgt operations on databaseEXCEPT
ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID
ENDCOMMIT
Cover story
8
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation
CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )
ASBEGIN
BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)
VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO
BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID
ENDEND
END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here
BEGININSERT INTO hellip
WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id
END
This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution
Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN
SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip
INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
WHEN ANY DOBEGIN
ROLLBACK TO SAVEPOINTS1
ERROR = REC_IDSUSPEND
ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well
FOR SELECT ID hellip INTOREC_ID hellipBEGIN
BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
ENDWHEN ANY DO
BEGINERROR = REC_IDSUSPEND
ENDEND
In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler
Therefore in virtually any case one
can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks
A bit of practice
Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear
Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as
SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1
work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above
1 savepoints constitute a stack andcan be canceled sequentially only
2 each block of PSQL-code with anexception handler is enclosed in a frame
Thus we arrive at a conclusion that anycode area of the following type
SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1
hellipWHEN
is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base
Thank God the InterBase developersdid not create such cardinal realization
and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in
this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler
However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample
Statement failed SQLCODE = -504Savepoint ltnamegt unknown
Cover story
9
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip
END
This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode
INSERT INTO TABLE1 hellipBEGIN
ROLLBACK TO SAVEPOINT S1WHEN ANY DO
EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon
SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is
ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)
neither available in Firebird nor in Yaffil
Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed
Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions
For this purpose three new API functions are introduced
As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to
Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)
Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)
Create savepoint isc_start_savepoint(status amptrans A)
Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)
Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)
all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)
stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316
Thus we can make up several conclu-sions
bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)
bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead
bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0
bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place
bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results
Tips
10
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Commit coordinated transaction isc_commit_transaction (status amptrans)
Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)
As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed
And that would be that way but forthe manually executed RELEASESAVEPOINT A
At first savepoint rollback was per-formed for the first connection and allchanges were canceled
Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible
The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints
That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process
Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole
Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API
Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means
The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability
It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature
InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used
ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain
[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]
As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source
The following experiment can be performed Create a table with structure as shownbelow
(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)
Type conversion throughCOMPUTED BY
CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))
Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be
AuthorIBSurgeon Research Labs researchib-aidcom
Database maintenance
11
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Embedded User Authenticationin InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version
Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database
In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base
1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server
2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one
3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)
As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted
This scheme is insufficient when used in
bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib
bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase
bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work
Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to
manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)
This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages
bullproduct DATABASE ACCESS is notlicensed
for databases in which EUA is enabledor disabled
bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)
for databases in which EUA was neverenabled
In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)
continued on page 12
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Cover story
6
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
In this case INSERT will not be canceleddue to the fact that the exception initiationof E results in rollback to savepoint 3 andsubsequent transfer of control to the errorhandler In order to cancel the INSERToperator in this case you should inhibit theexecution of the deletion handler
In addition a few words aboutexception handling adequacyregarding the SQL-standard
The standard allows three types ofhandlers in PSQL CONTINUEEXIT and UNDO With a CON-TINUE-handler the server mustrollback the erroneous operatorexecute the handler code and thencontinue the execution of the blockbeginning with the operator next tothe which one caused the error
An EXIT-handler requires finishingof the execution of the block rightafter exiting the handler code
An UNDO-handler requires a roll-back of all actions of the blockbefore entering the handler
Current versions of the server(InterBase as well as Firebird) donot support the explicit specifica-tion of the handler type and workaccording to the EXIT principle(however there is a possibility ofUNDO-behavior due to the anom-aly described above)
I suppose that in the future it
would be desirable to provide analternative to choose betweenUNDO- and EXIT- behavior of ahandler and repair the describedanomaly
Custom savepointsIn addition to the internalrealization of savepoints attransaction (and opera-torblock) levels the latestversions of servers (Inter-Base 71 Firebird 15 andYaffil 11) provide an SQL-interface developed for
this mechanism
Note savepointsrsquo syntax andsemantics are declared in the SQL-99 standard (see section 4371 ofthe specification)
Custom savepoints (also known asnested transactions) provide a con-venient business logic error han-dling method with no need to roll-back the whole transaction
Note rollback to a savepoint isalso sometimes called ldquopartialtransaction rollbackrdquo
New SQL operator (SAVEPOINT)was added to define a savepoint inthe transaction context to which arollback can be performed lateron
CREATE PROCEDURE PROC5ASBEGIN
INSERT INTO TAB (COL) VALUES (89)EXCEPTION E
END
a transaction If you attempt to createtwo savepoints with similar names thefirst savepoint is deleted and the speci-fied name is given to the second one
For rollback to a savepoint the follow-ing operator is used
SAVEPOINT ltnamegtltnamegt - the string identifier of a save-point As soon as a savepoint is creat-ed you can either continue transactioncommit (or cancel) the whole transac-tion or perform a rollback to a particu-lar savepoint Savepointsrsquo names (iden-tifiers) must be unique in the context of
Note the SAVEPOINT keyword isobligatory in InterBase 71
During execution of this operator thefollowing actions are performed
bullRollback of all changes made afterthe savepoint was set
bullAll savepoints set after this one aredeleted The current savepoint remainsunchanged and thus you can performseveral rollbacks to a savepoint Previ-ous savepoints remain unchanged aswell
Note Performing a rollback to save-point in InterBase 71 deletes the select-ed savepoint
bullAll explicit and implicit write locksoccupied after the savepoint was setare released At that other transac-tions which requested an access to therecords blocked by the transaction afterthe savepoint was set continue waitingfor the current transaction to be fin-ished Transactions which did notrequest access to the records may con-tinue and get access to them
Note This behavior refers to Firebird15 and can be changed in higher ver-sions
Since each savepoint uses certain sys-tem resources and also clogs thenamespace it makes sense to release
(delete) savepoints when they are no longer necessary This can be accom-plished using the following operator
RELEASE SAVEPOINT ltnamegt [ONLY]This command deletes the selected (and all following) savepoints fromtransaction context The ldquoONLYrdquo option is a switch to delete the selectedsavepoint only at that all following savepoints will be saved If a savepoint
was not released explicitly it will be automatically deleted assoon as the transaction is finished
Note The ldquoONLYrdquo option is non-standard extension and isnot supported by InterBase 71
ROLLBACK [WORK] TO [SAVEPOINT] ltnamegt
FastReport 3 - new generation of the reporting tools
Visual report designer with rulers guides andzooming wizards for base type reports exportfilters to html tiff bmp jpg xls pdf Dot matrix
reports support support most popular DB-engines
Full WYSIWYG text rotation 0360 degreesmemo object supports simple html-tags
(font color b i u sub sup) improved stretching(StretchMode ShiftMode properties)
Access to DB fields styles text flow URLs Anchors
httpwwwfast-reportcomenreadersphp5 discount for all our readers
Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers
At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail
First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint
It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set
through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71
A question emerges how would customsavepoints coexist with internal save-points managed by the server
Cover story
7
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title
A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation
Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative
In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO
LOOPTRY
SAVEPOINT DOClthellipgt single document handling commands
EXCEPTROLLBACK TO SAVEPOINT DOC
lthellipgt either log the error or display it
END END
CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name
Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)
START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY
lthellipgt operations on databaseEXCEPT
ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID
ENDCOMMIT
Cover story
8
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation
CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )
ASBEGIN
BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)
VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO
BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID
ENDEND
END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here
BEGININSERT INTO hellip
WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id
END
This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution
Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN
SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip
INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
WHEN ANY DOBEGIN
ROLLBACK TO SAVEPOINTS1
ERROR = REC_IDSUSPEND
ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well
FOR SELECT ID hellip INTOREC_ID hellipBEGIN
BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
ENDWHEN ANY DO
BEGINERROR = REC_IDSUSPEND
ENDEND
In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler
Therefore in virtually any case one
can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks
A bit of practice
Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear
Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as
SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1
work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above
1 savepoints constitute a stack andcan be canceled sequentially only
2 each block of PSQL-code with anexception handler is enclosed in a frame
Thus we arrive at a conclusion that anycode area of the following type
SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1
hellipWHEN
is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base
Thank God the InterBase developersdid not create such cardinal realization
and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in
this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler
However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample
Statement failed SQLCODE = -504Savepoint ltnamegt unknown
Cover story
9
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip
END
This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode
INSERT INTO TABLE1 hellipBEGIN
ROLLBACK TO SAVEPOINT S1WHEN ANY DO
EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon
SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is
ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)
neither available in Firebird nor in Yaffil
Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed
Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions
For this purpose three new API functions are introduced
As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to
Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)
Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)
Create savepoint isc_start_savepoint(status amptrans A)
Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)
Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)
all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)
stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316
Thus we can make up several conclu-sions
bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)
bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead
bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0
bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place
bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results
Tips
10
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Commit coordinated transaction isc_commit_transaction (status amptrans)
Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)
As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed
And that would be that way but forthe manually executed RELEASESAVEPOINT A
At first savepoint rollback was per-formed for the first connection and allchanges were canceled
Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible
The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints
That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process
Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole
Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API
Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means
The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability
It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature
InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used
ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain
[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]
As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source
The following experiment can be performed Create a table with structure as shownbelow
(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)
Type conversion throughCOMPUTED BY
CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))
Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be
AuthorIBSurgeon Research Labs researchib-aidcom
Database maintenance
11
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Embedded User Authenticationin InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version
Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database
In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base
1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server
2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one
3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)
As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted
This scheme is insufficient when used in
bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib
bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase
bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work
Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to
manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)
This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages
bullproduct DATABASE ACCESS is notlicensed
for databases in which EUA is enabledor disabled
bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)
for databases in which EUA was neverenabled
In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)
continued on page 12
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Savepoints in stored procedures and triggersNow let us consider the usage of custom savepoints in procedures and triggers
At first glance it looks very attractive and useful Originally this functionality is declaredin InterBase 71 Letrsquos consider the capabilities in detail
First of all savepoints must not break the atomicity of SQL-operators This means thatnone of the commands can be canceled partially Remember that EXECUTE PROCE-DURE is a legal SQL-operator and any operators updates may lead to trigger execu-tion Generally speaking any ldquosimplerdquo operator such as INSERT or UPDATE may resultin launching of a whole chain of triggers and procedures That is why we are to examinethe scope of a savepoint
It is obvious that in order to meet the atomicity requirement savepoint affected instruc-tions with in a procedure should not have an access to the transaction savepoint (set
through the SAVEPOINT global opera-tor) In addition procedure savepointsmust be local and their scope must bedefined by the procedure That is to saythere can be a savepoint named ldquoS1rdquoboth in the transaction and in proceduresand triggers executed in the context of thistransaction At that such savepoints willbe isolated from each other Note that thisvery method is used in InterBase 71
A question emerges how would customsavepoints coexist with internal save-points managed by the server
Cover story
7
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
InterBase Myths 2Record versions are created during readingVersions are created only when records are changed ordeleted (UPDATE or DELETE) On the contrary ifunnecessary versions of the same record are detectedthey are marked as trash (ie they would be deleted) Readas much as you want this will not result in creation of new versionsVice versa updating of a record creates a new version of the record inany case disregarding the fact that someone else reads this recordA myth sometimes becomes true versions of the records may be cre-ated during reading but only in Firebird 15 (and higher) in queriesselect for update with lock More detailed information will be pub-lished in the next issue of full title
A simple example of working with savepoints is given belowcreate table test (id int)commitinsert into test (id) values (1)commitinsert into test (id) values (2)savepoint ydelete from testselect from test -- returns empty setrollback to yselect from test -- returns two recordsrollbackselect from test -- returns one recordA custom savepointNow let us consider an example of how savepoints can be used in business logicAssume there is an operation of mass document handling in the application andit is necessary to display error messages (or save them for future presentation asa list) and let this bulk operation continue Since the document handling opera-tion is not atomic on the clientrsquos side it is better not to use regular exception han-dling since we cannot continue the transaction if we know that an exception per-formed a rollback of only half of the operation
Such a dilemma can be resolved by handling each document sequentially in aseparate transaction Nevertheless this does increase the consumption of inter-nal server resources (maximum number of records in TIP transaction counterincrement) and is therefore not the best alternative
In addition if there is a need to fix a set of documents during the handling process(for example by changing the transaction isolation mode or explicit blocking ofthe SELECT hellip WITH LOCK type) it would require using only one transaction forthe delta packet Using a savepoint the following algorithm would be used (inpseudocode)START TRANSACTION OPEN C FOR ( SELECT hellip )FOR ( C ) DO
LOOPTRY
SAVEPOINT DOClthellipgt single document handling commands
EXCEPTROLLBACK TO SAVEPOINT DOC
lthellipgt either log the error or display it
END END
CLOSE CCOMMITNote The use of savepoints in loopshas an additional advantage you donot need to call RELEASE each timesince resetting a savepoint automatical-ly deletes the previous savepoint withthe same name
Another example is undoubtedlyaudit For example you need to pro-vide a log record for each activity andat the same time if an error occurs therecord should remain in the audit log(with a corresponding note)
START TRANSACTION INSERT INTO AUDIT_LOG (ID EVENT STATUS) VALUES (ID EVENT 1)SAVEPOINT OPERTRY
lthellipgt operations on databaseEXCEPT
ROLLBACK TO SAVEPOINT OPERUPDATE AUDIT_LOG SET STATUS = 0 WHERE ID = ID
ENDCOMMIT
Cover story
8
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation
CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )
ASBEGIN
BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)
VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO
BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID
ENDEND
END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here
BEGININSERT INTO hellip
WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id
END
This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution
Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN
SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip
INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
WHEN ANY DOBEGIN
ROLLBACK TO SAVEPOINTS1
ERROR = REC_IDSUSPEND
ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well
FOR SELECT ID hellip INTOREC_ID hellipBEGIN
BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
ENDWHEN ANY DO
BEGINERROR = REC_IDSUSPEND
ENDEND
In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler
Therefore in virtually any case one
can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks
A bit of practice
Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear
Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as
SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1
work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above
1 savepoints constitute a stack andcan be canceled sequentially only
2 each block of PSQL-code with anexception handler is enclosed in a frame
Thus we arrive at a conclusion that anycode area of the following type
SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1
hellipWHEN
is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base
Thank God the InterBase developersdid not create such cardinal realization
and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in
this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler
However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample
Statement failed SQLCODE = -504Savepoint ltnamegt unknown
Cover story
9
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip
END
This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode
INSERT INTO TABLE1 hellipBEGIN
ROLLBACK TO SAVEPOINT S1WHEN ANY DO
EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon
SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is
ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)
neither available in Firebird nor in Yaffil
Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed
Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions
For this purpose three new API functions are introduced
As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to
Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)
Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)
Create savepoint isc_start_savepoint(status amptrans A)
Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)
Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)
all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)
stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316
Thus we can make up several conclu-sions
bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)
bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead
bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0
bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place
bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results
Tips
10
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Commit coordinated transaction isc_commit_transaction (status amptrans)
Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)
As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed
And that would be that way but forthe manually executed RELEASESAVEPOINT A
At first savepoint rollback was per-formed for the first connection and allchanges were canceled
Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible
The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints
That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process
Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole
Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API
Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means
The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability
It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature
InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used
ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain
[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]
As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source
The following experiment can be performed Create a table with structure as shownbelow
(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)
Type conversion throughCOMPUTED BY
CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))
Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be
AuthorIBSurgeon Research Labs researchib-aidcom
Database maintenance
11
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Embedded User Authenticationin InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version
Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database
In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base
1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server
2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one
3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)
As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted
This scheme is insufficient when used in
bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib
bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase
bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work
Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to
manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)
This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages
bullproduct DATABASE ACCESS is notlicensed
for databases in which EUA is enabledor disabled
bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)
for databases in which EUA was neverenabled
In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)
continued on page 12
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Cover story
8
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Some theoryLet us consider a simple example of savepoint usage in PSQL suggested byBorland in the InterBase 71 server documentation
CREATE PROCEDURE ADD_EMP_PROJ2 (EMP_NO SMALLINTEMP_NAME VARCHAR(20)PROJ_ID CHAR(5) )
ASBEGIN
BEGINSAVEPOINT EMP_PROJ_INSERTINSERT INTO EMPLOYEE_PROJECT (EMP_NO PROJ_ID)
VALUES (EMP_NO PROJ_ID)WHEN SQLCODE -530 DO
BEGINROLLBACK TO SAVEPOINT EMP_PROJ_INSERTEXCEPTION UNKNOWN_EMP_ID
ENDEND
END This example demonstrates how exceptional situations are handled whenusing savepoints That is to say when an exception with code -530 occurs(the violation of reference integrity by a foreign key) we cancel the insertoperation and initiate a user exception Actually this example is absolute-ly useless since we do not need a savepoint here
BEGININSERT INTO hellip
WHEN SQLCODE ndash530 DOEXCEPTION unknown_emp_id
END
This code will execute the same function since the server itself cancels theINSERT operation when an exception during its execution
Letrsquos consider a more complicated exampleFOR SELECT ID hellip INTO REC_ID hellipBEGIN
SAVEPOINT S1INSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellip
INSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
WHEN ANY DOBEGIN
ROLLBACK TO SAVEPOINTS1
ERROR = REC_IDSUSPEND
ENDENDHere we try to handle all documentsbut the program does not stop in caseof failure it only returns all unsuccessfulattempts at the end of the procedureServer standard logic would cancel theerror operator and control to the han-dler which in its turn cancels actions ofthe whole block Thus we can easilyturn the EXIT-handler to UNDO in caseif necessary Of course this functionali-ty can be obtained by standard meansas well
FOR SELECT ID hellip INTOREC_ID hellipBEGIN
BEGININSERT INTO TABLE1 hellipINSERT INTO TABLE2 hellipINSERT INTO TABLE3 hellipEXECUTE PROCEDURE helliphellip
ENDWHEN ANY DO
BEGINERROR = REC_IDSUSPEND
ENDEND
In this case all operators within theloop will be automatically canceled inthe event that an exception occurssince the operators are located in theatomic block by which the savepointframe for SQL-operators mechanismwas enabled After that the server willgo through the chain of embeddedblocks and will switch to the handler
Therefore in virtually any case one
can realize the same semantics usingthe servers standard mechanisms ieusing system savepoints instead of cus-tom ones at the cost of relativelyunhandy source code Thus savepointsin PSQL are nothing but an easy andcomprehensive alternative for the explic-it usage of BEGINhellipWHENhellipEND blocks
A bit of practice
Now let us return from theory to prac-tice and test this reasoning in InterBase71 The result is quite depressing none() of the given examples work anderror messages appear
Even the first example which was takenfrom the Release Notes () is not work-ing properly At the same time the mostprimitive examples such as
SAVEPOINT S1INSERT hellipROLLBACK TO SAVEPOINT S1
work correctly So whatrsquos the matter Ifwe investigate the situation more careful-ly the reason becomes obvious Remem-ber the two facts described above
1 savepoints constitute a stack andcan be canceled sequentially only
2 each block of PSQL-code with anexception handler is enclosed in a frame
Thus we arrive at a conclusion that anycode area of the following type
SAVEPOINT S1hellipBEGINhellipROLLBACK TO SAVEPOINT S1
hellipWHEN
is definitely invalid since to perform arollback to savepoint S1 it would benecessary to delete the system save-point created by the server for excep-tion handling in the ldquoBEGINhellipENDrdquoblock This would destroy the internalundo log and may corrupt the data-base
Thank God the InterBase developersdid not create such cardinal realization
and server attempts to cancel theprevious (last) savepoint directlyonly if its name matches Since sys-tem savepoints are unnamed in
this case such an attempt would failThis is proven by the above mentionederror messageThe above makes usarrive to the conclusion that working-with savepoints in PSQL is limited by thenesting level in case we are dealingwith blocks with a WHEN-handler
However it turned out that the mostinteresting thing is yet to come Theserverrsquos reaction to the error initiatedby the ROLLBACK TO SAVEPOINT orRELEASE SAVEPOINT operator isamusing Letrsquos illustrate this using anexample
Statement failed SQLCODE = -504Savepoint ltnamegt unknown
Cover story
9
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip
END
This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode
INSERT INTO TABLE1 hellipBEGIN
ROLLBACK TO SAVEPOINT S1WHEN ANY DO
EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon
SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is
ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)
neither available in Firebird nor in Yaffil
Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed
Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions
For this purpose three new API functions are introduced
As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to
Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)
Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)
Create savepoint isc_start_savepoint(status amptrans A)
Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)
Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)
all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)
stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316
Thus we can make up several conclu-sions
bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)
bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead
bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0
bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place
bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results
Tips
10
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Commit coordinated transaction isc_commit_transaction (status amptrans)
Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)
As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed
And that would be that way but forthe manually executed RELEASESAVEPOINT A
At first savepoint rollback was per-formed for the first connection and allchanges were canceled
Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible
The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints
That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process
Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole
Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API
Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means
The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability
It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature
InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used
ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain
[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]
As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source
The following experiment can be performed Create a table with structure as shownbelow
(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)
Type conversion throughCOMPUTED BY
CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))
Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be
AuthorIBSurgeon Research Labs researchib-aidcom
Database maintenance
11
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Embedded User Authenticationin InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version
Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database
In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base
1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server
2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one
3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)
As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted
This scheme is insufficient when used in
bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib
bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase
bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work
Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to
manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)
This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages
bullproduct DATABASE ACCESS is notlicensed
for databases in which EUA is enabledor disabled
bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)
for databases in which EUA was neverenabled
In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)
continued on page 12
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Cover story
9
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
BEGININSERT INTO TABLE1 hellipROLLBACK TO SAVEPOINT S1INSERT INTO TABLE2 hellip
END
This is an emulation of an error whichusually occurs if the required save-point cannot be found within a singlecode block As one would expect theexecution of the procedure returns thesame error But Procedure executiondoes not stop at this point Instead thesecond INSERT is executed (which youcan easily verify by substitutingINSERT with an operator of EXCEP-TION E_TEST type) The question iswhy It turns out that this error cannotbe handled in the procedure ie thecode
INSERT INTO TABLE1 hellipBEGIN
ROLLBACK TO SAVEPOINT S1WHEN ANY DO
EXCEPTION E_TESTENDdoes not throw the E_TEST exceptionas one might expect Even though thecode after ROLLBACK TO SAVE-POINT is executed nothing reallyhappens Which means that in casethe described error occurs in a proce-dure all changes made by this proce-dure will be unconditionally () can-celed This happens regardless ofwhich code was executed before orafter the command It would be inter-esting to find out how InterBase devel-opers explain this phenomenon
SummaryThere are some peculiarities in save-points logic which prevent realizationof their complete support by PSQL Theanalysis of InterBase 71 behaviorproves the point The rationale of that isthe presence of system savepointswhich interaction with custom ones islimited due to data integrity require-ments Thats why this functionality is
ISC_STATUS isc_start_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_release_transaction(ISC_STATUS status isc_tr_handle trans char name)
ISC_STATUS isc_rollback_transaction(ISC_STATUS status isc_tr_handle trans char name short option)
neither available in Firebird nor in Yaffil
Note As far as I understand it the same reasons prevent from using commitroll-back retaining in PSQL since in that case the savepoint-frame of a procedure wouldbe destroyed
Savepoint in distributed transactions InterBase 71 introduces the option to work with savepoints in coordinated transac-tions
For this purpose three new API functions are introduced
As you see these functions do nothave a connection descriptor (data-base handle) which means that corre-sponding SQL-commands are issued to
Connect to the database isc_attach_database(status 0 database1 ampdb1 0 NULL)isc_attach_database(status 0 database2 ampdb2 0 NULL)
Begin coordinated transaction isc_start_transaction(status amptrans 2 ampdb1 0 NULL ampdb2 0 NULL)
Create savepoint isc_start_savepoint(status amptrans A)
Executing database operations isc_dsql_execute_immediate(status ampdb1 amptrans 0 DELETE FROM TABLE1 1 NULL)isc_dsql_execute_immediate(status ampdb2 amptrans 0 DELETE FROM TABLE2 1 NULL) Delete the savepoint explicitly through the second connection descriptor isc_dsql_execute_immediate(status ampdb2 amptrans 0 RELEASE SAVEPOINT A 1 NULL)
Rollback to savepoint isc_rollback_savepoint(status amptrans A 0)
all databases used by the transaction This seems absolutely logical since formal-ly a savepoint is a part of a transaction and not of a connection However thereis one nuance here Letrsquos examine following program fragment (error handling isnot included)
stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316
Thus we can make up several conclu-sions
bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)
bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead
bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0
bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place
bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results
Tips
10
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Commit coordinated transaction isc_commit_transaction (status amptrans)
Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)
As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed
And that would be that way but forthe manually executed RELEASESAVEPOINT A
At first savepoint rollback was per-formed for the first connection and allchanges were canceled
Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible
The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints
That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process
Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole
Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API
Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means
The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability
It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature
InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used
ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain
[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]
As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source
The following experiment can be performed Create a table with structure as shownbelow
(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)
Type conversion throughCOMPUTED BY
CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))
Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be
AuthorIBSurgeon Research Labs researchib-aidcom
Database maintenance
11
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Embedded User Authenticationin InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version
Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database
In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base
1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server
2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one
3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)
As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted
This scheme is insufficient when used in
bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib
bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase
bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work
Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to
manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)
This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages
bullproduct DATABASE ACCESS is notlicensed
for databases in which EUA is enabledor disabled
bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)
for databases in which EUA was neverenabled
In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)
continued on page 12
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
stored as integersrsquo one and thereforewhen one enters 188 in NUMERIC(152) it would look as 188 though actual-ly (as a string) will turn out to be18799999952316
Thus we can make up several conclu-sions
bullreal numbersrsquo accuracy is boundedand therefore numbers stored as realshould never be used in equality (every-one knows that)
bullprecision of the FLOAT fields is quiteshort (similar to Delphirsquos single) That iswhy it is better to use DOUBLE PRECI-SION instead
bullnot all types are interconvertible theNUMERIC(15 2) field as INTEGERCOMPUTED BY will contain 0
bullinaccuracy should be taken intoaccount when processing real numbers(rounding aggregation comparisonadditionsubtraction and multiplica-tiondivision) Also do not forget thebookkeeperrsquos rule when multiplyingand dividing multiplication should becalculated in the first place
bullit is not recommended to use real typesas table primary keys Due to inaccuracyandor peculiarities of how clientrsquos andservers processors handle real numbersseemingly one and the same numbermay lead to different results
Tips
10
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Inside Savepoints
Commit coordinated transaction isc_commit_transaction (status amptrans)
Disconnecting the database isc_detach_database(status ampdb1)isc_detach_database(status ampdb2)
As a result of a rollback to savepointI expect rollback to be performed inboth databases I work with Then Icommit the transaction and after thatI need to see all data on their placessince DELETE operators were can-celed
And that would be that way but forthe manually executed RELEASESAVEPOINT A
At first savepoint rollback was per-formed for the first connection and allchanges were canceled
Then the same operation was accom-plished for second connection whilehellipoops hellip there is no savepoint any-more As a result the client receivesan error message But rollback of oneof the DELETE operators was success-ful () This is a situation when thecoordinated operation disintegratesitself and makes correct handling ofthe case impossible
The two-phase fixation of transactionmechanism which should bar fromsuch cases simply cannot deal withsavepoints
That is to say the new InterBase 71functions create appropriate SQL-commands and then cyclically exe-cute them for databases involved inthis process
Even if only one of them failsan error returns Generallythis error does not character-ize the current situation interms of correctness of theoperation as a single whole
Of course one might say thatonly one method of workingwith savepoints should beapplied ndash either throughSQL or through API
Now you know what it leadsto However for workingwith savepoints in InterBase71 a new API can be com-pletely substituted by serverrsquosstandard means
The server should either sup-press possibility to controlsavepoints in certain connec-tions in cases of coordinatedtransactions or refuse todeclare their workability
It is necessary to note thatFirebirdrsquos and Yaffilrsquos devel-opers have chosen this waypreferring not to provideusers with such an ambiguousfeature
InterBase has an interesting undocumented feature Usually when the COMPUTEDBY field is declared the following syntax is used
ltcol_defgt = col datatype | COMPUTED [BY] (lt exprgt) |domain
[DEFAULT literal | NULL | USER] [NOT NULL] [ ltcol_constraintgt] [COLLATE collation]
As you see syntax requires specifying either type of the column (datatype) or thecalculated expression (computed by) The eitheror directive is the | symbol Usu-ally type of the COMPUTED BY field is similar to the source one (which it is basedon) However it is possible to specify column type even if it would not coincide withthe source
The following experiment can be performed Create a table with structure as shownbelow
(the PRIMARY KEY definition may be omitted since in this case it is used only forDatabase Explorer tablesrsquo usability BDE does not allow update tables which donot have a primary key)
Type conversion throughCOMPUTED BY
CREATE TABLE TESTCOMP( t_data FLOAT NOT NULL PRIMARY KEY c_int INTEGER computed by (t_data) c_num NUMERIC(15 2) computed by (t_data) c_char CHAR(20) computed by (t_data))
Now in the table try to enter a record with the following T_DATA value 188 32351 (this test was done in dialect 1) You would see that the values the FLOAT fieldstores on disk differ from what you have entered The C_INT field contains the round-ed value of T_DATA The C_NUM field would contain either exact or rounded valueof T_DATA It depends on the parameter value of the BDE ENABLE BCD =TRUEFALSE alias At the same time C_CHAR would contain more precise value ofthe C_DATA real number When doing this trick it would be helpful to view theNUMERIC(15 2) values as strings The thing is that real numbersrsquo accuracy cannot be
AuthorIBSurgeon Research Labs researchib-aidcom
Database maintenance
11
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Embedded User Authenticationin InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version
Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database
In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base
1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server
2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one
3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)
As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted
This scheme is insufficient when used in
bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib
bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase
bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work
Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to
manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)
This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages
bullproduct DATABASE ACCESS is notlicensed
for databases in which EUA is enabledor disabled
bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)
for databases in which EUA was neverenabled
In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)
continued on page 12
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Database maintenance
11
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Embedded User Authenticationin InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
One of the most important new features in InterBase 75 is userauthentication which is embedded in the database Letrsquos con-sider the scheme used in the previous version
Standard schemeIn InterBase a separate special database isc4gdb is intended for user liststoring In InterBase 70 this database was renamed (adminib) and inaddition new ibconfig parameter was added (ADMIN_DB) which allowsto specify any name for this database
In isc4gdbadminib there is a basic ldquoUSERSrdquo table which contains user-name password and other parameters When a client connects to a data-base
1 the front-end encrypts the password by DES algorithm with data lossand then sends the username and encrypted password to the server
2 the server encrypts the received password once again by the same lossyDES and then calls isc4admin in the ldquoUSERSrdquo table finds the necessaryuser and then verifies the received password with the stored one
3 if the passwords are equal the userconnects to the database heshe speci-fied And if they do not the user isunable to connect to the database(wrong user name or password errorreported)
As you can see to access any databaseon this server a user must be specifiedin isc4admin only once In the futurein a particular database user access isdefined by the rights he is granted
This scheme is insufficient when used in
bullsingle-user applications It becomesnecessary to deploy both the databaseand adminib
bulldeployed or stolen databases Any-one can slip hisher own ldquoadminibrdquowith SYSDBAmasterkey to the serverand as a result completely control adatabase
bullsystems in which a user has to con-nect to only those databases withwhich one is allowed to work
Embedded UserAuthentication In InterBase 75 you can either refuseusing adminib (see below) or combineadminib with user control in the data-base For that purpose attributes ofseveral system tables were extendedand new SQL-operators were added to
manage this functionality (in ldquogsecrdquo aldquouser_databaserdquo option is added foruser management in such databases)
This functionality is supported only forODS 112 ie for the databases creat-ed or restored from backup in InterBase75 At that previous versions of Inter-Base for example 71 and belowwhen attempting to connect to suchdatabase will return two types of mes-sages
bullproduct DATABASE ACCESS is notlicensed
for databases in which EUA is enabledor disabled
bullinternal gds software consistencycheck (decompression overran buffer(179) file sqzc line 229)
for databases in which EUA was neverenabled
In other words there is no other way toconnect to the database but from Inter-Base 75 specifying a required pass-word (stored in the database) for a spe-cific user (we do not consider the possi-bility of ldquohackingrdquo such database ieediting it in HEX-editor)
continued on page 12
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Draw attention to the fact that one can ldquoenablerdquo and ldquodisablerdquo users by the alteruser xxx set inactiveactive command There is no such possibility in the standardadminib
Authentication orderIt is important to comment how exactly connections are performed in case EUA isactive in a database
bullThe server opens a specific database
1 EUA disabled ndash user authentication is accomplished from adminib
2 EUA enabled - user authentication (any user including SYSDBA) is accom-plished from rdb$users of this particular database
That is when enabling SYSDBA and changing password for SYSDBA it will be pos-sible to connect to this database under the ldquoSYSDBArdquo name only if the user speci-fies a correct password
Attention Adminib is mandatory in any case The server when trying to connectto a database requires presence of adminib regardless of whether database EUAis enabled or not
Database maintenance
12
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
Enabling EUAThere are two ways of enabling EUA in a database
1 When creating a database specify an extra option ndash WITH ADMIN OPTION ndashin CREATE DATABASE
2 For any ODS 112 database enter the following operator
ALTER DATABASE SET ADMIN OPTION INACTIVEand activated
ALTER DATABASE SET ADMIN OPTION ACTIVE During deactivation in RDB$USERS field RDB$USER_ACTIVE is set to N for alluser records (including SYSDBA) When activating it is performed converselyRDB$USER_ACTIVE is entered to Y for all users Doing that be careful since ifsome users were disabled before EUA deactivation as soon as EUA is activated allEUA users will be able to access the database (ie all EUA accounts will beenabled)
You can completely delete EUA together with all user records by the command
ALTER DATABASE DROP ADMIN OPTION This will clear the RDB$USERS table and restore functioning of the standardauthentication scheme (through adminib)
User managementIf EUA is enabled you can manage users
CREATE | ALTER USER SET
option PASSWORD [NO] DEFAULT ROLE
[NO] SYSTEM USER NAME [NO] GROUP NAME [NO] UID [NO] GID [NO] DESCRIPTION [NO] FIRST NAME [NO] MIDDLE NAME [NO] LAST NAME ACTIVEINACTIVE
Examples
CREATE USER TEST SET PASSWORD TEST NO LAST NAMEDEFAULT ROLE ABC
As a result a user TEST with ldquoTESTrdquo password will be created the LAST_NAME col-umn will be set NULL the default role will be ldquoABCrdquo (and rdb$user_privilege = 0ie not a database owner) The same can be performed by the following com-mand set
CREATE USER TEST SET PASSWORD TESTALTER USER TEST SET NO LAST NAME DEFAULT ROLE ABC
continued on page 13
ALTER DATABASE ADD ADMIN OPTION In any case among ODS 112 database system tables there always is theRDB$USERS table It is an equivalent to the USERS table from adminib(RDB$DEFAULT_ROLE RDB$USER_ACTIVE and RDB$USER_PRIVILEGE columnsare added)
When enabling EUA it becomes active right away and in the RDB$USERS table theSYSDBA user standard record appears (the password is encrypted ldquomasterkeyrdquo)and with RDB$USER_PRIVILEGE = 1 After that when connecting to a database theserver ignores presence (or absence) of the user in adminib as well as hisherpassword That is to say when EUA is enabled one can connect to a database onlyif usernamepassword combination stored in rdb$users is correctly specified
EUA can be temporarily deactivated by the command
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Database maintenance
13
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
InterBase Myths 3Database files (gdb) must be shared with usersNever do that InterBase ndash is not a file server andworks with databases independently A client onlyinforms the server which database heshe wants to workwith and what queries are to be executed
User schemes combiningThus in InterBase 75 two schemes ofuser management are supported stan-dard and EUA This allows building thefollowing schemes
1 standard all users included inadminib are allowed to access alldatabases Access rights to a specificdatabase are defined by grants
2 EUA the usernames for a particulardatabase are specified in this databaseonly Accordingly only these users canconnect to it
3 standard+EUA 1 SYSDBA every-where is the same (including pass-word) ie it administrates all databas-es on the server The server databasescan be divided into 2 sets the first setwithout EUA (public access fromadminib) and the second set with EUA(only the users specified in this particu-lar database are allowed to access it)
4 standard+EUA 2 All users are com-mon for all databases (if those forexample are copied from one source)but SYSDBA requires different pass-
words That is to say one SYSDBAmanages the databases which do nothave EUA while other SYSDBA con-trols the databases with EUA enabled
5 standard+EUA 3 SYSDBA user-names and passwords are different forall databases ndash with or without EUA
The picture illustrates an example ofhow two different users connect to dif-ferent databases
User 1 can connect to databases withEUA disabled To access DB1IB it isnecessary to create a new user(USER1) in this database and specifyeither the same or a different password(if needed)
User 2 can connect to DB1IB only Ifthis user is specified in ADMINIBheshe will be able to work with thedatabases in which EUA is disabled
BACKUPRESTOREAt the given moment in InterBase750174 the following behavior isdetected (there is no report about fixingthat problem in IB 75 SP 1)
After restore the rdb$user_privilege column of the rdb$users table has null valueEven though this is ldquounimportantrdquo for SYSDBA in cases when SYSDBA is not thedatabase owner (the owner is say the ldquoTESTrdquo user) that particular user as well asany other users cannot login to such database
The situation can be corrected if one logins to this database as SYSDBA setting ldquo1rdquovalue in the column instead of null for the database owner and ldquo0rdquo for all otherusers After this procedure EUArsquos workability will be restored
To date (16052005) in Borland it is considered as bug IB 750174
A workaround
disable EUA before backup (alter database set admin option inactive) after restoreis performed enable EUA (alter database set admin option active) However toavoid change of the owner (unless it is sysdba) there should be an owner of thedatabase with EUA in adminib
Other issuesSometimes for different purposes a database can be created by a user other thanSYSDBA for example in order to use a database owner as a backup user (at thatall objects are created and modified on behalf of SYSDBA and the owner cannotchange them) In this case a user who created the database is the database ownerand thus can perform backuprestore being an owner not only of the database butalso of all objects created by herhim There are several features of applying suchmethod when EUA is enabled
1 Create a database not as a ldquoSYSDBArdquo but as a ldquoTESTrdquo user As soon as aldquoTESTrdquo user is created it becomes a database owner At this point of course theTEST (with password ldquotestrdquo for example) user should be specified in adminib
2 Enable EUA in the database
ALTER DATABASE ADD ADMIN OPTION
In the RDB$USERS table a record about the TEST user with ldquotestrdquo password appears(the password is double-encrypted as in adminib) rdb$user_active = Y andrdb$user_privilege = 1
3 Add a local user USR
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
create user USR set password usr
4 All this leads to an interesting situation The ldquoTESTrdquo user can perform backup butwould it be a backup from adminib or from the database Letrsquos change the TESTrsquospassword in adminib Let the password be ldquottttrdquo
5 try to backup from TEST user through adminib
gbak -b dbib dbibk -v -user TEST -pass tttt
does not pass Try a user from EUA
gbak -b dbib dbibk -v -user TEST -pass test
it does pass That is only the user specified in eua can do backup (ie databaseowner)
6 So far it seems like one can delete the ldquoTESTrdquo user in adminib or completelydelete adminib But without adminib the server will not connect even to the data-bases with active EUA In addition restore should be done by a user specified inadminib since when restoring it is impossible to find out whether a database hasEUA or not
gbak -c dbibk 1db -v user TEST -pass test
does not pass as it was expected The TEST user has a different name in adminib
gbak -c dbibk 1db -v user TEST -pass tttt
restore is successfully accomplished
However as it was already said above the column rdb$user_privilege = NULL Thismakes impossible for any EUA user to connect to the restored database EUA(including the ldquoTESTrdquo user with password ldquotestrdquo)
Connect as TESTtttt set ldquo0rdquo instead of ldquonullrdquo in the rdb$users column of the TESTrecord and then disconnecthellip As a result EUA resumes work (see above an exam-ple of temporary solution of the problem)
ConclusionRegarding all that we can come to several conclusions
bull EUA in this version does not outlive backuprestore not only for the owner butfor SYSDBA as well SYSDBA is both king and god for databases and that is whyrdb$user_privilege = null is unnoticeable for a developer This becomes importantwhen one begins to use the database in operating mode
bull When performing restore there is a need to verify usernamepassword For reg-ular databases restoring with other usernames is performed when it is necessary tochange the database owner However since EUA is enabled for users within thedatabase restore can be performed in no other way but specifying a new user atgbak -c Actually SYSDBA from adminib may be a wrong user differing fromSYSDBA in the database with EUA if they have different passwords
Database maintenance
14
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Embedded UA in InterBase
bull Probably it would be better to create a flag in Header Page of the databasewhich would signify presence of EUA In this case users would be ignored duringrestore process This is up to InterBase 75 developers
bull In spite of apparent ldquoautonomyrdquo of EUA if there is no adminib it would be pos-sible neither to connect to the database nor perform restoring
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
closed after the following time intervalKEEPALIVE_ TIME+ ( KEEPALIVE_PROBES+1) KEEPALIVE_ INTERVAL
By default the parameters values arerather big and this makes use of themineffective For example the defaultvalue of KEEPALIVE_ TIME parameter isldquo2 hoursrdquo both in Linux and in Win-dows Actually 1-2 minutes would beenough to make a decision aboutforced disconnection of an inaccessibleclient On the other hand KEEPALIVEdefault settings sometimes cause forceddisconnections in Windows networkswhich are stay inactive during these 2hours (of course one may cast doubton necessity of such connections in theapplications but this is a different mat-ter)
Below adjustment of these parametersfor Windows and Linux operating sys-tems is described
Setting KEEPAILVE in LinuxKEEPALIVE parameters in Linux can bechanged either by file system directediting proc or by calling sysctl
IntroductionIn the systems within InterBase or Firebird databases which are intendedfor working in either real-time or near-real-time modes there is a problemof client connection status tracking on the server side and of forced discon-nection in case the client becomes inaccessible due to connection releaseIt is important to promptly release the resources busy with such phantomconnections especially when using servers with Classic architecture
If some users connect to the server through an unstable modem connectionthen the risk of disconnection becomes rather high
For instance a client saves a modified record set and after UPDATE is exe-cuted (while COMMIT is not) the connection is released
As a rule client applications in such situations reconnect to the server butthe client (as heshe continues working with the data after saving whichone received error message due to connection fail) will be unable to savechanges since heshe will receive a message about lockout conflict (ldquolockconflict on updaterdquo) The previous connection which opened the transac-tion (in the context of which UPDATE was executed while COMMIT was-nrsquot) still holds these records
Connection failures may occur in a local network too if the hardware (net-cards hubs commutators) is out of order or not adapted well andor dueto clutter in the network In Interbase and Firebird logs failures of tcp con-nections are displayed as error 10054 in Windows and 104 in Unix net-beui failures are displayed as 108109 errors
Hung connections control methodsIn InterBase and Firebird the mechanisms of DUMMY-packets orKEEPALIVE-sockets are used for tracking and disabling of such ldquodeadrdquoconnections
In InterBase 50 and higher the mechanism of DUMMY-packets is imple-mented at the application layer between an InterBase Firebird server anda gds32fbclient client library It is included in ibconfig firebird conf andis not examined in the present article
Note As we know from previous experience stability of the dummy-pack-
The TCP stack tracks the moment whenpackets stop transmit between the clientand the server by launching theKEEPALIVE timer As soon as the timerreaches the KEEPALIVE_ TIME pointthe server TCP stack would execute thefirst KEEPALIVE probe Probe is anempty packet with ACK flag sent to auser If everything is alright on the clientside then the TCP stack on client sidesends a response packet with ACK flagand the server TCP stack resets theKEEPALIVE timer as soon as it receivesa response
If the client does not response to theprobe the probes from the server con-tinue to be sent Their quantity equals tothe KEEPALIVE_ PROBES value theyare executed at the KEEPALIVE_ INTER-VAL time interval If the client does notrespond to the last probe then afteranother KEEPALIVE_ INTERVAL timeexpires the operating system TCP stackcloses the connection and the server (inthis case instance of InterBase or Fire-bird server) releases all resources busywith provision of this connection
Thus a failed client connection will be
et mechanism (the one implemented inInterBase 50 and repeatedly correctedin Firebird 15x) strongly depends onserverrsquos and clientrsquos operating systemstcp stack versions and many other con-ditions That is to say effectiveness ofsuch system in a real network tends tozero
KEEPALIVE-sockets are a more interest-ing mechanism Implemented in Inter-Base 60 and higher it is intended forconnection failure tracking KEEPALIVEis enabled by setting the SO_KEEPALIVE socket option at the open-ing Therersquos no need to manually set it ifyou use Firebird 15 or higher since it isimplemented in the program code ofthe Firebird server both for Classicand for Superserver
For Interbase and Firebird versionslower than 15 in the variant with Clas-sic architecture an additional setting isnecessary This setting is describedbelow
In this case the operating system TCPstack (instead of the Firebird server)becomes responsible for connectionstatus However to enable this mecha-nism one must adjust KEEPALIVEparameters
KEEPALIVE descriptionKEEPALIVE-sockets behavior is con-trolled by the parameter presented inthe following table
Database maintenance
15
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
Using KEEPALIVE-sockets to detect and release hungInterBase and Firebird client connectionsor how to avoid the 10054104 errors
Author Vasiliy Ovchinnikovovatkvcru
Parameter Description
KEEPALIVE_ TIME Time interval on expiry of which KEEPALIVE-probes start
KEEPALIVE_INTERVAL Time interval between KEEPALIVE-probes
KEEPALIVE_PROBES Number of KEEPALIVE-probes
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Substitute the lt valuegt word with necessary values
If you use version of Firebird Classic lower than 15 then in etcxinetdfirebirdthe following should be added
FLAGS=REUSE KEEPALIVE
Adjusting KEEPALIVE in Windows 9598ME Register branch
HKEY_ LOCAL_ MACHINE System CurrentControlSet Services VxD MSTCP
Everything about adjustment of TCP can be found here
httpsupportmicrosoftcomdefaultaspxscid=kben-us158474
Parameters
bullKeepAliveTime = milliseconds
Type DWORD
For Windows 98 type STRING
Defines connection inactivity time in milliseconds When it expires KEEPALIVE-probes start executing Default value is 2 hours (7200000)
bullKeepAliveInterval = 32-digit value
Type DWORD
For Windows 98 STRING type
Defines time between KEEPALIVE-probes (in milliseconds) As soon as the specified
KeepAliveTime interval expires after each KeepAliveInterval time (in millisec-onds) KEEPALIVE-probes are sent with maximum number of MaxDataRetries If noresponse comes the connection closes Default value is 1 second (1000)
bullMaxDataRetries = 32-digit value
Type STRING
Defines maximum number of KEEPALIVE-probes Default value is 5
Setting KEEPALIVE in Windows 2000NTXP Register branch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
Everything about TCP adjustment
2000 NT httpsupportmicrosoftcomkb120642
XP httpsupportmicrosoftcomkb314053
The MaxDataRetries parameter is substituted by
TCPMaxDataRetransmissions
All other parameters have the same names as in Windows 9x
Setting KEEPALIVE in Windows (for clients) This setting is optional but it possibly will reduce number of messages about con-nection failure if one uses unreliable communications channels Insert to the registerbranch
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters
parameter DisableDHCPMediaSense=1 See a description of this parameterhere
httpsupportmicrosoftcomscid =kb3Bru3B239924ampx=13ampy=14
Example
Letrsquos consider adjustment of Firebird SQL Server 152 CS under Linux OS
bullMake sure that the DUMMY-packets mechanism is disabled in firebirdconf (theparameter is commented-out)
helliphelliphelliphelliphellip
DummyPacketsInterval=0
helliphelliphelliphelliphellip
bullMake sure there is the etcxinetdfirebird configuration file
Database maintenance
16
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
For the first case the following lines should be edited
procsysnetipv4tcp_keepalive_time procsysnetipv4tcp_keepalive_intvlprocsysnetipv4tcp_keepalive_probes
For the second case the following commands should be executed
sysctl ndashw netipv4tcp_keepalive_time=value sysctl ndashw netipv4tcp_keepalive_intvl=valuesysctl ndashw netipv4tcp_keepalive_probes=value
Time value is expressed in seconds
For automatic setting of these parameters in case of server restarting add the fol-lowing lines to etc sysctl conf
netipv4tcp_keepalive_intvl = value netipv4tcp_keepalive_time = valuenetipv4tcp_keepalive_probes = value
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
bullConnect to any database on the server from any network client
bullCheck traffic on the server using any packet filter
If parameters specified as procsysnettcp_ keepalive_ within 15 seconds aftereverything stops in the channel the server creates a probe If the client is ldquoaliverdquothe server receives a response packet 15 seconds after that checking repeats andso on
bullIf a client is physically turned off (either the multiplexer or the modem unexpect-edly turns off ndash anything is possible) then the server does not receive a responseand the server begins to send probes with 10 seconds interval If the client does notrespond to the fifth probe then 10 seconds after that the server process dischargesand releases resources and blockings lockouts If the client gives any signals andresponses at least to the fifth probe (if worst comes to worst) then after another 15-seconds time-out the server will begin send probes And so on
Guidelines In conclusion we would like to give you some advice about how KEEPALIVE valuesshould be selected
Firstly determine necessary value of KEEPALIVE_ TIME The more the value is thelater KEEPALIVE-probes would start If you constantly see 10054104 errors in thelog of the server and you have to delete them manually it is recommended toincrease the KEEPALIVE_ TIME value
Secondly the values of the KEEPALIVE_ INTERVAL and KEEPALIVE_ PROBESshould meet your needs concerning before-the-fact release of already hung con-nections If your users connect to the server through unreliable channels then youprobably would want to increase number of probes and the interval between themin order to give the user a chance to detect the failure and reconnect to the serverIn case clients use a DSL connection to the Internet or access a SQL-server througha local network it is possible to decrease the interval between KEEPALIVE-probes
General recommendations if you for no particular reason receive from the clientsmany error messages concerning results saving due to lockout conflict (ie thereare no concurrent connections working with the same data) then you need toincrease systemrsquos reaction to the hung connections release Practically theKEEPALIVE_ TIME value may be above or equal 1 min You should yourself estimatethe time the longest transaction executes so that traffic would not be overloaded by
KEEPALIVE-checks of normallyworking connections whichlaunched long transactionsThe KEEPALIVE_ INTERVALvalue is above or equal 10seconds and the KEEPALIVE_PROBES value is above orequal 5 checks When manyusers work simultaneouslyremember that if you performchecking too frequently it mayconsiderably increase networktraffic
Also remember that in caseyour users actively changecommon data lockout errorswill occur as a result of opti-mum situation In this case youwould need a correct lockouterror handling in the clientapplications At the same timethe application should be ableto minimize occurrence of sucherrors
Examples of defaultconfigurationFinally here are some moreexamples of default configura-tions Downtime is the timewithin which users will beunable to update data (whichby that moment were updatedby the transaction opened bythe hung connection) Totaltime is the time on the expiryof which the hung connectionwill be closed
bullClients use modem connec-tions most of transactions inthe system are short downtimeis limited by 3 minutes
Database maintenance
17
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Using Keepalive
We kept everything unchanged as it was registered during installation Nothingneeds to be added
bullChange the TCP stack parameters
KEEPALIVE_TIME 1 minutesKEEPALIVE_PROBES 3 KEEPALIVE_INTERVAL 30 seconds TOTAL 3 minutes
InterBase Myths 4If I see the word NATURAL in the query itis a bad oneNothing terribleSome data retrieval (depending on query) is better by nat-ural table scan than by an index scanIts better to check how long time ago you refreshed statistics forindices in your database See documentation about SET STATISTICSINDEX ltindex_namegt statement
sysctl ndashw netipv4tcp_keepalive_time = 15 sysctl ndashw netipv4tcp_keepalive_intvl = 10 sysctl ndashw netipv4tcp_keepalive_probes = 5
bullClients use LAN connection most of transactions in the system are short downtimeis limited by 2 minutes
KEEPALIVE_TIME 30 sec KEEPALIVE_PROBES 5 KEEPALIVE_INTERVAL 10 sec TOTAL 90 seconds
bullClients use any connections downtime is not regulated
KEEPALIVE_TIME 15 minutes KEEPALIVE_PROBES 4 KEEPALIVE_INTERVAL 1 minutes TOTAL 20 minutes
bullClients use any connections continuous transactions are possible in the systemand downtime limit is 15 minutes
KEEPALIVE_TIME12 minutesKEEPALIVE_PROBES 7KEEPALIVE_INTERVAL 15 secTOTAL 14 minutes
We hope that the examples we have shown would be enough for correct adjustmentof TCP stack KEEPALIVE mechanism
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Developers area
18
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Working with temporarytables in InterBase 75
Author Dmitri Kouzmenkokdvib-aidcom
In InterBase 75 a new capability of working with temporary tableswas added Unlike system temporary tables (tmp$) these tables maybe created and used during applicationsrsquo work To this very day devel-opers had to store temporary data in ordinary tables and thatrequired constant table content tracking as well as specific organiza-tion of work with data
Surely most often temporary tables were necessary to those developers who hadbeen working with MS SQL before they started to use InterBaseFirebird
Letrsquos consider what temporary tables in InterBase 75 really are
MetadataOn the low system level temporary tables are implemented as permanent tablesThat is when you create these tables information about them is stored in theRDB$RELATIONS system table pointer page and other system pages are distrib-uted for them as for regular tables Moreover these tables not only will be stored inthe database constantly but also will outlive backuprestore (as distinct from anyother attempts to extend or change the structure of the rdb$ system tables)
Syntax of creation of temporary tables is as following
CREATE GLOBAL TEMPORARY TABLE lttablegt ( table-element-comma-list )
[ON COMMIT PRESERVE | DELETE ROWS] As you see temporary tables differ from standard ones by the global temporaryphrase Besides on commit is added In IB 75 in the RDB$RELATIONS systemtable there is RDB$RELATION_TYPE column It contains one of the following valuesRDB$RELATION_TYPE DescriptionPERSISTENT Standard tables (custom or system) in which
records are deleted only by delete+commit
GLOBAL TEMPORARY Temporary system tables which display server sta-tus connection to databases executed queriesand so on (TMP$DATABASE etc)
RDB$RELATION_TYPE Description
GLOBAL TEMPORARYDELETE
Temporary tables for which ON COMMIT DELETEROWS is specified ie the records which will beunconditionally deleted on the commit
GLOBAL TEMPORARYPRESERVE
Temporary tables for which ON COMMIT PRE-SERVE ROWS is specified ie the records whichwill be unconditionally deleted on disconnection
It is not recommended to modify this column manually this will not result in anythinggood That is to say it is impossible to turn a regular table into a temporary and viceversa
For a time of transactionGLOBAL TEMPORARY DELETE stores records only until any commit is per-formed (not only in the transaction which created them but also of any othertransaction within this connect) Such behavior resembles a bug since commit-ting of competitive transactions is not supposed to flush record view The temporarysystem tables work in exactly the same way ie they display updated informationas soon as any concurrent transaction executes a commit At the same time therecords created in the table are invisible to all but the current transaction A rollbackin this case is equivalent to a commit although it is clear that rollback would alsocancel all changes made in the regular tables In the case of commit the transactionchanges will be committed while the records in temporary tables would ldquodisap-pearrdquo
Letrsquos create such a table and try to work with it
CREATE GLOBAL TEMPORARY table TMPTRANS (ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT DELETE ROWS
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
and create FK from DETAIL to MASTER As a result (actually InterBase would notallow to create such FK) after creation of records in master and detail the first com-
For a time of connection GLOBAL TEMPORARY PRESERVE tables store records until current connection (dur-ing which they were added) is released At that they can be displayed only withinthe period of this connection
Letrsquos create a table
Developers area
19
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
Now you may create a procedure which would fill in the table with some data
CREATE PROCEDURE XTRANSASDECLARE variable I INTBEGIN
I = 0WHILE (I lt 10000) DOBEGIN
INSERT INTO detail VALUES (I asdfasdfasdfasdfasdf)I = I+1
ENDEND You may insert as many records as you need if all you want is to check the work100-10К would be enough If you want to test speed it is recommended to begineither from 100К records or million records (for example on my computer this pro-cedure loads 1 million records to a database with 4K within approximately 47 sec-onds)
Be careful do not commit after inserting records otherwise the records will be lostPerforming select from tmptran allows you to view the records After commit isperformed query iteration will return an empty table
TIP
In case you perform these operations using a tool with automatic transac-tion control (such as IBExpert) you would not see any temporary recordssince IBExpert executing any operator in SQLEditor performs startcom-mit of 3-4 another (hidden) transactions commit of which causes loss ofrecord view in the on commit delete table
At this moment one may ask a question where actually are these records Theanswer is despite of ldquotemporarinessrdquo of the records the temporary tablerecords are stored in the same way as in regular tables ie on a diskAt that after the records are inserted and commit is executed if one gathers statis-tics (for example with the help of IBAnalyst) it would look almost like the following
InterBase Myths 5
InterBase is designed for Windows and thereforeit is incompatible with Unix (Linux Solaris etc)
This is not true First version of InterBase was createdfor Unix and before the Windows-version was released there were15 ldquoportsrdquo for different Unix versions (AIX IRIX SCO HP-UX) Actu-ally the Windows version was released 7-8 years after the first ver-sion of InterBase
CREATE GLOBAL TEMPORARY table TMPCONN ( ID int not null NAME varchar(20) constraint PK_TMPTRANS primary key (id) )
ON COMMIT PRESERVE ROWS
Create a record in this table (it also can be done in IBExpert)
INSERT INTO TMPCONN VALUES (1 a)Perform commit Now within this connect the record will be visible from differenttransactions If another instance of IBExpert (or any other tool) runs and you exe-cute the same insert operator it would be executed with no PK or UNUQIE key vio-lation error
As soon as you close the current connection and open a new one the entered datawill be lost
Connections between temporary tablesIt is quite interesting that you can create Foreign Keys between temporary tablesbut this cannot be done between a temporary table and a constant one However
when creating FK one should take into consideration therecord view area in both tables For example you create twotables
Table Records RecLength VerLen Versions Max Vers Data Pages Slots Avg fillTMPTRANS 1000000 3100 000 0 0 51725 51725 68
MASTER on commit deleteDETAIL on commit preserve
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Developers area
20
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with temporary tables
mit would delete all records in master and that causes presence of records withmissing connections in DETAIL (in fact such type of connection as commit preserve-gt on commit delete is not permitted though you can perform the opposite)
To do that (and to change ldquotemporarinessrdquo type of the records) the ALTER TABLEoperator has the following extensionALTER TABLE lttablegt ON COMMIT PRESERVE | DELETE ROWSRESTRICT| CASCADE
This operator changes tablersquos type (preservedelete) and can also perform cascad-ing correction of type of the tables bound by FK in order to prevent the situationswith mismatch of recordsrsquo lifetime in master and detail The RESTRICT directive willinform about error if other temporary tables refer to this table
Temporary tables of all types cannot use FK pointing to constant tables
Garbage collecting
As already mentioned in spite of ldquotemporarinessrdquo of table content the on commitdelete and on commit preserve records are nevertheless stored on the disk asin ordinary tables Therefore the server sometime must remove them (as garbage)This happens when the following events occur
An example of the procedure which automatically fills a temporary table withrecords is given on purpose Tests were held using 1 million records For the tests2 IBExpert instances were launched and one IB_SQL was used Without going intodetails of the test we will list its results and conclusions
bull For ON COMMIT DELETE tables garbage is collected during first exclusive con-nection to the database Assume we have 10 working applications which fill in tem-porary tables To delete records in all temporary tables all 10 applications shoulddisconnect and at least one should connect Right at that moment garbage collect-ing in the ON COMMIT DELETE tables begins All connections which attempt toconnect to the server before garbage collecting is finished will hung
Resume
bull Working with ON COMMIT DELETE temporary folders may lead to fast grow ofthe database during a day since it is very seldom that during this period of time allusers disconnect from the database
bull The more garbage is collected in the ON COMMIT DELETE temporary tables thelonger will be the delay between the first connection and the working It takes
approximately 25 seconds for server todelete 1 million temporary records and~120 seconds to delete 3 million tem-porary records
bull For the ON COMMIT PRESERVEtables garbage is collected when dis-connecting the connection which creat-ed these records
Resume
bull The more records a connection cre-ates in temporary tables the longer theapplication would ldquohangrdquo when dis-connection is performed Deletion of 1million temporary records as well as inthe previous case takes ~25-35 sec-onds
SummaryTemporary tables InterBase 75 ndash arevery useful for applications which formcomplex reports and execute interme-diate calculations on the server How-ever due to strange behavior of ONCOMMIT DELETE it becomes possibleto use transaction context temporarytables only in the applications whichworks with only one transaction at atime Or they can work with severaltransactions on condition that a commitof competitive transaction is forbiddenuntil the transaction (which works withthe temporary table) performs a com-mit
Furthermore use of ON COMMITDELETE tables causes collecting ofgarbage records during multiuser work(since databasersquos size increases) andcollects garbage on first connection tothe database This can cause an unde-sirable delay in the beginning of theusersrsquo work
Table type When garbage is collected
ON COMMIT DELETE At first exclusive connection to the database
ON COMMIT PRESERVE When canceling the connection created the record
ON COMMIT PRESERVE is a more favorable way though the process ofdisconnection of applications would be more time-consuming (of courseunless these applications created records in temporary tables) In order toavoid usersrsquo complaints you will probably need to specially handle appli-cation disconnection and to display a message asking to wait some time
ps during the temporary tables test a spontaneous processor loading bythe IB75028 server was observed (though the applications were inactive)At that the loading appeared in certain order of transactionsrsquo starting andfinishing while they did not contain the executed operator The reason ofthis effect is currently being ascertained (with InterBase 75 SP1 also)
Registering for the Conference
Call for papers
Sponsoring the Firebird Conference
httpfirebird-conferencecom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
to a database specify the codepage which was used during database creation
In our case it is UNICODE_FSS This means that we want to get text data in UTF-8format from the database and the text will pass the text in the same format
Working with databaseExchange with server uses the following data categories
bullGeneral text fields with CHAR and VARCHAR types
bullBLOB text fields
bullArrays
bullSQL query text
UNICODE (the text format one usually deals with when programming for Win-dows) uses double-byte character set (UCS-2) As evident from the table above 1-3 bytes would be enough for presentation of the UNICODE characters with a codefrom 0x0000 to 0xFFFF in UTF-8
InterBase proceeds from the fact that UNICODE characters within the range[0x00000xFFFF] will be stored in UTF-8 Therefore when specifying size for stor-ing text data with UNICODE_FSS codepage the number of characters is multipliedby 3 bytes In addition all current versions of InterBase when working withCHARVARCHAR data in data presentation controls number of bytes not of char-acters That is why for example one can enter up to 9 single-byte characters to theCHAR(3) column with UNICODE_FSS codepage
Developers area
21
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
Working with UNICODEin InterBaseFirebird
Author Dmitri Kovalenko LCPIdimalcpilipetskru
What is UNICODE_FSSIt is an InterBase codepage (often called UTF-8) which displays double-byte and four-byte UNICODE characters (UCS-2 и UCS-4 respectively)in character strings from 1 to 6 bytes What is it intended for
bullIt provides transport for UNICODE texts based on regular ASCII text
bullData packing The characters with codes less than 128 are as usual representedas one byte
Interconversion of UTF-8 and UCS-2 (UCS-4) files is based on use of the followingtable
Bits Hex Min Hex Max UTF-8 Binary Encoding
7 00000000 0000007F 0xxxxxxx
11 00000080 000007FF 110xxxxx 10xxxxxx
16 00000800 0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
21 00010000 001FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
26 00200000 03FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
31 04000000 7FFFFFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
Table 1 Converting UCS-2 to UTF-8
Example
CREATE TABLE TESTUTF(TESTFIELD CHAR(3))INSERT INTO TESTUTF (TESTFIELD) VALUES(lsquo123456789rsquo) -- NO EXCEPTION
The main disadvantage of working with UTF-8 is the impossibility to detect the num-ber of symbols in a string without viewing it
That is why for better performance it is recommended to recode text data in UTF-8 format to a codepage with characters of fixed size (such as UCS-2)
Creation of database using UNICODE_FSSNothing out of the way everything is as usual
CREATE DATABASE DEFAULT CHARACTER SET UNICODE_FSS
Connecting to a databasewith a UNICODE_FSS codepageAs is well known when creating a database one can use one codepageand when connecting to a database can use a different codepage can beused Itrsquos partly true since a server when interacting with a client tries torecode text data to a coding the client wishes to use
However this mechanism has constraints and exceptions which are describedbelow That is why we recommend not to experiment with it When connecting
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
When recording text fields youconvert a text to UTF-8 and thenhandle it as an ordinary recordBuffer size and line length param-eters are expressed in bytes
If the field codepage differs from thecodepage of connection to databasethen the server performs conversion ofincoming and outgoing data
However for all that buffer size will becomputed according to the columncodepage Thus if the codepage of thewin1252 field and UNICODE_FSS con-nection is used and no other additionaloperations are launched then you willprobably receive the laquoCannot transliter-ate characters between character setsraquoerror message
Text fieldsWhen reading text fields the serverrequires providing a buffer (maxcharacter size)(number of charac-ters) This is the value which will beput in XSQLVARsqllen after dataaccess query text is prepared
A user does not have to worry aboutcalculation of size needed for textfield data in the clipboard We recom-mend using the value specified in thedefinition of the field However donot forget that for the VARCHARcolumns (SQL_VARYING type) oneshould add 2 bytes to the specifiedvalue in order to reserve some spacefor column length indicator Thereturned value will be expressed inbytes
The thing is that in win1252 all symbolsare single-byte and buffer for such col-umn will be required reasoning from 1byte for 1 character When recoding toUTF-8 characters with a code morethan 127 will become at least double-byte and this may result in overflow
So please consider it if you want to useconnections with several codepages inthe context of a single database
BLOB text fieldsAs is well known the link to data isstored in the InterBase record whichcontains a BLOB-field (subtype inde-pendent) The data are stored separate-ly and handled by InterBase API func-tions designed specially for workingwith BLOB
That is to say BLOB-fields data are readand written with separate calls of APIThat is why the rules of working with textinformation in BLOB differ from the onesof working with CHARVARCHAR
The basic rule is take care of yourselfBy default the server does not interpretcontents of BLOB-fields and treats themas ordinary binary information Thusthe client becomes responsible for datarecoding
Usually it is enough to useUNICODE_FSS coding on the clientand not use UNICODE_FSS and one-byte coding simultaneously since con-version to one-byte coding can causedata loss
ArraysUNICODE support in text arrays is simi-lar to the one for text fields
As in the text fields case the serverwhen working with text arrays operateson the byte-level not on the level ofcharacters Therefore the number ofcharacters in a string written in an arraycell may exceed the length specifiedwhen the text array column had beencreated
In exactly the same way the server sup-ports recoding of input and outputarrays data taking into account the con-nection codepage
SQL query textStrange as it may seem at first sight SQLquery text is also must use a codepagefor connection to a database The pointis that a SQL query text is one of themethods of parameter values explicittransfer
There are no severe limitations exceptfor general length of an SQL querywhich is 64K When converting a query
containing national coding characters toUTF-8 the resulting text may be largerthan the source one and thus it wouldexceed 64K limit
Access components forworking with UNICODE_FSSGenerally speaking support of a specif-ic codepage means that the access todatabase component is able to ensureclientrsquos work with other codepage Todo that it is enough to guarantee con-version of text data (at that UCS-2 anintermediate format is used) Conver-sion of text columns arrays and SQL-queries is not a problem at all but BLOBfields conversion is quite a laborioustask especially when accessing theBLOB field data through a stream mech-anism
Theoretically client application ofcourse should not depend on the code-pagesrsquo differences Therefore accesscomponents must block all possibleways of text data transfer and providenecessary information recoding
Developers area
22
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Working with UNICODEin InterBaseFireDird
TIP
By the way one of the undocumented features of InterBase is that for textfields in XSQLVARsubtype the number of the field codepage is specified inthe lower byte while the collate number is specified in the upper byte
InterBase Myths 6
Compiled procedures store query plans
Not a bit (unless the query plan is explicit)
This myth is based on the fact that the procedure after firstcall (that is the moment when query plans written in the procedureare computed) remains in the metadata cache until all clients whocalled this procedure disconnect In this case actually while the pro-cedure is in memory query plans do not change even if statistics ofthe indexes used by the plans change Read about this in InterBasedocumentation - DataDefpdf Chapter 9 section Altering and drop-ping procedures in use
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
When one uses either FirebirdYaffil Classic or InterBase 7175the following questions are oftenasked ldquowhat server to choose amultiprocessor or a single-processor onerdquo and ldquoshouldHyper-Threading be enabled onthe server or notrdquo
Itrsquos quite possible that if the server ispurchased from the hardware providersuch questions do not arise since insuch cases basic configuration is used(for example a dual-processor serverwith HT)
However different tests and commentswitness to the fact that it makes sense touse (keep enabled) the hyperthreadingtechnology only on single-processorworkstations () and not on servers
There are many reasons why Hyper-threading should be turned off on theserver
Note Each paragraph contains a link toa document which exemplifies ldquowhy it issordquo But this does not mean that the con-clusion in a particular paragraph isdrawn from that document You ofcourse will be able to find on the Internetcorroboration of the described facts andour experience We followed our ownexperience and tests results and wehave analyzed some reports (tests) ofother people as well as many differentdocuments concerning HyperThreading
1 Database-application (Firebird Inter-base Yaffil) is an application whichactively uses both the processor anddisk memory At the same time the per-centage of processor and disk chargemay vary
Total charge of several real (as well asvirtual) processors can occur only oncalculating applications
2 Enabling HT on a double-processorcomputer leads to appearance of fourprocessors (which are 2 physical and 2virtual) Thus when all processors areused the processor bus becomes moreloaded As a result instead of produc-tivity increase the system performancewould slowdown by 10-15
3 According to different reportsenabling HT slightly increases produc-tivity decreases it or does not affectproductivity at all Therefore the resultcan be seen only when analyzing aparticular combination of mother-board memory and processors
[conferences на newsgroups]
4 As Intel and Microsoft claim Win-dows 2003 is the only operating systemcertified for HyperThreading There-fore if one uses either Windows NT orWindows 2000 it is most likely that if HTis enabled productivity would be thesame or would decrease The same istrue for Linux ie HT should be enabledusing appropriate versions of this OS
Hardware
23
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Hyperthreading SMP andInterBase Firebird Yaffil
Author Dmitri Kouzmenkokdvib-aidcom
This is the first officialbook on Firebird mdash thefree independentopen source relationaldatabase server thatemerged in 2000
Based on the actualFirebird Project thisbook will provide youall you need to knowabout Firebird data-base development likeinstallation multi-plat-form configurationSQL language inter-faces and mainte-nance
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom
Hardware
24
The InterBase and Firebird Developer Magazine 2005 ISSUE 1
wwwibdevelopercomcopy2005 wwwibdevelopercom ltd All right reserved
Hyperthreading amp SMP + InterBaseFirebird Yaffil
Server Server type Operatingsystem
2 and moreprocessors
InterBase 60 SuperServer Windows No
Linux No
Solaris-SPARC No
Classic Linux Yes
InterBase 65 SuperServer Windows No
Linux No
Solaris-SPARC No
InterBase 7x SuperServer Windows Yes
Linux Yes
Solaris-SPARC Yes
Firebird 10 SuperServer Windows No
Linux No
Solaris-x86 No
HP-UX No
Classic Linux Yes
Solaris-x86 Yes
Solaris-SPARC Yes
FreeBSD Yes
MacOSX Yes
HP-UX Yes
Firebird 15 SuperServer Windows No
Linux No
Classic Windows Yes
Linux Yes
FreeBSD Yes
MacOSX Yes
Sinixz Yes
Solaris-x86 Yes
Yaffil SuperServer Windows No
Classic Windows Yes
httpwwwintelcomsupportplatformhtoshtmiid=ipp_htm+osamp
Note InterBase 71 tests on Windows2000 and Windows2003 with enabledHyperThreading have shown that Win2000rsquos productivity becomes worse Atthe same time it is similar for both OSrsquoif HT is disabled That is why the ibcon-fig ENABLE_HYPERTHREADING para-meter for IB 71 is disabled by default
5 Generally there are applicationswhich may work incorrectly (fail) onmultiprocessor computers if Hyper-Threading is enabled
(search Google for bsod+hyperthread-ing)
6 The HyperThreading technology isdesigned for increasing of multi-threadapplicationsrsquo productivity Thereforeexecution speed of two processes on
two virtual processors would be slowerthan it is on two physical processors Ifin addition to a database some appli-cations are active on the server thenenabling HT would cause decrease ofgeneral productivity (this does not con-cern single-processor systems in whicha slightly increase of productivity maybe)
httpwwwintelcomtechnologyhyperthread
SupplementWhile we hope you understood thatHyperThreading must be turned off onserver it is still not clear whether it isbetter to use a single-processor serveror a multiprocessor one The answer tothat question depends on architectureof the server and operating system
In the 2 and more processors column
No ndash not recommended or itmakes sense to tie a serverprocess to a specific processor(for example for Windowsthrough the ibaffinity utility orthrough the CPU_AFFINITYparameter in ibconfigfire-birdconf)
Yes ndash this server variant withthis operating system will useall processors In IB 7x it isrequired to purchase proces-sor license for each additionalphysical processor
For clarity sake yes variantsin the table are typed in bold
Subscribe now To receive future issues notifications send email to subscribeibdevelopercom