Enhancement of Communication Serverin the Composite Information

System/Tool Kit (CIS/TK) Prototype

Boniface Makatiani

May 1990 WP # CIS-90-17

ENHANCEMENT OF THE COMMUNICATION SERVER IN THE

COMPOSITE INFORMATION SYSTEM

by

Boniface Ayisi Makatiani

Submitted to the Department of Electrical Engineering and Computer Science on June 4, 1990 inpartial fulfillment of the requirements for the degree of Bachelor of Science.

Abstract

The Composite Information System Tool Kit (CIS/TK) is a prototype currently being developed at theComposite Information System Laboratory (CISL) at Sloan School of Management. The system merges fivedisparate databases into a single virtual database from which users can formulate queries. A communicationserver provides the physical interface between CIS/TK and the host databases. This thesis presents themodifications that have been made to the communication server to make it robust and more cost efficient.Earlier work on the communication server is summarized. The research into various means of modificationare discussed with comparisons between a multi-user, multi-database communication server model and single-user single-database communication server model. Aspects of both models are then used to build the finalmodel which is implemented. Insights gained in the process together with the final model's performance andC code are given and further modifications to future versions are suggested.

Thesis Supervisor:Title:

Professor Stuart E. MadnickProfessor of Management Science

-3-

Acknowledgements

I would like to thank my supervisor Stuart E. Madnick who from both his class and from this thesis hasopened me to a whole new career and way of thinking. Special thanks to Bertrand Rigaldies without whosecontinued help this thesis would have been more difficult. Last but not least I have to thank my mum whoknew that I could do it.

Table of Contents

Abstract 2Acknowledgements 3Table of Contents 4

List of Figures 5

1. Introduction 61.1 Background - Composite Information Systems/Tool Kit 61.2 Physical Connectivity - Communication Server 71.3 Goals of Thesis 71.4 Overview of Thesis 8

2. Communication Server 92.1 Existing Communication Server 92.2 Shortcomings of the existing communication server 122.3 Communications server solutions 12

3. Design and Architecture 153.1 Proposed models 15

3.1.1 Model 1--Logout/Login Communication Server 153.1.2 Model 2--Multi-User, Multi-database Communication Server 173.1.3 Discussion of the models 173.1.4 Final Model - Multi-lqp, multi-database communication server 17

3.1.4.1 Multi-user Communication Server 193.1.4.2 Startlqps 233.1.4.3 Finics 233.1.4.4 Communication Server 24

3.2 Implementation 26

4. Conclusion 284.1 Insights 284.2 Future Work 28

Appendix A. Startlqp.c code 30

Appendix B. Startfinics.c code - Starts the finics programs and the queue 33

Appendix C. Finics.c code - Queue maintaenance and CS invoker 34

Appendix D. Thesis2.c code - The Main Communication Server 42

-5-

List of Figures

Figure 1-1: Three Levels of Connectivity 1.1Figure 2-1: Alec Champlin's local query processor 2.1Figure 2-2: Francis Gan's local query processor 2.1Figure 2-3: Local query processor with the pipe to and from database shown 2.3Figure 3-1: Model 1 of the proposed communication server 3.1.1Figure 3-2: Model 2 of the proposed communication server 3.1.2Figure 3-3: A layout of the new communication server ' 3.1.4Figure 3-4: A finics and its components 3.1.4Figure 3-5: The startfinics with its child programs 3.1.4Figure 3-6: Flowchart of the Finics functions 3.1.4.3Figure 3-7: sorce code description 3.2

Chapter 1

Introduction

Information Technology (IT) is affecting how large organizations function today. Reduction in the cost ofobtaining, processing and transmitting data, has made IT more available and less expensive than a few yearsago.

The sudden growth of IT has led to a need or an opportunity to integrate disparate databases. An emergingstrategic application of IT consists of integrating disparate information systems and various databasemanagement systems (DBMS). A prototype aimed at integrating disparate information systems calledComposite Information System/Tool Kit (CISIK) has been developed at the Sloan School of Managementunder the direction of Professor Stuart E. Madnick.

1.1 Background - Composite Information Systems/Tool Kit

CIS[fK is a prototype developed to address the difficulties of integrating disparate databases. With theincreasing use of computer-based information systems, the difficulty of combining information and data fromvarious sources is becoming more common and has triggered large research efforts towards integratinginformation systems. This type of studies and systems is referred to as Composite Information Systems[Wang 88].

The problems which arise when building such a system include: physical connectivity, data connectivity andsemantic connectivity issues [Wong 891 as represented in figure 1-1 below.

Semantic Connectivity

Data Connectivity

Physical Connectivity

Figure 1-1: Three Levels of Connectivity

Physical connectivity is the ability to physically link and access information systems. Once connected, thedata that is retrieved has to be useful and correctly formatted. This ability is referred to as data connectivity.Data from multiple sources often has conflicts such as contradictions, ambiguity and incompleteness.Semantic connectivity refers to the ability to reconcile these inconsistencies using knowledge captured fromthe user about the assumptions underlying the data [Wong 89].

The goal of CIS/TK is to develop tools and techniques to support the entire spectrum of connectivity, with a

focus on semantic connectivity. The CISfTK approach [Wang 88] explicitly allows for the coexistence andusage of a variety of information systems while preserving their local autonomy. These information systemsare typically independently developed, hard to modify, and contain data that is dynamically changing.

1.2 Physical Connectivity - Communication Server

CISf[K is able to retrieve data from disparate databases. The particular modules within the CISITK systemdealing with physical connectivity issues are the Local Query Processors (LQPs) and the CommunicationServer (CS). The LQP provides a uniform interface for local applications allowing access to dissimilardatabases, handling the particularities of each local DBMS and its host system. The CS provides the physicalinterface between the LQP and the host database. These thesis focuses on providing better physicalconnectivity among various databases: allowing users to access various databases at physically distantlocations as if they were in a single virtual database. Another major objective is to provide a base andfoundation from which further research and design in physical connectivity can be investigated andimplemented.

1.3 Goals of Thesis

In order to improve the functions of the present communication server, the new communication server mustmeet the following goals which are explained in more detail in the chapters following:

(a) Provide the ability to log into a system and once the data collection is over, to wait for a default timebefore logging out of the external database,(b) Support many users and many databases simultaneously,(c) Provide the ability to handle errors,(d) Provide the ability to capture data from a remote database and then dynamically transform the query sentby an LQP using this newly obtained data.

1.4 Overview of Thesis

The focus of this thesis is the design of a modified communication server for CIS/TK.

Chapter 2 presents an overview of previous communication servers up to and including the presentcommunication server.

Chapter 3 presents an overview of the modified communication server architecture and also looks at twodesign considerations considered when at in developing the new CS.

Chapter 4 presents conclusions and recommentations for future work.

Chapter 2

Communication Server

2.1 Existing Communication Server

Originally, LQPs and communication servers were implemented in a single module. LQP first translatedqueries and then filtered data obtained from the remote databases. The initial architecture which is describedin detail in Alec Champlin's thesis [Chaplin 88], is shown in figure 2-1.

Francis Gan developed an architecture that allows all LQPs to share a single communication server [Gan 88].The communication server was able to cover various LQPs' queries without the redundancy of acommunication server in each LQP. His architecture is shown in Figure 2-1. Francis Gan's architecture hadthe following goals:

1. Implement a 'brighter' design that would be able to read the database prompts before sending a response.The original version by Alec Chaplin had the following format:

echo "string1"sleep X secondsecho "string2"sleep Y seconds

The next generation by Francis Gan had the following format:

prompt1, responsel, timeout1prompt2, response2, timeout2

promptN, responseN, timeoutN

The design had a timeout feature for incorrect prompt.

2. Solve the login/logout problem: when processing multiple data requests for a database, the login proceduretakes a significant amount of time and is expensive especially for commercial databases. Because thecommunication server has to log in and log out for every data request. A timer to keep the communicationserver logged into the external database for a specified default time was part of Francis Gan's design buthadn't been implemented. This problem will be referred to as login/logout pronlem. Next section discussesthe limitations of Francis Gan's communication server.

QueryTranslator

DBMSQuery

I

Global QueryProcessor

NkResult List

ResultReader

Standard"File Table

CommunicationsServer &Telnet/Cu Result

File

DBMS

Figure 2-1: Alec Champlin's local query processor

DataFilter

AbstratLocal Query

.0 - N6

-10-

Figure 2-2: Francis Gan's local query processor

-11-

2.2 Shortcomings of the existing communication server

Francis Gan's thesis still has the following limitations.

a) The inability to support the login/logout feature as explained in the last section.

b) The inability to handle errors. The communication server's language should include conditional functionslike 'IF...THEN...ELSE...'.

c) The inability to capture data obtained from a remote database, and then dynamically transform the querysent by an LQP using this newly obtained piece of data. For example in the Finsbury's dataline, the DatalineLQP could send a query along with a communication script to the Communication Server; this query wouldspecify which companies the LQP wants information on. Dataline is based on company codes, so thatpresently one has to go through a naming routine supported by the database to get the code of the companythey are interested in, then with the company code, they could query the database. Likewise, thecommunication server chould be able to go through the naming routines in order to collect the codesassociated with the companies' names given by the LQP and then use these codes to query the database.

d) Currently, only one LQP at a time can use the communication server. A multi-LQP coordinated systemthat allows multiple LQPs to share the communication server simultaneously is a highly desirable feature forthe following two reasons:

i) Independent sub-queries (all coming from the same global query) that can be sent in parallel. This featurewhich is covered in the multi-LQP/multi-database prblem classification in the chapter, will involve thesubdivision of quries so that the CS will process them as if from different LQPs.

ii) Multiple users could use the CIS[TK system simultaneously.

2.3 Communications server solutions

Francis Gan's communication server language is limited. The syntax of the script files (which are generatedby the specific LQPs) used to connect to the foreign host is as follows:

dial--comandprompt1, responsel, timeout1prompt2, response2, timeout2

promptN, responseN, timeoutN

Once generated, this script file is read by the communication server which then executes the first dial--command. It then connects to the remote database and in the mean time sets up two pipes called 'tocomm'and 'fromcomm'. For each subsequent line, it waits for a "promptX" to come in the 'fromcomm' pipe. If"promptX" comes through, the communication server sends a "responseX" string through the 'tocomm' pipe.If the "promptX" does not arrive, then the communication server disconnects after a period equal to"timeoutX" (see Figure 2-3). The limitations in this design which are discussed in the last section can besolved by adapting an alternative response for an unexpected prompt. To be able to handle this kind ofresponses and prompts, the CS language would have to be enhanced to allow for automatic adjustment toprompts that are received. This can be done by using conditional statements such as:

-12-

Figure 2-3: Local query processor with the pipe to and from database shown

-13-

if promptYresponseYelse if promptB = "system flooded"

send-message'GQP "database-not-available"

... etc...

-14-

Chapter 3

Design and Architecture

3.1 Proposed models

This chapter discusses some of the major design considerations in the development of the new communicationserver. First an overview of two models is given, followed by the final model that was implemented. AsCIS/TK project is one that is continuosly evolving, the ability to extend the major components within CIS/TKwithout major modifications is a critical design goal. In the final model, there was an attempt to have a CSdesign that would allow for modifications without major overhauls to the already existing versions. Thedesign should stimulate further ideas and changes and serve as a skeleton on which new addition could bemade.

3.1.1 Model 1--Logout/Login Communication Server

There is little difference between the current CS and this model except for the addition of a connector modulein the proposed model as shown in Figure 3-1 which handles the login/logout task discussed earlier in Chapter2. In this figure, the CS manager handles all the coordination between the connector and the LQP, and keepsthe status table for all the databases. The connector is a module that will setup up the pipes and execute thelogout after the database has received its data. The CS manager keeps the clock for the connector and alsodoes all the functions that Gan's CS was doing. The CS manager receives a request from an LQP after whichit looks at the status table to determine the communication link status between CISITK and the remote system.The status table has three possible conditions: inactive which means that the link does not exist; active meansthe pipes are functional and the link is good; broken means that there was some trouble linking to the remotedatabase and it is not functional. Some examples of possible reasons for a broken link are satellite down,computer down and similar problems that would cut the accessibility of the remote database.

If the link is inactive, the CS manager calls the specific connector that handles the requests belonging to thespecific LQP, which then sets up the link to the host database. The CS connector then takes over the link orpipes to the remote database and the rest of the procedure is the same as the old LQP [Gan 88]. Every LQPhas a connector that does the login/logout functions for it.

If an LQP sends a request while the status of the remote database is active, the request is channeled throughthe pipes to the host database. At the end of the session, when the LQP has received its results, the CSconnector keeps the link to the host database for some default time after which it breaks the link and removesthe pipes.

If there is a problem during setup, and connector cannot link then the status of the link is classified as broken.In this model, it is possible to have several host databases active even though only one can be functioning at atime. In case a request comes with the pipe shutdown, then the CS connector should set up the link andproceed with the call as described above. The CS manager keeps clocks for every database that exists and thisare accessed by means of a status table.

-15-

Connector 1

IfInactive

pipes

CS manager

Active Pipes

Many more ConnectorsOne Connector per LQP

Figure 3-1: Model 1 of the proposed communication server

Keeps status info.

Keeps timer info.

Calls connector ifpipes are inactive

LQP 2

-16-

3.1.2 Model 2--Multi-User, Multi-database Communication Server

This proposed model differs greatly from the present version of the CS. In this model, a request receivedfrom the LQP is put on a message queue from which the CS manager reads for its next process. The CSmanager which runs in the background is connected to a connector via pipes or another message queue. Inthis model, the connector does most of the timer, login/logout, and status upkeep functions as shown in Figure3-2. Every LQP has a unique connector associated with it. In this model, it is possible to have various remotedatabases linked at the same time.

3.1.3 Discussion of the models

Model 1 requires little change from the architecture of the CS that already exists in the CIS/TK. The littlechange required does not meet all the goals that this thesis set out to solve. It will solve the login/logoutproblem but it will not solve the multi-LQP/multi-database problem described in section 2.2., since the firstmodel does not have a queuing system which could handle more than one LQP or remote database.

Model 2 requires changes that will result in the construction of a message queue, and connector. In additionto the extra implementation work, it will use more memory than model 1 since the design requires severalconnectors at the same time. Due to the chain of processing required, Model 2 will be slower in setting up,but this should still be fast enough for the difference in setup times to negligible.

However, despite the disadvantages of model 2, it will solve the multi-LQP, multi-database problem.

3.1.4 Final Model - Multi-lqp, multi-database communication server

The final proposed model incorporates all the advantages from the above two models and the original CS. Itis designed to facilitate further work and expansion. Even though it borrows a lot of it concepts from model 2,it is different. It also solves the major shortcomings that this thesis set up to solve.

The final model has four components which are:a) Startlqp, an LQP interface to the queue (one per LQP),b) Finics, a CS interface to the queue (one per CS),c) Startfinics which starts the queue and the finics programs,d) CS program to link to the external host database,

Before discussing these modules individually in sub-sections 3.1.4.2, 3.1.4.3, and 3.1.4.4 a summary is givenbelow of how they function, so as to introduce the ideas that are used later in more detail. The specificfunctions used in these programs like "qputo", "qgeto" are also described in detail in those sections. Toillustrate the components are the following figure 3-3, figure 3-4, and figure 3-5.

Figure 3-3 illustartes the multi-LQP (multi-user)/multi-database feature. This will be implemented with theaid of Unix system's queuing mechanisms. Using a single queue for the communication server, an LQP sendsa message to the queue with the destination as certain database. The database, which has a distinct address,gets its messages from the queue on a first in first out basis. Likewise, a database message can be written to

the queue by use of the finics and startlqp can read from the queue, see Figure 3-3. The task involves theextension of the present communication server to include additional modules for queue maintenance, andoperation. It will also involve some redisigning of the present CS so as to handle the login/logout features thatwere explained earlier in section 2.1.

As illustrated in Figure 3-4 when the finics receives a request from the queue, it checks to see if the

-17-

DBMS1

setupnamed

connector 1 Pipes )

pipes or -Setup pipesmessage -keep status info.queue -keep timer info

-logout

connector 2

Data

one connector per LQP

connector N

IMS N

Figure 3-2: Model 2 of the proposed communication server

-18-

connection to the remote database is good, if there is no connection already running, it should set up theconnection with the database. This is the "1st function" in figure 3-4. Once the connection is up and runningthe finics module calls the communication server as the "2nd function" and which the reads the "scriptfile"and sends queries through the pipelines. The results it gets back are passed to the to the "datafile" which isread by the LQP or in case of an error it writes the appropriate error message to the "datafile". If some defaulttime after the last request to a database expires without a new request from the queue, the finics logs out of thedatabase. The timer function which is resident to Unix. In case a request for a database that is still connectedto the finics module comes through, the finics should turn off the timer and send the queries to the database asif it had initiated the connection.

Figure 3-5 shows the module "startfinics" which when called starts all the finics modules in the background.To shutdown all the finics modules, the task is handled by the "startlqp" which sents a message to all thefinics modules causing them to shutdown. This is explained in detail in section 3.1.4.2.

3.1.4.1 Multi-user Communication Server

The call to run Startlqp is executed from the LQP as illustrated in figure 3-3. It has as its functions thescriptfile-name and the datafile-name which are shown in figure 3-4. The scriptfile-name is the name of thescript file in which the queries to the remote database are written. This is prepared by the LQP. Likewise thedatafile-name is the name of the datafile that the LQP expects to find its data in. The ability to handle the twoarguments is an addition that Francis Gan's CS did not have. Startlqp places the request on the system queuewith its destination as the Finics associated with the requested database. A specific Finics gets the firstrequest on the queue that are addressed to it. The Finics then checks to see if the pipes already exists. If thepipes exist, the finics calls the CS, which then links to the database. The CS executes all the processingbetween the CISfIK and the remote system. The Finics modules keep track of time for the login/logoutfunction with the aid of a powerful Unix "Signal(function-to-brach-to, default-time)" function which has theability to branch to a given function after some default time given in seconds.

3.1.4.2 Startlqps

A Startlqp is called from the LQP by a system command. The system command, which is written in C-code, isof the form:

system ("Startlqp scriptfle datafile");

Startlqp will then use the "qput(mqueue, dest, msg)" function to place a message on the queue and"qget(mqueue, finics-name, msg)". "mqueue" is the queue to which the message "msg" should be put, "dest"is the destination to which the message "msg" should be delivered, and "finics-name" is the name of the finicsthat is to receive the message. After placing a message on the queue, startlqp then goes into waiting mode,from which it waits for response from the queue with the startlqp's Process Identification number (PID)number as the destination "dest" to be placed on the queue. Depending on the message placed on the queue,the startlqp should be able to tell what the results of the data request were. If the return message was brokenlink, then the startlqp should open up the datafile and write the beginning markers followed by the messageand then the end marker. If the message indicates a successful session, then the startlqp should exit andterminate by returning program execution to the LQP. Note that this termination does not kill the finicsmodule. To terminate the Finics module, startlqp has to run as follows:

system("startlqp SHUTDOWN");

This will kill all the Finics modules which would otherwise be running in the background. The startlqpprogram is a single program that can handle all the task described above for all the programs. Each LQP call

-19-

Message queue

Queue#, dest, origin

Queue 1,ipsharp,10

Queuel, 11,dataline

Queue1,Mit2e, 1

ethernet/CU

More startlqps as More finics modules

they are needed

Figure 3-3. The layout of the new communication server

-20-

Finics

-checks pipestatus

-Seefigure 3-6

-keeps trackof time afterlast login asexplained intext.

-closes pipes

pipe files/pipes

and logs outwhen default 2nd functionis over

Openpipes()1st function

ope ipes

comMUuicATIN SERVER ie

Francis Gan'sCore code

fscan()

fprintf()

Scriptfile

fscan()

Datafile

fprintf()

Figure 3-4. Finics and its components

CU/TELNET

RemoteDatabase

CU/TELNET

-21-

As many startlqps as are needed

Figure 3-5. Startfinics with its child programs which are started in the background .

IPSHARP FINICS

DATALINE FINICS

-22-

starts a new startlqp. This could be modified to shutdown only one Finics but CISfITK has no need for thatfunction at this stage. The startlqp has a list of current databases so that if a new database was to be added,the code would have to be modified to handle it. The startlqp will be modified in the future so as to be tabledriven.

3.1.4.3 Finics

Named Finics as a combination of the words "FINIsh" "CS", Finics is the most important module of the newcommunication server. Finics checks on the status of the pipes using the Unix function "access(pipename,status)" and opens them if they do not exist, (see figure 3-4). Finics keeps the clock in the implementation ofthe login/logout feature, using the Unix functions "Signal()" and executes the CS most of which consists ofFrancis Gan's communication server. When the CIS/TK system is set up, all the Finics modules are started inthe background. The Finics programs coordinate the flow of information from the Startlqp to the CS programusing the "qputo" and "qgeto" functions. Finics gets requests addressed to it from the queue and sendsresponses from the CS to the queue. As shown in figure 3-6 when a request comes through, Finics checks forthe status of that link. In order for a link to be active, the pipes (files) that pertain to that communication linkhave to exist. This pipes function like special files to which data can be written using the "C-code" functions"fscan(file, "format", information)" and read using the function "fprintf(filel, "format", information)". Thisfunctions are used to read/write information from/to specific files. When a script file is read, the informationis send to the remote database by writing it to the pipe that is assigned by Unix as the outgoing. Likewisewhen reading data from the remote database, the CS reads the data from the pipe assinged as the incomingpipes. If the pipes are active, Finics executes the CS and passes to it the script file-name, the pipenames, amessage to indicate which turn of the login/logout activity the current call is, i.e., first-time or not-the-first-time within the period assigned as default after the previous call, and data-filename. If the call is a first timecall, then the Finics executes the login procedure before calling the CS. The login script for the AT&T 3b2machines is of the form:

login: makatian 100Password: psd3dsp 100vt100): AM 100

The Finics then calls the CS with the script file name from the startlqp which the CS executes. If this was nota first time, then the Finics would call the CS with the script file name as an argument. The C-code commandis of the form:

CS scriptfilename pipetoname pipefromname whichtime datafilename

If the connection is inactive, then Finics sets up the pipes for communication link with the host database. Therest is as in the present CS. Timer information is controlled by the Finics; when the CS completes a process,the Finics keeps the pipes running for a default time. This is in case some other request comes through withinthe default time. If no request comes through, Finics erases the pipes associated with that CS. Finics is onlyshutdown when it receives a shutdown message from the startlqp.

3.1.4.4 Communication Server

The communication server (CS) is a modified version of the present CS. It however does not execute the firstdial-command as in the Francis Gan's CS. An example of the new script file follows:

cd /usr/students/xv578/makatian/test_1/test_2 100%/usr/cistk/bmakatia/work/tmp/connectl .tmp 100

-23-

Alarm alwaysignored when itgoes off duringprocessing ofrequest

Wait for request from queue orWait for alarm to go off then

break out of wait

NO

Figure 3-6: Flowchart of the Finics functions

i CS program: "system(scriptfilename pipetoefrom whichtimeafilename );"- YES

-24-

cat /usr/students/xv578/makatian/congrats 100%endcapture 100

Note that in Francis Gan's CS, the last line was "% logout 100" this has been eliminated in the present version

of the CS, to enable the login/logout feature. The CS gets its start from the Finics and uses the pipes provided

by finics to link to the external databases via telephone or ethernet links. At the end of communication link

process, which is indicated by the last line of the script file or an error, the modified CS will return execution

to the Finics.

3.2 Implementation

The C-code which is provided in the Appendix will be implemented on an AT&T 3b2/500 minicomputer and

will run in Unix System V.

-25-

Chapter 4

Conclusion

This thesis presented the design of a modified communication server to overcome major physical connectivityshortcomings. This was motivated by the goal of providing a more robust and cost efficient communicationserver. These chapter, presents how the modified communication server faired in satisfying this goal. It endswith a note on possible future work on extentions to the present communication server.

4.1 Insights

Several insights about the design of the CS were gained during the implementation of the new CS. There is ahuge number of possible errors that could occur in all the databases that we log in to. To make thecommunication server recognize every single error that could occur is not a worthwhile venture at this time inthe development of the CS. It would be more graceful to be able to categorize these errors in some groups forthe benefit of the LQP manager [Tung 90]. In these thesis, error handling was not solved as efficiently as wasthe initial goal but steps were made that could be extended on.

Separating the program into four modules proved to be a very powerful tool for tackling the various problemsthat had existed. To build any more complex system, would mean expanding on these modules as they havethe potential to handle more functions than they already are. For example, the startlqp could be built with itsdatabase which could be used to store information on the remote database like possible errors, with aninterpretation for their meanings in the local database, the various phone number, statistics of the possible linkproblems and other similar information.

4.2 Future Work

With the above insights, future work on the communication server would involve changes that should makefor a more robust and cost efficient communication server. The module in the new CS that sends character toand receives characters from the external remote databases could be handled by another program in whicherror handling routines could be incorporated. This will be very useful in the next generation ofcommunication servers if they are to handle errors which require conditional handling. Also this could serveas the starting point for the solution to the problem of dynamic processing of queries with informationreceived from a remote database. This will be useful especially with the dataline database.

Another beneficial modification would be to store the information specific to a certain database be stored inone file that would not require modification to the programs. Currently, even though the phone numbers foripsharp or dataline change are LQP objects, the CS does not have direct access to this objects and so when theobjects change, the CS programs have to be changed and recompiled. This is because of the fact that the LQPare written in LISP and the CS is done in C language. This is a function that would be easier if a file withconstants which the program could call existed, or the CS had access to the LQP objects via the system call tothe CS.

This thesis's aim of providing an efficient and robust communication server for the CIS/TK is almost obsolete

-26-

as far as the functionalities are concerned. A lot of work in this area has already been covered by thecommercial vendors of communication software such as the Procomm and Kermit PC software packageswhich could perform most of the tasks in this thesis with less coding in their script language. The biggestproblem is that most of these programs have been written for desktop computers with few working in the Unixenvironment. Also they do not provide source code from which this thesis could have borrowed programmingideas. However the CIS/TK provides an academic environment from which more basic and complex researchinto physical connectivity is being carried out. For this section of CIS/TK to be at the same pace with thefuture developments in the information technology academic and research community, the CS has to provideat least the functions provided by commonly available commercial communication software packages. Thisfunctions include an extended script language with powerful funtions, functions to wait for prompts, functionsto support conditional statements, funtions to break out of loops and redirect execution, functions to countcharacters received from remote database, functions to set up links to remote databases and many otherfuntions which in the present CS, are confined to specific programs. If they were all made into a languagethen it would be possible to access them from any other program much more easily. Nevertheless with thework in this thesis, a foundation has been laid that should now lead to more challenging work in the low levelcommunication facilities.

-27-

This page provides an introduction to the files, and a brief explanation of the task performed by eachfunction.

File Function Task Performed

Starfinics.c Takes no arguments. Initiates the beggining ofall the Finics programs. It is could be called bythe CIS/TK program

Startlqp.c This function takes 2 arguemnts. The 1st arg isthe script file name and 2nd is the data file name

Finics.c MainO Takes 1 arg, the name of the database to link to.Checks the status of the pipes and either callsopenpipes() and/or calls CS which then processesthe script file.

Openpipeso Is called by finics to open two pipes. It takes 2 argsthe 1st being the pipename for the pipe to- CU cmd.and 2nd being for the telephone links and telnet cmd.for ethernet.

timeupO Incase the alarm goes off before any request fromstartlqp, the program directs execution to this programwhich prints a message to the screen and returnsexecution back to the calling program.

Thesis2.c main() This is the core of Francis Gan's thesis. It takes 5args. The args are scriptfilename. pipe-to. pipe-from, NUMBER,datafile-name, The NUMBER is a tag to indicate the whetherthe call is a FIRST or NOTFIRST time call for the login /logoutfeature. The program links to a database by phone or ethernet thenuses the scriptfilename to query a database. The results from thedatabase are written into the datafilename. at the end of a script fileit returns execution to the finics.

-28-

Appendix AStartlqp.c code

-29-

/* STARTLQP - 5/21/90 Boniface Ayisi Makatiani *//* Take 2 arguments startlqp scriptfilename datafilename */

# include <stdio.h>

main(argc,argv)int argc;char *argv[];

int qnumber, mqueue; /* the queue number and queue identifier */int pid, finicspid; /* process ids of this process and finics */char msg[80]; /* the message from finics */char line[100];setbuf(stdout, 0); /* set stdout buffer size to 0 */

printf("startlqp: started..\n");qnumber=qgetnum(); /* get the queue number */printf("startlqp: using queue number %d\n",qnumber);mqueue=qopen(qnumber); /* open the queue */

pid=getpido; /* get process id for this process */

/* Loop to read FINICS commands and send them into finics for processingwhile (1)

sprintf(line,"%s %s",argv[1],argv[2]);if (strcmp(argv[1],"SHUTDOWN")==0)

qput(mqueue,1,line); /* send command to finics (dest=1) */qput(mqueue,2,line);qput (mqueue, 10, line);

elseif (strcmp(argv[1],"mit2e")==0)

qput(mqueue,1,line); /* send command to finics(dest=1) */

elseif (strcmp(argv[1],"mit2d")==0)

qput(mqueue,2,line); /* send command to finics(dest=2) */else

if (strcmp(argv[1],"ipsharp")==0)qput(mqueue,10,line); /* send cmd tofinics(dest=10) */

/* keep receiving from finics until "END!" is received *//* receive a message from finics with dest=pid */

finics_pid=qget(mqueue,pid,msg);if (strcmp(msg,"**BROKENLINK**")==0)

printf("\nstartcids:THERE ARE BROKEN LINKS TO THE DBS.\n");return; /* return to lqp */

if (strcmp(msg,"**END**")==0)

printf("\nstartLQP: finics has finished your job.\n");return(1); /* return to lqp */

/* if not END!, get next msg */} /* end of big while loop */

-30-

Appendix BStartfinics.c code - Starts the finics programs and the queue

I**** STARTFINICS/* Modified by Boniface Ayisi Makatiani 4/27/

/* *//* Takes no arguments */

/* *// ********************* *******

#include <stdio.h>

#define FIRST 1#define SECOND 2#define THIRD 3

main(){

int qnumber, mqueue; /* queue number and queue identifier*/char cmd[100];int i ; /* constant */setbuf(stdout, 0); /* set stdout buffer size to 0 */printf("\nStartfinics: started......\n");qnumber=qgetnumo; /* get the queue number */printf("Startfinics: using queue number %d\n", qnumber);qremove (qnumber);mqueue=qopen(qnumber); /* open the queue */

sprintf(cmd, "finics mit2e &");system (cmd) ;sprintf(cmd, "finics mit2d &");system (cmd) ;sprintf(cmd, "finics ipsharp &");system (cmd);

printf("\Startfinics: The finics programs are now working\n");

-31-

Appendix CFinics.c code - Queue maintaenance and CS invoker

-32-

/**** FINICS/* Modified by Boniface Ayisi Makatiani 4/27/90/* *1/* Takes one argument the name of the database to invoke/* Currently we have MIT2e, MIT2d, IPSHARP, DONNER, *//* DATALINE(FINSBURY) *//* *1/ *********************

#define NOMESSAGE 0#define FILEXIST 00#include <signal.h>#include <stdio.h>#include <stropts.h>

int oppid;int timetag;int debug=1;int number;int dbsnumber;char devpath[100];char data[50];char dbsname[100];

main (argc, argv)int argc;char *argv[];

int qnumber, mqueue; /* queue number and queue identifier*/int from_pid; /* process id of user that sent command */int pipestatus; /* to check the status of the pipes */int problem=0; /* problem with the csint t=1; /* time constant ... */int exityes=0;int mes=NOMESSAGE; /*to denote whether to close pipes or not*/extern int timetag;int timeupo;int timer=20; /* time to wait for logout/login feature */int r; /* return from open of pipes */int breakl(); /* function to break out of a loop */int res; /* return from run of thesisl.c */int number;FILE *fpq;FILE *fromcs, *tocs; /* file ids for named pipes */

char whichtime[10];char line[80]; /* line returned from CIDS */char msg[80]; /*message typed by the user(from startcids) */char cmd[200]; /* to hold system commands to unix */char telcom[150]; /* to hold the dial command */char command[25]; /*to hold command and first argument*/char message[50]; /* to hold message for another userchar scriptt[50], script[50]; /* of the script file */char pipeto[100],pipefrom[100];extern char dbsname[];

-33-

extern int dbsnumber;extern char devpath[];extern int op_pid;unsigned alarmo;

setbuf(stdout, 0); /* set stdout buffer size to 0 */strcpy(devpath,"/usr/cistk/bmakatia/work/tmp/");

/* check which the database name and react accordingly ****/

strcpy(dbsname,argv[1]);printf("\nfinics:copying dbsname '%s'\n",argv[1]);

strcpy(whichtime, "FIRST");sprintf(pipeto,"%sto%s",devpath,dbsname.name);sprintf(pipefrom,"%sfrom%s",devpath,dbsname.name);

/* safety: lets remove all old files and pipes */sprintf(cmd,"rm %s",pipeto);system(cmd);sprintf(cmd,"rm %s",pipefrom);system(cmd);

printf("finics: started......\n");qnumber=qgetnumo; /* get the queue number */printf("finics: using queue number %d\n", qnumber);mqueue=qopen(qnumber); /* open the queue */

/* Loop forever, until shutdown is read from demo */while(1) /* start of big while loop */

printf("finics: waiting for msg from startlqp\n\n");do

{frompid=qget(mqueue,number,msg);if (frompid !=O)break;

)while(mes==NOMESSAGE);sscanf(msg, "%s %s", script, data);

/* break msg into command and arg */if (strcmp(script, "SHUTDOWN")==)

qput(mqueue,from_pid,"**END**");if (debug==1)

printf("\nfinics: FINICS is shutting down\n");exityes=1;goto exitting;

if (debug==1)printf("\nFINICS: The pid number is '%d'.\n",from_pid);

sprintf(scriptt,"%s%s",devpath,script);if (!(access (scriptt, FILEXIST) == 0))

{printf("\nfinics: script

file %s does not exist\n",script);qput(mqueue,from pid,"**END**");

-34-

break;

if ((access (pipetoFILEXIST) == 0) &&(access (pipefrom,FILEXIST) == 0))

{printf("\nFINICS: The pipes are active.... \n");pipestatus = 1; /* The pipes are active */

else

pipestatus = 0; /* The pipes are inactive */if (dbsnumber > 10)r=openpcu(pipetopipefrom);

elser=openpipes(pipeto, pipefrom);

if (debug==l)

printf("\nFinics: r from openfiles = %d .\n",r);printf("\nFinics: The pid for

the telnet process is = '%d'\n",oppid);

op_pid++;

if (r != 0)

printf("\nThe pipes are broken\n");qput(mqueue,from_pid,"**BROKENLINK**");

if (! (fpq=fopen (data, "w")))printf("\nFinics:cannot open datafile

to write error message\n");fprintf(fpq,"*****BEGINNING MARKER*****\n");fprintf(fpq,"***CAN NOT OPEN CONNECTION****\n");fprintf(fpq,"********BEGINNING MARKER********\n");fclose(fpq);return(0);/**********************according to MIN I should put a message in theconnect.tmp file at this point

elsepipestatus =1;

if (debug==1)printf("\nFinics:the pipestatus is %d .\n",pipestatus);

while(pipestatus==l)

if (debug==l)printf("\nFinics:Processing the

inner loop.\n",pipestatus);

/* start the CS using a system command */

sprintf(cmd,"thesis2 %s%s %s %s %s %s%s",devpath, script, pipeto,pipefrom,whichtime, devpath, data);

-35-

if (debug==l)

printf("\nFinics: Starting the Comm. Server.\n");printf("\nFinics: this is the command line\n");printf("\n%s\n\n",cmd);

res = system(cmd);mes=NOMESSAGE;

/* it has used up the message that it had */if (debug==1)printf("\nFinics: Comm. Server

returncode is '%d'.\n",res);if ((res ==O) 1| (res == -1))

printf("\nFinics: There was a problem with CS\n");mes=NOMESSAGE;problem=l;

elseif (res==512) /* that is 2X256. This is a system used by

unix when it returns a value called by another function via SYSTEM() call.

The returned value is bigger by a factor of 256. The specific reason for this

is not available in the manuals that I have referred to yet */

printf("\nFinics: The CS read alogout response\n");

/* Wait untill thesisl.c is through with program *//* The output will be terminated by a clock timer */

printf("Finics:waiting for default time to end\n");qput(mqueue,from_pid,"**END**"); /* msgto startlqp */signal(SIGALRM,timeup);alarm(timer); /* set timer */timetag=1;if (problem==0)

do

frompid=qget(mqueue,number,msg);if (from_pid !=O)

if (debug==1)printf("\nFinics:There was a

message from que\n");mes=1;break;

if (t <= timer)

mes=NOMESSAGE;break;

}while(t < timer && mes==NOMESSAGE);

strcpy(whichtime,"NOTFIRST");

printf("\nfinics: got past the second QGET\n"); */

-36-

sscanf(msg, "%s %s", script, data);/* break msg into command and arg */

if (strcmp(script, "SHUTDOWN")==O)

qput(mqueue,from_pid,"**END**");

if (debug==l)printf("\nfinics: FINICS is shutting down\n");

exityes=1;goto exitting;

sprintf(scriptt,"%s%s",devpath,script);if (!(access (scriptt, FILEXIST) == 0))

{printf("\nfinics: script file %s does

not exist\n",script);qput(mqueue,from_pid,"**END**");break;

exitting:if (mes== NOMESSAGE 1| exityes==1)

/* never got a message */

/*close the pipes and the erase the temp files if time is overif (debug==1)printf("\nFinics:Logging

out of external system\n");if (strcmp(whichtime, "NOTFIRST")==0)

sprintf(command, "kill %d",oppid);system(command);system("sleep 2");sprintf(command,"rm %s",pipeto);system (command);sprintf(command,"rm %s",pipefrom);system (command);

pipestatus=0;problem=0;strcpy(whichtime,"FIRST");

printf("\nFinics: Getting out....\n");if (exityes==1)exit(1);

} /* end of loop makes a call assumingSTATUS ACTIVE */

/* end of loop that includes check for status *

int timeup() /* writes message to screen incase of an alarm *//* due to expiration of default time */

int timetag;printf("\nFinics:breakl: there was a break in qget\n");timetag=0;return;

-37-

openpipes(pipeto, pipefrom)char pipeto[100], pipefrom[100];{char telcom[150];char command[100];int timeupo;int tel pid;extern char dbsname.name;extern int op_pid;int id, status; /* the pid of the telnet process */unsigned alarmo;int timer=30; /* default timeout (sec) */

setbuf (stdout, 0);/* try openning the link *//* first put command into telcom */

printf("\nFinics:Openpipes:Databasename is '%s' \n",dbsname);

sprintf(command,"/etc/mknod %s p" ,pipeto);system (command) ;sprintf(command,"chmod +w %s", pipeto);system(conmand);sprintf(command,"/etc/mknod %s p",pipefrom);system(command);sprintf(command,"chmod +w %s",pipefrom);system(command);

/* launch the communication process by executing telcom */

sprintf(telcom," telnet %s < %s 2>&1 > %s &",dbsname.name, pipeto,pipefrom);

if ((id=fork()==0){printf("\nFinics:openpipes the tel_pid is '%d'\n",tel pid);execl("/bin/ksh", "ksh","-c",telcom, 0);

oppid=wait(&status);

if (debug==1)

printf("\nFinics: the phone connection should be good now.\n");printf("\nFinics:

The pid for the telnet process is = '%d'\n",op pid);

return(status);

openpcu(pipeto, pipefrom)char pipeto[100], pipefrom[100];

I

-38-

char telcom[150];char command[l00);int timeup();int telpid;extern int dbsname.dbsnumber;extern int oppid;int id, status; /* the pid of the telnet process */unsigned alarmo;int timer=30; /* default timeout (sec) */

setbuf (stdout, 0);/* try opening the link *//* first put command into telcom */

printf("\nFinics:Openpipes:Databasename is '%s' \n",dbsname);

sprintf(command,"/etc/mknod %s p" ,pipeto);system(command);sprintf(command,"chmod +w %s", pipeto);system(command);sprintf(command,"/etc/mknod %s p",pipefrom);system(command);sprintf(command,"chmod +w %s",pipefrom);system(command);

/* launch the communication process by executing telcom */

sprintf(telcom,"cu -s 1200 -t %d < %s 2>&1 > %s &",dbsnumber, pipeto,pipefrom);

if ((id=forkO)==O){printf("\nFinics:openpipes the tel pid is '%d'\n",telpid);execl("/bin/ksh", "ksh","-c",telcom, 0);

oppid=wait(&status);

if (debug==l)

printf("\nFinics: the phone connection should be good now.\n");printf("\nFinics: The pid for the telnet

process is = '%d'\n",op_pid);

return(status);

-39-

Appendix DThesis2.c code - The Main Communication Server

-40-

/*

FILE: /usr/cistk/demo/v3/lqp/comm/server/thesisl.c

Communication server program.Created in June 1989. Francis Gan

The pipes fromcomm and tocomm are created in -/demo/v3/lqp/comm/tmp/

November 8, 1989

this program is run by a command of the form"thesis2 scriptfilename pipetocs pipetfromcs INTERGER datafilename"the interger if a tag which can be either FIRST for first calling orNOTFIRST for any other calling of the program.This was to facilitate the login/logout feature

*/

/**** Thesis2.c/* Modified by Boniface Ayisi Makatiani 4/27/90

/* *//* Takes 5 arguments */

/* *//**********************

#include<stdio. h>#include<stropts.h>#include<malloc.h>#include<signal.h>

#define MAXLINES 100

FILE *tocomm, *fromcomm; /* file pointers to/from cu/telnet */FILE *fout; /* file pointer to result file */int capture; /* result capture toggle*/int debug=0; /* debug mode toggle */int timer=30; /* default timeout(sec) */int test=0;char outfile[50]; /* result filename */char devpath[100];char filename[100];char data[50];char whichtime[10]; /* to keep the times that we use the phone link */

main(argc,argv)int argc;char *argv[];

{FILE *fp, *fopen(;char scriptname[200];char prompt[MAXLINES][100];char c;char response[MAXLINES][1000);char telcom[150];char command[100];int temp,h;

-41-

extern char whichtime[10];extern FILE *tocomm, *fromcomn;

/* extern char devpath[], filename[];this are not required as they are defined in finics already */

int time[MAXLINES], x=O, y=O, status,endcapture; /* result endcapture toggle*/unsigned alarmo;void timeupo;

strcpy(scriptname, argv(1]);strcpy(whichtime, argv[4]);strcpy(data, argv[5]);endcapture=O;capture = 0;

if (debug==1)printf("\nThesis2 is now running.\n");

/* strcpy(devpath,"/usr/cistk/bmakatia/work/tmp/");this are not required as they are defined in finics already */

/* to open the data storage file. */if (!(fp=fopen(scriptname,"r"))){printf("Conmunication Server: Cannot open LQP

data file '%s'.\n", scriptname);return(0);

elseif (debug==1)printf("\nThe file %s was successfully opened\n",data);

/* to open the data storage file. */if (system(creat(data,0644)) < 0)

printf("Communication Server: Cannot open LQPdata file '%s'.\n", data);

return(0);

elseif (debug==1)printf("\nThe file %s was successfully opened\n",data);

while (x < MAXLINES && fscanf(fp,"%[^\t] %[A\t] %d%*c",prompt[x],response[x], &time[x]) !=EOF)

if(strcmp(response[x],"^M")==0)

response [x] [0]=0;if(debug==1)printf("%s\t%s\t%d\n",prompt[x], response(x], time[x]);

x++;

/*******strcpy(argv[4],"NOTFIRST"); *******

-42-

if (strcmp(argv[4],"NOTFIRST")== 0)

for ( h=0;h < x-2;h++)

temp=h+3;strcpy(prompt[h],prompt[temp]);strcpy(response[h],response[temp]);time [h]=time [temp];

if (debug==1)printf("%s\t%s\t%d\n",prompt[h], response[h], time[h]);

signal(SIGALRM,timeup); /* set alarm clock */alarm(timer); /* set timer */

/* tocomn is pipe to the communication script. */if (!(tocomm = fopen(argv[2],"w")))

printf("CS: Cannot open pipe to CU '%s'.\n",argv[2]);return(0);

}setbuf (toconm, 0);

/* fromcomm is the communication script. */if (!(fromcomm = fopen(argv[3],"r")))

printf("CS: Cannot open pipe from CU '%s' .\n",argv[3]);return(0);

)

setbuf (fromcomm, 0);

if (strcmp(argv[4],"NOTFIRST")== 0)

printf("\nCS:This is just before tocomm .\n");fprintf(tocomm,"\015");printf("\nCS:This is just before fromcomm .\n");c=getc(fromcomm);printf("\nCS:This is from fromcomm '%c' .\n",c);if (debug==1)printf("\nCS: just typed in '^M' to get a prompt\n");

if (debug==l)printf("\nCS:getting into the big DO LOOP \n");do

if (strcmp(response[y],"%endcapture") == 0)

endcapture = 1;

elseif (response[y][] ==(

-43-

capture = 2;endcapture = 0;strcpy (out file, data);

}else

status = lookfor(prompt[y],response[y],time[y]);if (status ==2)

{fclose (tocomnm);fclose (fromcomm);return(2);

}if (status == 1 && capture == 2)

{capture = 1;/* Openning of the file outfile */if (!(fout = fopen(outfile,"a")))

printf("Cannot open %s in mode \"a\".\n", outfile);fclose (tocomm);fclose(fromcomm);return(0);

}fprintf(fout,"*****BEGINNING MARKER*****");fclose(fout);

if (endcapture == 1)

endcapture=0;capture = 0;/* Openning of the file outfile */if (!(fout = fopen(outfile,"a")))

{printf("Cannot open %s in mode \"a\".\n", outfile);

fclose(tocomm);fclose (fromcomm);return(0);

fprintf(fout,"********END MARKER********\n");fclose(fout);

}

y++;

}while (y < x);

sprintf(conmmand,"chmod g+w %s",outfile);system (command) ;if (debug ==1)printf("\nCS:The CS program has finished execution\n");

return(1);

lookfor(prompt,response,time)char prompt[], response[];int time;

-44-

FILE *fpl, *log;char c, query[1000], *malloco;int i=O, t=O, nfds=l, poll();extern char filename[], devpath[];extern int capture;extern FILE *fout;

alarm(time); /* set timer */do

{c=getc (f romcomm);printf("%c",c);if (capture == 1)

if (!(fout = fopen(outfile,"a"))){return(0);

fprintf (fout, "%c", c);fclose (fout);

}/*

if (debug == 1)

{printf("%d %d %d %d\n", c, prompt[i], i, strlen(prompt));

*/if (c == prompt [i])

if (i == strlen(prompt)-1)

{alarm(timer); /* reset timer */if (response[O] ==

fpl = fopen(response+l,"r");fscanf(fpl,"%s",query);printf("\n%s\n",query);fprintf(tocomm, "%s\n",query);

else

if (strcmp(response, "logout")==0)

if (debug==lprintf("\nCS:Just read response logout

returning to finics\n");return(2);

}system("sleep 2");

fprintf(tocomm,"%s\n",response);/* printf("%s\n",response); */

return(1);

elsei++;

else

-45-

i = 0;while (t < time);

return (0);

void timeup()

fprintf(stderr,"Timed out\n");exit (0);

-46-

References

[Chaplin 88] Chaplin, Alec.Interfacing multiple remote databases in an object-oriented framework.Bachelor's thesis, MIT, May, 1988.

[Gan 88] Gan, Francis.An Architecture Design and implementation of a communication Server to Access

Disparate Databases.Bachelor's thesis, MIT, May, 1988.

[Tung 90] Tung, Min.Local Query Processor Manager for the Composite Information System / Tool Kit.Bachelor's thesis, MIT, June, 1990.

[Wang 88] Wang, Richard, Wong, T.K and Madnick, Stuart.Connectivity Among Information Systems.Composite Information Systems, Vol. 1. , September, 1988.

[Wong 891 Wong, Toon K.Data Connectivity for the Composite Information System/Tool Kit.Bachelor's thesis, MIT, June, 1989.