MQ - File Transfer User Guide v1.2

8/6/2019 MQ - File Transfer User Guide v1.2

1/37

State of Florida | Florida Voter Registration System (FVRS)

MQ/FTP Programming Guide


MQ File Transfer User Guide

Version 1.1

for

Florida Voter Registration System

prepared for

The State of Florida

Department of State

409 East Gaines StreetTallahassee, FL 32399-0250

August 15, 2005


2/37




Revision History

Name Date Reason For Changes Version

Gary Dischner 8/15/05 Initial Version 1.0

Stephanie Calaman 8/15/05 Review for formatting 1.0

Caroline OConnor 8/17/05 Edits 1.0

Caroline OConnor 9/12/05 Addition of API specs 1.1

Stephanie Calaman 9/14/05 Format of TOC 1.1

Caroline OConnor 9/15/05 Update to API Spec 1.2


3/37




Table of Contents

Section Page

1.0 Introduction to MQ ............................................................................................................................11.0.1 What is Message Queuing? ................................................................................................ 11.0.2 What is a Queue?................................................................................................................. 11.0.3 What is a queue manager?.................................................................................................. 21.0.4 What is a WebSphere MQ client?.......................................................................................2

1.1 Introduction to MQ File Transfer...................................................................................................... 2

1.1.1 Overview of WMQ Based File Transfer..............................................................................2

1.2 File Naming Conventions ................................................................................................................. 3

1.2.1 Required Format for File Names ........................................................................................31.2.2 List of Transactions That Can Exist as Part of a Batch File ............................................3

1.3 Installation Instruction for MQ Client .............................................................................................. 4

1.3.1 Installing WMQ Client ..........................................................................................................4

1.4 Scenario 1: The FVRS File Transfer Application.......................................................................... 14

1.4.1 Overview .............................................................................................................................141.4.2 Installing FVRS File Transfer Application .......................................................................141.4.3 Usage: Upload, Download, and List................................................................................221.4.4 County or Agency to FVRS (Upload) ...............................................................................231.4.5 FVRS to County or Agency (Download) .......................................................................... 231.4.6 County or Agency List Files Available for Download ....................................................24

1.5 Scenario 2: Directly interfacing with the MQ File Transfer API................................................. 24

1.5.1 Overview .............................................................................................................................241.5.2 Required System Variables............................................................................................... 251.5.3 mqftrcvc API ....................................................................................................................... 261.5.4 mqftrcvc Recommended Syntax ...................................................................................... 291.5.5 mqftsndc API ......................................................................................................................291.5.6 mqftsndc Recommended Syntax ..................................................................................... 301.5.7 Sample .NET wrapper ........................................................................................................ 301.5.8 Required Retry Logic Implementation ............................................................................. 30

1.6 System Availability and Support ...................................................................................................30

1.7 Contact Information ........................................................................................................................ 30

Appendix A: County/Agency Codes Reference.................................................................................... 32


4/37



MQ/FTP Programming Guide1

1.0 Introduction to MQ

1.0.1 What is Message Queuing?

Message Queuing (MQ) has been used in data processing for many years. It is most commonly usedtoday in electronic mail. Without queuing, sending an electronic message over long distances requires

every node on the route to be available for forwarding messages, and the addressees to be logged on

and conscious of the fact that you are trying to send them a message. In a queuing system, messages

are stored at intermediate nodes until the system is ready to forward them. At their final destination they

are stored in an electronic mailbox until the addressee is ready to read them.

Even so, many complex business transactions are processed today without queuing. In a large network,

the system might be maintaining many thousands of connections in a ready-to-use state. If one part of

the system suffers a problem, many parts of the system become unusable.

You can think of message queuing as being electronic mail for programs. In a message queuing

environment, each program from the set that makes up an application suite is designed to perform a well-

defined, self-contained function in response to a specific request. To communicate with another program,

a program must put a message on a predefined queue. The other program retrieves the message from

the queue, and processes the requests and information contained in the message. So message queuing

is a style of program-to-program communication.

Queuing is the mechanism by which messages are held until an application is ready to process them.

Queuing allows you to:

Communicate between programs (which might each be running in different environments)

without having to write the communication code.

Select the order in which a program processes messages.

Balance loads on a system by arranging for more than one program to service a queue when

the number of messages exceeds a threshold.

Increase the availability of your applications by arranging for an alternative system to service

the queues if your primary system is unavailable

1.0.2 What is a Queue?

A message queue, known simply as a queue, is a named destination to which messages can be sent.

Messages accumulate on queues until they are retrieved by programs that service those queues.

Queues reside in, and are managed by, a queue manager. The physical nature of a queue depends on

the operating system on which the queue manager is running. A queue can either be a volatile buffer area

in the memory of a computer, or a data set on a permanent storage device (such as a disk). The physical

management of queues is the responsibility of the queue manager and is not made apparent to the

participating application programs.


5/37




Programs access queues only through the external services of the queue manager. They can open a

queue, put messages on it, get messages from it, and close the queue. They can also set, and inquire

about, the attributes of queues.

1.0.3 What is a queue manager?

A queue manageris a system program that provides queuing services to applications. It provides an

application programming interface so that programs can put messages on, and get messages from,

queues. A queue manager provides additional functions so that administrators can create new queues,

alter the properties of existing queues, and control the operation of the queue manager.

For WebSphere(R)

MQ message queuing services to be available on a system, there must be a queue

manager running. You can have more than one queue manager running on a single system (for

example, to separate a test system from a live system). To an application, each queue manager is

identified by a connection handle (Hconn).

Many different applications can use the queue manager's services at the same time and theseapplications can be entirely unrelated. For a program to use the services of a queue manager, it must

establish a connection to that queue manager.

For applications to send messages to applications that are connected to other queue managers, the

queue managers must be able to communicate among themselves. WebSphere MQ implements a store-

and-forwardprotocol to ensure the safe delivery of messages between such applications.

1.0.4 What is a WebSphere MQ client?

WebSphere(R) MQ clients are independently installable components of WebSphere MQ products. A client

allows you to run WebSphere MQ applications, by means of a communications protocol, to interact with

one or more Message Queue Interface (MQI) servers on other platforms and to connect to their queue

managers.

1.1 Introduction to MQ File Transfer

1.1.1 Overview of WMQ Based File Transfer

The FVRS File Transfer Application allows you to send and receive files via the transport of WebSphere

MQ messaging. You can use the File Transfer Application to send and receive any type of file in any

format, for example: ASCII Linux format (with line feed characters), ASCII file Windows format (with

carriage return/line feed characters), binary (for example, image files, word processor files, spreadsheetfiles, or zip files), also reports, letters, memos and charts.

The File Transfer Application is available on both WebSphere MQ for Windows, and WebSphere MQ for

Linux (x86 platform) servers, and clients.


6/37




1.2 File Naming Conventions

1.2.1 Required Format for File Names

Field Field Length Description

County 3 This should be the 4character field. The fieldshould be populated withthe county code or in thecase of agencies should beHSMV, FDLE, OEC, BVS etc.

Separator 1 -

Transaction 10 A 10 character fieldindicating the transaction

type

Separator 1 _

Date 8 MM/DD/YYYY

Separator 1 _

Time 8 HH:MM:SS

1.2.2 List of Transactions That Can Exist as Part of a Batch File

Transaction Description

AB01 Absentee Update Transaction

BU01 Redistricting and Reprecincting

ST01 Clear Streets Information File

ST02 Streets Information

SY01 Registration Verification File

SY02 Request from County for full voter list

transfer

VH01 Voting History Update Inquiry

VH02 Voting History Clear TransactionAgencies Description

FDLE Florida Department of Law Enforcement

(FDLE

Office of Executive Clemency (OEC)


7/37




Bureau of Vital Statistics (BVS)

HSMV Death Data (IBM)

HSMV DL/SSN Verification (IBM)

HSMV Registrations (IBM)

1.3 Installation Instruction for MQ Client

1.3.1 Installing WMQ Client

Counties and Agencies must install the WebSphere MQ Client v6.0 software on any machine that will beusing the WMQ File Transfer. This code may be obtained on an FVRS website:

https://tfvrs.dos.state.fl.us:9090/MQDownload.html

Users should choose the appropriate installation for their Operating System, either Windows or Linux.The zip file should be downloaded and then unzipped to a temporary directory. Now the installation maybegin. Follow the steps below.

1. Logon to the computer with a local userid that has administrative privileges.


8/37




2. After the system is up click Start->Run on the Start menu. When the following dialogue isdisplayed type the path and name of the WebSphere MQ setup program, located in the root ofthe temporary directory where you unzipped the installation files. In the following picture, theinstallation files were unzipped to D:\. Enter the appropriate path and file name then click OK.

3. The image below is summoned. It is the language selection dialogue. The default is English.Simply press OK.


9/37




4. This is followed by a WebSphere splash screen. You need do nothing here. This screen will goaway with no action on your part.


10/37




5. This screen will disappear and be replaced by the WebSphere Welcome screen as shown below.Press Next.

6. The following dialogue requests that you acknowledge acceptance of licensing restrictions. Clickthe radio button that indicates acceptance and press the Next button.


11/37




7. After Next is pressed the following window appears. It is asking what type of setup is desired.Select Custom by clicking the Radio button by this choice, and press the Next button.

8. After Next is pressed on the previous dialogue the following window appears. This window allowsthe user to change the location of the installed software. Press the Change button.


12/37




9. After the Change button is pressed the following windows is presented. In the data entry fieldlabeled Folder Name: type C:\WMQCLIENT, and then press the OK button.

10. When OK is pressed this window is replaced by the following dialogue.When this window appears accept the contents and press Next.


13/37




11. When Next is pressed this window is replaced by the following one. Accept the contents of thewindow by pressing Next as shown below.

12. This window is then replaced by the following. When this window appears accept the contents bypressing the Next button.


14/37




13. This window is then replaced by the following one. Simply accept the provided default (install allthree features listed) by pressing the Next button.

14. The following screen repeats the choices made during this installation procedure. These shouldbe correct, but if your window does not resemble this one the install may be aborted or backed upto the desired step. Press the Install button to proceed.


15/37




15. The following window is displayed. It will take awhile to complete. When it is finished the nextwindow (step 16) will appear.

16. The following window will replace the previous one when the installation has completed. Simplypress the Finish button to complete. The next window will then replace this one.


16/37




17. Even though there is no need for additional processing the following Wizard dialogue appears.Press the Next button to proceed.

18. This is the final window in the installation process. When it appears uncheck the item labeledLaunch Notepad to view the release roles and press the Finish button. This screen willdisappear. This ends the installation of the WebSphere MQ Client code.


17/37




You have now installed the base functionality to allow communication with a WebSphere MQ Server. Toactually communicate with the FVRS WebSphere MQ Servers, you will need to proceed to either:

Scenario 1: The FVRS File Transfer Application

or

Scenario 2: Directly Interfacing with the MQ File Transfer API

1.4 Scenario 1: The FVRS File Transfer Application

1.4.1 Overview

The FVRS File Transfer Application is a prewritten application that handles all details of connecting to theFVRS MQ Servers for users, and presents them with a simple command line interface into one of threecommands: list, upload or download.

Counties and Agencies must install the FVRS File Transfer Application on any machine that will be usingthe prewritten FVRS File Transfer Application. Alternatively, counties and agencies may directly interfacewith the MQ File Transfer API, as described in section 1.5. Regardless of which approach is chosen, aprerequisite is the completion of section 1.3, Installation of MQ Client.

1.4.2 Installing FVRS File Transfer Application

If a county or agency chooses to use the prewritten FVRS File Transfer Application, this code may beobtained on an FVRS website:


Windows users should choose the link for FVRS File Transfer Package for Windows. The installationfile, FVRSINSTSIMPLE.msi, should be downloaded to a temporary directory. While logged on as anadministrator, double click this to start the installation of the FVRS File Transfer Package.


18/37




1. You will be greeted by the welcome screen. Click Next.


19/37




2. Next you will be prompted to select an Installation Folder. Users must accept the default folder ofC:\WMQCLIENT . Click Next.


20/37




3. The installation will now take place. Click Install.


21/37




4. Installation is now complete. Click Finish. There are 5 more steps.


22/37




5. Although installation of the FVRS File Transfer Application is now complete, there is a machine-specific Environment Variable that must be set manually. Right Click on My Computer and selectProperties.


23/37




6. Under the Advanced Tab, Select Environment Variables .


24/37




7. Under System Variables Select New:

8. Enter LOCATION as the Variable name and your 3-letter County Code or 4-letter Agency Codeas the Variable Value. In this example we have used LEO for Leon County. Click OK to exit theNew System Variable dialogue, OK to exit the Environment Variables window, and OK to exitthe System Properties window.

* For a list of County Codes and Agency Codes, please see Appendix I at the end of this document.


25/37




9. Test that your variable has been set correctly. Open an MS-DOS command window and type :

Echo %LOCATION%

The command returned LEO which is what I set the value of LOCATION to. I have successfully set myLOCATION environment variable.

Setup is complete.

1.4.3 Usage: Upload, Download, and List

There are two distinct operations that must take place in order to complete each file transfer. The file

must be uploaded to specified queue and queue manager first. Only then may the file stored in this

queue and queue manager be retrieved. In this sense this is a very simple utility. It supports file to queue

and queue to file transfer only. The complexity of these programs is internal and concerns error checking

and data integrity. These programs are designed to work with WMQ to provide a safe, recoverable,

secure and efficient way to move files.

The Upload command is used to move files into the WebSphere MQ managed storage using the

WebSphere MQ transport. The Download command is used to retrieve files from WebSphere MQ

managed storage using the WebSphere MQ transport. Additionally, a List command is provided to

browse the queue of files available for download.


26/37




There are two distinct types of movement needed to satisfy the file transfer requirements of the Florida

Voter Registration System (FVRS). The first type of file transfer is from county or agency to FVRS, and

the second from FVRS to county or agency.

1.4.4 County or Agency to FVRS (Upload)

Regardless of the direction of transfer between FVRS and county/agency the first step is to store a file on

a Queue. The second step is to retrieve this file from the queue. For the county or agency, a file that will

be uploaded must be placed in the directory:

C:\WMQClient\Upload

Once the file is stored in this directory, the provided upload utility can be called using the following syntax:

Upload filename

The file will be removed from the C:\WMQClient\Upload once it has been successfully uploaded. An

unsuccessful file transfer will not remove the file from the C:\WMQClient\Upload directory, and will also

return the message Cant connect to servers.

1.4.5 FVRS to County or Agency (Download)

Regardless of the direction of transfer between FVRS and county/agency the first step is to store a file on

a Queue. The second step is to retrieve this file from the queue. For the county or agency, a file that is

downloaded will be placed in the directory:

C:\WMQClient\Download

Once the file is stored in this directory, the provided upload utility can be called using the following syntax:

Download filename

County/Agency clients will know the file name from the results of the NT01 transaction. Alternatively, a

county/agency client can use the list function described below to see what files are available for

download. Once the Download command has been issued, the file can be retrieved from the

C:\WMQClient\Download if it has been successfully downloaded. An unsuccessful file transfer will not

create the file in the C:\WMQClient\Download directory, and will also return the message Cant connect

to servers.


27/37




1.4.6 County or Agency List Files Available for Download

An additional utility is provided for browsing the files available for a county/agency to download. The

provided list utility can be called using the following syntax:

List

Once the List command has been issued, the county/agency will receive in response a list of all files

available for download. In the event the List command cannot be successfully executed, the message

Cant connect to servers will be returned to the Client.

1.5 Scenario 2: Directly interfacing with the MQ File Transfer API

1.5.1 Overview

Should a county or agency choose not to use the prewritten FVRS File Transfer Application, they are freeto directly interface with the MQ File Transfer API. This will give greater control over the interaction, butwill require manual effort on their part. Detailed below is the API specification, as well as the FVRSspecific arguments that should be passed. There are also some prerequisite environment variables thatmust be set for the connectivity to be established.

A prerequisite to directly interfacing with the MQ File Transfer API is the completion of section 1.3,Installation of MQ Client.


28/37




1.5.2 Required System Variables

If a county or agency chooses to write their own interface code, they will need to set the following systemenvironment variables which would otherwise have been set via the FVRS File Transfer Application:

Name Value:

(Sample as set by FVRS File Transfer

Application)

Usage

MQCHLLIB C:\WMQCLIENT\BIN Tells MQ Client where to

look for your channel table

MQCHLTAB AMQCLCHL.TAB Tells MQ Client the nameof your client table (this

should not change)*

MQSSLKEYR C:\WMQCLIENT\SSL\key\key Tells MQ Client the

location of your SSL key

database (In this example

key.kdb is stored in

C:\WMQCLIENT\SSL\key

i.e. kdb name minus fileextension is the last part of

value)*

*These files have been provided in ConfigFiles.zip available at:



29/37




1.5.3 mqftrcvc API

mqftrcvc (receive file on client)

Use the mqftrcvc command to do one of the following:

Receive a file from a connected server.

View available files on a connected server.

Syntax

Required parameters

-q QueueName

The name of the target queue.

The format for a mqftrcvc queue name in FVRS will always be:

3-character county code/4Digit Agency Code_OUT

ex:

LEO_OUT

HSMV_OUT

* For a list of County Codes and Agency Codes, please see Appendix I at the end of this document.

-m QMgrName

The name of the queue manager that hosts the destination queue.

The Development Queue Manager Name for FVRS is:

FVRSDEVQM

-e

Extract a complete, or incomplete file. Messages associated with the file are not removed. If afile already exists of the same name, do one of the following:

Specify the -yparameter, so that the existing file is overwritten.

Specify the -rFileName parameter, so that the file is renamed.

Optional parameters

-c CorrelId

Select all files matching CorrelId. Selection can be combined with -s UserData, and -f

FileName.


30/37




-u MsgID

Select the message that has a message ID that matches MsgID. Used to select other

messages.

-s UserData

Select files by locating any occurrence of the character string UserData, in part or all of the

file's UserData. The comparison is case sensitive, and wildcard characters cannot be used.

Selection can be combined with -cCorrelId, and -fFileName.

-v

Return the CorrelId, and MsgIdof the file.

-a

List all files and messages, in the following order:1. Complete files, ordered by queue name2. Incomplete files, ordered by queue name3. Other messages, ordered by queue name

This is the default value.

-l

List all complete files, ordered by queue name.

-i

List all incomplete files, ordered by queue name.

-o

List all other messages, ordered by queue name.

-r FileName

Assign new file name and/or file location upon download.

Used to rename, or to relocate a file. The file is assigned the name specified in FileName. A

fully qualified file name can be specified to relocate the file. If the file name, or path, containsembedded spaces, it must be specified in double quotes. One file can be specified only, andyou cannot use wildcard characters.

-y

Replace an existing local file of the same name. Used with optional parameter -e.

-fFileName


31/37




Select all files matching FileName. The fully qualified file name can be specified. If the file

name contains embedded spaces, it must be specified in double quotes. You cannot usewildcard characters.

Selection can be combined with -cCorrelId, and -s UserData.

Return codes


32/37




0 Successful operation

36 Invalid arguments supplied

40 Queue manager not available

69 Storage not available

71 Unexpected error

163 Queue name required

164 Cannot open queue

165 Cannot open file

166 Cannot put to queue

167 No file name specified (Send)

168 Message length is too small to send data

169 Sending file has changed

170 Cannot get from queue

171 Cannot write to file

172 CorrelId is invalid

173 MsgId is invalid

174 No messages to receive

175 File for delete is not unique

180 File not found

Examples

This command lists all files and messages on the queue, LEO_OUT, located onQueue Manager FVRSDEVQM:

mqftrcvc -q LEO_OUT m FVRSDEVQM -a

This command gets the first complete file on the queue, LEO_OUT, located onqueue manager FVRSDEVQM:

mqftrcvc -q LEO_OUT -m FVRSDEVQM -e

This command gets the complete file, named Mydocument.txt, on the queue,

LEO_OUT, located on QM FVRSDEVQM:

mqftrcvc -q LEO_OUT m FVRSDEVQM -e -f "Mydocument.txt"

This command gets the complete file, named Mydocument.txt, also markedURGENT, on the queue, LEO_OUT, located on queue manager FVRSDEVQM :

mqftrcvc -q LEO_OUT -m FVRSDEVQM -e -f "Mydocument.txt" -s

"URGENT"

1.5.4 mqftrcvc Recommended Syntax

To receive a file:

mqftrcvc -q LEO_OUT -m FVRSDEVQM -f C:\yourFile.txt -rC:\DownloadDirectory\yourfile.txt -y -e


33/37




1.5.6 mqftsndc Recommended Syntax

mqftsndc -q LEO_IN -m FVRSDEVQM -f C:\YourFile.txt -l 3000 -s "your comments"

1.5.7 Sample .NET wrapper

Some clients have requested a .NET wrapper to allow their code to interface with the MQ File Transfer

API. Here is the code for one such program written by Yong Zhao of IBM.

PrivateSub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles

Button1.Click

Dim myProcess As System.Diagnostics.Process

myProcess = System.Diagnostics.Process.Start(txtCommand.Text, txtArguments.Text)

myProcess.WaitForExit()

MessageBox.Show(myProcess.ExitCode, "Exit Code")

EndSub

1.5.8 Required Retry Logic Implementation

NOTE: THIS cannot be 100% tested in the development environment as there is only one MQ Server in

Development.

For counties or agencies that will be directly interfacing with the MQ File Transfer API, they will need to

implement retry logic similar to that of the prewritten FVRS File Transfer Application. The retry logic is

implemented as follows:

1. Try to initiate mqftrcvc or mqftsndc on first queue manager defined.

2. Catch response code. If response code indicates that communication could not be established

with the queue manager, retry on secondary queue manager.

3. Additionally, large distributions should alternate as to what they set their primary and secondary

queue managers to (reverse order for half to distribute load).

1.6 System Availability and Support

System availability will be based on the availability of the MQ Server as well as the FVRS network and will

not be unavailable except for announced maintenance windows. Support for the pilot environment will be

provided during normal business hours Mon Friday 8am et 6PM ET.

1.7 Contact Information

If there are any questions or problems users are encouraged to report the problem via the DOS message

board. The URL for this facility is:

http://207.156.23.12:8080/~FloridaVoterRegistrationSystem/


34/37




The first time a user accesses the DOS message board they will need to establish a user ID. Once this is

done, the user can navigate to the conference that is applicable to your issue. Once there, please review

the threads and see if a colleague has reported the same or similar issue. Then add your comments in to

the appropriate new or existing thread. If you need to add a thread, please be as descriptive as possible

in the title to help others find similar issues easily.

The IBM team will monitor the message board. Problems will be addressed in accordance with the

support model defined above. IBM will disseminate all information regarding the disposition of a

particular problem via replies on the message board.


35/37




Appendix A: County/Agency Codes Reference

County/Agency Codes

Description County/Agency ID

Alachua County ALA

Baker County BAK

Bay County BAY

Bradford County BRA

Brevard County BRE

Broward County BRO

Calhoun County CAL

Charlotte County CHA

Citrus County CIT

Clay County CLA

Collier County CLL

Columbia County CLM

DeSoto County DES

Dixie County DIX

Duval County DUV

Escambia County ESC

Flagler County FLA

Franklin County FRA

Gadsden County GAD

Gilchrist County GIL

Glades County GLA

Gulf County GUL

Hamilton County HAM

Hardee County HAR

Hendry County HEN

Hernando County HER

Highlands County HIG

Hillsborough County HIL

Holmes County HOL

Indian River County IND

Jackson County JAC

Jefferson County JEF

Lafayette County LAF


36/37




County/Agency Codes


Lake County LAK

Lee County LEE

Leon County LEO

Levy County LEV

Liberty County LIB

Madison County MAD

Manatee County MAN

Marion County MRN

Martin County MRT

Miami-Dade County DAD

Monroe County MON

Nassau County NAS

Okaloosa County OKA

Okeechobee County OKE

Orange County ORA

Osceola County OSC

Palm Beach County PAL

Pasco County PAS

Pinellas County PIN

Polk County POL

Putnam County PUT

St. Johns County STJ

St. Lucie County STL

Santa Rosa County SAN

Sarasota County SAR

Seminole County SEM

Sumter County SUM

Suwannee County SUW

Taylor County TAY

Union County UNI

Volusia County VOL

Wakulla County WAK

Walton County WAL

Washington County WAS

Highway Safety and Motor VehiclesHSMV


37/37



County/Agency Codes


Florida Dept of Law Enforcement FDLE

Documents

MQ - File Transfer User Guide v1.2