1

Software and Data Integration Strategy Alaska Oil Production and Distribution Regulatory System

Business Object and Service Oriented Architecture

Veronica Gardner July 2016

Contents Realized Proficiencies ................................................................................................................................... 2

Thesis Statement .......................................................................................................................................... 2

Economy of Scale within the Oil Industry ..................................................................................................... 2

Legacy Database Applications ....................................................................................................................... 2

Integrated System Architecture .................................................................................................................... 3

Non Tank Vessel Contingency Plan Application ............................................................................................ 3

Decision Support Capability .......................................................................................................................... 5

Integration Strategy: Service Oriented Architecture ................................................................................... 6

Type Mapping: Cross Application Data Usage .............................................................................................. 7

Entity Types and Open Backend ................................................................................................................... 8

Dynamic SQL ................................................................................................................................................. 9

Business Object Population and Data Access Layer ................................................................................... 10

Summary and Project Review ..................................................................................................................... 11

2

Realized Proficiencies Tactical and strategic Decision Support System capabilities. Improved data quality, accessibility, and accuracy. Reduction of the code footprints and data redundancy. Decrease in the number of technical personnel required to support the system. Increase the response time for rolling out bug fixes and new software functionality. Improve communications by ensuring everyone interacts with the same accurate information.

Thesis Statement This white paper communicates the software design and implementation strategy used for the system

integration of disparate oil industry regulatory database applications. The integration of both data and

code functionality was accomplished through an infrastructure of standards, autonomous components

and well defined business requirements. The code and SQL snippets, data structures, diagram and

images portrayed within this document are actual working elements of the first phase integration

initiative. Technical discourse encompasses the Service Oriented Architecture (SOA) and ‘System of

Record’ infrastructure with discussion of the backend data structure and data access mechanism. The

Non Tank Vessel Contingency Plan (NTV), is the business application built during the first phase

implementation and is presented in this document to help clarify the information.

Economy of Scale – Oil Industry The ‘Economy of Scale’ of the oil industry plays a significant role in the feasibility for integrating

disparate information systems and data. Regulating the oil industry of Alaska has a fairly low transaction

frequency when compared to other types of industries. The regulations are governed by law and the

certifications of approval for operations are usually valid for years at a time. The high cost of doing

business in the oil industry also limits the number of competing companies and changes to operating

procedures can generally be expensive so are minimized.

Legacy Environment The legacy infrastructure encompasses standalone database applications built for a specific regulatory

function. Some of these applications are developed using recent technology such as Asp.Net with MVC

and Entity Framework. Others are legacy MS Access database applications. These isolated applications

and databases often maintain the same business attributes but cross system comparisons are risky if

even feasible to do so. Often times the data value do not match so linking the same attribute (data

column) across systems is problematic. Many of the text field that should contain the same data value

are spelled different and have varying abbreviations. The date attributes often times differed across

systems so investigations were necessary and the accuracy of the systems became uncertain. This

disparity was a real hindrance to a streamlined information system and our potential for correctness

was greatly diminished. Attempting to use a query command to match across applications using

nonstandard business names at minimum required wildcard searches which can be slower with a

greater chance for erroneous data being returned.

3

Integrated System Infrastructure The infrastructure used for the integration initiative is common throughout the industry and allows for a

secure implementation. Most of the regulatory information is public domain data and is not regulated

under HIPAA or proprietary constraints. The front end applications are outside of the firewall and

accessible by the public. Inside the DMZ resides the reusable services. The services communicate with

the Data Access layer to obtain the data from the data store.

Non Tank Vessel Contingency Plan Application The Non Tank Vessel Contingency Plan (NTV) is the first regulatory application developed with the new

integrated architecture in mind. This application supports the regulatory requirement that all vessels

with greater than 450 GRT that is not a Barge or Tanker and operates in Alaska waters have a

documented and operational contingency plan. These plans cover numerous vessels that can only

operate in designated agreed upon regions of the state. Core integrated components built for the

integration initiative and used by the NTV application include: the Affiliation Tracking Module (ATM);

Document Management System (DMS), Facility Management System (FMS) and a Common Utilities

component which handles user management and encompasses the automated report generation and

distribution functionality.

The User Interface is not a topic of this technical discussion so only two user interface diagrams are

included. They have been added to provide a little bit of visual into how a user can interact with the

integrated components and services. The dashboard for the NTV Plan application is shown in Figure 1.

This user interface provides easy access to tactical level information and record access is configured per

user login. It is important to note that the data shown for the vessels, affiliations and plans are from

Figure 1 - General System Infrastructure

4

completely independent objects and services. They all exist and operate independently of each other yet

can be associated through business rules and logic of NTV or other regulatory applications. Please be

mindful that all data included in this dissertation are example ‘Test’ data and cannot be taken as true or

accurate information.

Figure 3 depicts the Plan Manager page designed for the maintenance of the contingency plan data. It is

possible to control the data displayed for the various roles and authority of the current user. This page

example represents a manager’s authority which includes viewing and editing of all or any plan data. A

Plan Holder or Response Planning Facilitator (RPF) managing their NTV Plan from the public side of the

system will only see the Plans they have created under a predesignated login. Most of the NTV

contingency plan data is public information and law states the data must be made available for viewing

by the public. The user interface shown in this document is for maintenance of the data and caution is

taken to ensure correct authority for any amendments or action taken on a plan.

Figure 2- Dashboard of NTV Internal Application

5

Decision Support Capability The Facility, Regulation and Affiliations are autonomous objects available through different service projects yet related to each other through business logic and associations. This integrated infrastructure can quickly provide the tactical information necessary for accurate operation forecasting and decision making. Examples of tactical queries are listed below:

Which NTV Contingency Plan Holder operates the most vessels in Alaska?

Which country has the highest fuel capacity operating under an Alaska NTV Plan?

Get all NTV Contingency Plan Holders that use ‘SEAPRO’ as their cleanup contractor.

Get all the Vessels that Company One owns and operate in Regions 1, 2 and 3.

Get all the Vessels where the Plan Holder is different than the vessel owner.

Integrating an oil spill response tracking application with the ATM will add a new dimension to the reporting capabilities of the infrastructure. The ATM integration is easy with minimal effort. All CRUD operations are complete, stable and ready for use. A few queries are listed below but this is a small subset of the potential questions that could be answered by further object integration.

What is the total spillage of all vessels under a NTV Plan since 2012?

Tell me which NTV contingency Plan holder is responsible for the most spill incidents.

List all countries and how many NTV Plans for each. What are the gallons spilled for each?

What is the maximum oil remediated per Cleanup Contractor? Incident Management Team?

Figure 3 - Plan Management Page

6

Integration Strategy: Service Oriented Architecture Autonomy, Autonomy, Autonomy

The ‘System of Record’ infrastructure used for the integration initiative promotes the autonomy of both

the data elements and functionality within the system. The authoritative data source is stored in one

physical location and there is one set of functional methods that provide the CRUD operations for that

data. This infrastructure dictates keeping physical control of an object separate from the business

functionality and data rules. Primary and Foreign Key constraints are used to link data tables. These

identifiers can be used to associate and disassociate business objects on the front end but otherwise

have no business value and cannot be directly modified. The logic associations of the objects is

implemented through business rules, functional requirements and the Type Mapping strategy discussed

later in this paper.

To help solve the complex problems of integrating application code across various regulatory entities and requirements a Service Oriented Architecture (SOA) is central. The software structure is organized on a functional business perspective so the various business programs can use the built-in and standard regulatory functionality. The service architecture was designed with a separation of concerns premise to helps ensure the autonomy of information and business processes. A short summary of the the WCF Service projects built for the integration effort is provided below.

Affiliation Tracking Module (ATM): The ATM service project provides all relevant functionality to create and maintain Affiliation records including the contact, phone, and address data. An application is easily registered with the ATM which requires that the application identifier along with the record identifier be input parameters when saving and retrieving affiliation records. The ATM gives every opportunity to associate and reuse affiliation and contact information across application configured to use the system. This mechanism works well within the oil industry because the economy of scale limits the number of interested parties.

Regulation Services: The Regulation Service Project houses objects and functionality relating to any regulatory applications. Contingency plans, financial responsibility, oil spill, and contaminated sites tracking are just a few of those regulations. The first Regulation service implemented in this phase of the integration is the Non Tank Vessel Contingency Plan. Other regulatory applications will be systematically integrated according to the priorities of the business decision makers.

Facility Services: There are numerous oil industry facilities and operations that are regulated by the state of Alaska. The list of facility types include but are not limited to vessels, underground storage tanks, terminal tank farms, production and exploration wells. This list is quite lengthy but facilities have one thing in common: their operation can cross regulatory boundaries and are currently tracked in numerous applications and databases.

o The ‘Vessel: System of Record’, is the first Facility service built within the Facility Services project. The Vessel Services provides all CRUD functionality for Vessel particulars. Maintaining the relationship to a vessel is the responsibility of the parent or subscribing application. See the VesselAssociation table in Figure 6 of this document.

Document Management Services (DMS): The DMS provides the function to store, retrieve, and display uploaded document relating to a specific application record. The DMS can be used with

7

any application as long as the application identifier and the unique record identifier are provided and passed into the service methods along with the document itself.

Common Utility Services: This project is built to provide services for generic utilities whose functionality is needed in numerous applications and components. The Common Utility service project includes the automated report generation and distribution for plan activity and changes. These reports and correspondence documents are systematically uploaded and associated to the plan records.

Type Mapping: Cross Application Data Usage A good Decision Support System is dependent on standard data from all relevant heterogeneous

sources. To maintain these data standards it was necessary to manage the usability of data across

varying requirements and applications. A good example is the use of a Vessel Component in varying

business contexts. An NTV Plan has strict data rules that require a regulated Vessel’s details be

populated before legal operation can commence. For marine oil and hazardous spill tracking, the

available details of the vessel may be unknown especially when the vessel is on the bottom of the

ocean. Type Mapping, a form of a declarative language is used to accommodate the varying data rules

across the multiple applications. Figure 4 shows some of the standard Data Types used to integrate data

across the various regulations and applications. Some of these types have a corresponding Type

Mapping which provides the separation of concerns.

Integrated Data Types Entity Type Entity Identifier: NTV Plan, Spill, Vessel, Oil Production Platform…

Contact Type Primary Contact, 24 Hour Contact, Office Contact…

Affiliation Type Plan Holder, Owner, Operator, Cleanup Contractor, Response Planning Facilitator…

Organization Type Public, Private, Nonprofit, Individual

Facility Type Vessel, Terminal Tank Farm, Production Wells, Exploration Wells…

Vessel Type Fishing, Passenger, Cargo, Chemical…

Plan Status Type Standard operational status – WIP, Active, Terminated…

Status Action Type Used to track actions that change status of a plan: Activate, Terminate, Renew…

Document Type Approval Letter, NTV Application, Certificate for Operation, Reason for Decision…

By mapping the business context to a standard Type list the system can provide application specific

needs yet support the requirement for standardization of data. The Type Mapping mechanism makes it

quick and easy to configure which Type values will be shown in the user interface lists and combo boxes

for any context. This implementation created great advantage for consistency, accuracy and data access

Figure 4 - Dropdown List Types – Representative Only

8

optimization. To Type Map the information it was necessary to ensure autonomy of meaning for each

Type created. The different business functions use different data values for the same Data Type. The

Barge and Tanker Contingency Plans only need their own type, Barge or Tanker. The NTV Plan users

need to see only vessels under their regulation (Cruise Ship, Ferry, Fishing, Chemical…), while a Spill

tracking application interface will need access to the entire Vessel Type list.

The Type Map table that controls which Contact Type can be associated to particular Affiliation Type

within a specified Entity Type (application/facility) context is shown in Figure 5 - Type Mapping Table.

NTV mapping examples include a Primary Contact of an Owner of a Vessel, a Primary Contact of a Plan

Holder of an NTV Plan, or a 24 Hour Contact of a Cleanup Contractor of a NTV Plan. Most of the

regulatory applications (Entity Types) will require and utilize the mapping for a Primary Contact of the

Responsible Party affiliation. These lists are controlled through the column MapTypeID which is 1 for

required or 0 for optional. If the type value is not available for a given Entity Type then that value is not

an option for that business context. The ContactTypeCount column signifies how many times the

‘contact type’ can be used within the current operating context.

Entity Types and Open Backend Each integrated database is designed to allow CRUD operation for non-complete record sets. The

flexibility of the over all system is greatly enhanced because this open backend architecture keeps

business logic separate from the data storage. Throughout the system the operating pair of EntityTypeID

and EntityID is used extensively for the integration of data and functionality. Technically, the

EntityTypeID is an identifier assigned to a database table but logically, it signifies a business objects, i.e.

a regulation or a particular facility. Currently in use are the EntityTypes of Contingency Plans and the

Facility Vessel. Each of these applications utilize the ATM, DMS, and/or the FMS services by passing in

their own assigned Entity identifier along with the record identifier (EntityID). Any EntityID parameter

used without it’s corresponding Entity Type identifier will be ignored. Figure 6 - NTV Integrated Table

Relationships, is a scaled down version of the table structure of the NTV Plan database integrated with

the Affiliations, Documents and Vessels data stores. Each object controls it’s own CRUD operations but

Figure 5 - Type Mapping Table Structure

9

it’s the applications than govern their own rules to add/update/remove a relationship with other

business objects.

Dynamic SQL Dynamic SQL is used to help build flexibility into the integrated system by returning datasets for a

variety of criteria. Dynamic SQL is a little slower due to the compilation so thought and caution was

taken during the implementation to ensure quick response from the data store. The stored procedures

never apply business logic and always use the identity columns in the relational joins among the tables.

Care was taken using this implementation to ensure proper use of returned datasets. Data returned

within a transactional context (CRUD functionality) generally need far less data records returned as

compared to usage of data records for reporting purposes. The stored procedure NTVPlan_SELECT is

shown in Figure 7 and is a simple representation how the dynamic SQL is implemented.

Figure 6 - NTV Plan Integrated Table Relationships

10

CREATE PROCEDURE [dbo].[NTVPlan_SELECT] @NTVPlanID INT = NULL, @PlanNumber varchar(10) = null, @PlanStatusID int = null, @EntityTypeID int = null AS BEGIN DECLARE @WhereClause varchar(max) SET @WhereClause = '' IF(@NTVPlanID > 0) BEGIN SET @WhereClause = N'WHERE p.NTVPlanID = ' + CAST(@NTVPlanID AS varchar(20)) END ELSE BEGIN IF (@PlanNumber != '') BEGIN SELECT @PlanNumber = RTrim(LTrim(@PlanNumber)) SET @WhereClause = N'WHERE p.PlanNumber = ' + '''' + @PlanNumber + '''' END ELSE BEGIN IF(@PlanStatusID > 0) BEGIN SET @WhereClause = N'WHERE p.PlanStatusID = ' + CAST(@PlanStatusID AS varchar(20)) END IF(@EntityTypeID > 0) BEGIN IF(@WhereClause = '') BEGIN SET @WhereClause = N'WHERE p.EntityTypeID = ' + CAST(@EntityTypeID AS varchar(20)) END ELSE BEGIN SET @WhereClause = @WhereClause + N'AND p.EntityTypeID = ' + CAST(@EntityTypeID AS varchar(20)) END END END END EXEC ('SELECT p.NTVPlanid, p.PlanNumber, ps.PlanStatusType, pt.EntityType, p.PlanActivationDate, p.PlanExpirationDate, p.VesselInAlaskaDate, p.ApplicantTypeID, p.PlanStatusID, p.UserID, p.EntityTypeID,sde.UserName, p.RPFTypeID, p.ApplicationSubmittedByID, p.ApplicationCertifiedById, p.CreateDate, p.ModifyDate FROM DB1.NTVPlan p inner join DB1.PlanStatusType ps on p.PlanStatusID = ps.PlanStatusTypeID LEFT JOIN DB2.dbo.EntityType pt on p.EntityTypeID = pt.EntityTypeID LEFT JOIN DB5.dbo.Users user on user.UserID = p.UserID ' + @WhereClause

Business Object Population and Data Access Layer All data in the business objects are maintained via that objects CRUD methods. This includes the data

access and conversions of the List<T> child objects a component may have. Each access method is

autonomous along with each object conversion method. The data access layer receives record result

sets in either data table or data set format. The database result sets are converted to the corresponding

object structure and all business rules and processing thereafter is applied to that object. Figure 8 is the

code snippet for converting an NTVPlan data table result set from the NTVPlan_SELECT method into the

NTVPlan object structure. The List<t> child properties of a NTV plan (i.e. List<VesselAssociation>,

List<Affiliation>) have their own conversion and data access methods therefore are not included in the

conversion method for the plan itself. There is no business logic within these conversion methods other

than the annullable attributes.

Figure 7 Dynamic SQL within the Stored Procedures

11

List<models.NTVPlan> NTVPlanList = new List<models.NTVPlan>

( NTVPlanDataStoreResults.AsEnumerable().Select(O => new models.NTVPlan { NTVPlanID = O.Field<int>("NTVPlanID"), PlanNumber = O.Field<string>("PlanNumber"), EntityTypeID = O.Field<int>("EntityTypeID"), EntityType = O.Field<string>("EntityType"), PlanStatusID = O.Field<int>("PlanStatusID"), PlanStatus = O.Field<string>("PlanStatusType"), PlanActivationDate = O.Field<DateTime?>("PlanActivationDate"), PlanExpirationDate = O.Field<DateTime?>("PlanExpirationDate"), VesselInAlaskaDate = O.Field<DateTime?>("VesselInAlaskaDate"), PlanHolderTypeID = O.Field<int?>("ApplicantTypeID"), UserID = O.Field<int>("UserID"), CreateDate = O.Field<DateTime?>("CreateDate"), ModifyDate = O.Field<DateTime?>("ModifyDate") }));

Summary and Project Review The strategy used to integrate data and functionality across the various applications has been extremely

successful. The integration eliminates redundant code, improves data accuracy, and promotes DDS

capabilities. The SOA service projects and components so far include the Affiliation Tracking Module, the

Document Management System, the Facilities Services (Vessel system of record), and the Common

Utilities services. The NTV Plan functionality within the Regulation Services is the first of the regulatory

service structures to be implemented. The ATM is one of the more complex integrated component

therefore is the thesis of the next paper covering this integration initiative.

All of the technical discourse, code, SQL snippets, diagram, and images within these documents were

designed and implemented for the integration initiative and the development of the NTV Plan

application. The initiative took three developers dedicating at minimum 50% of their time approximately

a year and a half to complete the first phase development. It was the first time the team worked

together on a project and all members had other duties assigned throughout the duration of the

initiative.

Software Implementations and Tools that Worked Well

Separation of Concerns greatly improves the flexibility for adding features and functionality.

Open backend infrastructure fully supports the services reuse across varying business rules.

The Types and Type Mappings Declarative Markup Language provides the foundation for much

of the decision support capabilities and improves the front end user experience.

SOA architecture is key to this integration initiative and is the corner stone of the

implementation.

Figure 8 Convert NTV Plan Data Access Results to NTV Plan Business Object

12

Coding and Naming Standards significantly improve enhancement capabilities, troubleshooting

efforts, and the ability for developers to understand and maneuverer the system.

Improvements and Considerations

Needed more developers (or less work) to take the time to design the NTVPlan and NTV

Certificate tables as a more generic implementation that could accommodate other regulations.

Note: NTV Plans are not governed through the same regulations as all other contingency plans.

Some say SQL IDENTITY columns should not be used for primary / foreign key implementation.

Would it be better to use some other type of identifier? These Identity columns make

development of the backend so easy.

Some database programmers do not like the lack of database referential integrity for the

relationship between the Plan, Affiliation, Document and Facility databases.

Would have liked to create the service and model layout that fully accommodates a Restful

architecture. No team expert on Restful and too little time to prove a good ROI for its

implementation.