GAA Presents "goMongo" and HayStack

© Copyright GAA CorporationGAA CONFIDENTIAL

www.gaaconsulting.com

Building a Smarter Enterprise In a Services and Knowledge-based Economy

Migrating Legacy Data: goMongo

Dr. Naveen KapoorGAA [email protected]

March 2013

GAA CONFIDENTIAL © Copyright GAA Corporation


2 March 2013

Objectives and Topics

Objectives Demonstrate how GAA is converting legacy databases from SQL to

mongoDB. Discuss lessons learnt using multiple conversion projects Share knowledge of embedding and linking Maintaining referential integrity in mongoDB

Topics Background on GAA

Experience of transformation from relational to mongoDB

GAA’s solution to ease the transformation process

goMongo: Conversion tool from relational databases

Joins, embedding and linking in mongoDB

Appendix

Background on migration projects

GAA Solutions

AAS Audit Graphs

On Demand Reporting (ODR)

Dashboard



Who is GAA ConsultingWe are a small business with over four years of experience serving Federal and Commercial clients


Our People• Services ranging from architecture, system support, on demand reporting and analytics• Resources skilled in Oracle, Java, Web services, SOA, Javascript, Web development

• Dedicated team that supports its clients on a 24X7 basis

Our Clients

Department of CommerceUS Economic CensusGSA, USGS, VA, DOT

Our Differentiators• Proven, well established production level systems, system support

• Seven (7) outstanding ready-to-integrate solutions at various Federal Agencies• Highly skilled resources to support legacy and new application builds

• Dedicated team to support 24X7 data center operations

JAN 2013JAN 2013

GAA Integration, Architecture and On Demand Reporting

Company Overview

3

http://www.gaaconsulting.com/



Authentication and Authorization Solutions

March 2013

Authentication- AAS

Authorization- CAMS

Enterprise- SSO

GAA4

Converted to mongoDB from

Oracle and MySQLUsing goMongo



Demonstrations

CAMSRelational Database

mongoDB

goMongoConversion

Embedding and Linking

5 March 2013



Converting CAMS: Our experiences with goMongo

March 2013

CAMS is written using JavaScript, PHP, AJAX, Web Services

Data Storage: Oracle RAC

High Availability: Yes

Current Loads: 50K transactions/day

JavaScript: Over 4000 lines

PHP Script: Over 2000 lines

CAMS on mongoDB

Data Storage: mongoDB

High Availability: Replicated

Current Loads: New release

JavaScript: Over 4000 lines – made 10 changes

PHP Script: Less than 500 lines

Time to Convert: 1 week

6



Transformation: GAA provides goMongo to ease the transformation process

March 2013

Step1: Transfer all relational content to mongoDB

Step2: Identify PK/FK for tables in mongoDB

Step3: Convert and Test using “Embed or Link” PK/FK relationships

7



goMongo – Web based tool

March 2013

• Connect to the database• Oracle, mySQL• Provide u/p• Select DB

Select tables

• Convert the tables to mongoDB

• _id’s are generated automatically

Apply Node/Leaf

Rule • Generate foreign associations as embedded array elements

• Generate foreign associations as links

Test the result

8



Connect to Relational Database (Oracle)

1. Select Connection

stored in CAMS

2. Tables loaded from

the database

3. Tables to collections

4. Click Convert.Done!!!

Conversion maintains data types for strings,

integers, floats and date time.

March 20139



View the mongoDB

1. View all mongoDB databases

and collections

2. Select a collection to

view documents

3. Click to edit it in

Document Editor

10 March 2013



goMongo Document Editor

JSON is loaded. Click on any field to view and

edit

11 March 2013



Embedding and Linking with goMongo

Preserving the capability for:Self referencing

Application has sub-applications

1-many relationships A Classroom has many students

Many-many: would like to embed or link all users in the group collection. USER (ID)

GROUP (ID)

USERGROUP(USERID, GID)

12 March 2013



goMongo addresses all typesOver 4K embeds or linkages in under a second

1. Identify Primary,

Foreign and Brokerage Collections

3. Identify Brokerage Collection

fields

2. Identify Fields in

Primary and Foreign

Collections

4. Select option for embed or

link and label in FC

13 March 2013



Node/Leaf Rule: 2 Simple rules for converting Associated entities

14 March 2013

Rule 1: If entities are leaf then Embed the associationsRule 2: If entities are Nodes Link the associations

ApplicationAID = 1

PID = 1

Application Name = ODR

Permissions

ID = 1Permission = Create RoleID = 1

Permission = Create User

mongoApplication

_id : 1

PID : [{“0”:{“id”:0,”perm”:”create role”}, “1”:{“id: 1,”perm”:”create user”}}]

Application Name : ODR

Embed the Permissions EntitySince it has no sub entities.Permissions entity is a leaf



Node/Leaf Rule: 2 Simple rules for converting Associated entities

15 March 2013

Rule 1: If entities are leaf then Embed the associationsRule 2: If entities are Nodes Link the associations

ApplicationAID = 1

RID have multiple roles

Application Name = ODR

RolesRID

GID have multiple Groups

mongoApplication

_id : 1

roles : [_id_role1,_id_role2]

Link the Roles EntitySince it has sub entities ie Groups.

Groups

GID = 1

RID = 1

Group Name = Admin Group

Groups

GID = 2

RID = 1

Group Name = Admin Group

mongoRoles_id _role_1

groups: [_id_group1,_id_group2]

mongoGroups_id _group1

Group name: Admin Group



Joins in MongoDB

March 2013

Function: JoinTables

No. of Arguments: 6Arg Name Type Description

1 $PT String Primary Table which has the unique set of values

2 $FT String Secondary Table which has the repeating set of values

3 $IF String Link Field. This is the string value that is used in the $FT to link to the $PT. The $FT will have an array of values which is stored under the $lf label

4 $matchArray

Array Matching value for example array('USERNAME'=>'kapoo304'). The USERNAME field is in the $PT collection

5 $PTC Array Array of columns that you want from the $PT. for example: array('USERNAME'=>'User') (Actual table column and display column)

6 $FTC Array Array of columns that you want from the $FT. for example: array('GROUPSNAME'=>'Group') (Actual table column and display column)

Response: An array of results which shows the values that match the matchArray. Use the response from this to pass to the first argument of the joinArray function.

16



Joins in MongoDB – Cont…

March 2013

Function: JoinArray

No. of Arguments: 6Arg Name Type Description

1 $PA Array Primary Array which has the unique set of values

2 $FT String Secondary Table which has the repeating set of values

3 $IF String Link Field. This is the string value that is used in the $FT to link to the $PT. The $FT will have an array of values which is stored under the $lf label

4 $columnMatch

String Name of column from the $PA that matches in value in the lined values in $FT under the $lf label

5 $PTC Array Array of columns that you want from the $PA. for example: array('USERNAME'=>'User') (Actual table column and display column)

6 $FTC Array Array of columns that you want from the $FT. for example: array('GROUPSNAME'=>'Group') (Actual table column and display column)

Response:

17



Data Abstraction Layer and GAA driver for mongoDB: A logical representation of mediation to support noSQL

18 March 2013

Data Abstraction Layer

Database Specific DriverExample: mySQL driver GAA driver for goMongo

MySql mongoDB

Application Server



AppendixNode/Leaf Rules

GAA Products using goMongo

19 March 2013



AAS – Authentication and Authorization Service

March 2013

AAS Server

Session Information

Authorization Information

LDAP Server

Manage SSO

Verify users credentials and get info

SOAP Service requests

Web Browser

APP/DBServer

SOAP Servicerequests

RCP

Request Pages

CAMSWeb

based

Manage Authorization

Manage Authorization

Forms/PL/SQL

AdHocQueries

20



Single Sign-On using SAML and Kerberos Certificates

March 2013

Value Creation:1) Application Users can

authenticate against single enterprise source of identity.

2) Provides high levels of security3) No passwords are transferred

over the wire4) Integrate SSO easily with new

and legacy applications using .jar, and web services that can be leveraged across multiple software languages.

5) One call to authenticate and authorize user.

6) Sessions are maintained in the database as opposed to files; easing scaling and high volume traffic

7) User transactions can be logged and integrated with analytics engine

8) CAMS provides the historical charts on transactions by user, by application and time periods

2121



Authorization - Increasing reusability with CAMSGovernance, Security, Control, Monitoring, Management • As an enterprise how do we manage the growing interconnections amongst applications?• What if I move the end point of the service providers? • Rapid provisioning of new business applications, services and service providers. • Monitoring, Security, Flexibility and Resiliency needs to be ensured

22 March 2013 MARCH 2013

Deliver architecture using services• Provide Security, Governance, Monitoring, Auditing and Control• Load balancing to support a Federated service delivery platform

• Identify and deliver new services by leveraging SOA suite tools: BPEL

Business Entities LDAPAuthentication Service

SOA Governance1. Security Policies (HTTPS, SLA)2. Business Rules (Content Based Routing)3. MonitoringMonitoring and rapid growth:4. Monitoring SLAs5. Integrate business services through ESB’s exposed end points to

create new business servicesBusiness Applications

Legacy applications

Service Registry & LogsStores WSDLs , End Points for HA, LB, DRM & Logs

ESBRoute dynamicallyTransport formats

Mediate and DR, LB, HA

Load balanced, highly available authorization

information

Logs

AASAuthentication and

Authorization Services



CAMS: Business Owners can manage and govern consolidations, applications and entitlement

23 March 2013

Click user to add to the group

Click a groupto manage Roles,

RLP, Users

Click on user to remove from group

Value Creation:1) Multiple applications and their

owners can assign permissions to users via groups and roles.

2) Application space can be further broken down into sub-applications

3) Supports single sign-on.4) Supports web services with

adapters using REST or SOAP protocols

5) Configurations for applications can be centralized

6) CAMS as an application introspects to provide access to manage privileges for other applications

7) Business Owners manage who can do what

8) Rapid deployment9) Business owners, developers,

testers and deployment staff are consumers of this application

10) Proven solution



AAS Web services that provide flexible integration across the enterprise. Supports both REST and SOAP based protocols.

24 March 2013

Value Creation:1) Supports single sign-on for

various applications written in Java, PHP, Python, JavaScript, Oracle PL/SQL

2) Core web services based technology that supports both SOAP and REST based web services call

3) Supports authentication by integrating with LDAP servers

4) Load balanced and highly available to address disaster and performance issues

5) Sessions are managed in MongoDB for high performance as opposed to a file based management system

6) Scales up to 100,000+ transactions per day



GAA Solutions

March 201325



AAS audit graphs: Monitor and track usage of applications

March 201326

• View delivered services

• Governance• Management• SLA

Compliance• PDF Reports• Authorization

Network



CAMS delivers On-demand reporting: Mashup content from diverse platforms while delivering single-view to the Client

March 2013 MARCH 2013

Problem:No centralized reporting system that will integrate with systems and applications

Solution:On demand reporting that integrates with applications, LDAP, Directory and servers

Value Creation:1. Users can formulate their own

request for information via an email

2. Information can be retrieved on an on-demand basis.

3. Integrates with any applications, system support, Remedy, COTS products.

4. Services both mobile and standard platforms

27

Documents

GAA Presents "goMongo" and HayStack