DataPortal - Database Sharing In the CloudA simpler way to transfer complex data

Connection Concepts, Inc.2141 Industrial Parkway, Suite 202

Silver Spring, MD, 20904-7824, USA860 729-3117

con2inc.comdata-portal.biz ; directtodb.com

Contact: Gary Whitten, Pres., CIOwhitteng@con2inc.com

2

Data Sharing Today

Sharing structured, complex data (e.g. Relation Database Data)

is too burdensome

Goal: “YouTube for data”● Push-button sharing of complex data over the Web● Agile - Instant, Effortless Publishing, Retrieval

Data Complexity/ Value

Ease ofTransfer

PracticalLimit

Simple: Convert to flat files Structure, Information and Value is lost

Difficult: Legacy EDI or similar (Electronic Data Interchange) • Requires development• Rigid, Inflexible• Requires Conversion for data Provider AND Consumer• Requires Firewall Modification

Current Options:

3

Database Sharing Needs

Experts call for sharing data in its Experts call for sharing data in its originaloriginal form: form: (from CNE(from CNET January 27, 2009 http://news.cnet.com/8301-13739_3-10150699-46.html)

David Robinson Associate Director of the Center for Information Technology Policy at Princeton University:"(no) one person or organization could possibly anticipate all the ways that Americans will want to analyze, reuse, or cross-reference the information that Recovery.gov will offer. And no one person or organization needs to do so, as long as the data itself is readily available."

In 2008, Robinson and his colleagues at Princeton published a paper calling for the government to provide open access to the raw data used by all federal Web sites. The highly influential paper has been widely circulated among technology policy circles in recent months.

Jim Harper Director of Information Policy Studies at the Cato Institute: ...the entire back-end database should be made available."This is a little tricky, because people have to settle on a format, and then require submissions in that format from contractors and state and local entities, etc.,"

4

Database Sharing Needs

Experts call for easier, more agile data sharing:Experts call for easier, more agile data sharing: From Information Week, June 21, 2010, cover story (page 26) From Information Week, June 21, 2010, cover story (page 26) “Share”,“Share”, “Why IT Needs To Push Data Sharing Efforts” “Why IT Needs To Push Data Sharing Efforts” http://www.informationweek.com/news/services/integration/showArticle.jhtml?articleID=225700546

... one of the single best things (IT departments) can do for their businesses is enable effective data sharing. … Yet data sharing, particularly automated systems that give your external business partners access to your data when they want it, are not ubiquitous or easy, and the level of data sharing of any kind is surprisingly low.

*******************************

When it comes to what frustrates data sharing efforts, the classic culprit, budget limitations, tops the list of survey respondents, followed by complaints about the multiple sets of tools and the care and feeding required by legacy connections.

*******************************

A bigger problem is if IT simply doesn't have the ability to respond quickly to new requests. Too many companies can't move fast enough.

5

Database Sharing Today

Need for more agile Supply Chain managementNeed for more agile Supply Chain management

Real world example - Real world example - Sikorsky Aircraft Company:Sikorsky Aircraft Company: Resistance from potential new suppliers to burdensomeResistance from potential new suppliers to burdensome EDI process to share order data EDI process to share order data

6

DataPortalDatabase Sharing In the Cloud

DataPortalDataPortal transfers business data transfers business datainin database form database form over the over the WebWeb

Appropriate when source and destination data reside in a Appropriate when source and destination data reside in a relational database systemrelational database system

• No development• Works across database vendors• Minimal setup and configuration (no manual installation for Web client)• Serves a wide audience• Instant database migration

• SSL encryption• Multiple layers of password protection• No firewall modification• Data NOT exposed directly to network• No unnecessary conversion at either end• Maintains database value and complexity

DataPortal - “Push-button” solution for instantlysharing complex database data in the Cloud

7

DataPortal Operation

Client requests a DataSource (published database)

DatabaseSystem

DataPortalClient

DataPortal ClientHost

Web

HTTP

ClientRequest

DatabaseSystem

DataPortalServer

DataPortal ServerHost

8

DataPortal Operation

DatabaseSystem

DataPortalClient

DataPortal ClientHost

Web

HTTP

DatabaseSystem

DataPortalServer

DataPortal ServerHost

Snapshot of requested DataSourceis transferred over the Web, throughunmodified firewall, to user's preferreddatabase system

available for immediate processing using existing infrastructure

ServerResponse

9

DataPortal Write Data Modes

Creates and populatesnew database – replacesAny pre-existing databasewith same name

Append

OriginalData

Data to beWritten

ResultingData

Edit

OriginalData

Data to beWritten

ResultingData

OriginalData

ReplaceDatabase

Data to beWritten

ResultingData

Writes new dataover old

Appends new dataAfter old

10

Shared Data Selection Options

Shared Data Selection Options

• Share Full Database Structure and Content

• Share Only Selected Tables

• Share Data Filtered by Authenticated ID

11

Data Sharing Usage Example

Broadcast – central database is distributed to many partners (e.g. - Replace Database mode)

DataPortalClient

DataPortalClient

DataPortalClient

DataPortalClient

DataPortalClient

DataPortalClient

DataPortalClient

DataPortalClient

DataPortalServer

12

Data Sharing Usage Example

Merge and Integrate – many partners add data to central database (e.g. - Append mode)

DataPortalServer

DataPortalServer

DataPortalServer

DataPortalServer

DataPortalServer

DataPortalServer

DataPortalServer

DataPortalServer

DataPortalClient

13

DataPortal Security

DataPortal Security

• Uses Web standard SSL (Secure Socket Layer) for• Encryption• Authentication• Other Web security standards can be applied when available

• Mulitple levels of password authentication• Web level• Application level• Database level

• Data filtering based on authenticated ID

• Database never exposed to network (stays behind firewall)

14

DataPortal Applet Client

● No installation required

● Uses standard Java-enabled Web browser

Usage:● URL selects DataPortal Server● Selection Fields

DataSource Data Destination DB Vendor Host/Port User/Password Initial DB (if required) New DB Name Info/Status Display

DataPortal Clientwithin Web browser

*** standalone client also available *** standalone client also available **

15

DataPortal StandaloneClient Application

● Includes all functionality of Web client

● Defines and saves data transfers

● Web listener transfers data based on authenticated Web requests from programs or browsers

DataPortal ClientStandalone Application

Transfer Control View

DataPortal ClientStandalone Application

Listener View

16

DataPortal Server ManagementPublishing a DataSource

Add DataSource Web Form DataSource Form Parameters DB Vendor Type

Host/Port

DB Name

DataSource Name (as seen by client)

Database User/Password

Min/Max Number of Connections

Require Client User/Password for DataSource access

17

DataPortal Security Design

DataPortal Security Diagram

DataPortalServer

DataPortal ServerHost

RDBMSDataPortalClient

DataPortal ClientApplet

Applet GUI

ManualSubmission

RDBMS

RDBMS

DataPortalClient

DataPortal ClientApplication

Application GUI

Manual Submission

HTTP/SSLListerner(s)

Manual(Browser)

Submission

ProgramaticURL

Request

Web FormURL

Request

Standard Data Path

SSLData Path

18

DataPortal Transfer Integrity – Implicit Verfication

● DataPortal transfer requires many Java component actions

● Each action is monitored for material exceptions

● Data corruption without exception is highly unlikely(supported by testing)

● Transfer is treated atomically - If any action throws any material exception, the entire transfer is aborted– Transfer results reported– Transfer can be repeated, in its entirety, until successful

DataPortal Data Transfer Integrity:Implicit Verification

19

DataPortal Transfer Integrity – Explicit Verification

● Element Size at Destination is compared to Size at Source – can be done inline, not currently included in standard transfer (overhead not justified)

● DataPortal infrastructure is used to compare structure and data of two DataBases, across DB vendors – Compare 2 DB separate utility

DataPortal Data Transfer Integrity:Explicit Verification

20

DataPortal Data Transfer Integrity:Database Compare Utility

DataPortal infrastructure is used to compare structure and data of two DataBases, across DB vendors

Example:In a DB of ~150 MBs, an improper handling of the “ ' ”

character by the database was discovered

21

DataPortal Deployment

DataPortal Deployed by loadingWAR (Web ARchive) File intoServlet Engine

Client Applet JAR (Java Archive) Fileis contained within WAR File

Client is automatically downloaded andinstalled into Web browser

DataPortal ServerHost

Web Server

Servlet Engine

WAR File(Server)

JAR File(Client)

22

DataPortal Demo

23

Patented DataPortal Technology

DataPortal IP Status:DataPortal IP Status:Connection Concepts awardedConnection Concepts awardedU.S. Patent 7,346,635 for DataPortal technologyU.S. Patent 7,346,635 for DataPortal technologyConnection Concepts owns patentConnection Concepts owns patent

24

DataPortal Advantage

Web

CSV,XML...

Export

Convert

Dilute

DatabaseData Source

CSV,XML...

Setup

Convert

Import

DatabaseData Destination

DatabaseData Source

DatabaseData Destination

Web

DataPortalDataPortal

Traditional Database Data Transfer Over the Web

Direct Database Data Transfer Over the Web with DataPortal

Data Complexity/ Value

Ease ofTransfer

PracticalLimit

DataPortal

Data Complexity/ Value

Ease ofTransfer

PracticalLimit

• With DataPortal, data is read/written directly from/to database without conversion to intermediate formats

• More complex, high value data can be transferred with the ease of a simple flat file

25

DataPortal Value

Users can share complex database data over the Web...

● directly from/to their databases (provider and consumer) without conversion to intermediate forms

● immediately● effortlessly (“push-button” ease)● without development effort or need for programmers● directly from the browser so consumer does not need to install

software● without limit on data complexity (value)

e.g. tables, primary/foreign keys, images, docs... ● while serving a wide audience● no firewall modification

Complex data can be shared among millions of users within minutes

26

Standards-Based Solution

DataPortal uses standards based technologiesDataPortal uses standards based technologies● Relational Database Systems – standard for storing dataRelational Database Systems – standard for storing data

(e.g. Oracle, DB2, Access, MySQL, SQL Server...) (e.g. Oracle, DB2, Access, MySQL, SQL Server...)● WebWeb● SQL (Structured Query Language)SQL (Structured Query Language)● Standard port – no firewall modificationStandard port – no firewall modification● Client runs in browser – no software installationClient runs in browser – no software installation● SSL (Secure Socket Layer) for authentication and encryptionSSL (Secure Socket Layer) for authentication and encryption● JavaJava● Server deployed as Java Servlet within Web serverServer deployed as Java Servlet within Web server● Java Applet runs in Web browserJava Applet runs in Web browser● JDBC – Java standard for database connectivityJDBC – Java standard for database connectivity

27

DataPortal Applications

DataPortal applicationsDataPortal applicationsDataPortal is a potential solution whenever parties want toDataPortal is a potential solution whenever parties want toquickly and easily share database dataquickly and easily share database data

● Many existing or new EDI applications (e.g. Supply-chain management, Many existing or new EDI applications (e.g. Supply-chain management, Order Processing or Status reporting...)Order Processing or Status reporting...)

● Putting catalog, product and pricing info on the WebPutting catalog, product and pricing info on the Web● Supplying sales force with current data in the fieldSupplying sales force with current data in the field● Government data sharingGovernment data sharing

● Providing users original data for data-centric public facing Web sites (data.gov, Providing users original data for data-centric public facing Web sites (data.gov, usaspending.gov, recovery.gov..)usaspending.gov, recovery.gov..)

● Public data reporting to government (e.g Recovery funds usage reporting)Public data reporting to government (e.g Recovery funds usage reporting)

● Inter-agency/department data sharingInter-agency/department data sharing

DataPortal Potential: Standard for Sharing Business and Government Data

28

DataPortalServer

DataPortal ServerHost

DatabaseAbstract DB Structure, Data

Representation Serialized Objects(DB Vendor Neutral)

DB Structure, DataRequests

DB Structure, DataInfo

ToDataPortal

Client

DB Vendor

“A”

DataPortalClient

DataPortal ClientHostCreate DB, Tables

fromDataPortal

Server

Database

Populate Tables

DB Vendor “B”

DataPortal Design

29

Competitive Advantages

Non-Web based competition:EDI (Legacy), Database Development ToolsEDI (Legacy), Database Development Tools● Difficult to use – intended for programmersDifficult to use – intended for programmers● Restrictive – must conform to specific formats or database Restrictive – must conform to specific formats or database

vendors/versionsvendors/versions● Requires firewall modification,Requires firewall modification,

network programing network programing ● Client must install softwareClient must install software● Must add in security and authenticationMust add in security and authentication● Persistent connections use network resourcesPersistent connections use network resources● Intended for narrow audienceIntended for narrow audience

30

Competitive Advantages

Web based competitionWeb based competition● CSV, Excel filesCSV, Excel files

– Requires conversionRequires conversion– Limits complexity/value of dataLimits complexity/value of data

● XMLXML– Requires conversionRequires conversion– Requires programmingRequires programming– Limits complexity/value of dataLimits complexity/value of data– Inefficient – high data overheadInefficient – high data overhead

● APIAPI– Requires programming, expertise, time, effortRequires programming, expertise, time, effort

Data Complexity/ Value

Ease ofTransfer

PracticalLimit

DataPortal

CSV

XML

API

31

Where Can DataPortal Be Used?

Provide Standard Tool for Sharing Complex, Structured DataProvide Standard Tool for Sharing Complex, Structured Data Within or Across:

● DepartmentsDepartments● OfficesOffices● PartnersPartners● SuppliersSuppliers● CustomersCustomers● Sales ForceSales Force● Government (Federal, State, City, Local)Government (Federal, State, City, Local)● ……....

Contact Information:

Connection Concepts, Inc.2141 Industrial Parkway, Suite 202Silver Spring, MD, 20904-7824, USA

http://data-portal.biz ; http://directtodb.comhttp://con2inc.com

Gary Whitten, President., CIOwhitteng@con2inc.com860 729-3117

Database Sharing In the CloudA simpler way to transfer complex data