BP307: IBM Lotus Domino 7 and DB2Effective Use of Access and Query Views

Dr. Ludwig W. NastanskyIngo Erdmann

Dr. Ludwig W. Nastanskyprofessor of business information systems

head of Groupware Competence Center (GCC)

in Lotus Notes and Domino research & business since Release 1

founder of several university spin-off companies

co-founder of German Notes User Group (DNUG)

Ingo Erdmannresearch assistant

9 years of Lotus Notes experience

certified instructor, developer, admin

Speakers

Agenda

1 Introduction2 DB2 Access Views (DAV)

scenariolive demo

3 Query Viewsscenariostechnology, tipsfederated data paradigmlive demos

4 Summary

1 Introduction

Session focus on query view and integration scenariosGoals

put Lotus Notes and Domino DB2 integration in right contextlearn about innovative efficient integration optionsenvision new approaches for application developmentstart thinking about required changes in infrastructuretake home tips for first implementations

No NSFDB2 basics will be covered

Topic for presentation: Specific aspects of NSFDB2The new RDB-functionalities provided by DAV & query viewsRDB functionalities in Notes have been an issue since Release 2Many successful approaches up to now

DECSLEISAP connector

NSFDB2 is the smoothest and most radical integration everBesides that, even the DB2 aficionados will finally admit that Domino is based on a real & powerful database engine …… the core issues are unchanged

integration is not (only) about technologybut (much more) about application domains, IT-strategies, top-down system architectures, functionality concepts

Notes/Domino & DB2 Integration, finally… ?

Documents vs. Relational Data

© Delphi Research

Lotus Notes andDomino World DB2/RDB World

Lotus Notes and Domino World

Documents vs. Relational Data

strategic orientation & communication centricknowledge & information managementtool paradigm on objects – code re-usecompound documents, semi-structured, very flexible data typesmultimedia, links, embedded methodsdecentralized, buttom-up, user workplace & collaboration centricreplication, information sharing, robust distribution, redundancy, message objectsindex engine optimized for static access to unstructured datasupport of mobile, nomadic and disconnected user workplace

DB2/RDB World

operational orientation & data centrictransaction processing & high volumeautomation paradigm on data – code efficiencyrecords, tables, structured data, restricted flexibility, strict formatstransactions, dynamiccentral organization, top-down, system centric

access coordination, referential integrity, redundancy elimination, 2-phase commit, ACIDindex engine optimized for dynamic access to structured datastatic office-based and server-connected workplace

Integration, Cooperation, Federation

Challenge: bring together the best of both worldsas much as needed and as much as makes sense from a business value perspective

Position: Domino RDB integration is just one instance of the innumerable system integration tasks taking place currently

The integration is between equal partnersstay cool, no religion involved, don't demonize

Purposeintegration of data across system boundaries

cooperation of separate applications

federation of Domino applications with DB2-data and, vice versa, DB2 applications with Domino data

NSFDB2 brings RDB/SQL functionalities to Notes and Domino

Let us start with one principal data modelling challenge using a typical scenario

Solution in Domino is clumsyWe need to hard-code relational dependancyby duplicating data in documents and thuscreating redundancy- Option #1: Lookup "LastName" in "Employees" view and save in Room-form- Option #2: Lookup "Room" in "Room list" view and save in Employee-form- [Option #3: Enforce consistent user entry in related documents]

#1 #2

Start: twoindependentlists

Goal: modelEmployee-Roomrelation

Scenario 1 - Model Employee–Room Relation

Domino challenge: Model Employee–Room Relation

Problem:View "Employee in room" cannot be generated without duplicating "Room"-field and "LastName"-field in "Employee"-document and "Room"-document respectively

reason: @LookUp not allowed in view columns

Synchronization has to be modelled

Modelling the synchronization is possible, but clumsy, e.g.use scheduled agents to collect & update changes

set up (very) disciplined user entry infrastructure enforcing consistent update over involved document collection

Avoiding redundancy of field entries would help considerably

Solution with new options of NSFDB2: Data duplication is not necessary using DB2 Access Views (DAV) and query views

Remark

Slides marked with the "mouse" symbolwill be covered during live demo sessions

Demo 1: Relations

SQL JOIN to find employee in room

Show room documents

Room documents do not contain the employee name

Use SQL join to find "employee in room"

DB2 Access Views and Query Views

NSFDB2

DB2

update, insert, deleteupdate, insert, delete

Domino data flowDAV related data flowControl

Domino

Domino Managed Data

SQLApplications

redundantlystore

Access Views

Access Views

Access Views

Access Views

DAV

read

read

read

DB2Access

DB2 data flowAccess Views

Access Views

form basedselection

user basedselection

DB2 Table

DB2 View

DB2 Managed Data

NotesQueryView

Notes Client

Agenda

1 Introduction

2 DB2 Access Views (DAV)scenariolive demo

3 Query Viewsscenariostechnology, tipsfederated data paradigmlive demos

4 Summary

DB2 Access Views

DAV - Overview

Redundantly store Domino data in DB2 table

DB2 view that corresponds to the table is used for SQL operations

Each row represents one Domino document

DAV is not a Notes UI element, i.e. not Notes standard view

Expose Domino data toSQL applications

JDBC, ODBC

DB2 tools and applications

relational reporting tools (Crystal Reports etc.)

query views

No DECS, LEI, connectors needed

DAV – Field Definition

Field 6

DAV DesignDocument

Field 5

Field 4

Field 3

Field 2

Field 1 Fields are elementsof forms

Fields are itemsin notes

New fields, that do notexist in Domino database yet

etc. …

DAV – Architecture Summary

Using SQL, you canread Domino data with security semantics enforcedfrom an SQL perspective, this adds "row level security" to DB2 datainsert, update, delete with full Domino semantics

DB2 handles security for read operationsDB2 Access for Lotus Domino handles Domino security on insert, update, delete

DAV calls DB2 Access serverDomino handles replication conflicts, document locking etc.Domino handles ACL, reader fields etc.user mapping required for security(Domino Administrator)

Scenario 2: Integration #1

Application and data integration demands for Domino and DB2

analysis using MISreporting using Crystal ReportsHR uses existing DB2 applicationsalesforce uses mobile Notes applicationsalesforce orders are processed by DB2 applicationcustomer uses J2EE application with DB2 backend to file orders

Challengeintegration of applicationssynchronization of data

Scenario 2: Integration #2

Solution: integration via DAVmake document based data available for corporate SQL applications via DAV:

expose Domino data to DB2 for read and writepopulate documents with DB2 based data from SQL applications via DAV

make DB2 data available for documentsmake DB2 data available in flexible Notes view context

Benefitsenrich Notes and Domino environment with DB2 based application optionsenrich DB2 environment with Notes and Domino based application options

Scenario 2: Mobile Salesforce

Salesforce uses mobile Notes applicationOrders are replicated into central NSFDB2DB2 application works on data exposed via DAV

Notes adds offline functionality to DB2 via replicationTransactions are processed in DB2

Salesforce replicates transaction status back to mobile device

Demo 2: DAV Design, Use & Remarks

Updated by DB2 application

Look at DAV elements in DB2views

tables

data

Modify SQL data from Notes

DAV Use

DAV Remarks

Structure clash between Domino flexibilityand restricted DB2 tables

DB2 requires fixed column/field lengthespecially text fields have to be considereddo not modify truncated data

Think about multi value handlingList fields

reduce DB2 column lengthsum length for multi value fieldsuse alias if possible

DAV Summary

Domino Designer as a development tool for DB2Expose Domino data to SQL applicationsMake Notes and Domino functionalities easily available for DB2 applications, examples:

Domino semantics offer additional value to SQL applications("row level security")enrich DB2 with disconnected options

Potential to lower cost for application integration withexisting infrastructure

Agenda

1 Introduction2 DB2 Access Views (DAV)

scenariolive demo

3 Query Viewsscenariostechnology, tipsfederated data paradigmlive demos

4 Summary

Query Views

Query Views - Overview

Notes views which are enabled forSQL queriesSQL queries are used in view selection formula contextUsage examples

filter documents (dynamic view selection formula)add DB2 data to a Notes viewuse retrieved DB2 data in column formulascombine Domino data with external DB2 data

Query Views – Data Federation #1

Rows can be populated bydocuments in the current Domino database (via DAV only)

DAV data in DB2 not in the current Domino database

DB2 data

combinations between these via SQL JOIN

Thus query views can display data fromcurrent Domino database exposed by DAV

other Domino databases exposed by DAV

Domino documents exposed by DAV, with additional data via SQL JOIN

Native (non Domino originating) DB2 data

Each row does NOT necessarily represent one document

Keep this in mind during clicking rows

Query Views – Data Federation #2

"Federated data" do not originate from current Domino databaseA row can contain document values plus federated data

"normal" Domino field contenthas to be included in DAVhas to be included in SQL selection

result of a column formula calculationDB2 data objects retrieved using SQL Query"double click" yields opening of document

Query Views – Data Federation #3

A row can be defined by federated data onlyall values result from DB2 data objects"double click" on row makes no sense (in most cases) and yields error messagecan be used for data consolidation from different NSF files

same applicationdifferent applicationsee "Scenario 5"

Query Views – SQL Query Rules

Query is defined in Notes formula language contextQuery supports standard SQL

SQL JOINSQL UNIONORDER BYetc.

Queries that do not produce a result set are not allowedsecurity mechanismprevents deletion/update from view

Query Views Are Dynamic

No persistent view index involved

Efficient DB2 indexing is used

Queries can be user specific

Parameterized and personalized

Lookups are allowed to collect Domino data for queryconstruction

Context can be considered when embedded

Scenario 3 – Federated Data in Views

HR uses Notes and Domino to manage resource dataEveryone can view employee name and phone numberOnly HR is allowed to view salarySecurity challenge

field encryption requires complex key managementmechanisms such as "hide when" are not security features@DBLookup is not supported in column formulas

Solutionstore data in different database or documentuse query views and SQL JOIN to add federated data

NSFDB2 EmpUSA.nsf

DAVEMPLOYEE

NSFDB2 Salary.nsf

DB2

DAVSALARY

EMPID,NAME,PHONE

EMPID,SALARY

Query View D1

NAME PHONE SALARY EMPIDRecord Set 1Record Set 2Record Set n

B

A

SQL JOIN

Result Set of:“SELECT

A.NAME , A.PHONE , B.SALARY, A.EMPIDFROM

EMPUSA.EMPLOYEE A LEFT OUTER JOIN SALARY.SALARY B

ON A.EMPID = B.EMPIDORDER BY

A.NAME”

Demo 3

Design a query viewJOIN data from other Domino database

Data from other database

Remarks

In most scenarios, LEFT OUTER JOIN is most applicable

ensures all documents are displayedleaves column empty if federated data does not match

Do not include multiple NoteID fields in result setuse specific selection instead of *wrong document might be openedmultiple rows with equal NoteIDs will show as single row

Scenario 4 – Dynamic Selection

Workflow application

Very large number of documents

Users only need a small subset of documents

Users want to select which documents to display

Users want profile based personal selection settings

Challenge (in Notes context)@DBLookup in selection formula is not possible

@UserName only works in private views

bad performance due to large view index

Solutionquery views don't have a static view index

query statement(s) can be based on dynamic Notes formula mechanism

User selected data

Demo 4

Design a query view

Dynamic selection using @Prompt

Dynamic user preference

Query Views – SQL UNION application

Dynamic formulas allow user specific aggregationsAggregate documents from multiple Dominodatabases into "single point of access"Returns federated data sets and documentsAllows for design of aggregated views

DB2

SQL UNION – Aggregated Views

SQL Query View

Notes Client

"Select A.Lastname, A.Firstname from Asia.Employee AUNIONSelect B.Lastname, B.Firstname from EMEA.Employee BUNIONSelect C.Lastname, C.Firstname from USA.Employee C"

NSFDB2 EMEA.nsf

DAVEmployee

NSFDB2 USA.nsf

DAVEmployee

NSFDB2 Asia.nsf

DAVEmployee

Scenario 5 – Aggregated Views

User tasks are dispersed over multiple workflowand project management applicationsUser is active in multiple teamroomsChallenge

user has to open many databases to get his job doneuser is not up to date if new tasks appear

Solutiona single view displays all documents a user needs from multiple databasesuse query view and SQL UNION to display federated (virtual) "documents"retrieving originating (real) documents accessible needs specific attention…

Demo 5

Design an aggregated view

Display external documents from multiple Domino databases in query view

Implement access mechanism to open external documents

Retrieve Documents From Federated Data - #1

View includes rows with federated data not related to documents of current database

which originate from (real) documents of external databases

Thus, document related Domino objects are not accessibledocument collection

document context

caret note ID not available

View entry objects (rows) originating from external documents doexist

Use caret category to identify row contextrequires unique value in sort column

limit of caret category is 99 characters

Include necessary data to access externaldocument to (hidden) designated column(s)

UNID / Note IDservernamepath or replica ID

Intercept QueryOpenDocument eventCreate backend document object that representsexternal documentOpen external document in Notes Client UI

Retrieve Documents From Federated Data - #2

Federated Data - Remarks

Dynamic resort has to be consideredwould change caret category

Wishlist for subsequent releasesidentifier for row selection context(other than caret category)provide data of external document automatically(servername, path or replica ID in addition to UNID)support response hierarchies

Enterprise Infrastructure Requirements

Prerequisites for aggregated viewsRequirements for applications

follow data structure policiesuse common set of application templates

Evaluate using DB2 as data store

Scenario 6 – Dynamic Column Data

Application has many fieldsUser needs are highly individualChallenge

many columns slow down view updatescustomize view functionality cannot be integrated in application designprivate views raise new issues

Solutionapplication lets the user specify data to displaya query view displays only the data specified by the user

Demo 6

Show field mapping

Show formula to alias DAV fields

Scenario 7 – Filter Categorized View

Application has views with complexcategory structuresUser needs complex and dynamicfilter mechanismsChallenge

full-text search breaks up category stucture

Solutionimplement user interface to set filtersa query view displays the filtered documents based on SQL selectcategory structure is retained

Demo 7

Show view update

Show profile document

Show formula to filter documents

Demo 7 Remarks

Make extensive use of filters to limit number of rows

customized query builderspersonalized viewsuser specific preferencessaved default values

Demo 7 Remarks

Consider multi value fieldsuse "like" operatoruse Notes multi value separator as separator for DAV

Scenario 8 – GCC Activity Manager

User needs complex relational filter mechanismsChallenge: portal like user experience

to collect relevant datato render relevant data in UI

Solutionimplement complex user interface to set filtersa query view displays the filtered documents based on SQL selectcategory structure is retained

Demo 8

Show GCC Activity ManagerExplain Architecture

Query Views - Performance

Performance lower than NSF for "as is" usageDB2 result sets

possible advantage for small result sets out of large document collectionsno view index storedlive SQL query on open or refresh

For certain scenariosquery views could help to handle performance issues –appropriate redesign has to be donequery views do not work locallylimit the number of records for result sets

Query View Remarks

Does NOT work locallyDB2 user mapping necessary for securityAlways use @DB2Schema to determine the schema name for queryDefault for maximum number of returned rows is 500 at server levelFederated data not available locallyFederated data must be available in DAV or DB2 tableQuery views with federated data do not update automatically

query is not re-executeduse F9 to update

Query View Summary

JOINS solve basic Notes problem with relations

New features offer the potential to create applications thatenable users to work more efficiently

Agregated views offer a new way to create a single point of access to all documents a user needs

Potential to address current problem scenarios with respect to performance/scalability

Keep the result set small

Agenda

1 Introduction2 DB2 Access Views (DAV)

scenariolive demo

3 Query Viewsscenariostechnology, tipsfederated data paradigmlive demos

4 Summary

Conclusion/Summary

Start thinking about scenarios in your companyStart consolidating your application landscapeNew integration options in Workplace contextLook into DB2 administrationGet skills in SQL programmingPrepare applications to utilize DAV and query viewsMake extensive use of filters

Related Lotusphere Activity

SessionsAD210 - Application Development and Deployment for DB2-Enabled IBM Lotus Domino ServerAD211 - Using IBM Lotus Domino 7 with DB2: A Real-World ExampleID105 - DB2 Setup and Administration for IBM Lotus Domino Server

BOF and Hands onHND202 - IBM Lotus Domino NSFDB2: Development and Integration Hands-OnMTG611 - Sharing Data Access View and Query View Development Experience

Q & A / Download

Questions and discussion welcome"Meet the Speaker" Swan Toucan 2 after the session"Gurupalooza"Sample code and updated slideshttp://gcc.upb.de/k-pool/LS2006_BP307

Contact Information

Dr. Ludwig W. Nastanskymailto: Ludwig.Nastansky@notes.upb.deIngo Erdmannmailto: Ingo.Erdmann@notes.upb.deUniversity of PaderbornGroupware Competence Centerhttp://gcc.upb.dePlease remember to fill out your evaluation!

BP307: IBM Lotus Domino 7 and DB2�Effective Use of Access and Query ViewsSpeakersAgenda1 IntroductionNotes/Domino & DB2 Integration, finally… ?Documents vs. Relational DataDocuments vs. Relational DataIntegration, Cooperation, FederationScenario 1 - Model Employee–Room RelationDomino challenge: Model Employee–Room RelationRemarkDemo 1: RelationsDB2 Access Views and Query ViewsAgendaDB2 Access ViewsDAV - OverviewDAV – Field DefinitionDAV – Architecture SummaryScenario 2: Integration #1Scenario 2: Integration #2Scenario 2: Mobile SalesforceDemo 2: DAV Design, Use & RemarksDAV UseDAV RemarksDAV SummaryAgendaQuery ViewsQuery Views - OverviewQuery Views – Data Federation #1 Query Views – Data Federation #2Query Views – Data Federation #3Query Views – SQL Query RulesQuery Views Are DynamicScenario 3 – Federated Data in ViewsSQL JOINDemo 3RemarksScenario 4 – Dynamic SelectionDemo 4Query Views – SQL UNION applicationSQL UNION – Aggregated ViewsScenario 5 – Aggregated ViewsDemo 5Retrieve Documents From Federated Data - #1Retrieve Documents From Federated Data - #2Federated Data - RemarksEnterprise Infrastructure RequirementsEnterprise Infrastructure RequirementsScenario 6 – Dynamic Column DataDemo 6Scenario 7 – Filter Categorized ViewDemo 7Demo 7 RemarksDemo 7 RemarksScenario 8 – GCC Activity ManagerDemo 8Query Views - PerformanceQuery View RemarksQuery View SummaryAgendaConclusion/SummaryRelated Lotusphere ActivityQ & A / DownloadContact InformationBackup SlidesSQL UNION RemarksDAV Design - #1DAV Design - #2DAV Design - #3DAV Design - #4Create new DAV for each formDAV DeploymentUser Mapping for SQL SecuritySequence of a SQL Query to DAVDAV JDBC AccessDAV JDBC AccessCreditsType section title hereAgendaTable of contentsLegal information