The ]po[ Data Model Frank Bergmann, 2006-05-22

This guide contains overview information that is useful if you want to write reports or if you need to extract information from the API.

It is not ment as a developer guide. Please check the OpenACS manual for more detailed information.

SoftwareDevelopment

Templates

Trans-lation

CRMFinance

Controlling

HR

System

Collaboration,Content & KM

OOFrame Security

Calendar

OpenACSPermission

Web Server

Database

FinanceBase

Payroll

Skill Database

Filestorage

OnlineDiscussions

IncidentWorkflow

BasicAuthentication

LDAPAuthentication

WorkflowEngine

Chat

PackageManager

AutomaticSoftwareUpdates

PostgreSQL Oracle 8i, 9i, 10g

PageContracts

SQLTemplates

OO Model

ObjectMetadata

LocalizationFramework

ContactMgmt.

ReportingEngine

Wiki

Oracle Intermedia/TextTSearch2

Linux Solaris BSDWindows+ CygWin

Mac OSOperating

System

SearchEngine

AOLServer PoundRevers Proxy

DynField Object Extensions

PlatformServices

Profiling & Performance

DebuggingSystem

ApplicationModules

FormBuilder

Portal &Components

Mail ServerIntegration

ContentManagement

ApplicationServices

DB-APITCL

Quotes &Invoice

Payments

FinancialReporting

Mail ServerIntegration

CustomerWeb Reg.

MarketingCampaigns

CRMTracking

AutomaticAudits

ISDN TelIntegration

ProjectMgmt.Project &Subprojects

ProjectControlling

TranslationWorkflow

TMIntegration

FreelanceInvoicing

TimesheetInvoicing

AvailableDocumentation

TimesheetMgmt. AutomaticInvoicing

]po[ Overview

FinanceGuide

Unix InstallGuide

OpenACSInst. Guide

Operations& Maint.Guide

OpenACSDeveloperGuide

ConfigurationGuide

FilestorageGuide

ForumGuide

TranslationWorkflow

Guide

Full-TextSearch

AutomaticTesting

BigBrotherSys Mgmt.

OtherRoomReservation

E-Commerce

CMS

WebDAV

SOAP &XML-RPC

Surveys

GlossaryWeb-Mail

Blog

RecruitingWorkflow

CVS

Postfix/Sendmail

Database Replication

MondrianData-

Warehouse

Contents

Other Documentation General Conventions "Categories" ]po[ Objects and Types Main Classes Financial Classes Auxilary Tables

Other Documentation

This is a limited overview document. For more detailed information please see:

]po[ Overview Diagram: http://www.project-open.org/ OpenACS Developer Guide: http://www.openacs.org/doc/ ]po[ Whitepapers: http://www.project-open.com/whitepapers/ ]po[ List of modules:

http://www.project-open.org/product/modules/ ]po[ Developer Documentation: http://www.project-

open.org/doc/

General System Architecture

]po[ consists of three main parts:– A powerful object-relational data model– TCL pages that render the contents of the

database and– TCL libraries that provide auxilary functions

to the TCL pages The Data Model is particularly important

in ]po[ because:– It is object-oriented– The TCL layer does not contain objects. It

merely renders the content of the Data Model The Data Model is the location where you

need to go if you want to extract information from the system

Data Model

TCL Library

TCL Pages

General Conventions

Tables names are always in plural form and written in lower case Tables are held in general in 2nd normal form Denormalized table columns are named with a "_cache" postfix. All main tables have an integer primary key <table_name>_id.

This primary key references the table acs_objects which contains the type of the object.

Most object->object mappings are handleled by the "acs_rels" table and its subclasses (see below). Other mapping tables are called something_something_map

Boolean fields have a "_p" postfix and contain 't' or 'f' char values

All main objects include "object_type_id" and "object_status_id" fields. "Type" is used to create minor subclasses of the objects, such as the distinction of im_companies such as "customer" and "provider". "Status" is used to deal with the object's lifecycle, such as "potential", "active", "closed" and "deleted".

Normal arrow means referential integrity relation-ship between tables.

Big arrow tip means inheritance relation-ship. An inheriting "object" references its parent class with its object_id.

“Business Objects”

Data Model Overview – Business Objects

Most data in the ]po[ data model are structured as “Business Objects” (BizObj).

A “BizObj” is just a database table following a number of conventions

– A integer primary key called <object>_id representing a unique object ID

– The primary key references the “acs_objects” table. acs_objects includes object meta-information such as object type, creation date, creation user etc.

– Every ]po[ BizObj includes a field “object_status_id” re

object_idobject_type

acs_objects

project_id

im_projects

project_lead_id

project_name

project_status_idproject_type_id

supervisor_id…

project_nrcompany_id

Data Model Overview – Business Objects

The ]po[ data model is based on the notion of “Business Objects”

A “BizObj” is just a database table following a number of conventions

– A primary key representing a unique object identifier

– The primary key references the “acs_objects” table. acs_objects includes object meta-information such as object type, creation date, creation user etc.

– Every ]po[ BizObj includes a field “object_status_id” re

object_idobject_type

acs_objects

project_id

im_projects

project_lead_id

project_name

project_status_idproject_type_id

supervisor_id…

project_nrcompany_id

category_id

name

im_categories

description

category_type

persons

person_idfirst_nameslast_name

rel_idobject_id_one

acs_rels

object_id_two

persons

person_idfirst_nameslast_name

Data Model Overview – Business Objects

The ]po[ data model is based on the notion of “Business Objects”

A “BizObj” is just a database table following a number of conventions

– A primary key representing a unique object identifier– The primary key references the “acs_objects” table.

acs_objects includes object meta-information such as object type, creation date, creation user etc.

– Every ]po[ BizObj includes a field “object_status_id” re

Hierarchical Queries

Main Project

Sub-Project 1

Sub-Project 1.1

Sub-Project 1.2

Sub-Project 2

Sub-Project 2.1

Sub-Project 2.2

"Categories"

Conventional/ traditional database designnormally includes a lot of foreign keytables defining the type and status of an object.

]po[ takes a different approach in order tominimize the number of database tables andmaintenance screens. We use a single"im_categories" table for all types of status andtype information:+ The total number of DB tables is reduced to a

third or fourth+ A single maintenance screen is responsible

for managing categories.+ Built-in features:

+ Localization+ Hierarchical categories+ Common GUI widgets

+ Referential Integrity is enforced- It is possible to assign the wrong Category to

a field+ This has never happened in practice yet.

object_type_idobject_status_id

object

object_type_idname

object_type

description

object_status_idname

object_status

description

namedescription

...

object_type_idobject_status_id

object

namedescription

...

category_id

name

im_categories

description

category_type

The "Classical" DB-Design: Every table has it's own tables for type, status and similar information.

]po[ DB-Design: All type and status information is stored in a single "im_categories" table.

parent_category_id

im_category_hierarchy

child_category_id

Categories

Select Everything About Categoriesselect c.*, im_category_from_id(aux_int1) as aux_int1_cat, im_category_from_id(aux_int2) as aux_int2_cat, h.parent_id, im_category_from_id(h.parent_id) as parent from im_categories c left outer join im_category_hierarchy h on (c.category_id = h.child_id) where c.category_type = 'Intranet Cost Types' order by category_type, category_id;

Sample SQL

Categories Definition-- We use categories as a universal storage for business -- object states and types, instead of a zillion of tables -- like 'im_project_status' and 'im_project_type'.

create sequence im_categories_seq start 100000;create table im_categories (

category_id integerconstraint im_categories_pkprimary key,

category varchar(50) not null,category_description varchar(4000),category_type varchar(50),category_gif varchar(100) default 'category',enabled_p char(1) default 't'

constraint im_enabled_p_ckcheck(enabled_p in ('t','f')),-- used to indicate "abstract"-- super-categorys that are not -- valid values for objects.-- For example: "Translation -- Project" is not a

project_type, -- but a class of project_types.

parent_only_p char(1) default 'f'constraint im_parent_only_p_ckcheck(parent_only_p in

('t','f')));

Category Hierarchy-- Optional system to put categories in a hierarchy.-- This table stores the "transitive closure" of the-- is-a relationship between categories in a kind of matrix.-- Let's asume: B isa A and C isa B. So we'll store-- the tupels (C,A), (C,B) and (B,A).-- This structure is a very fast structure for asking:---- "is category A a subcategory of B?"---- but requires n^2 storage space in the worst case and-- it's a mess retracting settings from the hierarchy.-- We won't have very deep hierarchies, so storage complexity-- is not going to be a problem.

create table im_category_hierarchy ( parent_id integer constraint im_parent_category_fk references im_categories, child_id integer constraint im_child_category_fk references im_categories, constraint category_hierarchy_un unique (parent_id, child_id));

Extract Categories Without Join-- A helper functions to make our queries easier to

readcreate or replace function im_category_from_id

(integer)returns varchar as 'DECLARE p_category_id alias for $1; v_category varchar(50);BEGIN select category into v_category from im_categories where category_id = p_category_id;

return v_category;end;' language 'plpgsql';

-- Example:-- select im_category_from_id(48);Create a New Category Entryinsert into im_categories (

category_id, category, category_type,category_description, enabled_p,aux_int1, aux_int2,aux_string1, aux_string2

) values (:category_id, :category, :category_type,:category_description, :enabled_p,:aux_int1, :aux_int2,:aux_string1, :aux_string2

);

]po[ Objects and Types

All major ]po[ tables are "Objects" Objects are identified by an object_id. Information about object type is stored in the

acs_objects table, together with metadata and creation information.

Advantages of "being" an object: Permission system:

Associates objects with "privileges", including all necessary maintenance screens

SQL metadata system: Allows the administrators to dynamically extend object types with new attributes

Full-Text search:Indexes all objects and produces generic URL for the search results

Configurable workflow:Allows to manage the status of any object. The WF includes a graphical WF editor.

Generic Object Relations:acs_rels allows you to define generic relationships between objects.

object_type_idobject_status_id

object

namedescription...

object_idobject_type

acs_objects

acs_object_types

object_typesupertype

table_nameid_column

name_method

attribute_id

table_name

acs_attributes

column_name

object_typepretty_name

]po[ Objects and Types

acs_attributes:– Each row in acs_attributes table defines an attribute of the specified object

type.– The contents of the table is managed by the "intranet-dynfield" package that

contains a SQL metadata editor. acs_privileges:

– Contains permissions "tokens" (=> privileges). These privileges allow a user to perform a certain operation in the system.

acs_permissions:– Defines a mapping between acs_objects and acs_privileges.– The mapping is per user group ("grantee_id"). Groups can be hierarchical.

Main Classes

object_id

im_biz_objects

rel_idobject_id_one

acs_rels

object_id_two

object_idobject_type

acs_objects

group_idgroup_name

groups

profile_idprofile_gif

im_profiles

users_contact

<Home Address><Work Address>

im_employees

<Payroll Info><Recruiting Info>

im_freelancers

<Freelance Info>

parties

party_idemailurl

persons

person_idfirst_nameslast_name

users

user_idusernamepasswordsaltauth_token

company_id

im_companies

main_office_id

company_namecompany_status_id

company_type_id

primary_contact_idaccounting_contact_id

notevat_number

office_id

<Address>

im_offices

office_status_idoffice_type_id

company_idproject_id

im_projects

project_lead_id

project_name

project_status_idproject_type_id

supervisor_idproject_budget

budget_hoursbudget_currency

project_nrproject_pathcustomer_id

descriptionnote

start_dateend_date

Main Classes

im_biz_objects:– This abstract class defines a number of functions to manage "horizonal

permissions" (=> Please see other ]po[ documentation) for project, companies and offices

– "Horizonal" permissions currently include roles such as "Project Manager" of "Full Member".

parties, persons, users:– These three tables form an inheritance hierarchy. There is a view "cc_users"

that unifies the information from these tables. im_freelancers, im_employees, users_contact:

– These are "extension tables" for "persons" and define additional information for users with particular profiles.

im_profiles:– This table defines the groups that are relevant for ]po[, as opposed to

potentially hundereds of groups of the underlying OpenACS system.

Financial Classes

Cost TypesAll costs have an entry in im_costs, plus:

Financial Docs(im_invoices)

– Customer Invoice– Quote– Provider Bill– Purchase Order– Delivery Note

Simple Costs(only im_cost)

– Timesheet Cost– Employee Salary

Expense Item(im_expenses)

Expense Bundle(im_expense_bundles)

cost_center_idcost_center_name

im_cost_centers

cost_center_codecost_center_type_idcost_center_status_iddepartment_p

im_costs

cost_idcost_name

cost_nr

project_idcustomer_idprovider_id

investment_idtemplate_id

cost_status_idcost_type_id

effective_date_idpayment_days

amountcurrency

paid_amountpaid_currency

taxvat

descriptionnote

cost_center_id

invoice_idcompany_contact_id

im_invoices

invoice_nrinvoice_office_id

item_iditem_name

im_invoice_items

project_idinvoice_id

item_unitsitem_uom_idprice_per_unitcurrencysort_orderitem_type_iditem_status_iddescription

payment_idcost_id

im_payments

company_idprovider_idreceived_datepayment_type_idpayment_status_idamountcurrencynote

object_idobject_type

acs_objects

Inheritance!

expense_idext._company_name

im_expenses

ext._comp._vat_num.billable_preimbursablebundle_id

Financial Classes (2)

The Problem:Cost and Projects can have a N:M relationship: One project may have more then one invoice/cost item. One cost item may “relate” to more then one project (in particular: project + sub-project, but also

the item may be split between two main projects). An invoice may actually not belong to any project at all.

Solution: In general, the relationship between projects and costs is defined as an N:M relationship by acs_rel. im_costs.project_id points to im_projects ONLY in the case that one cost item relates to exactly one

project

Watch out: Cost Items can be related to a main-project or to a sub-project or any level. So in order to calculate

the main project’s total Profit & Loss, you need to sum up the financial items through the full project tree (hierarchical query)

im_costs

cost_idcost_name

cost_nr…

project_id…

im_projects

rel_idobject_id_one

acs_rels

object_id_two

Relationship between Costs and Projects

project_id

select distinct c.cost_name, im_category_from_id(c.cost_type_id) as cost_type, c.amount * im_exchange_rate(c.effective_date::date, c.currency, 'EUR') as amount_convertedfrom im_projects main_p, im_projects sub_p, acs_rels r, im_costs cwhere main_p.project_id = :main_project_id and sub_p.tree_sortkey between main_p.tree_sortkey and tree_right(main_p.tree_sortkey) and r.object_id_one = sub_p.project_id and r.object_id_two = c.cost_id

A sample query to extract all financial items related to a certain main-project.

Financial Classes (3)Example

project_id…

im_projects

project_idparent_id

im_projects

project_idparent_id

im_projects

…

…

im_costs

cost_idcost_name

cost_nr…

rel_idobject_id_one

acs_rels

object_id_two

project_id

Main project

Sub-project with cost item

Sub-project with cost item

Financial Classes

im_costs:– This is the main table for all cost items.– All financial elements in the system are stored in this table.

The table also receives updates from timesheet (cost related to users working on a project) etc.

im_cost_centers:– Is currently not used. The table has been included in the

current datamodel for smooth upgrade in future versions. im_invoices:

– This table contains financial documents such as "Invoices", "Quotes", "Bills" and "Purchase Orders" (the name "im_invoices" is a bit misleading).

im_invoice_items:– These are the individual lines of an invoice.

Relationship Classesobject_idobject_type

acs_objects

rel_id

im_biz_object_members

object_role_id

rel_id

acs_rels

object_id_oneobject_id_two

rel_type

rel_id

membership_rels

member_staterel_id

admin_rels

rel_id

composition_rels

rel_id

group_rels

rel_typegroup_id

Defines the relationships between an im_biz_object and some users. The membership type corresponds to "horizontal permissions" and usually includes "full member", "project manager", "key account" and user-defined roles.

rel_type

acs_rel_type

object_type_tworole_one

object_type_one

role_two

min_n_rels_onemax_n_rels_onemin_n_rels_twomax_n_rels_two

Relationship Classes

acs-rels:

im_biz_object_members: – Defines the member of a im_biz_object and their roles (project

manager of full member). – Such a membership has an impact on the access rights of members to

the business object. membership_rels: Defines group->supergroup relations

The acs_rels table is essentially a generic mapping table for acs_objects. Once we come up with a way to associate attributes with relationship types, we could replace many of the ACS 3.x mapping tables like user_content_map, user_group_map, and user_group_type_modules_map with this one table. Much application logic consists of asking questions like "Does object X have a relationship of type Y to object Z?" where all that differs is X, Y, and Z. Thus, the value of consolidating many mapping tables into one is that we can provide a generic API for defining and querying relationships. In addition, we may need to design a way to enable "type_specific" storage for relationships (i.e., foreign key columns for one-to-many relationships and custom mapping tables for many-to-many relationships), instead of only supporting "generic" storage in the acs_rels table. This would parallel what we do with acs_attributes.

Translation Classesobject_idobject_type

acs_objects

im_costs

im_invoices

im_trans_invoices

price_id

uom_id

im_trans_prices

company_idtask_type_idsource_lang_idtarget_lang_idsubject_area_id

currencypricenote

im_trans_tasks

task_idtask_name

task_filename

project_id

source_language_idtarget_language_id

invoice_idquote_id

task_status_id

task_type_id

task_unitsbillable_units

trans_idedit_id

proof_idother_id

match_xmatch_repmatch_100

match_95

task_uom_id

match_85match_75match_50

match_0

end_date

note

subject_area_id

report_id

allowed_errors

im_trans_quality_reports

task_idreport_datereviewer_idsample_size

total_errorscomments

report_id

im_trans_quality_entries

quality_category_idminor_errorsmajor_errorscritical_errors

object_id

im_trans_trados_matrix

match_xmatch_repmatch_100match_95match_85match_75match_50match_0

Translation Classes

im_trans_tasks:– This tables contains the description of the main activity related to translation

projects.– Includes information about Translation Memory repetitions– Inludes information about the translator, editor, proof-reader and "other"

assigned to perform this task (static workflow) im_trans_invoices:

– This is currently just a flag (binary information) to identify invoices that have been created based on im_trans_tasks. Translation invoices are referenced by im_trans_tasks.invoice_id in order to make sure that every trans_task is included in exactly one invoice. These references need to be removed if a translation invoice is deleted.

im_trados_matrix:– Contains the discounts per customer and translation task type for different

Translation Memory matches.– The trados_matrix associated with the "internal" company contains the site-

wide defaults for customer translation prices.

Timesheet Classesobject_idobject_type

acs_objects

im_costs

im_invoices

im_ts_invoices

material_id

description

im_materials

material_namematerial_nrmaterial_type_idmaterial_status_idmaterial_uom_id

task_id_one

im_ts_task_deps.

task_id_twodependency_type_iddifferencehardness_type_id

project_id

im_projects

project_name

...

customer_id

start_dateend_date

task_id

invoice_id

im_timesheet_tasks

material_iduom_idplanned_unitsbillable_unitscost_center_id

priority

price_id

uom_id

im_timesheet_prices

company_idtask_type_idmaterial_id

currencypricenote

project_type_id

Consulting Classes

im_timesheet_tasks:– This tables contains the description of the main activity related to projects.

im_timesheet_invoices:– This is currently just a flag (binary information) to identify invoices that have

been created based on im_timesheet_tasks. Timesheet invoices are referenced by im_timesheet_tasks.invoice_id in order to make sure that every task is included in exactly one invoice. These references need to be removed if a timesheet invoice is deleted.

im_timesheet_prices:– Contains a price per material and customer. – The timesheet_prices associated with the "internal" company contains the site-

wide default price list.

Workflow

Workflow

Workflow

]po[ uses a Petri-Net based workflow A workflow consists of “Places”

(states) and “Transitions”, linked by “Arcs” that might carry “Guard” expressions.

WF “Transitions” can be linked to pages acting on object’s fields

Transition

Transition

Place

Start-Place

Place

End-Place

[Guard] [Guard]

WF @ Runtime

During runtime, a similar structure is created representing a specific “Case” (a specific instance of a WF):

– The “case” corresponds to a WF in action.

– Transitions become “tasks”, with specific assignments to users.

– Places may include “tokens” that move through transitions to other places. A single token in a WF place can be thought as “the WF is in status XXX”.

Petri-Nets allow for more then one token and more then one token per place, but this option is rarely used in ]po[.

Case

Task

Task

Place

Start-Place

Place

End-Place

[Guard] [Guard]

StaticAssignments

Workflow Case

WF Tables

place_key

wf_places

workflow_key

transition_key

wf_transitions

direction

workflow_key

wf_workflows

case_id

wf_cases

workflow_keyobject_id

task_id

wf_tasks

case_idtransition_key

transition_key

wf_arcs

place_keydirection

token_id

wf_tokens

case_idplace_key

role_key

wf_roles

workflow_key

state

state

state

workflow_keyrole_key

TransitionStates

WF “Callbacks”

“Callbacks” are Pl/SQL databaseprocedures that determine the behaviorof a transition: A number of “Outgoing Events” allow

to customize the behavior of WF transitions:

– Enable: Is called when the transition is enabled (a token is placed in one of its input places)

– Unassigned: Is called if the transition hasn’t been statically (see below) assigned to a user or a group.

– Fire: Is called once the transition gets fired

A number of transition’s parameters can be determined by external routines:

– Time:– Deadline:– Hold Timeout– Notification

Enable

Start

Finish Cancel

Enabled

Started

Finished Canceled

<autom>

“Enable”

“Fire”

“Fire”

Time

Deadline

Hold Timeout

Notification

“Unassigned”

TransitionStates

WF Assignments

“Static Assignments”:– Specified during the definition of the WF– Specified using WF “roles”. Each role can be

assigned to one or multiple parties (groups or users)

– Specified as “roles” using wf_transition.role_key and then mapped to parties using wf_context_assignments.

– The specific assignments at runtime are held in wf_task_assignments.

– This construction might seem a bit complex, but it is necessary, as you will find out when defining your first real WFs.

“Dynamic Assignments”– Use the “Unassigned” callback to define the

assigned parties.– This option allows you to specify the assignee

as a function of object characteristics etc.– The results of dynamic assignments are

stored in wf_task_assignments.

Enable

Start

Finish Cancel

Enabled

Started

Finished Canceled

<autom>

Timesheet & Controlling

Timesheet & Controlling

Controlling uses caches per project in order to maintain aggregated Profit & Loss (P&L) information.

The project caches are updated via triggers Reporting on portfolios & customers is handled

by the reporting engine.

CustomerGroup

SubProject

MainProject

Customer“Timesheet”

Task

“Forward” PropagationCosts are cached. Caches are updated via triggers

“Backward” ReportingValues are summarized via Reports

Requirements and Architecture

Timesheet & Controlling

Projects contain a set of “xxx_cache” fields that contain the aggregated cost elements per type.

CUD (create, update, delete) operations on costs (timesheet hours, expenses, …) does NOT update the caches, for performance reasons. Instead, triggers reset the “cost_cache_dirty” to NULL.

There is a “sweeper” process that periodically checks “cost_cache_dirty” and updates the cache. This might be a nightly activity, or every few minutes.

project_id

im_projects

…

cost_quotes_cachecost_invoices_cache

cost_timesheet_planned_cachecost_purchase_orders_cache

cost_bills_cachecost_timesheet_logged_cachecost_expense_planned_cache

cost_expense_logged_cachecost_delivery_notes_cache

reported_hours_cache

cost_cache_dirty

“Sweeper”Process

Controlling cache architecture

Timesheet & ControllingControlling cache architecture

user_id

im_hours

project_idday

hoursbilling_rate

note

cost_idconf_object_id

billing_currrency

im_costs

cost_id…

conf_id

im_ts_conf_objects

conf_project_idconf_user_id

start_dateend_date

conf_type_idconf_status_id

Project reportedhours

cost_tslogged

cache_dirty

Main Project (9)

Sub Project1 (4)

Task1.1 4

Task1.2

SubProject2 (5)

Task2.1 3

Task2.2 2

The hours are aggregated from tasks up to main

projects

between

conf_project_id refers to the main

project. All sub projects and tasks

are included.

Auxiliary Tables

Auxilary Tables

GUI Define what "plugins" (these grey boxes) should appear

on what page

Defines the hierarchical menu structure of the system

Define the columns for ListPages in the "core". The ListPages need to be extensible at runtime because they might have to accomodate new columns from add-on modules

The list of all countries in the world

The list of all currencies

View

view

ViewColumn

Currency

Country

im_component_plugins

Menu

Auxilary Tables

GUI Define what "plugins" (these grey boxes) should appear

on what page

Defines the hierarchical menu structure of the system

Define the columns for ListPages in the "core". The ListPages need to be extensible at runtime because they might have to accomodate new columns from add-on modules

The list of all countries in the world

The list of all currencies

View

view

ViewColumn

Currency

Country

im_component_plugins

Menu

Frank Bergmannfrank.bergmann@project-open.com

www.project-open.com