Natural Rule Language - · PDF fileNatural Rule Language •An open language for...

Natural Rule Language

Christian Nentwich, Model Two Zero Immo Hüneke, Zühlke Engineering

Presented at the British Computer Society

8th of June 2011

Agenda

• Issues in System Integration

• The goal of executable specifications

• Quick intro to the Natural Rule Language

• Project Review: Soft Trades Synchronizer (Immo)

• Additional experience report from financial services

Area of Concern

• We are dealing with large-scale system integration

ESB

Problem Context

• Hundreds or thousands of systems

• Lots of variety

– Off-the-shelf

– In-house

– “Tactical” (hacks)

• Great complexity

– Tens of thousands of data items

– Complex processes

Widespread Practice

Business Requirement

“We could save money by pre-populating SMALL SYSTEM from BIG SYSTEM

every day. Do it, we are allocating £200,000 to it.”

Analysis

• Analyse the data and semantics in SMALL SYSTEM and BIG SYSTEM

• Write a big specification

• (Usually extremely data centric, with some process on top)

Implementation

Testing and Go-Live

• Sometimes performed by separate teams

• Errors found here are guaranteed to delay the schedule

• Interpret and implement the specification

• Done by programmers using proprietary or open source tools

Readability By Non-Technical Users

• Critical for gap analysis on future projects

• Important for bridging communication gap between developers / business

(Not readable)

Precise and Formal Semantics

• Plain English is too ambiguous

• Must control that you are talking about right things in system context

> ird-12 (Mandatory)

> English Description:

> Context: CalculationPeriodDates (complex type)

>The frequency specified in calculationPeriodFrequency must divide the [period] precisely.

> This means that by stepping through the period from the start date at the specified frequency,

> it must be possible to reach the end date.

What does happen with stubs? Shouldn't the rule include the cases with stubs?

Microsoft Word/Excel

Are you sure

this exists?

What does

“gives any

value” mean?

“Is any of this

currently in

production??”

Is this as good as it gets?

• Poor processes

• Huge maintenance problems

– No transparency of data flows

– Specification useless after go-live

– Too much code to maintain

• Inefficient use of differently skilled people

• Communication barriers

– Off-shoring problems

• System integration is a specialised discipline – but discipline is what is lacking

Changing Direction

Instead of writing informal specifi-

cations and then code, create

executable specifications.

Executable Specifications

• Critical features:

– Readable by non-technical users

– Independent of a proprietary tool

– Precise and formal semantics

– Easy to write code generators or interpreters

• Desirable features:

– Writeable by non-technical users

– Amenable to easy analysis using tools

Microsoft Word/Excel

• Readable by non-technical users

• Independent of a proprietary tool

• Precise and formal semantics

• Easy to write code generators or interpreters

• Writeable by non-technical users

• Amenable to easy analysis using tools

Java / XSLT / Schematron

• Readable by non-technical users

• Independent of a proprietary tool

• Precise and formal semantics

• Easy to write code generators or interpreters

• Writeable by non-technical users

• Amenable to easy analysis using tools

Natural Rule Language

• An open language for expressing:

– Validation rules (constraints)

– Action rules (e.g. Enrichment rules)

– Transformation / mapping

• Aimed at the core problem areas in integration

• Goals

– Read like an English sentence wherever possible

– Require no customisation to get going

– Offer symbolic and textual alternatives

• Specification: http://nrl.sourceforge.net

The Origins: Validation

public void validate(Transaction transaction) {

if (transaction.getDebit() != null && transaction.getCredit() != null) {

if (transaction.getDebit().getValue() == transaction.getCredit()

.getValue() && getChangeInAmount() != null) {

exceptions.add(new ValidationError(

"Must not specify changeInAmount if a debit is equal to a credit"));

}

}

}

context: Transaction

inv: (self->debit = self->credit) implies (self->changeInAmount->empty())

<rule context="ns:transaction">

<assert test="ns:debit != ns:credit or not(ns:changeInAmount)">

Must not specify changeInAmount if a debit is equal to a credit

</assert>

</rule>

Java

OCL

Schematron NRL

Context: Transaction

Validation Rule "Our Sample Rule"

If a debit is present and a credit is present then

no changeInAmount is present

Report 'Must not specify changeInAmount if a debit is equal to a credit'

NRL

A quick practical introduction!

How to use this?

• NRL started out as a “Swiss army knife”

– “Use it for anything”

• Lesson: don’t build Swiss army knives!

• Here is where we got to eventually...

Proprietary

Format Validation

(NRL Constraints)

Mapping (NRL Actions)

Enrichment (NRL Actions)

Canonical

Format

POJO Classes

Spring Mule ESB

WebSphere

Logical Adapter

Physical Adapter

Weblogic

Camel

JBoss

NRL Parser

• The NRL parser is free and open source, also on sourceforge

• Designed for processing / code generation

ConstraintRuleDeclaration

IfThenStatement

ExistsStatement ExistsStatement

1. Text in NRL concrete syntax 2. Parser-generated AST

3. Code generators (from AST)

Model Specifications

Executable Specification using NRL

UML

XML

Schema

Rule Specifications

CSV

Fixed

Width

XML

SWIFT

MT

NRL

Constraints

NRL

Actions

*SWIFT SDK

Code

Guaranteed Consistency

Plain Java

Datastage

Drools

Project Review

Over to Immo!

What else have we done?

• Almost entire take-up is for mapping – Central reference data publishing service of a bank

– Corporate action processing

– SWIFT messaging

– 60+ smaller projects, world-wide

Lesson: Dates are Important

• 83 out of 207 (40%) of FpML validation rules are about dates

• 220 out of 751 rules (29%) in reference data export adapter about dates

• Very important to treat dates as a primitive type

If DebtInstrumentCharacteristics.LegalMaturityDateTime is present

then set to DebtInstrumentCharacteristics.LegalMaturityDateTime;

Lesson: Rationale is External

Context: InterestRateStream

Validation Rule "ird-2"

paymentDates.paymentFrequency [is an integer multiple

of] calculationPeriodDates.calculationPeriodFrequency

Why???

Pragmatics != Semantics

Lesson: Testing

• How to test this?

• People still want to do TDD, even with executable specs

• Tests form part of the specification!

• We’re looking into it

Context: InterestRateStream

Validation Rule "ird-2"

paymentDates.paymentFrequency [is an integer multiple

of] calculationPeriodDates.calculationPeriodFrequency

Lesson: Analysis

• There are big benefits to be had from metadata management

– See session this afternoon!

• Turing-complete languages are too hard to analyse

• Think hard about how expressive your language needs to be

Traceability Both Ways

Natural

Rule

Language

UML XML

Schema

Generated

Code

Metadata

Tools

Published

Specifications

Lesson: Analysis

public boolean validate(CalculationPeriodDates context) throws Exception {

CalculationPeriodDates current = context;

return current.getTerminationDate().getUnadjustedDate().getValue()

.toGregorianCalendar().compareTo(current.getEffectiveDate().

getUnadjustedDate().getValue().toGregorianCalendar()) > 0;

}

Context: CalculationPeriodDates

Validation Rule "ird-14"

terminationDate.unadjustedDate is after effectiveDate.unadjustedDate

Lesson: Don’t Abuse Rule Engines

• We wrote a code generator for the JESS engine

• Hand-wrote NRL

• Artificially split rules into left-hand and right-hand side for RETE matching

• Didn’t work too well (execution model too hard, and doesn’t fit)

Lesson: Programmers...

... like to turn things into code!

If ([getStaticData] 'getSystemDate') = 'Y' then

Set header.DateOfCreationAmendment to

([padField] ([currentDate]'yyyyMMdd') and 8 and 'B' and '0')

Else

Set header.DateOfCreationAmendment to

([padField] headerrecord.DateOfCreationAmendment and 8 and 'B' and '0')

Camel-case instead of English sentence

Mixing syntax and semantics

Lesson: Discipline

• What makes one model better than another?

• How do a good data architects distinguish themselves?

• What is the best way to gather tacit rules?

• What are best practices for large-scale modelling?

• What is the standard ontology of artefacts and activities in integration?

Lesson: Discipline

• Some inroads

• System integration still broadly lacks discipline

• Many topics are off the radar of computer science curriculum

• Good luck finding the best people!