62
June 1998 CHAIMS 1 Compiling High-level Access Interfaces for Multi-site Software Stanford University Objective: Investigate revolutionary approaches to large-scale software composition. Approach: Develop & validate a composition-only language. Contributions and plans: Hardware and software platform independence. Asynchrony by splitting up CALL-statement. Performance optimization by invocation scheduling. Potential for multi-site dataflow optimization. www-db.stanford.edu/CHAIMS CHAIMS: Mega-Programming Research CHAIM S

C ompiling H igh-level A ccess I nterfaces for M ulti-site S oftware

  • Upload
    shiloh

  • View
    19

  • Download
    0

Embed Size (px)

DESCRIPTION

CHAIMS: Mega-Programming Research. C ompiling H igh-level A ccess I nterfaces for M ulti-site S oftware Stanford University Objective : Investigate revolutionary approaches to large-scale software composition . - PowerPoint PPT Presentation

Citation preview

Page 1: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 1

Compiling High-level AccessInterfaces for Multi-site Software Stanford University

Objective: Investigate revolutionary approaches to large-scale software composition.

Approach: Develop & validate a composition-only language.

Contributions and plans: • Hardware and software platform independence.• Asynchrony by splitting up CALL-statement.• Performance optimization by invocation scheduling.• Potential for multi-site dataflow optimization.

www-db.stanford.edu/CHAIMS

CHAIMS: Mega-Programming Research

CHAIMS

Page 2: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 2

Participants• Support

– DARPA ISO EDCS program (1996-1999)– Siemens Corporate Research (1996-1998)– DoD AFOSR AASERT student support (1997-1999)– Sloan Foundation - computer industry study (1996-97)

• People – Gio Wiederhold (Prof. Res) PI Marianne Siroker (Administration)

– Dorothea Beringer (postdoc EPF Lausanne) since Dec.1997

– Ron Burback (CS PhD cand.) Neil Sample (CS PhD Student)

– Laurence Melloul (CS MS) Woody Pollack (CS MS)

– MS and BS CS graduated: Joshua Hui, Gaurav Bhatia, Prasanna Ramaswami, Kirti Kwatra, Pankaj Jain, Mehul Bastawala, Catherine Tornabene, Wayne Lim (I.E.), Connan King (E.E.).

– Louis Perrochon (postdoc ETH Zurich) Fall quarter 1996

Page 3: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 3

Gio Wiederhold: Personal Background

• 1936 born Varese, Italy

• 1957: Learned programming at NATO SHAPE ADTC

• 1958-1975 Programmer and software engineer – at IBM, UC, Stanford, Index, MaSCOR

• 1963 - now Consultant for government, Industry

• 1974-1976 PhD on Database Design at UC SF

• 1976- now Professor Stanford– Computer Science, Medicine, Electrical Eng., Business School

• Elected fellow ACMI, IEEE, ACM

• Innovations:– solid rocket fuel combustion A-formatincremental compilers – timeshared real-time data acquistion time-oriented databases – database design knowledge-based system concepts – object creation from relationsmediatorssecurity filters.

Page 4: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 4

Dorothea Beringer: Personal Background

• Masters in Computer Science: hybrid-monitoring tool for debugging and software performance analysis for distributed software

• Software engineer: telecommunication systems

• Consultant: software methodologies, quality assurance, project management, CASE-tools

• PhD: Modeling scenarios in object-oriented analysis

• Teaching: Fusion

• Now: CHAIMS -- large-scale software composition, distributed systems

Page 5: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 5

Presentation

• Motivation and Objectives

– changes in software production

– basis for new visions and education

• Concepts of CHAIMS

– CHAIMS language

– CHAIMS architecture and composition process

– Scheduling

– Dataflow optimization

• Status, Plans, Conclusions

Page 6: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 6

Coding

Integration

1970 1990 2010

Shift in Programming Tasks

Page 7: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 7

Languages & Interfaces• Large languages intended to support coding

and composition have not been successful– Algol 68– PL/1– Ada– CLOS

• Databases are being successfully composed, using Client-server, Mediator architectures

– distribution -- exploit network capabilities

– heterogeneity -- autonomy creates heterogneity

– simple schemas -- some human interpretation

– service model -- public and commercial sources

Page 8: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 8

Typical Scenario: LogisticsA general has to ship troops and/or various material from San Diego NOSC to Washington DC:

– different kind of material: criteria for preferred transport differ

– not every airport equally suited

– congestion, prices

– actual weather

– certain due or ready dates

Today: calling different companies, looking up information on the web, reservations by hand

Tomorrow: system proposes possibilities that take into account various conditions

» hand-coded systems

» composition of processes

Page 9: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 9

Scaling alternatives

?

Page 10: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 10

C H A I M S

Megamodules

Megaprogram for composition, written by domain programmer

CHAIMS system automates generation of client for

distributed system

Megamodules, provided by various megamodule

providers

CHAIMS

Page 11: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 11

Megamodules - DefinitionMegamodules are large, autonomous, distributed,

heterogeneous services or processes.• large: computation intensive, data intensive, ongoing processes

(monitoring services)

• distributed: to be used by more than one client

• heterogeneous: accessible by various distribution protocols (not only different languages and systems)

• autonomous: maintenance and control over recourses remains with provider, differing ontologies ( ==> SKC)

Examples:– logistics: “find best transportation route from A to B”,

reservation systems– genomics: easier framework for composing various processing

tools than ad-hoc coding

Page 12: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 12

Challenge: Fat Clients Domain expert

Client computer

Control &Computation

Services

I/O

a bc

de

Wrappers to resolve

differences

I/O

DataResources

Page 13: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 13

Challenge: Thin Clients Domain expert

Client workstation

ComputationServices

IO module

MEGA modules

IO module

a

b

cd

e

DataResources

Sites RT

S U T

C

Page 14: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 14

Challenge: Heavy-weight Services

Services are not free for a client:• execution time of a service

• transfer time for data

• fees for services

What we need:==> monitoring progress of a service

==> possibility to choose among equivalent services based on estimated waiting time and

fees

==> parallelism among services

==> preliminary overview results, choosing level of accuracy / number of results for

complex processes

==> novel optimization techniques

Page 15: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 15

Challenge:Empower Non-technical Domain Experts

Company providing services:• domain experts of domain of service (e.g. weather)

• technical experts for programming for distribution protocols, setting up servers in a middleware system

• marketing experts

“Megaprogrammer”:• is domain expert of domain that uses these services

• is not technical expert of middleware system or experienced programmer,

• wants to focus on problem at hand (=results of using megaprogram)

• e.g. scientist, logistics officer

Page 16: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 16

Challenge: Purely Compositional Language Possible?

Which languages did succeed?– Algol, ADA: integrated composition and computation

– C, C++ focus on computation

Why new language?– complexity: not all facilities of a common language

(compare to approach of Java),

– inhibiting traditional computational programming (compare C++ and Smalltalk concerning object-oriented programming)

– focus on issue of composition, parallelism by asynchrony, and optimization

Page 17: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 17

CHAIMS “Logical” Architecture

Customer

Megaprogramclients(in CHAIMS)

Network/Transport(DCE, CORBA,...)

Megamodules(Wrapped or Native)

Page 18: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 18

CHAIMS Physical Architecture

Network

DCE, CORBA, JAVA RMI, DCOM...

MegaprogramClients in CHAIMS

Megamodules (wrapped, native) each supportingsetup, estimate, invoke, examine, extract, and terminate.

Page 19: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 19

Decomposing CALL statements

Copying Code sharing

Parameterized computation

Objects with overloaded method names

Remote procedure calls to distributed modules

Constrained (black box) access to encapsulated data

progressin

scale ofcomputing

ExtractInvokeEstimate ExamineSetup

CHAIMSdecomposes CALL functions

CALL gainedfunctionality

Page 20: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 20

CHAIMS Primitives

Pre-invocation:SETUP: set up the connection to a megamodule

SET-, GETATTRIBUTES: set global parameters in a megamodule

ESTIMATE: get estimate of execution time for optimization

Invocation and result gathering:INVOKE: start a specific method

EXAMINE: test status of an invoked method

EXTRACT: extract results from an invoked method

Termination:TERMINATE: terminate a method invocation or a connection to

a megamodule

Control: Utility:WHILE, IF GETPARAM: get default parameters

Page 21: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 21

Megaprogram Example: Overview

InputOutput- Input- Output

RouteInfo- AllRoutes- CityPairList- ...

AirGround- CostForGround- CostForAir- ...

Routing- BestRoute- ...

RouteOptimizer- Optimum- ...

General I/O-megamodule» Input function takes as parameter a default

data structure containing names, types and default values for expected input

Travel information:» Computing all possible routes between

two cities

» Computing the air and ground cost for each leg given a list of city-pairs and data about the goods to be transported

Two megamodules that offer equivalent functions for calculating optimal routes

» Optimum and BestRoute both calculate the optimum route given routes and costs

» Global variables: Optimization can be done for cost or for time

Page 22: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 22

Megaprogram Example: Codeio_mmh = SETUP ("InputOutput")route_mmh = SETUP ("RouteInfo")...best2_mmh.SETATTRIBUTES (criterion = "cost")

cities_default = route_mmh.GETPARAM(Pair_of_Cities)input_cities_ih = io_mmh.INVOKE ("input”, cities_default)WHILE (input_cities_ih.EXAMINE() != DONE) {}cities = input_cities_ih.EXTRACT()...route_ih = route_mmh.INVOKE ("AllRoutes", Pair_of_Cities = cities)WHILE (route_ih.EXAMINE() != DONE) {}routes = route_ih.EXTRACT() …

IF (best1_mmh.ESTIMATE("Best_Route") < best2_mmh.ESTIMATE("Optimum") ) THEN {best_ih = best1_mmh.INVOKE ("Best_Route", Goods = info_goods, Pair_of_Cities = cities, List_of_Routes = routes, Cost_Ground = cost_list_ground, Cost_Air = cost_list_air)}ELSE {best_ih = best2_mmh.INVOKE ("Optimum", Goods = info_goods, …...best2_mmh.TERMINATE()

// Setup connections to megamodules.

// Set global variables valid for all invocations // of this client.

// Get information from the megaprogram user // about the goods to be transported and about// the two desired cities.

// Get all routes between the two cities.

//Get all city pairs in these routes.//Calculate the costs of all the routes.

// Figure out the optimal megamodule for// picking the best route.

//Pick the best route and display the result.

// Terminate all invocations

Page 23: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 23

Operation of one Megamodule

• SETUP

• SETATTRIBUTES provides context

• ESTIMATE serves scheduling

• INVOKE initiates remote computation

• EXAMINE checks for completion

• EXTRACT obtains results

• TERMINATE I / ALL

M handle

M handle

M handle

M handle

I handle

I handle

I handle

M handle

I handle

Page 24: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 24

CHAIMS Megaprogr. Language

Purely compositional:– no primitives for arithmetic ==> math megamodules

– no primitives for input/output ==> general and problem-specific I/O megamodules

Splitting up CALL-statement:– parallelism by asynchrony in sequential program

– novel possibilities for optimizations

– reduction of complexity of invoke statements

• higher-level language (assembler => HLLs, HLLs => composition/megamodule paradigm)

Page 25: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 25

Architecture: Runtime

ed

a

b

c

Distribution System (CORBA, RMI…)

CSRT(compiled megaprogram)

MEGA modules

Page 26: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 26

Architecture: Composition Process

ed

a

b

c

MEGA modules

CHAIMS Repository

adds information to

MegamoduleProvider

wraps non-CHAIMScompliant megamodules

Wrapper Templates

e

Page 27: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 27

writes

Architecture: Composition Process

Megaprogrammer

CSRT(compiled megaprogram)

Megaprogram(in CHAIMS language)

CHAIMS Compiler

generates

CHAIMS Repository

information

information

Page 28: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 28

writes

Architecture: Overview

e

Megaprogrammer

d

a

b

c

Distribution System (CORBA, RMI…)

CSRT(compiled megaprogram)

Megaprogram(in CHAIMS language)

CHAIMS Compiler

generates

MEGA modules

CHAIMS Repository

adds information to

MegamoduleProvider

wraps non-CHAIMScompliant megamodules

information

information

Wrapper Templates

Page 29: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 29

Architecture: CHAIMS-Language and CHAIMS-Protocols

Megaprogram

Megaprogrammer

M e g a m o d u l e s

CHAIMS-language

CHAIMS-protocols

CORBA-idl DCE-idl Java-class

CHAIMS API defines interface between megaprogrammer and megaprogram; the megaprogram is

written in the CHAIMS language.

The CHAIMS protocols define the calls the mega-modules have to understand. These protocols are slightly different for the different distribution protocols, and are defined by an idl for CORBA, another idl for DCE, and a Java class for RMI.

Page 30: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 30

Name of Person

Architecture: GentypeMinimal Typing in CHAIMS:

Integer, boolean only for controlAll else is placed into an ASN.1 bag, transparent to compiler :

A Gentype is a triple of name, type and value, where value is either a simple type or a list of other gentypes (i.e. a complex type).

Simple types: given by ASN.1, the ASN.1-conversion library for C++, our own conversion routines.

Example: Person_Information complex

First Name string Joe Last Name string Smith

Personal Data complex Address

Date of Birth date 6/21/54 Soc.Sec.No string 345-34-345

Page 31: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 31

Wrapper: CHAIMS Compliance

• CHAIMS protocol - support all CHAIMS primitives

• State management and asynchrony: » clientId (megamodule handle in CHAIMS language)

» callId (invocation handle in CHAIMS language)

» results must be stored for possible extraction(s) until termination of the invocation

• Data transformation: » all parameters of type blob (BER-encoded Gentype)

must be converted into the megamodule specific data types (combination hand-coding/decoding routines

Page 32: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 32

Architecture: Three Views

Transport View moving around data blobs and CHAIMS messages

Composition View (megaprogram)

- composition of megamodules

- directing of opaque data blobs

Data View - exchange of data - interpretation of

data - in/between

megamodules

CHAIMS Layer

Distribution Layer

Objective: Clear separation between composition of services, computation of data, and transport

Page 33: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 33

execution of a remote method

synchronous

invoke a methodie extract results

setup / set attributes s

s

e

i

tim

e

decomposed(no benefit for one module)

asynchronous

s,i

tim

e

eti

me

available for other methods

e

s,i

Scheduler: Decomposed Execution

Page 34: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 34

Optimized Execution of Modules

M1 M4(<M1+M2)

M5

M2

M3 (>M1+M2)

i1

e1

e4e3

e2

i3i4

i5

i2

e5

tim

e

M1

M4

M5

M2

M3

i1

e1

e2

e3

e4

e5

i2

i3

i4

i5

tim

e

data dependenciesexecution of a module

non-optimized

optimized by scheduleraccording to estimates

invoke a methodie extract results

Page 35: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 35

Decomposed Parallel Execution

tim

e

M1M4

(<M1+M2)

M5

M2

M3<M1+M2)

optimized by scheduleraccording to estimates

invoke a methodextract results

set up / set attributes

Long setup timesoccur, for instance,when a subset of a large database hasto be loaded for asimple search, sayTransatlantic fightsfor an optimal arrival.

Page 36: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 36

M1M4

(<M1+M2)

M5

M2

M3 (>M1+M2)

Decomposed Optimized Execution

M1

M4(<M1+M2)

M5

M2

M3 (>M1+M2)

optimized by scheduleraccording to estimates

invoke a methodextract results

set up / set attributes

tim

e

pri

or

tim

e

Page 37: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 37

M1M4

(<M1+M2)

M5

M2

M3 (>M1+M2)

Repeated invocations

M1

M4(<M1+M2)

M5

M2

M3 (>M1+M2)

optimized by scheduleraccording to estimates

invoke a methodextract results

set up / set attributes

tim

e

pri

or

tim

e

Page 38: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 38

M1M4

(<M1+M2)

M5

M2

M3 (>M1+M2)

Repeated Extractions

M1

M4(<M1+M2)

M5

M2

M3 (>M1+M2)

optimized by scheduleraccording to estimates

invoke a methodextract results

set up / set attributes

tim

e

pri

or

tim

e

Page 39: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 39

Scheduling: Simple Example

1 cost_ground_ih = cost_mmh.INVOKE ("Cost_for_Ground", 1 List_of_City_Pairs = city_pairs,Goods = info_goods)

2 WHILE (cost_ground_ih.EXAMINE() != DONE) {} 3cost_list_ground = cost_ground_ih.EXTRACT()

3 cost_air_ih = cost_mmh.INVOKE ("Cost_for_Air", 2 List_of_City_Pairs = city_pairs,Goods = info_good)

4 WHILE (cost_air_ih.EXAMINE() != DONE) {} 4cost_list_air = cost_air_ih.EXTRACT()

order inunscheduledmegaprogram

order in automaticallyprescheduled megaprogram

Page 40: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 40

Scheduling: Possible ActionsINVOKES: call INVOKE’s as soon as possible

» may depend on other data

» moving it outside of an if-block: depending on cost-function (ESTIMATE of this and following functions concerning execution time, dataflow and fees (resources).

EXTRACT: move EXTRACT’s to where the result is actually needed

» no sense of checking/waiting for results before they are needed

» instead of waiting, polling all invocations and issue next possible invocation as soon as data could be extracted

TERMINATE: terminate invocations that are no longer needed (save resources)

» not every method invocation has an extract (e.g. print-like functions)

Page 41: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 41

Compiling into a Network

Mega Program

Module A

Module B

Module CModule E

Module DModule F

current CHAIMS systemMega Program

Module DModule F

control flow data flow

with distribution dataflow optimization

Mega Program

Module A

Module B

Module C Module E

Module DModule F

Page 42: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 42

CHAIMS Implementation• Specify minimal language

– minimal functions: CALLs, While, If *

– minimal typing {boolean, integer, string, handles, object}

» objects encapsulated using ASN.1 standard

– type conversion in wrappers, service modules*

• Compiler for multiple protocols (one-at-time, mixed*)

• Wrapper generation for multiple protocols

• Native modules for I/O, simple mathematics*, other

• Implement API for CORBA, Java RMI, DCE usage

• Wrap / construct several programs for simple demos

• Schedule optimization *

• Demonstrate use in heterogeneous setting *

• Define full-scale demonstration * in process

Page 43: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 43

Status• Definition of architecture for Megaprogramming

– bottom up assessment of code to be generated

• examples: room reservation, shipping

– primitives

– handles for parallel operation

– heterogeneity -- common features of distribution protocols

• Minimal language that can generate the code– no versus very few types -- ASN.1 for complex types

– natural parallelism -- still a major research issue

• Awareness of novel optimizations– information flow constraints -- scheduling

– direct data flow between megamodules

Page 44: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 44

Focus for Future

• Finishing basic infrastructure and demo examples.

• CHAIMS interpreter to complement compiler.

• Dynamic scheduling of invocations and extractions.

• Flexible interaction with megamodules; extracting and handling overview results.

• Direct dataflows between megamodules – (future project).

Page 45: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 45

Upcoming Changes to Architecture: PreCompiler + Interpreter

megaprogramin CHAIMS language

client code in C, C++, Java,

stub code

executableclient (CSRT)

user

Compiler:

CHAIMS executionmachine

(interpreter andscheduler)

userInterpreter:

networkCHAIMS-protocol

completemegaprogram

in CHAIMS language

some CHAIMSstatements

user

serves as input to

serve as input to

CHAIMS compiler,simple scheduler

Idl-file generatorand compiler

C++, Java compiler

and linkernetwork

Page 46: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 46

Interpreter• Dynamic scheduler:

» Parsed input is stored in an executable dependency graph.

» Execution machine (interpreter / scheduler) works through the graph and makes appropriate calls:

– estimate-calls are inserted to get necessary run-time information for scheduling (cost-function)

– every invocation is issued as soon as possible (data-flow) and reasonable (according to cost-function)

– all invocations for which the CSRT waits for results are polled regularly, and results extracted and new invocations issued as soon as possible

CSRT would still be sequential!

• Overview results, flexible interactions:» megaprogrammer can program statement by statement and get results

immediately; results will influence what he/she does next

» like ftp, web

Page 47: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 47

CHAIMS

Conclusion: Research Questions

• Is a Megaprogramming language focusing only on composition feasible?

• Can it exploit on-going progress in client-server models and be protocol independent?

• Can natural parallelism for distributed services be effectively scheduled?

• Can high-level dataflow among distributed modules be optimized?

• Can CHAIMS express clearly a high-level distributed SW architecture?

• Can the approach affect SW process concepts and practice?

Page 48: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 48

Other Research ProjectsRelated by common issue: Large-Scale Interoperation

• Mediation -- modules in 3-tier Information Systems– {acess, abstraction, integration, summarization, delivery}

– maintenance management is a major benefit

• Security and Privacy Mediators – filter results to complement access control

– for healthcare privacy / manufacturing collaboration

• Scalable Knowledge Composition– develop algebra ( over ontologies

– articulate distinct distinct domains to create user contexts

• Image databases– rapid search by match using wavelets

– identifying pornography

– extracting text from images and icons for privacy/search

Page 49: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 49

Paying for SW ServicesYou can not run an effective (SW) business and not be

reimbursed for it. How? Four approaches:

– Sell Software sell oilfield to customer

– Lease copy / usage rights lease well

– Time / user limited access fill tank

– Charge by use instance provide bus

General problems, effects differ

– IP protection?

– keeping SW updated

– billing for est.value

– performance effect

poor some fair good

poor ok good good

simple simple awkw. hard

no no little some

Buy Lease Limit Use

protect

update

bill

perform

Page 50: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 50

Conclusion: Questions not addressed

• Will one Client/Server protocol subsume all others?

– distributed optimization remains an issue

• Synchronization / Concurrency Control

– autonomy of sources negates current concepts

– if modules share databases, then database locks may span setup/terminate all for a megaprogram handle.

• Will software vendors consider moving to a service paradigm?

– need CHAIMS demonstration for evaluation

Page 51: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS

Integration Science

IntegrationScience

IntegrationScience

ArtificialIntelligence

knowledge mgmtmodels

uncertainty

ArtificialIntelligence

knowledge mgmtmodels

uncertainty

Systems Engineering

analysisdocumentation

costing

Systems Engineering

analysisdocumentation

costing

Databasesaccessstoragealgebras

Databasesaccessstoragealgebras

Page 52: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 52

CHAIMS

Page 53: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 53

Backup slides

Page 54: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 54

Composition of Processes...

• versus composition and integration of Data» data-warehouses

» wrapping data available on web

• versus composition of Components» reusing small components via copy/paste or shared

libraries locally installed

» large distributed components within same “domain” as composition, e.g. within one bank or airline

CHAIMS: » processed information

» composing autonomous execution threads

Page 55: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 55

Summary• CHAIMS requires rethinking of many common

assumptions

– gain understanding via simple examples

• Work focused on CALL statement decomposition

– to accomplish integration of large services

– exploit inherent asynchrony

• First version of architecture and language drafts are completed; basic infrastructure partially available (compiler, wrapper templates).

• More demos will come soon.

Half-way through a four year project.

http://www-db.stanford.edu/CHAIMS

Page 56: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 56

CHAIMS proves that...• We can do composition in a high-level language.

» same language for Java-RMI-invocations and CORBA-invocations (and DCE, DCOM, TCP/IP protocols)

» (single megaprogram can deal with multiple protocols simultaniously)

» multiple megamodules can run in parallel

• Large-scale composition can be automated.» in contrast to manual non-software composition (e.g. telephone,

cut&paste)

» in contrast to fixed programs for one specific problem (e.g. transporting military goods within US)

• We can do schedulings of programs in a way right now only smart logistics officers can do, avoiding unnecessary waits.

» Scheduling of invocations can be optimized.

Page 57: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 57

Long-term Objectives of CHAIMS1 Implementing a system for a simple and purely

compositional language hiding differences of diverse protocols

2 Automatic optimized scheduling of invocations (taking advantage of inherent parallelism and estimate-capabilities of megamodules, hence splitting up of CALL-statement)

3 Decision-making support (direct) interaction with megamodules, based on overview and incremental results (fixed flow, not yet interactive changes to megaprogram)

4 Automatic dataflow optimization (direct dataflows between megamodules), not yet

Page 58: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 58

Assumptions, Additional Constraints• Heterogenous legacy modules ==> wrapping of modules, mixing

protocols on client side or in wrappers.

• Parallelism of megamodule-methods not through multithreading on client side but through splitting up CALL-statement (==> sequential program on client side); this leads to useful parallelism because we deal with coarse-grain parallelism.

• CHAIMS-compliancy for megamodules is achieved by wrapper-templates, for new native megamodules as well as for legacy ones (CHAIMS-compliancy is more than just knowing CHAIMS-protocol!).

• No reliance on existence of one specific higher level protocol like CORBA, DCOM, RMI ==> implementing an independent data-encoding and marshalling with ASN.1, instead of using one of them and then having converters in the wrappers.

• Interfaces of megamodules match <==> no investigation into opaque datablobs on client side necessary.

• Thin client, client should be able to run anywhere (not quite fulfilled right now - we need local ORB, DCE, JavaVirtual-machine).

• Clear seperation client - server, minimal repository.

Page 59: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 59

Non- (not yet)-Objectives of CHAIMS• No commercial product.

• No specific controls over ilities (security, name-serving, etc.) that they are normally present in distributed systems.

• No sophisticated front-end, no graphical programming/composition, no browser for repository, no higher-level language as input (not yet).

• Not solving all problems of megamodule composition that are mentioned in the various CHAIMS-papers (e.g. differing ontologies, non-matching interfaces of megamodules), only the ones mentioned in objectives and additional conditions.

Page 60: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 60

Short-term Objectives of CHAIMSRest of 1998:

» Basic infrastructure (fixing most severe flaws, moving to consistent architecture, all primitives, types, associative lists with handling it, having CORBA) ==> conceptual and implementation work -- CONSOLIDATION

» More examples (descriptions of scenarios as well as implemented demos), wrapping one (maybe two) additional suites of megamodules. ==> implementation work -- CONSOLIDATION

» Mixing of protocols in client (CORBA, RMI) or/and TCP/IP-three-tier architecture

» Preparing for more capable scheduler (examples with current scheduler, reading about other scheduler-problems and implementations, redesigning architecture of compiler (interpreter?), designing scheduler algorithm and architecture, writing paper about all this…) ==> lots of conceptual work, some implementation -- looking ahead for better Scheduler

1999 (depending on where we are at the end of 1998):» Scheduler

Page 61: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 61

Upcoming Changes to Architecture:Other Approach to Heterogeneity

Client (megaprogram)

nativeserver

1

nativeserver

3

nativeserver

2

chaimscompliantmodule

chaimsI/O

module

RMIwrapper

TCP/IP socketsCHAIMS protocol

CORBA RMIserver-specific protocols

site

s o

f s

erve

rs

clientsite

differentwrapper site

RMIwrapper

CORBAwrapper

Page 62: C ompiling  H igh-level  A ccess I nterfaces for  M ulti-site  S oftware

June 1998 CHAIMS 62

Reasons for an Alternative ArchitectureOverall:

• Simpler architecture: fewer wrappers, just one protocol on client side

Server-side:

• No direct linking with legacy code also for CORBA-wrappers, different sites for wrapper and legacy megamodule possible

• All native CHAIMS-megamodules will be built using wrapper templates ==> no reason for several protocols, they can all use TCP/IP.

• Dataflow-optimization: direct messages between megamodules/their wrappers necessary (without bridges)

Client-side:

• Thin client that could run everywhere (TCP/IP is available everywhere, but not CORBA or DCE, RMI also is easily available everywhere).

• CSRT could be implemented by interpreter instead of compiler, maybe also possible with current architecture, but more complex.

• We use just transport-facility (really true? what about native CHAIMS-types like string, integer, boolean?) of CORBA, RMI, DCE (for data we have ASN.1); this is already offered by TCP/IP ==> no unnecessary overkill

Drawback: missing one of the current funding objectives (heterogeinity on client side).