Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics

Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems

MTA-BME Lendület Research Group on Cyber-Physical SystemsBudapest University of Technology and Economics

Fault Tolerant Systems Research Group

Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics

Gábor Szárnyas, Zsolt Kővári, Ágnes Salánki, Dániel Varró

Motivation

Research Community

Problems of experimental evaluation of MDE papers

Difficult to find real industrial model

Tool Providers

Test generation for modeling tools

Scalability evaluationand stress testing of MDE tools

Smart CPS

Synthesis of prototypical test context/environment

Testing of autonomous robots(R3COP project)

Motivation

Research Community

Problems of experimental evaluation of MDE papers

Difficult to find real industrial model

Tool Providers

Test generation for modeling tools

Scalability evaluationand stress testing of MDE tools

Smart CPS

Synthesis of prototypical test context/environment

Testing of autonomous robots(R3COP project)

How to automatically synthesize graph models…?

Research Question and Objectives

• All well-formedness constraints satisfied

• Designated seed fragments included Consistent

• How to characterize realistic models?

• How to distinguish real and generated models?Realistic

• Guaranteed test coverage

• Required for tool qualificationDiverse

• Performance benchmarks

• Stress testing of tools and control algorithmsScalable

How to automatically synthesize graph models which are...

Research Question and Objectives

• All well-formedness constraints satisfied

• Designated seed fragments included Consistent

• How to characterize realistic models?

• How to distinguish real and generated models?Realistic

• Guaranteed test coverage

• Required for tool qualificationDiverse

• Performance benchmarks

• Stress testing of tools and control algorithmsScalable

How to automatically synthesize graph models which are...

Performance Experiments

Performance Experiments

„I would like to benchmark my tool on real models”

Performance Experiments

„I would like to benchmark my tool on real models”

o Industrial models are difficult to obtain.

Performance Experiments

„I would like to benchmark my tool on real models”

o Industrial models are difficult to obtain.

Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”

Performance Experiments

„I would like to benchmark my tool on real models”

o Industrial models are difficult to obtain.

Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”

o Great, but what does that imply for real use cases?

Performance Experiments

„I would like to benchmark my tool on real models”

o Industrial models are difficult to obtain.

Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”

o Great, but what does that imply for real use cases?

Workaround #2: Implement a custom benchmark

Performance Experiments

„I would like to benchmark my tool on real models”

o Industrial models are difficult to obtain.

Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”

o Great, but what does that imply for real use cases?

Workaround #2: Implement a custom benchmark

o Again, what does that imply for real use cases?

Performance Experiments

„I would like to benchmark my tool on real models”

o Industrial models are difficult to obtain.

Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”

o Great, but what does that imply for real use cases?

Workaround #2: Implement a custom benchmark

o Again, what does that imply for real use cases?

Qualitative description of models is required

How to Obtain Models for Benchmarking?

• Difficult to obtain

• Obfuscated modelsIndustrial

• Quality of models?Student work

• Good quality models

• Small in sizeTutorial

• How realistic are these models?Generated

What Makes a Model Realistic?

How to decide if a model is realistic

without domain-specific knowledge?

Statecharts with Attributes

RedRed &

Orange

GreenOrange

RedRed &

Orange

GreenOrange

Statecharts with Attributes

RedRed &

Orange

GreenOrange

RedRed &

Orange

GreenOrange

Statecharts

S1 S2

S3S4

S1 S2

S3S4

Statecharts

S1 S2

S3S4

S1 S2

S3S4

Typed Graphs of the Models

S1 S2 S3 S4

T1 T2 T3 T4 T5

E S4 S1 S2 S3

T1 T2 T3 T4 T5

E

Typed Graphs of the Models

S1 S2 S3 S4

T1 T2 T3 T4 T5

E S4 S1 S2 S3

T1 T2 T3 T4 T5

E

Which is the graphof a real model?

Graph Metrics

Use graph metrics for characterizingthe graph of the model.

Graph Metrics

Graph Metrics

Number of vertices

Graph Metrics

05

10152025

Number of vertices

Graph Metrics

05

10152025

Number of vertices

Number of edges

Graph Metrics

05

10152025

Number of vertices

0

10

20

30

40

Number of edges

Graph Metrics

05

10152025

Number of vertices

0

10

20

30

40

Number of edges

Average shortestpath

Graph Metrics

05

10152025

Number of vertices

0

10

20

30

40

Number of edges

Average shortestpath

0123456

Graph Metrics

05

10152025

Number of vertices

0

10

20

30

40

Number of edges

Average shortestpath

0123456

Graph Metrics

05

10152025

Number of vertices

0

10

20

30

40

Number of edges

Average shortestpath

0123456

Clusteredness

One-Dimensional Graph Metrics

05

10152025

Number of vertices

0

10

20

30

40

Number of edges

Average shortestpath

0123456

Clusteredness

00.20.40.60.8

1

Graph Metrics

05

10152025

Number of vertices

0

10

20

30

40

Number of edges

Average shortestpath

0123456

Clusteredness

00.20.40.60.8

1

Centrality

Graph Metrics

05

10152025

Number of vertices

0

10

20

30

40

Number of edges

Average shortestpath

0123456

Clusteredness

00.20.40.60.8

1

Centrality

00.20.40.60.8

1

Graph Metrics

S1 S2 S3 S4

T1 T2 T3 T4 T5

E S4 S1 S2 S3

T1 T2 T3 T4 T5

E

Which is the graphof a real model?

Graph Metrics

S1 S2 S3 S4

T1 T2 T3 T4 T5

E S4 S1 S2 S3

T1 T2 T3 T4 T5

E

Which is the graphof a real model?

Graph Metrics

S1 S2 S3 S4

T1 T2 T3 T4 T5

E S4 S1 S2 S3

T1 T2 T3 T4 T5

E

They are isomorphic.

Which is the graphof a real model?

Graph Metrics

S1 S2 S3 S4

T1 T2 T3 T4 T5

E S4 S1 S2 S3

T1 T2 T3 T4 T5

E

They are isomorphic.

Which is the graphof a real model?

Related finding: simplegraph metrics are unable topredict query performance

Network Theory

Mid ‘90s, László Albert-Barabási et al.

o Preferential attachment: „the rich gets richer”

Scale-free networks (web, power grid, etc.)

Most approaches only consider untyped graphs.

Network Theory

Mid ‘90s, László Albert-Barabási et al.

o Preferential attachment: „the rich gets richer”

Scale-free networks (web, power grid, etc.)

Most approaches only consider untyped graphs.

S1

S2

S3

S4

T1

T2

T3

T4

T5

E

Network Theory

Mid ‘90s, László Albert-Barabási et al.

o Preferential attachment: „the rich gets richer”

Scale-free networks (web, power grid, etc.)

Most approaches only consider untyped graphs.

S1

S2

S3

S4

T1

T2

T3

T4

T5

E S4 S1 S2 S3

T1 T2 T3 T4 T5

E

„Evaluation of Multidisciplinary Graph Metrics”

Typed graph (computer science)

Multi-layered networks (social network analysis)

Multidimensional networks (network theory)

Multiplex networks (physics)

Source: Wikipedia, Multidimensionalnetwork

Multidimensional Metrics

Dimensional degree distributions

Node dimension connectivity

o ratio of nodes in the that belong to a dimension

Multiplex participation coefficient

o the connections of v are uniformly distributed among D

Node activity & pairwise multiplexity

o the ratio of nodes, which are active in both d1 and d2

Methodology

RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

2. Data Cleansing: removeRed

Red-Orange

Green Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

2. Data Cleansing: remove

o layout information RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

2. Data Cleansing: remove

o layout information

o attributes

RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

2. Data Cleansing: remove

o layout information

o attributes

RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

2. Data Cleansing: remove

o layout information

o attributes

o object types

RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

2. Data Cleansing: remove

o layout information

o attributes

o object types

RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

2. Data Cleansing: remove

o layout information

o attributes

o object types

o small models

RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

2. Data Cleansing: remove

o layout information

o attributes

o object types

o small models

o derived references

RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

2. Data Cleansing: remove

o layout information

o attributes

o object types

o small models

o derived references

3. Calculate graph metrics

RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Methodology

1. Collect models

2. Data Cleansing: remove

o layout information

o attributes

o object types

o small models

o derived references

3. Calculate graph metrics

4. Analyze results

o Statistical + exploratory

RedRed-

OrangeGreen Orange

T1 T2 T3 T4 T5

Entry

Domains

AutoFOCUS

Building Information Model

Capella

JaMoPP

Train Benchmark

Yakindu

real

real

tutorial

synthetic

tutorial

tutorial

Domain 1

Statistical Analysis

Domain 1

Statistical Analysis

0

1

0 2 4 6

0

1

0 2 4 6

Domain 1

Domain 2

Statistical Analysis

0

1

0 2 4 6

0

1

0 2 4 6

0

1

0 2 4 6

0

1

0 2 4 6

Domain 1

Domain 2

Statistical Analysis

0

1

0 2 4 6

0

1

0 2 4 6

0

1

0 2 4 6

0

1

0 2 4 6

0

1

0 2 4 6

Statistical Analysis

0

1

0 2 4 6

Homogeneity

Statistical Analysis

0

1

0 2 4 6

0

1

0 1 2 3 40

1

0 2 4 6

Homogeneity

Statistical Analysis

0

1

0 2 4 6

0

1

0 1 2 3 40

1

0 2 4 6

Homogeneity

Statistical Analysis

0

1

0 2 4 6

0

1

0 1 2 3 40

1

0 2 4 6

Kolmogorov-Smirnov distance

Homogeneity

Statistical Analysis

0

1

0 2 4 6

0

1

0 1 2 3 40

1

0 2 4 6

Kolmogorov-Smirnov distance

Homogeneity

Distinctiveness

Statistical Analysis

0

1

0 2 4 6

0

1

0 1 2 3 40

1

0 2 4 6

0

1

0 2 4 6

Kolmogorov-Smirnov distance

Homogeneity

Distinctiveness

Statistical Analysis

0

1

0 2 4 6

0

1

0 1 2 3 40

1

0 2 4 6

0

1

0 2 4 6

Kolmogorov-Smirnov distance

Dimensional Clustering Coefficients

Dimensional Clustering Coefficients

KS distance

Findings

1. Metamodel-level information is insufficient

Findings

1. Metamodel-level information is insufficient

1. The ratio of containment edge types in the Capella metamodels: 75%

2. The ratio of containment edges in the Capella models: 42–50 %

Findings

1. Metamodel-level information is insufficient

2. Containment edges dominate distributions

Findings

1. Metamodel-level information is insufficient

2. Containment edges dominate distributions

3. Many edges follow the locality principle

Future Directions

Use metrics for

o Instance model generators

o Query optimization

Improve performance of calculating metrics:incremental calculation

o https://github.com/ftsrg/model-analyzer

oWorks for both EMF and RDF models

All analysis results & code are available online:

o http://docs.inf.mit.bme.hu/model-metrics/

The Train Benchmark

SOSYM paper – The Train Benchmark: Cross-Technology Performance Evaluation of Continuous Model Validation

o 6 queries, 12 transformations

o EMF, property graphs, RDF, SQL

o 12+ tools

o Automated visualization & reporting

http://github.com/ftsrg/trainbenchmark

Ω