Correlation Metric

Seoul National University

Seoul National University System Health & Risk Management

2017/2/25 ‐ 1 ‐

Correlation metric

Seoul National University System Health & Risk Management

Jungho Park*

*[email protected]

Seoul National University ‐ 2 ‐

Validation metric

Validation metric : a mathematical operator that measures the difference between a

system response quantity (SRQ) obtained from a simulation result and one obtained

from experimental measurement.(verification and validation in scientific computing)

Figure reference : Verification, validation, and predictive capability in computational engineering and physics, Oberkampf et al. ,Applied mechanics(2004)


Validation metric 의종류

1. Classical hypothesis testing

‐ 평균및분산에대한가설을세우고, 얻어진실험결과로부터가설검정실시

‐ 장점 : 모델의적합도여부를결정가능

‐ 단점 : 실험의개수가적을때는이용불가능

Liu, Yu, et al. "Toward a better understanding of model validation metrics."Transactions of the ASME‐R‐Journal of Mechanical Design 133.7 (2011): 071005.



2. Bayes factor

‐ Bayesian hypothesis testing 에서유래

‐ Null, alternative 가설의 posterior distribution 의비에의해결정


B=bayes factor



3. Frequentist’s metric

‐ Hypothesis 로부터모델의적합도를 ‘yes ‘ or ‘no’를결정하기보다는실험과시

뮬레이션값의차이를정량화


tane estimated predictionerrors estimated s dard devidationN numberof physicalobservation

Estimated error in the predictive model with a confidence level of 100(1‐ α)% that the true error is in the interval =

e



4. Area metric

‐ Mean, variance 같은moment 가아닌시험, 시뮬레이션분포의전체적모양을

비교

‐ 시험, 시뮬레이션개수가적을때사용가능

‐ U‐pooling method 와함께자주쓰임



Error metric(or correlation metric )의종류

1. Vector norms

2. Average residual and Its Standard Deviation

3. Coefficient of correlation and cross relation

Limitations: Not able to distinguish error due to phase from error due to magnitude

Limitations: Positive and negative differences at various point may cancel out

2

11

( )

1

N

i

N

R RS

N

( )i iRi a b

Limitations: Sensitive to phase differenceNot able to distinguish error due to phase from error due to magnitude

1 1 1

2 2 2 2

1 1 1 1

( )( )

( ) ( ) ( ) ( )

N n N n N n

i i n i i ni i i

N n N n N n N n

i i i n i ni i i i

N n a b a bn

N n a a N n b b

*Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics, H.Sarin, M.Kokkolaras, G.Hulbert, P.Papalambro, S.Barbat, R.‐J.Yang, Journal of Dynamic Systems, Measurement, and Control(2010)



4. Sprague and Geers metric

5. Russel’s error measure

1&G

1 cos ( ),ABS

AA BB

P

& 1,

2 2& & &S G S G S GC M P

2 2

1 1 1, , ,

N N N

i i i ii i i

AA BB AB

a b a b

N N N

Characteristics: Phase error portion consideredLimitations: lumped the entire information into , ,

Magnitude :

Phase :

Total :

10( ) log (1 )AA BBR AA BB

AA BB

M sign

Characteristics: Phase error portion consideredLimitations: lumped the entire information into , ,

No magnitude error




6. Normalized Integral Square Error(NISE)

7. Dynamic Time Warping

2 ∗ 2∗

2 ∗ 1 ∗

12

Phase : Magnitude: Shape :

Total :

Characteristics: Shape error portion consideredLimitations: Magnitude portion can be negative. (which mean magnitude portion can decrease overall error)

Characteristics: Algorithm for measuring discrepancy between time history




8. Weighted Integrated Factor (WIFac)

1

max , ⋅ 1 max 0, ⋅max ,

max ,0 1

1∑∑


Correlation metric and validation metric

Validation metric : a mathematical operator that measures the difference between a

system response quantity (SRQ) obtained from a simulation result and one obtained

from experimental measurement.(verification and validation in scientific computing)

0 5 10 15 20 25 300

50

100

150

200

250

300

350Simulation

Time(ms)

Res

ulta

nt A

cc(g

)0 0.2 0.4 0.6 0.8 1

0

10

20

30

40

50

60

70

80

90

WIFac

Den

sity

Exp :Sim :

Time(s)Ac

c(g)

0 0.005 0.01 0.015 0.02 0.025 0.030

20

40

60

80

100

120

140

160

180

Meanof exp.

:

0 5 10 15 20 25 300

50

100

150

200

250

300

350Simulation

Time(ms)

Res

ulta

nt A

cc(g

)

0 5 10 15 20 25 300

50

100

150

200

250

300

350Simulation

Time(ms)

Res

ulta

nt A

cc(g

)

0 5 10 15 20 25 300

50

100

150

200

250

300

350Simulation

Time(ms)

Res

ulta

nt A

cc(g

)

+3σ +1.5σ

‐3σ ‐1.5σ

WIFac

Derivation of WIFac for Simulation(0.5275, 0.5133,0.5293,0.5183)

0 5 10 15 20 25 300

50

100

150

200

250

300

350Experiment

Time(ms)

Res

ulta

nt A

cc(g

)

HIC 324HIC 565HIC 347HIC 290

Derivation of WIFac for Experiment(0.7738, 0.6186, 0.7648, 0.7308)

WIFac is not Validation metric, but Area metric is

0 0.5 10

0.5

1Funi

FuCDF

Um = 0.2641

Engineering

Correlation Metric