The Application of Partial Least Squares to Non-linear Systems in the Process Industries

Centre for Process Analytics and Control Technology (CPACT)University of Newcastle, UK

The Application of Partial Least Squares to Non-linear Systems

in the Process Industries

Elaine Martin and Julian Morris

Centre for Process Analytics and Control Technology

CPACT

School of Chemical Engineering and Advanced Materials

University of Newcastle, England


Overview of the Presentation

Motivation for the Application of “Data Mining” in Non-linear Process Systems

Process Modelling and Analysis of Non-linear Systems

Constrained Partial Least Squares

Local Linear Modelling

Prediction Intervals for Non-linear Partial Least Squares

Conclusions


Data Rich Information Poor

Enhanced Profitabilityand

Improved CustomerSatisfaction Modern Process

ControlSystems

Process Optimisation

Process Monitoring for Early Warning

andFault Detection


Process Modelling

Mechanistic models developed from process mass and energy balances and kinetics provide the ideal form given:

process understanding exists time is available to construct the model.

Data based models are useful alternatives when there is:

limited process understanding process data available from a range of operating

conditions.

Hybrid models combine the two approaches.


Process Modelling

Traditionally two types of variables have been used in the development of a process model/process performance monitoring scheme:

Process variables (X) Quality variables (Y)

In practice, a third class exists:

Confounding variables (Z).

A confounding variable is any extraneous factor that is related to, and affects, the two sets of variables under study (X) and (Y).

It can result in a distortion of the true relationship between the two sets of variables, that is of primary interest.


Global Process Variation

Confidence ellipse including confounding

variation

Trajectory of confounding

variable

Confidence ellipse excluding confounding variation

X

X X X X XX

Mal-operation


Partial Least Squares

ET TPX X-block outer relationship (monitoring)

FT UQY Y-block outer relationship (monitoring)

BTU Inner relationship (prediction)

X and Y-block scores are calculated recursively

TptXE 111

TqtbYF 1111

•

•


Constrained PLS

To exclude the nuisance source of variability, a necessary condition is that the derived latent variables, , and , are not correlated with the confounding variables:

and for .

The idea of constrained PLS is to apply the constraints to ordinary PLS.

0tZ hT 0uZ h

T Ah ,,1

ht hu

2T minarg hhh twEtt

2T minarg hhh uqFuu

2T minargT

hhh twEt0tZ

2T minargT

hhh uqFu0uZ


Constrained PLS

Standard constrained optimisation techniques can be used to solve the equations in each iteration.

An algorithm has been developed that enhances the efficiency of the constrained PLS algorithm.

The other steps of constrained PLS are as for ordinary PLS.

The resulting latent variables can then be used for process monitoring with the knowledge that they are not confounded with the nuisance source of variability.

Any unusual variation detected from these latent variables can then be assumed to be related to abnormal process behaviour.


Industrial Application

An industrial semi-discrete batch manufacturing operation is used to illustrate the advantages of the constrained PLS algorithm over ordinary PLS.

The process involves the production of a variety of products (recipes), some of which are only manufactured in small quantities to meet the requirements of specialised markets.

The objective of the analysis was to build a monitoring scheme to detect the onset of subtle changes in production and final product quality.


An Industrial Application

For simplicity, three recipes are selected to demonstrate the methodology.

A total of thirty-six process variables, including flow rates, pressures and temperatures, are recorded every minute, whilst five quality variables are measured off-line in the quality laboratory every two hours.

A nominal process monitoring scheme was developed using both ordinary PLS and constrained PLS from 41 ‘ideal’ batches.

A further 6 batches, A4, A10, A29, A35, A38 and B32 were used for model validation. These batches were known to lie outside the desirable specification limits.


Industrial Application Ordinary Partial Least Squares

-10 -5 0 5 10

-6

-4

-2

0

2

4

6

A

C

A

A A

B

A

B

A

B

C

B

A A

A A

C C

A

C C C

A

A A

A

C A A A

A

A

B

A

A

B

A

A A A

t1

t2

-6 -4 -2 0 2 4 6

-6

-4

-2

0

2

4

6

A

C

A A A B

A

B

A

B

C

B A A

A A

C C

A C C C

A A A A

C

A

A A A A

B

A A

B

A

A A

A

t3

t4

Latent variable 1 V Latent variable 2 Latent variable 3 V Latent variable 4


Industrial Application Ordinary Partial Least Squares

0 10 20 30 400

5

10

15

20

T2 s

tatis

tic

0 10 20 30 400

10

20

30

40

50

SP

E

-10 -5 0 5 10

-6

-4

-2

0

2

4

6

A

C

A

A

A A

B

A

B

A

A

B

C

B

A A

A A

C C

A

C C C C

A

A A

A

A

C

B

A A

A

A

A

A

A

B

A

A

B

A

A A A

4

10

29

32

35 38

t1

t2

Bivariate Scores Plot Hotelling’s T2 and SPE



-10 -5 0 5 10

-6

-4

-2

0

2

4

6

A

C

A

A

A A

B

A

B

A

A

B

C B

A

A

A A

C

C

A

C C C

C

A

A A

A A

C

B

A

A A

A A

A

A

B

A A

B A

A

A

A

35 38

4 10

29

32 35 38

t1

t2

-5 0 5-15

-10

-5

0

5

10

15

A C A

A

A A B A

B

A

A

B

C B

A A A

A C C A

C C

C

C

A A A

A

A

C

B

A A

A

A A

A

A

B A

A

B A A A

A

4

10

29 32 35

38

t3

t4

LV 1 versus LV 2 LV 3 versus LV 4




0 10 20 30 400

100

200

300

T2 s

tatistic

0 10 20 30 400

200

400

600

800

1000

SP

E

Hotelling’s T2

Squared Prediction Error



Constrained PLS - Conclusions

Constrained PLS possesses the following important

characteristics:

It removes that information correlated with the confounding variables.

The information excluded by constrained PLS contains only variation associated with the confounding variables.

The derived constrained PLS latent variables achieve optimality in terms of extracting as much of the available information as possible contained in the process and quality data.


Local Linear and Non-linear

Multi-way Partial Least Squares Batch Monitoring


Batch Process Modelling and Monitoring

Batch processes exhibit non-linear, time variant and dynamic behaviour.

These characteristics challenge the linear multivariate statistical technique of multi-way Partial Least Squares (PLS) that has traditionally been applied in batch process performance monitoring.

A local model based approach has been developed to overcome these limitations.


Local Model Approach

Batch processes often exhibit distinct phases of process operation thus instead of modelling a non-linear time variant batch process as a global model, batch trajectories are sub-divided into individual operating regions.

A local linear PLS model is then developed for each operating region

Each model can comprise a different number of latent variables.

A validity function then creates a smooth transition between the local models to build a global non-linear model.


Validity Function

The validity function determines which operating region the process lies within at each time point:

Identification of the most appropriate local model Weighting of local models if two or more are applicable

The validity function is based on a fuzzy logic rule based function:

Rules based on process variable behaviour

IF x1 is LOW AND x2 is HIGH THEN model 1 is valid


Dynamic Feature Addition

Batch process variables also exhibit serial and cross correlation.

Auto Regressive with eXogenous inputs (ARX) structure is a time series structure used to model such data

Including past input and output process variables into the X data matrix of a PLS model encapsulates some of the dynamic features within the model.

j)x(tb....)x(tbk)y(ta....)y(ta(t)y jk 11 11


A fed-batch fermentation process is used to demonstrate local model performance monitoring.

17 batches with good operating conditions and high yield were selected for the nominal model.

30 batches with standard operating conditions but mid to low yield were used to assess the monitoring charts.

A model was developed using local dynamic PLS and global dynamic PLS.

Application to an Industrial Process


Operating Region Specification

Operating regions specified using process knowledge

4 operating regions identified

Regions based on conditions within the fermenter

Operating region 1: initial start up of the fermenter before optimum conditions are reached

Operating region 2: initialisation of product growth Operating region 3: maximum growth rate of product Operating region 4: reactions are complete


Operating Region Specification

Addition rate of chemical A

0 50 100 150 200 250 300 350 400-5

0

5

x1

0 50 100 150 200 250 300 350 400-5

0

5

x6

0 50 100 150 200 250 300 350 400-2

0

2

time

y1

pH

Potency


Validity Function

Fuzzy logic rules used to determine movement between operating regions

Rules applied to Power, Substrate Addition Rate, Respiration Rate

0 50 100 150 200 250 300 350

0

0.2

0.4

0.6

0.8

1

time

validity

model 1 model 2 model 3 model 4


Global Dynamic PLS

0 50 100 150 200 250 300 350 4000

10

20

30

40

50

60

70

80

observation number

potency Predicted and

Actual Values of Potency

0 10 20 30 40 50 60 70-4

-3

-2

-1

0

1

2

3

Residuals of Global Dynamic PLS Model


Prediction using Local Dynamic PLS Model

0 50 100 150 200 250 300 350 4000

10

20

30

40

50

60

70

80

observation number

po

ten

cy

0 10 20 30 40 50 60

-1.5

-1

-0.5

0

0.5

1

resid

uals

potency

model 1model 2model 3model 4

Predicted and Actual Potency for Each Model

Residuals of Local Dynamic PLS models


Performance Monitoring and Fault Detection

0 50 100 150 200 250 300 350 4000

2

4

6

8

10

12

14

16

18

0 50 100 150 200 250 300 350 4000

5

10

15

20

25

Local SPE chart - varying control limit

Global SPE chart - constant control limit


Fault Detection

0 50 100 150 200 250 300 350 4000

50

100

150

loca

l t2

batch 3

0 50 100 150 200 250 300 350 4000

10

20

30

40

glob

al t

2

batch 3

False alarm

Process fault detected Local SPE

chart

Global SPE chart


Conclusions

Inclusion of dynamic behaviour improves model performance through the removal of process structure within the model

Fuzzy model rule based validity function approach allows batch specific movement between model

Local model approach to performance monitoring leads to control charts with improved model limits

Local model monitoring charts detect faults and process deviations earlier than the global model equivalent


Non-linear Partial Least Squares

Prediction Intervals and Leverage



A simple approach to non-linear PLS has been to extend the input matrix (X) by including non-linear combinations of the original variables (such as logarithms, square values, cross-products, etc.) and then performing linear PLS.

If there is no a priori knowledge, then there is no limitation as to the number (and kind) of transformation that might be applied.

Thus by pre-treating data sets in this way, the number of non-linear terms can increase excessively, resulting in large input and output matrices and the results become difficult to interpret.



A more structured approach to the development of a non-linear PLS model is to modify the NIPALS algorithm by introducing a non-linear function that relates the output scores u to the input scores t, without modifying the input and output variables:

Wold et al (1989) proposed a non-linear PLS algorithm which retained the framework of linear PLS but that used second order polynomial (quadratic) regression:

uj = c0j+ c1j tj + c2j tj 2+ ej

eXet wffu ,


Prediction Intervals for Non-linear PLS

As for every regression technique, a measure for assessing the reliability of the predicted values is required.

A common approach is through the use of prediction intervals. These are the upper and lower confidence limits of the predicted values.

The larger the magnitude of these intervals, the less precise is the prediction.

A methodology used to evaluate prediction intervals for neural network models has been extended to linear and non-linear partial least squares algorithms.


Calculation of Prediction Intervals

The prediction intervals are computed using a first order Taylor series expansion and the Jacobian matrix of the functional mapping provided by the PLS algorithms.

Given a set of input and output training data, X and Y, respectively, a PLS regression model is built and the Jacobian matrix F is computed for the same set of training data

When the PLS regression model is used to predict a new output value, corresponding to a new sample of input variables, the vector of partial derivatives is computed and the prediction interval is evaluated

*

1*21,** .1, fFFf TT

pn styyPI


Case Study

The data were generated from the simulation of a pH neutralisation system.

Samples were collected under steady state operating conditions, thus no time correlation existed between any two consecutive samples.

The data included four input variables (flowrates of the inlet and outlet streams of the neutralisation tank) and one output variable (pH value measured in the outlet stream) and were noise free.


pH Neutralisation Process

q4

q2

q1 q3

pH

h


Radial Basis Function PLS

An error based up-dating partial least squares radial basis function PLS model was built using 350 data samples.

It was constructed from one latent variable with twenty one nodes included in the inner radial basis function model.

In excess of 99% of the total variance of the output variable was captured by this representation.



Time Series Plot for the Test Data with Predictions

20 40 60 80 100 120 140 160 180 2001

2

3

4

5

6

7

8

9

10

11

Sample number

pH


Leverage

The quantity

is similar in form to leverage.

It can be used to provide an additional metric for assessing the quality of the regression model.

This is achieved by computing the critical value of the chi-square distribution with degrees of freedom, for predefined confidence levels, e.g. 95% and 99%, and plotting the value of for each sample and the critical value of the distribution divided by (n-p).

*

1* .1 fFFf TT


Leverage

When the ‘leverage’ is smaller than the critical value, the corresponding predicted value is considered to be reliable with the predefined confidence level and vice versa, when the ‘leverage’ is larger than the limit, the predicted value is considered to be unreliable.



20 40 60 80 100 120 140 160 180 2000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Sample number

Ab

so

lute

Va

lue

of

Pre

dic

tio

n E

rro

r

Leverage for the Test Data Prediction Intervals

20 40 60 80 100 120 140 160 180 2001

2

3

4

5

6

7

8

9

10

11

Sample number

pH


Conclusions - PLS Prediction Intervals

A methodology proposed for prediction intervals in neural network modelling was extended to non-linear PLS algorithms.

This approach was known to give approximate, but generally reliable, results whilst being less computationally expensive than other more mathematically precise approaches such as the likelihood, lack-of-fit, jackknife and bootstrap.

The development of the algorithm led to the definition of a metric, the leverage, which can be used in conjunction with, or as an alternative to, prediction intervals.


Conclusions

DATA RICH INFORMATION POOR

DATA

INFORMATION

KNOWLEDGE


Acknowledgements

EBM acknowledges Dr Pino Baffi, Dr Baibing Li, Miss Nicola Fletcher and colleagues in CPACT for the many stimulating discussions.

EBM acknowledges colleagues at BASF Ag. for stimulating the research, in particular Gerhard Krennrich and Pekka Teppola.

EBM acknowledges Pfizer for providing the data.

Documents

The Application of Partial Least Squares to Non-linear Systems in the Process Industries