Feature Extraction for Modeling Patients’ Outcomes: an … · rais, como os valores maximos e m´ ´ınimos medidos ate ao momento da avaliac¸´ ao atual. Finalmente, a˜ abordagem

Feature Extraction for Modeling Patients’ Outcomes:

an Application to Readmissions in ICUs

Rita Domingues Viegas

Thesis to obtain the Master of Science Degree in

Mechanical Engineering

Supervisor: Dr. Susana Margarida da Silva Vieira

Examination Committee

Chairperson: Prof. João Rogério Caldas PintoSupervisor: Dr. Susana Margarida da Silva Vieira

Member of the Committee: Prof. João Miguel da Costa Sousa

June 2015

To my family, the best teachers,

to the friends who are family,

and to the teachers who became friends.

i

ii

Abstract

The aim of this dissertation is to compare the performance of different approaches in the prediction of

patients’ readmissions in intensive care units (ICUs). To do so, fuzzy modeling techniques are applied to

data comprising patient demographics and measurements of physiological variables collected through

the patients’ hospitalization period.

Initially, patients are divided in two groups, readmitted and not readmitted, where the former corre-

sponds to patients readmitted to the ICU within 24 to 72 hours after discharge. Each patient is then

evaluated with a sampling time of 24 hours, so that new samples relative to each one of the hospital-

ization days can be obtained, which is necessary to develop models capable of predicting the risk of

readmission at any stage of the hospitalization.

In a first approach, demographic variables and the current values of each physiological variable

are used directly to develop fuzzy models. Following, a feature extraction approach is implemented

where additional information contained in the time series variables is included, as the maximum and

minimum values measured until the current evaluation moment. Finally, the feature extraction approach

is combined with a multimodel approach. Multimodels are hence implemented where different feature

selection and decision criteria are considered.

Overall, an increase in performance was achieved with the implementation of the proposed ap-

proaches, having the best model been obtained with the feature extraction and multimodel approach.

Keywords: Feature Extraction, Multimodel Approach, ICU, Readmissions, Fuzzy Clustering

iii

iv

Resumo

A presente dissertacao tem como objetivo a comparacao do desempenho de diferentes abordagens

na previsao de readmissoes de doentes em unidades de cuidados intensivos (UCIs). Para tal, sao

aplicadas tecnicas de modelacao fuzzy a dados demograficos e a medicoes de variaveis fisiologicas

recolhidas durante a hospitalizacao dos doentes.

Inicialmente, os doentes sao divididos em readmitidos e nao readmitidos, onde os primeiros

correspondem aos doentes readmitidos na UCI 24 a 72 horas apos a alta. Cada doente e avaliado

com um tempo de amostragem de 24 horas, de forma a obter novas amostras relativas a cada dia

da hospitalizacao, necessarias para o desenvolvimento de modelos capazes de prever o risco de

readmissao em qualquer momento da hospitalizacao.

Numa primeira abordagem, sao utilizadas as variaveis demograficas e os valores atuais de cada

variavel fisiologica diretamente para construir modelos fuzzy. Em seguida, e implementada uma abor-

dagem de extracao de variaveis, onde sao incluıdas informacoes adicionais contidas nas series tempo-

rais, como os valores maximos e mınimos medidos ate ao momento da avaliacao atual. Finalmente, a

abordagem de extracao de variaveis e combinada com uma abordagem de multimodelos. Deste modo

sao implementados multimodelos, onde diferentes criterios de selecao de variaveis e de decisao sao

testados.

De uma forma global, verificou-se um aumento no desempenho com a implementacao das abor-

dagens propostas, tendo a abordagem multimodelo com extracao de variaveis resultado no melhor

modelo.

Palavras-chave: Extracao de Variaveis, Abordagem Multimodelo, UCI, Readmissoes, Fuzzy

Clustering

v

vi

Acknowledgments

At first, I must address a special thank you to my supervisor, Professor Susana Vieira, for her constant

support and guidance through the development of this thesis, her knowledge on this area of research

has truly inspired me.

At second, I must thank Catia Salgado for her never-ending availability to discuss ideas and for

providing me so many thoughtful suggestions and words of encouragement through the work.

To my remaining lab partners, Lucia Cruz, Marta Ferreira, Carlos Azevedo, Hugo Proenca and Hugo

Santos, whose company made this such a pleasant task, I must also address a big thank you for all the

brainstorming and cheering up through the work.

Also to all the remaining friends I’ve met through my journey in IST, specially Joana Correia, Beatriz

Lopez, Ana Angelino, Zita Carreira, Tomas Hipolito, Diogo Ruivo, Joao Caldeira and David Santos, who

accompanied me in countless moments, heard my complains in difficult times and always believed in

me and cheered me up, I must address the biggest thank you. These last years would have been much

more difficult without them.

Also to my family, without whom I wouldn’t have made it to this point, I must address a huge thank

you for all the unconditional love and support, for being such inspirational role models and for all the

priceless values they have taught me.

At last, I would like to take the opportunity to thank all the amazing friends, who even often at a

distance have supported me for endless years and to the great teachers I had the opportunity of meeting

through my academic path for believing in me and having inspired me to be better.

vii

viii

Contents

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

Resumo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii

Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Readmissions in ICUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.4 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Knowledge Discovery 5

2.1 Knowledge Discovery in Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.4 Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.4.1 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4.2 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4.2.1 Sequential Forward Selection . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.5 Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.5.1 Model Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.6 Evaluation / Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 Statistical Analysis of the Readmissions Source Data 13

3.1 Readmissions Source Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.2 Age and Gender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.3 ICU Length of Stay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.4 Number of Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.5 Elapsed time between admission and first measurement . . . . . . . . . . . . . . . . . . . 18

ix

4 Readmissions Data Processing 23

4.1 Data Acquisiton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.3 Output Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.4 Prime Features Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.5 Extracted Features Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.6 Data Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5 Modeling 29

5.1 Fuzzy Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.1.1 Takagi-Sugeno Fuzzy Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.2 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.2.1 Fuzzy C-Means . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

5.2.2 Gustafson-Kessel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5.3 Multimodel Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

5.3.1 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

5.3.2 Decision Criterion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

6 Results 37

6.1 Daily Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

6.1.1 Single Model with Prime Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

6.1.2 Single Model with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . 39

6.1.3 Multimodel with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . 40

6.1.3.1 Areas Under the Sensitivity and Specificity Curves . . . . . . . . . . . . . 41

6.1.3.2 Sensitivity and Specificity at the Intersection Threshold . . . . . . . . . . 42

6.1.3.3 Sensitivity and Specificity Close to the Intersection Threshold . . . . . . . 43

6.1.4 Results Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

6.1.5 Model Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6.2 Discharge Day Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

6.2.1 Single Model with Prime Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

6.2.2 Single Model with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . 51

6.2.3 Multimodel with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . 52

6.2.3.1 Areas Under Sensitivity and Specificity Curves . . . . . . . . . . . . . . . 53

6.2.3.2 Sensitivity and Specificity at the Intersection Threshold . . . . . . . . . . 54

6.2.3.3 Sensitivity and Specificity Close to the Intersection Threshold . . . . . . . 55

6.2.4 Results Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

7 Conclusions 59

7.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

x

Bibliography 67

Appendix A Daily Evaluation A1

A.1 Single Model with Prime Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A1

A.2 Single Model with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A3

A.3 Multimodel with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A5

Appendix B Discharge Day Evaluation B1

B.1 Single Model with Prime Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B1

B.2 Single Model with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B2

B.3 Multimodel with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B3

xi

xii

List of Figures

2.1 Steps of the KDD procedure, adapted from [7] . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2 Schematic representation of the sequential forward selection process. . . . . . . . . . . . 10

2.3 Different ROC curves with the corresponding AUC value. . . . . . . . . . . . . . . . . . . 12

3.1 Age distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.2 Age and gender distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.3 ICU length of stay. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.4 Heart rate measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.5 Temperature measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.6 Platelets measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.7 NBP mean measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.8 SpO2 measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.9 Lactic acid measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.10 Creatinine measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.11 Troponin measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.12 Elapsed time between admission and first heart rate measurement. . . . . . . . . . . . . 19

3.13 Elapsed time between admission and first Temperature measurement. . . . . . . . . . . . 19

3.14 Elapsed time between admission and first platelets measurement. . . . . . . . . . . . . . 19

3.15 Elapsed time between admission and first NBP mean measurement. . . . . . . . . . . . . 20

3.16 Elapsed time between admission and first SpO2 measurement. . . . . . . . . . . . . . . . 20

3.17 Elapsed time between admission and first lactic acid (0.5-2.0) measurement. . . . . . . . 20

3.18 Elapsed time between admission and first creatinine (0-1.3) measurement. . . . . . . . . 21

3.19 Elapsed time between admission and first troponin measurement. . . . . . . . . . . . . . 21

4.1 Scheme of the data acquisition and data preprocessing steps. . . . . . . . . . . . . . . . 25

4.2 Output evolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.3 Scheme of samples distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.4 Features acquired from the time series variables. . . . . . . . . . . . . . . . . . . . . . . . 27

5.1 Configuration of fuzzy systems configuration, adapted from [44] . . . . . . . . . . . . . . . 30

5.2 Multimodel configuration with a posteriori decision, adapted from [25, 26] . . . . . . . . . 34

xiii

6.1 AUC, accuracy, sensitivity and specificity obtained with the different approaches imple-

mented. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

6.2 Real and predicted outcomes of the not readmitted patient with ID number 30062. . . . . 48


6.4 Real and predicted outcomes of the readmitted patient with ID number 31651. . . . . . . 48







6.11 AUC, accuracy, sensitivity and specificity obtained with the different approaches imple-

mented. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

xiv

List of Tables

2.1 Possible combinations of predicted and real outcomes. . . . . . . . . . . . . . . . . . . . . 11

3.1 List of initially considered variables from the MIMIC II database and corresponding total

number of measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4.1 Final variables considered from the MIMIC II database. . . . . . . . . . . . . . . . . . . . 24

4.2 Physiological ranges of the variables considered from the MIMIC II database. . . . . . . . 24

6.1 Best combination of parameters obtained with prime features after 5-fold cross-validation. 38

6.2 Best subsets of features obtained by performing SFS with 5-fold cross-validation with

prime features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

6.3 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-

validation in the MA subset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

6.4 Best combination of parameters obtained with extracted features after 5-fold cross-validation. 40


extracted features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40


validation in the MA dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

6.7 Best subsets of features for the model highlighting sensitivity, obtained by performing

SFS with 5-fold cross-validation with extracted features and the first set of feature selection

criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

6.8 Best subsets of features for the model highlighting specificity, obtained by performing


criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

6.9 Mean and standard deviation of the best results obtained after performing 5 × 5-fold

cross-validation in the MA dataset for the first set of feature selection criteria. . . . . . . . 42


SFS with 5-fold cross-validation with extracted features and the second set of feature

selection criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42




xv


cross-validation in the MA dataset for the second set of feature selection criteria. . . . . . 43


SFS with 5-fold cross-validation with extracted features and the third set of feature selec-

tion criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43



tion criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44


cross-validation in the MA dataset for the third set of feature selection criteria. . . . . . . . 44

6.16 Mean and standard deviation of the AUC values obtained for the best multimodels devel-

oped. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

6.17 P-values between the best multimodels developed. . . . . . . . . . . . . . . . . . . . . . . 46

6.18 Performance comparison of different models. . . . . . . . . . . . . . . . . . . . . . . . . . 47

6.19 Features selected for the sensitivity subset 1 and the specificity subset 3. . . . . . . . . . 48

6.20 Best combination of parameters obtained with prime features after 5-fold cross-validation. 50


prime features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51


validation in the MA subset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

6.23 Best combination of parameters obtained with extracted features after 5-fold cross-validation. 52


extracted features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52


validation in the MA dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52



criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53



criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53


cross-validation in the MA dataset for the first set of feature selection criteria. . . . . . . . 54







xvi


cross-validation in the MA dataset for the second set of feature selection criteria. . . . . . 55



tion criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55



tion criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56


cross-validation in the MA dataset for the third set of feature selection criteria. . . . . . . . 56

A.1 Prime features acquired from the MIMIC II database. . . . . . . . . . . . . . . . . . . . . . A1

A.2 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-

validation in the FS dataset with prime features, varying the number of clusters between

2 and 10 and the degree of fuzziness between 1.1 and 1.5. . . . . . . . . . . . . . . . . . A2



2 and 10 and the degree of fuzziness between 1.6 and 2. . . . . . . . . . . . . . . . . . . A2

A.4 Prime features obtained for each subset, after SFS with 5-fold cross-validation. . . . . . . A2

A.5 Extracted features acquired from the MIMIC II database. . . . . . . . . . . . . . . . . . . . A3


validation in the FS dataset with extracted features, varying the number of clusters be-

tween 2 and 10 and the degree of fuzziness between 1.1 and 1.5. . . . . . . . . . . . . . A4



tween 2 and 10 and the degree of fuzziness between 1.6 and 2. . . . . . . . . . . . . . . A4

A.8 Extracted features obtained for each subset, after SFS with 5-fold cross-validation. . . . . A4

A.9 Extracted features obtained for each subset, by performing SFS with the first set of feature

selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . . . . . A5

A.10 Extracted features obtained for each subset, by performing SFS with the second set of

feature selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . A5

A.11 Extracted features obtained for each subset, by performing SFS with the third set of fea-

ture selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . . . A5


validation in the MA dataset for the first set of feature selection criteria. . . . . . . . . . . . A6


validation in the MA dataset for the second set of feature selection criteria. . . . . . . . . . A7


validation in the MA dataset for the third set of feature selection criteria. . . . . . . . . . . A8

xvii

B.1 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-


tween 2 and 10 and the degree of fuzziness between 1.1 and 1.5. . . . . . . . . . . . . . B1



2 and 10 and the degree of fuzziness between 1.6 and 2. . . . . . . . . . . . . . . . . . . B2

B.3 Prime features obtained for each subset, after SFS with 5-fold cross-validation. . . . . . . B2



tween 2 and 10 and the degree of fuzziness between 1.1 and 1.5. . . . . . . . . . . . . . B2



tween 2 and 10 and the degree of fuzziness between 1.6 and 2. . . . . . . . . . . . . . . B3

B.6 Extracted features obtained for each subset, after SFS with 5-fold cross-validation. . . . . B3

B.7 Extracted features obtained for each subset, by performing SFS with the first set of feature

selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . . . . . B3

B.8 Extracted features obtained for each subset, by performing SFS with the second set of

feature selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . B4

B.9 Extracted features obtained for each subset, by performing SFS with the third set of fea-

ture selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . . . B4


validation in the MA dataset for the first set of feature selection criteria. . . . . . . . . . . . B5


validation in the MA dataset for the second set of feature selection criteria. . . . . . . . . . B6


validation in the MA dataset for the third set of feature selection criteria. . . . . . . . . . . 7

xviii

Notation

Symbols

α Feature selection threshold

βj Degree of activation of the jth rule

δ Final multimodel threshold

µAjn(x) Membership function of Ajn

µij Membership degree of sample j to the ith cluster

Ai Positive definite symmetric matrix

Ajn Fuzzy set of the jth rule and the nth feature

aj Parameter vector of rule Rj

bj Scalar offset of rule Rj

cs Sensitivity feature selection performance criterion

csp Specificity feature selection performance criterion

dij Distance from sample j to the ith cluster prototype

h Height

J Number of rules

Jm Objective Function

K Number of cross-validation folds

m Degree of fuzziness

N Total number of features

Ns Number of samples

nc Number of clusters

Pi Covariance matrix of cluster i

q Total number of features

Rj Fuzzy rule

sα Sensitivity value at threshold α

spα Specificity value at threshold α

t Threshold

tadm Number of days since admission in the ICU

tmax Time elapsed between admission and maximum value recorded until the evaluation point

tmin Time elapsed between admission and minimum value recorded until the evaluation point

U Partition matrix

V Prototypes matrix

vf Value of the first recorded measurement

vi Prototype of the ith cluster

vl Value of the last recorded measurement

xix

vm Mean of all values recorded until the evaluation point

vmax Maximum value recorded until the evaluation point

vmin Minimum value recorded until the evaluation point

w Weight

wβ Minor weighting factor of the model selection criterion

wφ Major weighting factor of the model selection criterion

wi Weighting factor of model i

X Dataset

x Sample input vector

xfmaxMaximum value for the f th feature

xfmin Minimum value for the f th feature

xif Prime value of sample i for feature f

xifnormNormalized value of sample i for feature f

y Sample output

yδ Multimodel final output

y1 Sensitivity model output

y2 Specificity model output

yc Continuous output

yj Consequent function of the jth rule

xx

Acronyms

ACC Accuracy

APACHE II Acute Physiology and Chronic Health Evaluation

AUC Area Under the Curve

BMI Body Mass Index

FCM Fuzzy C-Means

FN False Negative

FP False Positive

FPR False Positive Rate

FS Feature Selection

GK Gustafson-Kessel

ICU Intensive Care Unit

KDD Knowledge Discovery in Databases

MA Model Assessment

MIMIC II Multi-Parameter Intelligent Monitoring for Intensive Care

NBP Non-Invasive Arterial Blood Pressure

PCA Principal Component Analysis

ROC Receiver-Operating Characteristic

SFS Sequential Forward Selection

SpO2 Peripheral Capillary Oxygen Saturation

TN True Negative

TNR True Negative Rate

TP True Positive

TPR True Positive Rate

TS Takagi-Sugeno

xxi

xxii

Chapter 1

Introduction

1.1 Motivation

The amount of data available is growing at an exponential rate. As of 2012, around 2.5 exabytes of

data were created each day, and that number is doubling nearly every 40 months [1].

Such huge amounts of information available lead to equally huge amounts of knowledge opportu-

nities [2], which results in an increasing interest towards the potentialities of data in a variety of fields.

From financial applications [3] to healthcare [4], storm detection [5] to marketing and sales [6], every-

where data is being studied and analyzed in an attempt to extract and infer as much novel and useful

information as possible about the complex systems that surround us.

However, it is known how quantity does not necessarily stand for quality. Knowing how to make

sense of the data available, how to separate the relevant from the non-relevant information or how to

spot erroneous information that can otherwise lead to misleading results, represents a huge and time

consuming challenge.

From the beginning of time humans have dealt with this challenge, trying to understand the world

around them and to extract knowledge from the information available, first in an empirical way and

later resorting to manual analysis. However, when the quantity of data available is as massive and

rapidly growing as nowadays, such outdated tools are not enough, resulting in an urgent need for a new

generation of computational theories and tools [7].

Towards this need, began in recent decades the development of a new area of knowledge and

research, data mining. This area attempts to make the most of the computational power at our disposal,

being defined as the process of discovering patterns in data in an automatic or semi-automatic way [2].

One field where data mining can be extremely useful is the healthcare industry, an extremely data-

rich field where such technological tools are not yet as popularized as in other industries [8]. Neverthe-

less, the achievements that can be obtained through data mining techniques in this area are endless

and more and more that awareness is growing.

Resorting to it, and for example to the data mining technique of clustering where objects are classified

and organized in groups according to their similarities [9], specific patterns can be spotted in different

1

types of patients. Discovering such patterns, as well as the characteristics that make each patient belong

to a specific group, can then be used to a variety of purposes, as diagnosing different cancer types [10]

or identifying high-risk diabetes patients [4].

A topic in healthcare that can much benefit from data mining techniques is preventing patients from

being readmitted into intensive care units (ICUs), a highly relevant issue, as such patients have an

increased risk of death, longer length of stay and represent higher costs [11].

ICUs are subject to increasing pressures regarding the management of care and resources which

leads to the implementation of strategies seeking to rapidly free expensive beds. Such scenarios place

clinicians in difficult situations, as they must decide on the discharge of apparently stable patients to

make room for more urgent ones, risking the discharged patients to be readmitted in the short term [12].

The present study aims to help in this decision-making, by presenting a feature extraction and mul-

timodel approach combined with fuzzy modelling to predict patients’ readmissions into ICUs, based on

patient demographics and a set of physiological variables measured along the hospitalization period.

1.2 Readmissions in ICUs

A clinician’s decision on the discharge of a patient is always a delicate task. But when such deci-

sion takes place in ICUs where patients’ health conditions are extremely fragile and the case of early

discharge and consequent readmission can result in severe consequences, an even greater amount of

responsibility is at stake.

ICU readmissions are a commonly used quality measure but, despite it and the decades of research,

such events continue to occur at rates relatively unchanged over the years [13]. These readmissions

are of particular concern as readmitted patients have much worse prognoses than not readmitted ones.

Previous studies have shown how ICU readmissions can cause significant distress to patients and result

in considerable financial costs [14] or how in-hospital death rates can be two to ten times greater in

patients readmitted to ICUs than in not readmitted ones [15]. However, the amount of data collected

in ICUs has grown considerably in the last decades making it a particularly well-suited setting to imple-

ment data-driven systems to discover underlying patterns and relationships in intensive care data [16].

Considering this, several studies have been conducted in past years making use of the data collected in

ICUs to develop models to predict patients’ readmissions. Traditionally, such models have been based

on the clinical gold-standard method, the Acute Physiology and Chronic Health Evaluation II (APACHE

II) score [17], a severity of disease classification system [18]. Other studies have achieved slightly better

results applying logistic regressions to variables considered predictive of readmissions, as admission

diagnosis, treatment status, age, length of ICU stay or ICU mortality [19, 20].

More recently, different approaches were followed resorting to more complex methods. In [11], a

fuzzy model combined with tree search feature selection was developed based on physiological vari-

ables recorded in the last 24 hours before discharge, showing how it is possible to predict patients’

readmissions using a small number of variables. In [21], the application of fuzzy probabilistic systems

was proposed, having been achieved results comparable to the gold standard method, yet using a sig-

2

nificant lower amount of variables.

However, when it is intended to develop models based on records from physiological variables as-

sessed over time, some considerations must be taken. Such information, in the form of time series,

cannot be used straightforward as if they were constant features, like age or length of ICU stay.

To deal with it, several approaches can be followed. The arithmetic mean of each time series can

be computed, as in [11], or the procedure presented in [22, 23] can be followed where, additionally to

the arithmetic mean, other features as minimum, maximum, trends or first and last data points were

extracted from the time series variables to predict mortality in ICUs.

Finally, one additional approach has shown promising results in ICU applications, the multimodel ap-

proach. In [24], significantly better results were obtained using a multimodel scheme than a general one

to predict the necessity of vasopressors in sepsis patients and in [25] the same physiological variables

of [11] were used, but better results were obtained in the prediction of patients’ readmissions due to the

implementation of multimodels. Also in [26], the multimodel approach was addressed and considerably

well succeeded this time to predict the survival of ICU septic shock patients. In that work, a combined

strategy was implemented where initially separate models were developed, one aiming to maximize

sensitivity and the other to maximize specificity, being afterwards combined into a final output.

1.3 Contributions

The aim of this thesis is to combine two promising approaches to predict patients’ readmissions in

ICUs, feature extraction and multimodeling, and comparing them with the alternative approaches, the

use of prime features and single models.

A set of initial time series variables is considered based on a previous work [11] where such variables

were found to be the most predictive of ICU readmissions.

A feature extraction approach is then applied to the time series variables, aiming to retain more of

the information contained in the time series, in an alternative to the procedure presented in [11] where

solely the arithmetic mean of the values collected in the last 24 hours before discharge were used.

A multimodel is then developed based on the work of [25], but adding the concept of initially devel-

oping separate models, one highlighting the specificity and the other the sensitivity, as is suggested in

[26].

Furthermore, this thesis introduces one additional contribution to what was done in previous works

seeking to predict patients’ readmissions into ICUs. Instead of solely considering the last day of a

patients’ hospitalization, each patient is evaluated at different stages of his/her stay. In this way, more

information about the patients’ hospitalization is considered and a more suitable model for real world

applications is developed. Such suitability derives from the fact that in real world a clinician cannot know

a priori which is the last day of a patient’s hospitalization to decide on his discharge, making this a more

realistic approach.

3

1.4 Outline

Chapter 2 presents an overview of the main procedure followed as guideline for the generality of the

work, the Knowledge Discovery in Databases (KDD) procedure. The main steps that compose it are

described and a summary of methods that can be used in each step is presented.

In Chapter 3, the source readmissions data is presented and a statistical analysis is performed.

Patients are divided into two categories, readmitted and not readmitted, and the distributions of different

features per category are presented.

Chapter 4 presents the transformations applied to the initial data analyzed in Chapter 3 in order to

obtain the final datasets where to perform modeling. The topics of data acquisition, data preprocessing,

output definition and considered features are briefly explained and their application to the dataset is

presented.

In Chapter 5, the theoretical aspects of the modeling developed in the work are presented. Concepts

as fuzzy modeling, clustering algorithms and multimodel approach are introduced and their relevance to

the present work is presented.

In Chapter 6, the main results are summarized and compared. Results are presented for different

approaches, initially using a single model and prime features, afterwards a single model with extracted

features and, at last, a multimodel with extracted features.

Finally, in Chapter 7, the results of the different approaches are summarized and conclusions are

drawn. The benefits and limitations of each approach are discussed and directions for future research

are presented.

4

Chapter 2

Knowledge Discovery

This Chapter presents an overview of the main procedure followed throughout this thesis, the KDD

procedure. Its main constituent steps are described: selection, preprocessing, transformation, data

mining and evaluation.

2.1 Knowledge Discovery in Databases

The amount of data available is increasing at an enormous speed. In many fields huge amounts of

data are being collected, becoming essential to be able to deal with it and, specially, to know how to take

advantage of the knowledge opportunities that rise with it.

Considering the huge dimensions of this now available data, it becomes obvious how old techniques

based on human analysis have become obsolete and how the development of new procedures, re-

sorting to computational techniques and tools, has become mandatory, being a popular subject among

researchers. Towards this need, in 1996 the KDD procedure was firstly introduced [7], a procedure

commonly used in medical applications [27].

In [7], the KDD procedure is explained in detail and the set of steps that should to be taken when it is

intended to extract knowledge from raw data are presented. The KDD procedure has been defined as

a non-trivial process of identifying valid, novel, potentially useful and ultimately understandable patterns

from large collections of data.

According to [7], to extract knowledge from data the following main steps illustrated in Figure 2.1

should be taken:

- Selection;

- Preprocessing;

- Transformation;

- Data Mining;

- Evaluation / Interpretation.

5

Figure 2.1: Steps of the KDD procedure, adapted from [7] .

Nevertheless, it should be considered that even with an established procedure as this one, following

each of the steps is not always straight forward and the repetition of some of the steps in an iterative

and recursive way can be required, until satisfactory results are achieved.

In this work, the KDD procedure is followed as a guideline, being the description of each of its steps

introduced below.

2.2 Selection

Selection, also referred to as data acquisition [7], is the process of choosing the target data and

selecting the set of data samples and variables for the discovery process to focus. To be successful in

it, one should first understand the domain of application of the problem and clearly define the goal of

the KDD process, so that the more accurate and relevant target data can be defined. Once this has

been clarified, it involves the entire process of acquiring and storing the target data extracted from the

selected database, taking into account the inclusion and exclusion criteria defined for the data samples

and variables to be obtained.

In the present work, the selection step plays an important role, as one of the hypothesis of the

thesis regards selection of data and comparison with other approaches, where different data has been

selected.

2.3 Preprocessing

Data preprocessing is an essential step when dealing with virtually all real-world databases. Due

to their usual enormous dimensions and their likely origin from multiple and varied sources, these

databases are often incomplete, noisy, lacking attribute values, containing errors or outlier values and

inconsistent [2].

In the medical world, as in the majority of fields, there are countless potential sources of contami-

nation of the recorded data. It can be due to human errors, as misunderstandings or distractions, or to

machine ones, such as equipment malfunctions or lack of calibration, both contributing to the creation

6

of inaccurate data. Therefore, data preprocessing techniques, as handling missing data, noise and out-

liers, are essential to enable the improvement of the quality and reliability of the data, as well as the

reduction of the chances of biasing the knowledge extracted from it.

According to [2], to deal with missing values the following options can be chosen:

1. Ignore the tuple;

2. Input the missing values manually;

3. Use a global constant to input the missing values;

4. Use the attribute mean to input the missing values;

5. Use the attribute mean for all samples belonging to the same class as the given tuple;

6. Use the most probable value to input the missing values.

However, it should be noted that methods 3 to 6 can bias the data, as the imputed values may not be

the correct ones.

When faced with noise, a random error or variance in a measured variable, several methods, as

binning and regressions, can be used to remove it and thus smooth the data [2].

Binning methods work by sorting the data into a number of bins and replacing their values according

to the values inside each bin, using the mean, the median or the closest boundary value [2].

Alternatively, regressions can be used to smooth the data, as linear regressions where the best line

to fit two attributes is found so that one can be used to predict the other. In more complex cases the

same principle can be used, resorting to multi linear regressions instead where more than two attributes

can be involved [2].

Finally, to deal with data discrepancies or outliers, one should make use of the knowledge available

about the properties of the data, referred to as metadata [2]. This knowledge can include knowing the

domain, the data types or the acceptable values of each attribute, which can then be used to detect

values that do not respect it, outliers, and thus remove them. Different procedures can also be used, as

identifying as possible outliers values that are more than two standard deviations away from the mean

of a given attribute.

2.4 Transformation

Transformation and dimensionality reduction methods are extremely important, particularly when

dealing with high dimensional data sets, as these can increase substantially the size of the search

space and the chances of a data-mining algorithm finding erroneous patterns not valid in general [7].

Hence, the purpose of transformation and dimensionality reduction methods is obtaining a reduced yet

relevant set of features to represent the data, which can bring huge benefits both to the computational

time as to the performance of the resulting models.

7

There are two main categories in which these methods can be divided, feature extraction and feature

selection [28]. Feature extraction methods have the purpose of obtaining new features out of the original

variables while feature selection methods seek to select the most useful subset of features from an initial

set, while discarding the non-relevant ones.

2.4.1 Feature Extraction

Feature extraction techniques address the problem of finding the transformation of features contain-

ing the greatest amount of useful information, while attempting to achieve a reduced size set of features

[29]. Many techniques can be used to obtain such result, from complex linear and nonlinear statistical

procedures to simple mathematical operations.

One popular method is the principal component analysis (PCA) [30], a linear transformation that

enables dimensionality reduction, where a linear mapping of the data into a lower-dimensional space is

performed in a way that the variance of such data is maximized. This transformation results from the

eigenvectors corresponding to the largest eigenvalues of the covariance matrix of the data, which in turn

can be used to reconstruct a large fraction of the variance of the original data [28].

However, when dealing with time series variables, as is the case of the records of variables for each

patient along their stay, other methods can be more suitable to retain as much information as possible,

while diminishing substantially the dimensions of the dataset used. For example, for each time series,

the mean, the minimum, the maximum or the trend can be computed [22], as well as other properties

that can help to accurately represent the information contained in the initial time series.

2.4.2 Feature Selection

Feature selection can bring a variety of benefits, as diminishing the processing and storing require-

ments or facilitating the data visualization and understanding, by reducing the dimensionality of the

dataset [31]. However, it operates based on a sensitive compromise, as it is attempted to minimize the

number of selected features while retaining, as much as possible, the overall prediction information.

There are many methods used to perform feature selection but, in general, the main idea is finding a

subset of features that maximizes a specified criterion, while discarding the ones that present the worst

performances. In health care applications, as in other fields, this process brings also the advantage

of pointing out sets of features that might not have been initially chosen as the most relevant to the

prediction of an outcome by the human judgement.

According to [29], there are four main categories of feature selection methods: filters, wrappers,

hybrids and embedded.

Filter and wrapper methods both make use of search strategies to explore the space of all possible

combinations of features in order to select the best subsets. However, they differ substantially in the

evaluation criterion used. Filter methods use criteria that do not involve machine learning algorithms to

evaluate each feature or combination of features, as measurements of entropy, variance or relevance

indexes based on correlation coefficients among features. On the contrary, wrapper methods use the

8

performance of machine learning algorithms as criterion searching for the most predictive subset of

features, according to the chosen criterion.

In an attempt to take the most of the two previous methods, there are hybrid methods where first a

filter method is used to obtain a reduced set of features so that a wrapper method can then be used to

select the most relevant ones, according to the performance of a selected machine learning algorithm.

Finally, there are embedded methods that differ from the previous ones, as they incorporate simulta-

neously the generation of subsets of features and their evaluation in the model training process.

In terms of finding the candidate feature subsets to be evaluated, also different approaches can be

followed. Greedy search strategies are computational advantageous, since only a reduced area of the

search space is explored. They can be used in two ways, forward selection or backwards elimination.

In forward selection, variables are progressively incorporated into larger subsets, whereas in backward

elimination one starts with the set of all variables and progressively eliminates the least promising ones

[31]. Alternatively, more complex and computationally heavy search strategies can be used, as meta-

heuristics where the computational time maybe longer, but a greater amount of possibilities is considered

[32].

In this thesis, due to the complexity and large size of the dataset, a greedy search strategy is imple-

mented in a wrapper way, following the Sequential Forward Selection (SFS) approach.

2.4.2.1 Sequential Forward Selection

In a wrapper context, SFS works by building and evaluating models with increasingly complex sub-

sets of features, following the increase in performance of the developed models [33].

Initially, a performance criterion is chosen based on the objectives of the problem, which will allow

the comparison between the results obtained for each candidate feature subset. Once the criterion has

been selected, a model is built for each of the available features and its performance is evaluated. The

feature that returns the best performance score is kept and, in a second round, combined with each of

the remaining features. Once again, a model is built and evaluated for each of the new combinations

of features and only the best subset is kept. This process is repeated successively, always increasing

the number of selected features, as illustrated in Figure 2.2, until there is no increase in performance, in

which case the process is stopped, and the best subset of features found so far is kept.

This is a very popular method, having as advantages its simplicity and the possibility of an easy

graphical representation from which can easily be understood the performance of each candidate fea-

ture subset. However, it has the disadvantage of being susceptible to finding subsets of features corre-

sponding to local optima.

In the present work, this approach was chosen considering its acceptance and popularity in the

medical community, as well as the computational simplicity, extremely advantageous when dealing with

high dimensional datasets.

9

Figure 2.2: Schematic representation of the sequential forward selection process.

2.5 Data Mining

Data mining is a fundamental step of the KDD process, as it is the phase in which a primordial

knowledge starts being extracted from the data. It often involves repeating iteratively the use of one or

more data-mining methods in order to achieve the desired goals [7], which can be an extensive and time

consuming procedure.

Data mining can be used particularly to achieve two kinds of knowledge discovery goals: verification

and discovery. In verification it is intended to verify a user’s a priori hypothesis, being mostly used

common methods of traditional statistics. In discovery, the aim is to find new patterns from the data that

can be used to predict the future behaviour of a given system or to generate a description of the data, in

a more human-understandable form [7].

For description purposes, data mining involves fitting models to, or determining patterns from ob-

served data, so that knowledge can be inferred, like assigning data samples to classes as is the purpose

of the present work. Many methods can be used to obtain this result, but most of them are based on

tried and tested techniques from machine learning, pattern recognition and statistics, as classification,

clustering or regression [7].

The use of these methods can be associated to two major types of problems: unsupervised learning

and supervised learning [34].

In unsupervised learning the problem consists in discovering the underlying structure of the dataset,

if there is any. This means that the aim is finding if the data can be gathered in different groups according

to their similarities, and what these characteristics are that make objects similar within each group but

different among others. This is a complex problem, as there are no objective criteria equally accepted

to measure the results being the indication of the goodness of the results often subject to the user’s

estimate [34].

In supervised learning, on the contrary, each data sample is preassigned to a class label. The goal

is then to train a classifier to understand and model the relationships between the attributes of each

sample and the corresponding label so that if new data is introduced the model can, as accurately as

possible, assign the new samples to the correct label. This is the case of the present study where it is

10

intended to classify patients into the readmitted or not readmitted classes, by first training a model and

then testing it. It represents also a complex problem, but when compared to unsupervised learning has

the advantage of having several popularized objective criteria that can be used to evaluate the results,

as the ones presented in the following section.

2.5.1 Model Assessment

In supervised learning, several performance measures can be used to access the performance of a

given model. One of the most popular in classification problems is the area under the receiver-operating

characteristic (ROC) curve, the AUC [35], very common in the performance evaluation of classifiers in

medical problems [36].

To understand it, it is necessary to first be aware of how in the case of binary classifiers (as the

one being implemented in the present work) each sample can belong to one of the classes, positive

or negative and, similarly, the output of the classifier can belong to one of the two classes. Should be

noted though, that to obtain binary values from a classifier, generally with the values of 0 or 1, it is first

necessary to compare the initially continuous output with a value chosen a priori, the threshold, used as

a delimiter of the two classes.

Taking into account the combination of both real and predicted classes, four possible combinations

arise, presented in Table 2.1.

Table 2.1: Possible combinations of predicted and real outcomes.

Predicted ClassPositive Negative

Real ClassPositive True Positive (TP) False Negative (FN)

Negative False Positive (FP) True Negative (TN)

In this way, TP corresponds to samples correctly classified in the positive class, FP to samples incor-

rectly classified in the positive class, FN to samples incorrectly classified as belonging to the negative

class and TN to samples correctly classified in the negative class.

With this notation other important performance measurements can then be defined, as:

- Sensitivity, or true positive rate (TPR), that measures the quantity of positive samples correctly

classified as such:

Sensitivity = TPR =TP

TP + FN. (2.1)

- Specificity, or true negative rate (TNR), that measures the quantity of negative samples correctly

classified as such:

Specificity = TNR =TN

TN + FP. (2.2)

11

- Accuracy (ACC), that measures the proportion of correctly classifications over the total number of

observations:

Accuracy =TN + TP

TN + TP + FP + FN. (2.3)

Finally, the ROC curve can be introduced, as it is obtained by plotting the TPR, sensitivity, against

the false positive ratio (FPR), one minus the specificity, for varying thresholds between 0 and 1. The

AUC is then obtained by integrating this curve over all thresholds, such that AUC values between 0 to

1 can be obtained, where 0.5 corresponds to a random classifier and 1 to a perfect one. Values bellow

0.5 mean the classifier is performing better for the inverse problem, requiring an exchange of classes.

Figure 2.3 presents examples of different ROC curves and their corresponding AUC value.

AUC = 1

0,5 < AUC < 1

AUC = 0,5

FPR

TPR

0 0.2 0.4 0.6 0.8 1.0

1.0

0.8

0.6

0.4

0.2

0

Figure 2.3: Different ROC curves with the corresponding AUC value.

2.6 Evaluation / Interpretation

The last step of the KDD process is evaluation and interpretation. Even being the last, it is also a

crucial step, as it requires interpreting and evaluating critically the knowledge obtained, considering its

validity, impact, usefulness and novelty. At this point, it is also required to consider the present results

and to choose whether or not to return to any of the previous steps in order to achieve better or more

valid results.

In the present thesis, all of the mentioned steps of the KDD procedure are covered, since all are

necessary and vital to successfully extract knowledge from raw data. However, a special emphasis is

given to the steps of data acquisition, data transformation and data mining. In terms of data acquisition,

a novelty is introduced as the target data and prime variables selected differ from previous approaches

for the application problem. In data transformation, feature extraction is applied, which combined with a

multimodel approach implemented in the data mining step, makes the present a novel approach to the

problem of predicting patients’ readmissions in ICUs.

12

Chapter 3

Statistical Analysis of the

Readmissions Source Data

In this Chapter, the database used and the initial dataset considered are presented. In a first ap-

proach, a statistical analysis of this dataset is performed with the aim of better understanding the influ-

ence of different variables in the two categories of patients, readmitted and not readmitted to any ICU

within 24 to 72 hours after discharge.

3.1 Readmissions Source Data

To conduct the present work, an initial dataset was obtained from the Multi-Parameter Intelligent

Monitoring for Intensive Care (MIMIC II) database [37], a public research archive of data collected from

over 32,000 ICU patients. This large database has been de-identified by removal of all Protected Health

Information and contains data collected at the Beth Israel Deaconess Medical Center in Boston from

patients admitted to a variety of ICUs from 2001 to 2008. It comprises clinical information, as patients’

demographics, laboratory test results, vital signs recordings, fluid and medication records, charted pa-

rameters and free-text reports as nursing notes, imaging reports and discharge summaries.

In a first approach, the adult patients, that is, older than 15 year, that survived within one year of

discharge and that were admitted to the ICU for a minimum of 24 hours were selected from the original

database. Patients who returned to any ICU of the same medical center between 24 and 72 hours were

classified as readmitted and the ones that did not return to any ICU classified as not readmitted, resulting

in a set of 12,080 not readmitted patients and 775 readmitted ones, corresponding to 6.4 % of the total

of not readmitted patients. Being readmitted between 24 and 72 hours after discharged is considered an

early readmission [38]. The choice of 24 hours as the lower bound relies on the fact that in the MIMIC

II database patients readmitted in less than 24 hours are considered as belonging to the same ICU stay

and the upper bound, 72 hours, is based on medical advice obtained in previous works [11].

Data regarding patient demographics, gender and age, and measurements from the variables pre-

sented in Table 3.1 were then collected from this initial set of patients. The choice of these variables,

13

except creatinine and troponin, is based on [11], where their high predictive power was shown. The

choice of creatinine and troponin as variables of interest resulted from medical suggestions, as they are

important indicators of ICU patients’ health.

Table 3.1: List of initially considered variables from the MIMIC II database and corresponding totalnumber of measurements.

Variable ID Variable Name Total Number of Measurements

211 Heart Rate 1,407,781

677 Temperature 305,230

828 Platelets 75,213

456 Non-Invasive Arterial Blood Pressure (NBP) Mean 616428

646 Peripheral Capillary Oxygen Saturation (SpO2) 1,374,595

818 Lactic Acid (0.5-2.0) 25,799

791 Creatinine (0-1.3) 76,393

851 Troponin 815

Once obtained the initial dataset, a statistical analysis was performed with the aim of better charac-

terizing each set of patients, readmitted and not readmitted, as presented in the following sections.

3.2 Age and Gender

In the present section, the distributions of each set of patients according to their age and gender are

presented. Figure 3.1 presents the distributions of each set of patients, readmitted and not readmitted,

according to their age.

(a) Not readmitted patients (b) Readmitted patients

Figure 3.1: Age distribution.

In terms of age distribution, there is a slight increase in the number of readmitted patients aged

between 70 and 90 years old and an even slighter increase in the number of not readmitted patients

between 50 and 70 years old.

The distributions of each set of patients, now considering both their age and gender, is again pre-

sented in Figure 3.2.

14


Figure 3.2: Age and gender distributions.

Regarding the gender distribution, there is mostly a predominance in the number of male patients but

can not be spotted any obvious differences between the readmitted and not readmitted patients cases,

apart from an increase in the number of readmitted female patients aged between 70 and 85 years old.

3.3 ICU Length of Stay

The total amount of time a patient is hospitalized in the ICU can be an indicator of the severity of his

medical condition. Considering this, Figure 3.3 shows the ICU lengths of stay distributions, in hours, for

each set of patients.


Figure 3.3: ICU length of stay.

For both cases, a higher incidence of patients hospitalized between 24 (the minimum imposed) and

60 hours is observed, lacking the existence of a clear distinction between readmitted and not read-

mitted patients. Additionally, a cyclic behaviour of peaks can be spotted in both cases, with about 24

hours of interval, that most likely corresponds to the daily procedure of rounds where more patients are

discharged.

3.4 Number of Measurements

The number of times a given variable is measured can reflect a patient’s health situation. In this

section, Figures 3.4 to 3.11 present the distributions of number of measurements per patient for each of

15

the variables listed in Table 3.1, for readmitted and not readmitted patients.


Figure 3.4: Heart rate measurements.

In terms of heart rate measurements, in Figure 3.4 it is observable a comparable distribution for

readmitted and not readmitted patients. However, it can be noted that there are more not readmitted

than readmitted patients having a number of measurements inferior to one hundred and that the opposite

happens for superior numbers of measurements, although in a less obvious way.


Figure 3.5: Temperature measurements.

Regarding temperature measurements, Figure 3.5 shows that the number of patients having null or a

low amount of measurements is considerably higher for not readmitted patients than it is for readmitted

ones. Additionally, it is depicted that most of the patients with high numbers of measurements belong to

the readmitted class.


Figure 3.6: Platelets measurements.

In terms of platelets measurements, a clear reduction in the total amount of measurements is pre-

16

sented in Figure 3.6. Furthermore, it can be noted again, a slightly higher incidence of patients having

null or a low amount of measurements in not readmitted patients.


Figure 3.7: NBP mean measurements.

Figure 3.7 shows a substantially higher amount of not readmitted patients having less than fifty NBP

mean measurements and a specially higher number of not readmitted patients where no measurement

was made.


Figure 3.8: SpO2 measurements.

Regarding the SpO2 numbers of measurements, Figure 3.8 shows a rather similar distribution for

both sets of patients. However, it can be noted a slightly higher amount of not readmitted patients

having a reduced number of SpO2 measurements.


Figure 3.9: Lactic acid measurements.

In terms of lactic acid measurements, is shown in Figure 3.9 how the number of measurements for

this variable is substantially low when compared to the previous cases. However, although small, it can

17

be noted a higher incidence of not readmitted patients with null measurements.


Figure 3.10: Creatinine measurements.

Regarding the creatinine measurements, Figure 3.10 shows, once again, a slightly higher incidence

of not readmitted patients having a reduced number of measurements.


Figure 3.11: Troponin measurements.

Finally, for the case of troponin, presented in Figure 3.11, it can be noted how most patients do not

present any measurement, independently of their group. In fact, the lack of measurements from this

variable is so high that the variable needed to be excluded for the construction of the final dataset.

Overall, it can be concluded that most of the selected variables present differences, even if small, in

their behaviour for readmitted and not readmitted patients.

3.5 Elapsed time between admission and first measurement

The elapsed time between a patient’s admission and the first time a variable is measured can be a

strong indicator of the condition in which a patient is admitted to the ICU. In this section, Figures 3.12

to 3.19 present the distributions of the elapsed times between admission and the first measurement of

each of the variables listed in Table 3.1 for each group of patients.

18


Figure 3.12: Elapsed time between admissionand first heart rate measurement.

In terms of heart rate, it can be seen in Figure 3.12 how there is a considerably higher amount of

readmitted patients having the first measurement made less than an hour after being admitted to the

ICU.


Figure 3.13: Elapsed time between admissionand first Temperature measurement.

Regarding the time of the first measurement of temperature, it is depicted in Figure 3.13 how both

sets of patients seem to have similar distributions. Nevertheless, it can be seen that a slightly bigger

amount of readmitted patients have their temperature measured in the first hour of their admission.


Figure 3.14: Elapsed time between admissionand first platelets measurement.

Figure 3.14 shows how the distributions of time until the first measurement of platelets is rather

similar for both sets of patients. Slight differences do exist, but the percentage of patients considered is

too low for these differences to be significant.

19


Figure 3.15: Elapsed time between admissionand first NBP mean measurement.

In terms of time elapsed until the first measurement of NBP mean, Figure 3.15 shows how there is

a higher amount of readmitted patients with this measurement taking place in the initial hours of their

admission.


Figure 3.16: Elapsed time between admissionand first SpO2 measurement.

Also in terms of the time elapsed until the first measurement of SpO2, it is depicted in Figure 3.16 a

higher incidence of readmitted patients with the first measurement taking place in the first hour of their

admission.


Figure 3.17: Elapsed time between admissionand first lactic acid (0.5-2.0) measurement.

Regarding the elapsed time until the first lactic acid (0.5-2.0) measurement, in Figure 3.17 is shown

how the distribution is rather similar for both readmitted and not readmitted patients.

20

A-PDF Page Crop DEMO: Purchase from www.A-PDF.com to remove the watermark


Figure 3.18: Elapsed time between admissionand first creatinine (0-1.3) measurement.

For the case of creatinine (0-1.3), presented in Figure 3.18, as in the previous case there is no obvi-

ous distinction between the time elapsed until the first measurement for readmitted and not readmitted

patients.


Figure 3.19: Elapsed time between admissionand first troponin measurement.

At last, for the case of troponin case, the number of measurements is so low that it is hard to obtain

a good characterization of its distribution.

Although some differences have been spotted in the elapsed times until the first measurements of the

several variables considered, it can be concluded that the use of the analysis conducted in this Chapter

alone is insufficient to fully characterize and distinguish the two categories of patients. Therefore, it

can be understood how the problem of successfully predicting patients’ readmissions is complex and

extensive, requiring more complex approaches.

21

22

Chapter 4

Readmissions Data Processing

This Chapter presents an overview of the steps taken to obtain the final datasets used to implement

the data mining phase of the KDD procedure.

Initially, the steps of data acquisition and data preprocessing are covered, followed by a description

of the assumptions made and the procedure followed to obtain the totality of samples and respective

outputs. The two approaches considered to derive the final datasets are described, the use of prime

and extracted features, and the data normalization step is explained.

4.1 Data Acquisiton

As discussed in Section 3.1, a dataset containing information regarding the measurements of the

variables presented in Table 3.1 was initially extracted from the MIMIC II database. This dataset con-

cerned patients that survived within one year of discharge and that were ICU inpatients for a minimum of

24 hours. Patients who returned to any ICU of the same medical center between 24 and 72 hours were

classified as readmitted patients and the ones that did not return to any ICU, classified as not readmitted

patients, resulting in a set composed by 12,080 not readmitted patients and 775 readmitted patients.

Having been shown in the previous analysis the lack of available measurements of the variable

Troponin, data concerning it has not been included in the following steps. Hence, the set of initial patients

was restricted to the ones containing a minimum of two measurements of the variables presented in

Table 4.1, resulting in a set comprising 2,645 not readmitted patients and 241 readmitted patients.

23

Table 4.1: Final variables considered from the MIMIC II database.

Variable ID Variable Name

211 Heart Rate

677 Temperature

828 Platelets

456 NBP Mean

646 SpO2

818 Lactic Acid (0.5-2.0)

791 Creatinine (0-1.3)

From this set of patients were then extracted the patients’ gender, age, height, weight, time of admis-

sion in the ICU and time and value of each measurement of the variables listed in Table 4.1, as well as

an indication of the patient’s class, 0 for not readmitted patients and 1 for readmitted ones.

4.2 Data Preprocessing

In large databases, as the present one, outliers can be rather common, being necessary to handle

them so that the remaining data points do not become corrupted [2]. With this purpose, a method in

which values outside an interval of valid physiological ranges are eliminated was applied to the present

dataset. The maximum and minimum values considered for the physiological ranges of each variable

can be found in Table 4.2 [39].

Table 4.2: Physiological ranges of the variables considered from the MIMIC II database.

Variable ID Variable Name Units Minimum Maximum

211 Heart Rate beats/min 0 250

677 Temperature oC 25 42

828 Platelets cells ×103/µL 3 1,000

456 NBP Mean mmHg 10 187

646 SpO2 % 60 100

818 Lactic Acid (0.5-2.0) mg/dL 0 10

791 Creatinine (0-1.3) mg/dL 0.1 9

Additionally, some preprocessing was required regarding the patients’ age, given that in the MIMIC II

database patients older than 95 years have had their age register modified to exaggerated high values,

so that their identity remained protected. Consequently and to prevent the remaining values from being

influenced by such exaggerated records, these values have been once again modified to the maximum

of 95 years old.

Once the preprocessing was complete, the dataset was again restricted to patients with at least two

measurements of each of the variables listed in Table 4.1 and with a valid record of gender, age, height

and weight, resulting in a final dataset comprising 1499 not readmitted patients and 105 readmitted

24

patients, corresponding to 7% of the total of not readmitted patients.

A scheme describing the data acquisition and data preprocessing steps is presented in Figure 4.1.

Figure 4.1: Scheme of the data acquisition and data preprocessing steps.

4.3 Output Definition

One of the main goals of this thesis is to develop models capable of predicting patients’ readmissions

at any stage of their hospitalization in an attempt to mimic the real-world situation in which clinicians do

not know a priori when is the discharge day and have to evaluate continuously their patients until the

decision to discharge is made. To achieve such goal, it is first necessary to obtain samples representing

the different stages of hospitalization and to assign a corresponding output to each one.

Taking this into account, a sample time of 24 hours was chosen and each patient was evaluated

once a day along his stay in the ICU and at the moment of discharge. As a patient’s health condition

days before discharge is worse than that at the moment he or she leaves the ICU, all samples obtained

more than 24 hours before discharge were assigned to the readmitted patients’ class. This decision was

made considering that such instants represent moments in time where a patient is not fully treated and

would have to be readmitted in case he or she was discharged. Figure 4.2 presents the evolutions of

25

the outputs considered for samples belonging to not readmitted and readmitted patients.

discharge

1

0

Output

Days discharge

1

0

Output

Days

24h24h

(a) Not readmitted patients

discharge

1

0

Output

Days discharge

1

0

Output

Days

24h24h

(b) Readmitted patients

Figure 4.2: Output evolution.

After the sampling process, a final dataset was obtained comprising 1,144 samples from originally

readmitted patients and 10,911 samples from originally not readmitted patients, distributed by the two

classes as is shown in Figure 4.3.

105 readmitted

patients

1,144 samples classified as

readmitted patients

8,007 samples classified as

readmitted patients

2,904 samples classified as not

readmitted patients

1,499 not readmitted

patients

evaluated every 24 hours and at discharge

evaluated every 24 hours and at discharge

Figure 4.3: Scheme of samples distribution.

4.4 Prime Features Dataset

In an initial approach, a prime features dataset was constructed where the current value of each of

the physiological time series variables listed in Table 4.1 was considered for each sample.

Additionally, the number of days since admission in the ICU, tadm, the age, the gender and the body

mass index (BMI) of the belonging patient were added to each sample, resorting to Equation 4.1 to obtain

the BMI. The BMI is a measure of relative size based on the weight, w, and height, h, measurements

of an individual that can provide a better indication of a patient’s constitution than the use of either

measurement alone, having been associated in previous studies to the mortality of adult patients in

ICUs [40].

BMI =w

h2[kg/m2] (4.1)

In Table A.1 of Appendix A the set of 11 features considered for the prime features dataset is pre-

26

sented. This dataset was developed with the aim of being used as a comparison basis to other ap-

proaches.

4.5 Extracted Features Dataset

Feature extraction is a powerful technique of feature transformation, used to obtain a more relevant

representation of a given dataset [28].

With the aim of making further use of the information contained in the time series of physiological

variables, a feature extraction approach was taken. A new set of features was obtained based on medical

advice in an attempt to reproduce the information considered by clinicians when deciding on discharge.

In order to better retain the information contained in the original time series, the following features

were extracted from the series of records of each variable for each sample:

- Value of the last recorded measurement, vl;

- Value of the first recorded measurement, vf ;

- Maximum value recorded until the evaluation point, vmax;

- Time elapsed between admission and maximum value recorded until the evaluation point, tmax;

- Minimum value recorded until the evaluation point, vmin;

- Time elapsed between admission and minimum value recorded until the evaluation point, tmin;

- Mean of all values recorded until the evaluation point, vm;

Additionally to these features, the number of days since admission in the ICU, tadm, the age, the

gender and the BMI were added to each sample, resulting in a total of 53 features per sample, listed in

detail in Table A.5 of Appendix A.

A scheme comparing the features considered from the time series variables for the prime and ex-

tracted features datasets is presented in Figure 4.4.

Evaluation

point

Evaluation

point

𝑣𝑚

𝑣𝑚𝑎𝑥

𝑣𝑚𝑖𝑛𝑡𝑚𝑖𝑛

𝑡𝑚𝑎𝑥

𝑡𝑎𝑑𝑚

𝑣𝑙𝑣𝑓

𝑡𝑎𝑑𝑚

𝑣𝑙

Value Value

Time Time

(a) Prime features

Evaluation

point

Evaluation

point

𝑣𝑚

𝑣𝑚𝑎𝑥

𝑣𝑚𝑖𝑛𝑡𝑚𝑖𝑛

𝑡𝑚𝑎𝑥

𝑡𝑎𝑑𝑚

𝑣𝑙𝑣𝑓

𝑡𝑎𝑑𝑚

𝑣𝑙

Value Value

Time Time

(b) Extracted features

Figure 4.4: Features acquired from the time series variables.

27

4.6 Data Normalization

After the datasets are defined, the normalization of the data is required in order to have all data

values placed within a similar range.

To do so, a commonly used method in engineering and biomedical applications was chosen, the

minimum-maximum normalization [34, 41].

In this method, firstly, the minimum and maximum values of each feature f are computed, repre-

sented by xfminand xfmax

, respectively. Afterwards, for each sample i and feature f , the new normalized

value, xifnorm, is obtained from the prime value, xif , through Equation 4.2.

xifnorm=

xif − xfmin

xfmax − xfmin

(4.2)

28

Chapter 5

Modeling

In this Chapter, the theoretical background of the models developed is presented. Initially, the con-

cept of fuzzy modeling is introduced and the Takagi-Sugeno (TS) fuzzy models implemented are de-

scribed. Following, the technique of clustering is explained and two clustering algorithms are presented,

the Fuzzy C-Means (FCM), one of the most widely used and the Gustafson-Kessel (GK), the one chosen

for the models developed. At last, the multimodel approach is described and the set of feature selection

and decision criteria considered in the implementation of multimodels are presented.

5.1 Fuzzy Modeling

Traditional set theory requires elements to be either part of a given set or not [42]. While some

success has been achieved based on this theory, most real-world problems cannot be solved by such

approaches where the domain is mapped exclusively into two-valued variables. It is often necessary

to introduce a certain amount of fuzziness, i.e. it may be necessary to assign elements to sets with

a degree of certainty, similarly to what is done in human reasoning, when common sense is used.

Fuzzy modeling attempts to mimic such reasoning producing what is often referred to as approximate

reasoning [42]. Resorting to fuzzy sets, elements are assigned to sets in a certain degree, indicating the

certainty of their membership. Fuzzy logic is then used to deal with such uncertainties in order to infer

new facts with a respective degree of associated uncertainty.

Static or dynamic systems that make use of fuzzy sets and fuzzy logic are referred to as fuzzy

systems [43]. They can be used in different fields and for a variety of purposes, as modeling, data

analysis, prediction or control.

The main components of a fuzzy system, as presented in Figure 5.1, are a fuzzification interface, a

knowledge base, an inference engine and a defuzzification interface [44].

The fuzzification interface, or fuzzifier, is responsible for making the conversion between the inputs

and the fuzzy system. Resorting to it, the inputs are converted into linguistic values, represented by

fuzzy sets.

The knowledge base contains the main relationships between inputs and outputs. It is composed

29

by a data base where membership functions for linguistic terms are defined and a rule base, generally

represented by if-then statements.

The inference engine is responsible for computing the fuzzy output of the system, using the informa-

tion contained in the rule base and the given input value to produce an output.

Finally, the defuzzification interface, or defuzzifier, is responsible for making the connection between

the fuzzy system and the output. It is generally used to convert a fuzzy output into a required crisp one,

as in the case of binary classifiers where the output must take one of two possible values.

Input Output

Fuzzification Inference Defuzzification

Data base

Knowledge base

Rule base

If 𝒙𝟏 is 𝑨𝒋𝟏 and

… 𝒙𝑵 is 𝑨𝒋𝑵then 𝒚𝒋 = 𝒂𝒋

𝑻𝒙 + 𝒃𝒋

Figure 5.1: Configuration of fuzzy systems configuration, adapted from [44] .

5.1.1 Takagi-Sugeno Fuzzy Models

TS fuzzy models [45] consist of fuzzy rules that translate the relationships between inputs and out-

puts using fuzzy logic. In the present work, first order TS fuzzy models [44] were used, in which each

discriminant function is composed by rules of the type:

Rj : If x1 is Aj1 and ... xN is AjN

then yj = aTj x+ bj ,

where j = 1, ..., J corresponds to the rule number, x = (x1, ..., xN ) is the input vector, N is the total

number of features, Ajn is the fuzzy set for rule Rj and the nth feature, yj is the consequent function of

rule Rj , aj is the parameter vector of rule Rj and bj a scalar offset of rule Rj . The degree of activation,

βj , for the jth rule can be computed according to Equation 5.1:

βj =

N∏n=1

µAjn(x), (5.1)

where µAjn(x) : R→ [0, 1].

The final output is then determined through the weighted average of the individual outputs obtained

by each rule, resorting to Equation 5.2.

yc =

∑Jj=1 βjyj∑Jj=1 βj

=

∑Jj=1 βj(a

Tj x+ bj)∑J

j=1 βj(5.2)

The number of rules, J , is defined by the number of groups objects are sorted into and the antecedent

fuzzy sets, Ajn, are determined using fuzzy clustering in the product space of the input and output

30

variables [44]. The consequent parameters for each rule are obtained as a weighted ordinary least-

square estimate.

Considering that this is a classification problem, a final binary output y ∈ {0, 1} is required, where

0 corresponds to not readmitted patients and 1 to readmitted ones. However, the previous is a linear

consequent that results in a continuous output, yc ∈ [0, 1], being necessary to apply a threshold, t, to

transform it. In the present work such threshold will be determined in the training phase by varying its

value from 0 to 1 and accessing the values of specificity and sensitivity obtained for each threshold.

The threshold that results in the minimum difference between the two measures will be selected, as it

represents the best compromise between true positive and negative rates. Once the threshold has been

selected, the final binary output y can be determined by:

y =

0 , yc < t

1 , yc ≥ t(5.3)

5.2 Clustering

Clustering techniques seek to divide objects into groups or clusters, according to their similarities

[9]. They belong to the unsupervised learning methods, as the division is performed without making

use of prior information about the class of each object. The goal is to divide the dataset in a way that

objects belonging to the same cluster are as similar as possible and as dissimilar as possible to objects

belonging to different clusters. Such similarity measures can be defined in different ways, according to

the purpose of the problem, but are often expressed in terms of a distance norm that can be measured

between the data vectors or between the data vectors and the centers of the clusters, the prototypes.

These prototypes, as the partitioning of the data into each cluster, are unknown a priori and are sought

out by cluster algorithms in an iterative way. Clustering algorithms are useful in situations where few prior

knowledge about the data is available, having great potential to reveal underlying structures in it. They

work in an attempt to reconstruct these unknown structures so to obtain the best division of the data into

clusters representing different categories. They can be used for classification and pattern recognition

purposes, but also to help reduce the complexity of modeling and optimization problems [9].

There are several types of clustering algorithms that can be roughly distinguished into two categories

according to the way the data is partitioned: hard and fuzzy clustering [9]. Hard clustering makes use

of the classical set theory, requiring objects to be either part of a cluster or not. This implies that the

data is partitioned into mutually exclusive subsets. On the other hand, fuzzy or soft clustering enables

the division of the data into several clusters simultaneously, with different degrees of membership. Their

values range from 0 to 1, indicating the degree of memberships to each cluster. In many cases this

approach can be more suitable than the previous, as objects on the boundaries between several classes

are not required to belong exclusively to one of the classes.

31

5.2.1 Fuzzy C-Means

Fuzzy clustering algorithms based on objective functions have been widely used for different pur-

poses, as pattern recognition, data mining, image processing and fuzzy modeling [46]. Such methods

partition the datasets into overlapping groups in order to obtain clusters describing its underlying struc-

tures. When dealing with these methods, a number of aspects should be considered, namely the shape

and volume of the clusters, the initialization of the algorithm, the distribution of the data patterns and the

number of clusters.

There is a wide variety of clustering algorithms. One of the most popular and commonly used is the

FCM algorithm, based on the minimization of an objective function [47], as it will be discussed further

on.

In the FCM algorithm, a predefined number of clusters, nc , is computed, where each cluster is

characterized by a prototype, vi, defined by:

vi = [v1, ..., vq], (5.4)

where i = 1, 2, ..., nc and q represents the total number of features considered.

Each data sample, xj , is assigned with a normalized membership degree, µij , to a cluster i, being

the membership degrees contained in a fuzzy partition matrix, U :

U =

µ11 · · · µ1Ns

.... . .

...

µnc1 · · · µncNs

, (5.5)

where Ns is the total number of samples in the dataset.

The prototypes, vi, represent the centers of the clusters and can be computed according to:

vi =

∑Ns

j=1 µmijxj∑Ns

j=1 µmij

, (5.6)

where m ∈ [0,∞[ is the weighting exponent that determines the degree of fuzziness of clusters. High

values of m mean soft boundaries between clusters, while lower values stand for harder boundaries.

In the present work, each sample is assigned to each cluster with a certain degree of membership.

This degree is proportional to the distance between the sample and the cluster prototype, which in a

general way can be computed as:

d2ij(xj , vi) = ‖xj − vi‖2 = (xj − vi)TAi(xj − vi), (5.7)

where Ai is a positive definite symmetric matrix, usually equal to the identity matrix in the FCM algorithm.

Once the distances are computed, the referred membership degrees can be obtained through:

µij =1∑nc

k=1(dijdkj

)2

m−1

(5.8)

32

It should be noted that µij ∈ [0, 1], where zero implies that the sample j does not belong at all to

cluster i, in opposition to one that implies that a sample j completely belongs to cluster i. Additionally, it

must be considered that the sum of all membership degrees of any sample to all clusters must be equal

to one, according to:

nc∑i=1

µij = 1, ∀j, (5.9)

and that the total sample memberships to any cluster must be bigger than zero and smaller than one

according to:

0 <

Ns∑j=1

µij < Ns, ∀i (5.10)

The problem of assigning a dataset, X, into nc clusters is considered optimal when the previous

mentioned FCM objective function is minimized. This objective function is given by:

Jm(X,U, V ) =

nc∑i=1

Ns∑j=1

µmijd2ij(xj , vi), (5.11)

where V is a matrix containing all cluster prototypes vi.

Such minimization represents a nonlinear optimization problem that can be solved using Picard iter-

ation through the first-order conditions for stationary points of the objective function, Jm.

5.2.2 Gustafson-Kessel

As mentioned previously, in the FCM algorithm, the matrix Ai used to compute the distance in Equa-

tion 5.7 is the identity matrix. This causes the distance measure used to be reduced to the Euclidean

distance, making it suitable for clusters with spherical shapes [48].

However, the clusters obtained for a given dataset can have different geometrical shapes and orien-

tations. Considering these factors, the GK algorithm [49] can be a good alternative, since it resorts to

the Mahalonibis distance [46], where Ai is represented by Equation 5.12:

Ai = P−1i , (5.12)

Pi is the covariance matrix of cluster i and one additional volume constrain, |Ai| = ρi, must be added.

The objective function is then identical to the one used in the FCM algorithm, Equation 5.11, as

the update equations for the cluster centers, Equation 5.6, and the update equations for the member-

ship degrees, Equation 5.8. However, the alteration introduced in the distance measure improves the

adaptability to different cluster shapes, making it the fuzzy clustering algorithm chosen.

33

5.3 Multimodel Approach

The problem of modeling is usually associated with selecting the best model and being conditioned to

produce inferences from it [50]. However, a different approach can be taken by selecting more than one

model and producing inferences based on the entire set of models. Such approach is named multimodel,

or multiple model approach [51].

In recent decades, many studies were developed considering this approach. In a variety of areas, as

weather predictions [52], predicting the necessity of vasopressors in sepsis patients [24] or predicting

patients’ readmissions into ICUs [25], better results were achieved using multimodels than the traditional

single model.

In this work, a multimodel with a posteriori decision will be used to predict patients’ readmissions,

following the procedure presented in [24].

A posteriori decision means that the aggregation into the final output is carried out only after testing

each model independently. In a general way, it can be said that the models are tested separately, each

predicting an output. The set of predicted outputs is then combined into one single output, according to

a preselected decision criterion.

The present multimodel is composed by two independent models, one developed with the aim of

maximizing the sensitivity, i.e. the amount of readmitted patients correctly classified and the other with

the aim of maximizing the specificity, the amount of not readmitted patients correctly classified.

Figure 5.2 presents an overall scheme of the multimodel implemented to perform the patients’ clas-

sification.

Figure 5.2: Multimodel configuration with a posteriori decision, adapted from [25, 26] .

5.3.1 Model Selection

Selecting models that maximize sensitivity and specificity can be a complicated task, since many

different criteria can be used and, to the best of the author’s knowledge, there is none considered as

the most correct one. Hence, with the aim of exploring different possibilities, the following three sets of

feature selection performance criteria are used to develop the models featuring the multimodel:

1. Areas under the sensitivity and specificity curves

2. Sensitivity and specificity at the intersection threshold

3. Sensitivity and specificity close to the intersection threshold

34

In the first set, the specificity and sensitivity values for the test samples are computed for each

threshold α, sα and spα, respectively, being the threshold varied between 0 and 1 and the overall sum

of sensitivity and specificity values considered. In the model highlighting sensitivity, a bigger weighting

factor, wφ, is given to the sum of the sensitivity values and a smaller one, wβ , to the sum of the specificity

values, being the opposite for the model highlighting specificity. In this way, more importance is given

to the measure aimed to highlight, but at the same time, a compromise is found so that the opposite

measure does not present too poor results. Hence, the performance criteria used in feature selection

for the specificity and sensitivity models, cs and csp, are calculated respectively through Equations 5.13

and 5.14, where the values found for the weighting factors wφ and wβ were, respectively, 0.8 and 0.2.

cs = wφ

1∑α=0

sα + wβ

1∑α=0

spα (5.13)

csp = wφ

1∑α=0

spα + wβ

1∑α=0

sα (5.14)

In the second set of criteria, the threshold is again varied between 0 and 1, but this time in the

training phase. The sensitivity and specificity values of the training samples are calculated for each

threshold, as well as the difference between the two. The threshold, α, corresponding to the minimum

difference is chosen and kept for the test phase. In this set of criteria, solely the values of sensitivity

and specificity corresponding to that threshold are computed and, once again, more weight, w1, is given

to the sensitivity value in the model highlighting sensitivity and the opposite for the model highlighting

specificity. In this way, it is aimed to obtain the models that result in the best values of sensitivity and

specificity at the intersection point of both measures. The performance criteria used to perform feature

selection for the specificity and sensitivity models, are calculated respectively through Equations 5.15

and 5.16.

cs = wφsα + wβspα (5.15)

csp = wφspα + wβsα (5.16)

Finally, in the third set of criteria considered, the threshold is found in the same way as in the previous

one. However, when accessing the values of sensitivity and specificity in the test phase, a slight variation

of the threshold is chosen. For the model highlighting sensitivity the threshold α becomes the one

selected less 0.01 and in the one highlighting specificity, 0.01 is added to the initial threshold instead.

Such criteria were chosen with the aim of intensifying the importance given to each measure, since

values of sensitivity increase as the threshold decreases to zero and the opposite for the values of

specificity. The overall expressions to calculate the sensitivity and the specificity criteria are the same

as those used in Equations 5.15 and 5.16, respectively.

At last, it should be added that the weighting factors wφ and wβ chosen for the last two criteria were,

respectively, 0.7 and 0.3. These values differ from the first ones as the sole values of sensitivity and

specificity obtained for the selected thresholds were much more uneven than the ones obtained through

35

the overall sums, thus being necessary to adjust the corresponding weighting factors, obtained in an

empirical way throughout the work.

5.3.2 Decision Criterion

To obtain the final output of a multimodel is necessary to apply a decision criterion to combine, or

select, the outputs obtained by each of the integrating models. In the multimodel implemented, again

with the aim of testing different possibilities, three distinct decision criteria are considered to produce the

patients’ classification.

The first one consists of averaging the outputs of each model. That is, for an output y1 of the

sensitivity model and an output y2 of the specificity model, the output yδ provided by the decision criterion

is given by Equation 5.17.

yδ =y1 + y2

2(5.17)

The second criterion consists of choosing the predicted outcome with the lower uncertainty asso-

ciated, as it is presented in [25]. For each model i a threshold αi is found, according to the feature

selection criteria used. For the case of the first set of criteria where the overall sums of sensitivity and

specificity are considered, the threshold chosen is the same as the one used in the second set of criteria

where the intersection of sensitivity and specificity is sought. The differences between the outputs yi

and the respective threshold are then computed. The output corresponding to the maximum difference

corresponds to a predicted outcome closer to its predicted class and, consequently, with a lower degree

of uncertainty associated. In this way, the decision is performed according to:.

yδ = maxi|αi − yi| (5.18)

The third criterion consists of combining the two predicted outcomes with different weights wi, in-

versely proportional to their degree of uncertainty. In this way, more importance is given to the outcome

predicted with a lower degree of uncertainty. The weights, wi, can be computed as:

wi =

|αi − yi|αi

, yi < αi

|αi − yi|1− αi

, yi ≥ αi(5.19)

And the output yδ is given by:

yδ = w1y1 + w2y2 (5.20)

It should be noted, however, that the output yδ resulting from the decision criterion is a continuous

output yδ ∈ [0, 1], being still necessary to find a final threshold δ to transform it in the binary output

y ∈ {0, 1}. To do so, again the threshold that results in the minimum difference between the sensitivity

and specificity values of the training samples is chosen.

36

Chapter 6

Results

In this Chapter, the main results obtained for the different methods and respective approaches con-

sidered are summarized and compared.

Two distinct methods are implemented, one performing a daily evaluation where the complete set of

samples obtained in Section 4.3 is used and another carrying out a discharge day evaluation where only

samples obtained at discharge days are used in the testing sets. The daily evaluation is carried out with

the aim of developing suitable models for the prediction of patients’ readmissions at any stage of their

ICU hospitalization. The discharge day evaluation approach is in line with previous works, where patients

were evaluated solely at their discharge day. In particular, in [11], an AUC of 0.72±0.04, an accuracy of

0.71±0.03, a sensitivity of 0.68±0.02 and a specificity of 0.73±0.03 were obtained, whereas in [25] an

AUC of 0.74±0.06, an accuracy of 0.74±0.07, a sensitivity of 0.74±0.16 and a specificity of 0.74±0.09

were obtained.

For each evaluation, results are presented in the following way: initially, the prime features dataset

is used in a single model configuration. Then, the same model configuration is kept but the extracted

features dataset is used instead, so that conclusions can be drawn regarding the use of prime and

extracted features. At last, the extracted features are used in a multimodel configuration, so that the

influence of this configuration can be studied and compared with the one resulting from the use of a

single model with the same features.

In the development all the tested approaches, the same methodology was followed in order to obtain

comparable results. For the same reason, the type of models developed was also kept the same, having

been used TS fuzzy models where the number of rules and the antecedent fuzzy sets were determined

using the GK fuzzy clustering in the product space of the input and output variables.

In terms of the methodology followed, initially the readmitted and not readmitted patients and their

respective samples were divided randomly among two subsets, one for feature selection (FS) and the

other to perform model assessment (MA). This division was carried out in a way that approximately the

same number of readmitted and not readmitted patients, along with their samples, was assigned to each

subset .

The FS subset was firstly used to decide on the parameters of the model to be developed, the number

37

of clusters, nc, and the degree of fuzziness, m, by creating models and using the respective AUC as

performance criterion. Once the model parameters selected, this subset was used to perform feature

selection with the SFS technique, resorting to the AUC as criterion in the single model case and the

criteria presented in Section 5.3.1 in the multimodel one.

Having selected the model parameters and best subsets of features, the MA subset was used to

develop models and evaluate their performance in terms of AUC, accuracy, sensitivity and specificity. In

this way , different data from the one where the parameters and features were selected was used, so

that the results obtained are independent from the previous phase and comparable with each other.

With the aim of reducing the variability of the results and to assure their independence from the

training and test groups selected, the K-fold cross-validation methodology was used in all steps of the

model development and assessment. With it , the patients and their respective samples are separated

into K folds, in a way that an approximate equal proportion of readmitted and not readmitted patients is

kept in each fold. Once the data is divided into the K folds, the process is repeated K times, each time

using a different fold as testing set.

6.1 Daily Evaluation

The totality of samples obtained in Section 4.3 for the different hospitalization days was used with

the purpose of developing models capable of predicting patients’ readmissions at any stage of their

hospitalization, in an attempt to mimic the real ICU situation.

6.1.1 Single Model with Prime Features

As a first approach, a single model was developed using the prime features dataset introduced in

Section 4.4.

Parameter Selection

To select the nc and m parameters, 5-fold cross-validation was used in the FS subset. The nc

parameter was varied between 2 and 10, simultaneously with the degree of fuzziness m, varied between

1.1 and 2, with a step of 0.1. Models were created, and the AUC was used as performance measure.

The complete set of results obtained can be consulted in Tables A.2 and A.3 of Appendix A, where the

best combination of parameters is presented in Table 6.1.

Table 6.1: Best combination of parameters obtained with prime features after 5-fold cross-validation.

nc m AUC

4 1.6 0.71 ± 0.02

38

Feature Selection

To select the best subset of features, the SFS method was used, together with 5-fold cross-validation.

The AUC was used as performance criterion along the SFS, and the entire process was repeated 10

times. The three subsets that presented the best results are listed in Table 6.2, being the features that

integrate each subset listed in Table A.4 of Appendix A.

Table 6.2: Best subsets of features obtained by performing SFS with 5-fold cross-validation with primefeatures.

Subset 1 Subset 2 Subset 3

Number of Features 7 7 8

AUC 0.74 ± 0.01 0.73 ± 0.01 0.73 ± 0.02

Model Assessment

For MA, the three subsets presented in Table 6.2, together with the entire initial set composed by 11

features, were used to create models and evaluate their performance using different criteria. The results

obtained for AUC, accuracy, sensitivity and specificity after performing 5 times 5-fold cross-validation are

presented in Table 6.3.

Table 6.3: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA subset.

Initial Set Subset 1 Subset 2 Subset 3

AUC 0.69 ± 0.02 0.71 ± 0.02 0.72 ± 0.02 0.72 ± 0.03

Accuracy 0.64 ± 0.02 0.66 ± 0.02 0.66 ± 0.02 0.66 ± 0.03

Sensitivity 0.64 ± 0.03 0.66 ± 0.03 0.66 ± 0.04 0.66 ± 0.04

Specificity 0.64 ± 0.04 0.66 ± 0.04 0.66 ± 0.04 0.66 ± 0.04

6.1.2 Single Model with Extracted Features

In a second approach, single models were developed with the features obtained through feature ex-

traction, explained in detail in Section 4.5.

Parameter Selection

Initially, models were created and evaluated using 5-fold cross-validation in the FS subset to select

the nc and m parameters. The two parameters were varied simultaneously, the nc between 2 and 10 and

the m between 1.1 and 2, with a step of 0.1, being used the AUC as performance measure. The best

combination of parameters achieved is presented in Table 6.4 and the results obtained for the remaining

combinations are available in Tables A.6 and A.7 of Appendix A.

39

Table 6.4: Best combination of parameters obtained with extracted features after 5-fold cross-validation.

nc m AUC

3 1.5 0.74 ± 0.02

Feature Selection

To select the best subset of features, 5-fold cross-validation was carried out, using the SFS approach.

The AUC was used as a performance criterion through the SFS and the entire process was repeated 10

times. The three subsets that presented the best results are listed in Table 6.5 and the lists of features

that compose each subset are available in Table A.8 of Appendix A.

Table 6.5: Best subsets of features obtained by performing SFS with 5-fold cross-validation withextracted features.



AUC 0.78 ± 0.01 0.78 ± 0.01 0.77 ± 0.02

Model Assessment

For MA, the three subsets presented in Table 6.5 were evaluated, as well as the entire initial set com-

posed by 53 features. The results obtained for AUC, accuracy, sensitivity and specificity after performing

5 times 5-fold cross-validation are presented in Table 6.6.

Table 6.6: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset.


AUC 0.73 ± 0.03 0.75 ± 0.02 0.76 ± 0.02 0.76 ± 0.02Accuracy 0.65 ± 0.03 0.67 ± 0.03 0.67 ± 0.02 0.68 ± 0.02Sensitivity 0.62 ± 0.04 0.66 ± 0.04 0.65 ± 0.04 0.67 ± 0.04Specificity 0.70 ± 0.04 0.71 ± 0.04 0.72 ± 0.03 0.70 ± 0.04

6.1.3 Multimodel with Extracted Features

At last, the extracted features dataset presented in Section 4.5 was used in the implementation of

a multimodel, following the methodology and criteria presented in Section 5.3. As the initial features

considered were the same as the ones used in the single model with extracted features, the parameters

nc and m were kept the same, as presented in Table 6.4, 3 and 1.5 respectively.

40

6.1.3.1 Areas Under the Sensitivity and Specificity Curves

Feature Selection

To select the best subset of features for each model, 5-fold cross-validation was used and initially the

first set of feature selection criteria presented in Section 5.3.1 was considered. Again the SFS process

was carried out 10 times, being the best subsets obtained for the models highlighting sensitivity and

specificity presented, respectively, in Tables 6.7 and 6.8. The features that integrate each subset can be

consulted in Table A.9 of Appendix A.

Table 6.7: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the first set of feature selection criteria.



0.8∑1α=0 sα + 0.2

∑1α=0 spα 0.72 ± 0.01 0.71 ± 0.01 0.71 ± 0.01

Table 6.8: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the first set of feature selection criteria.



0.8∑1α=0 spα + 0.2

∑1α=0 sα 0.44 ± 0.01 0.44 ± 0.01 0.43 ± 0.01

Model Assessment

To conduct the MA, 5-fold cross-validation was performed 5 times. The subsets of features obtained

for the model highlighting sensitivity, Table 6.7, and the one highlighting specificity, Table 6.8, were

combined and the three decision criteria presented in Section 5.3.2 were used. The results obtained

for the best three combinations of subsets are presented in Table 6.9 and the ones obtained for the

remaining combinations are available in Table A.12 of Appendix A.

41

Table 6.9: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the first set of feature selection criteria.

Weighted distance Maximum distance Average

AUC 0.77 ± 0.02 0.76 ± 0.01 0.77 ± 0.01Sensitivity subset 1 Accuracy 0.69 ± 0.02 0.70 ± 0.02 0.69 ± 0.02

and specificity subset 2 Sensitivity 0.69 ± 0.03 0.70 ± 0.03 0.68 ± 0.03

Specificity 0.70 ± 0.03 0.69 ± 0.03 0.71 ± 0.03



Specificity 0.71 ± 0.03 0.69 ± 0.03 0.73 ± 0.03



Specificity 0.70 ± 0.03 0.68 ± 0.03 0.73 ± 0.03

6.1.3.2 Sensitivity and Specificity at the Intersection Threshold

Feature Selection

In a different approach, the second set of feature selection presented in Section 5.3.1 was used

with 5-fold cross-validation to select the best subsets of features for each model. SFS was performed

10 times and the best three subsets obtained for each model were kept. Table 6.10 presents the best

subsets of features obtained for the model highlighting sensitivity and Table 6.11 the best subsets of

features obtained for the model highlighting specificity. Information about the features that compose

each subset can be found in Table A.10 of Appendix A.

Table 6.10: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the second set of feature selection criteria.



0.7 sα + 0.3 spα 0.70 ± 0.02 0.69 ± 0.02 0.68 ± 0.03

Table 6.11: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the second set of feature selection criteria.



0.7 spα + 0.3 sα 0.69 ± 0.01 0.69 ± 0.02 0.69 ± 0.02

42

Model Assessment

To conduct the MA, 5-fold cross-validation was performed 5 times. The subsets of features obtained

for the models highlighting sensitivity and specificity, Tables 6.10 and 6.11, were combined and the

three decision criteria presented in Section 5.3.2 were used. The results obtained for the best three

combinations of subsets are presented in Table 6.12. Additionally, the results obtained for the remaining

combinations are available in Table A.13 of Appendix A.

Table 6.12: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the second set of feature selection criteria.




Specificity 0.69 ± 0.03 0.67 ± 0.03 0.72 ± 0.03



Specificity 0.71 ± 0.03 0.68 ± 0.03 0.73 ± 0.03



Specificity 0.70 ± 0.03 0.67 ± 0.03 0.73 ± 0.03

6.1.3.3 Sensitivity and Specificity Close to the Intersection Threshold

Feature Selection

At last, the third set of feature selection criteria presented in Section 5.3.1 was used, being repeated

10 times the SFS procedure with 5-fold cross-validation. Table 6.13 presents the three best subsets of

features obtained for the model highlighting sensitivity and Table 6.14 the best three subsets of features

obtained for the model highlighting specificity. The list of the features that integrate each subset can be

found in Table A.11 of Appendix A.

Table 6.13: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the third set of feature selection criteria.



0.7 sα + 0.3 spα 0.69 ± 0.02 0.68 ± 0.03 0.66 ± 0.02

43

Table 6.14: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the third set of feature selection criteria.



0.7 spα + 0.3 sα 0.70 ± 0.02 0.70 ± 0.02 0.70 ± 0.03

Model Assessment

To conduct the model assessment, 5-fold cross-validation was performed 5 times. The subsets of

features obtained for the model highlighting sensitivity, Table 6.13, and the one highlighting specificity,

Table 6.14, were combined and the three decision criteria presented in Section 5.3.2 were used. Ta-

ble 6.15 presents the results obtained for the best three combinations of subsets and Table A.14 of

Appendix A the results for the remaining combinations.

Table 6.15: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the third set of feature selection criteria.




Specificity 0.70 ± 0.03 0.66 ± 0.03 0.74 ± 0.03



Specificity 0.70 ± 0.03 0.66 ± 0.03 0.73 ± 0.03



Specificity 0.70 ± 0.03 0.66 ± 0.03 0.74 ± 0.04

6.1.4 Results Comparison

Starting with the single model with prime features, Section 6.1.1, a slight improvement can be noted

in Table 6.3, specially in terms of AUC, resulting from the use of subsets of features selected by SFS

instead of the entire initial set. Although different subsets were tested, probably due to the close number

of selected features, the results obtained were very similar for the three subsets, being of about 0.72 for

the AUC and 0.66 for the remaining performance measures.

In Section 6.1.2, single models were developed using extracted features. Again, in Table 6.6 it can

be noted the positive influence of the use of feature selection, as the results obtained for the three

subsets of features were better than the ones obtained for the initial set. Comparing the performance of

44

these subsets to the ones obtained using prime features also some improvements can be noted, having

the greatest improvement been achieved in the measures of AUC and specificity. The values of AUC

improved from an average of 0.72 to 0.76 and of specificity improved from an average of 0.66 to 0.71.

In terms of sensitivity, the average values remained the same, about 0.66, but in terms of accuracy also

a slight improvement was achieved, from 0.66 to an average of 0.67.

In Section 6.1.3 multimodels were developed based on the extracted features, using different feature

selection and decision criteria. Overall, it can be concluded that the set of feature selection criteria that

presented the best results was the first one, using the areas under the sensitivity and specificity curves,

Table 6.9, followed by the second one, using the sensitivity and specificity values at the intersection

threshold, Table 6.12.

In terms of the decision criteria used, a similar performance was obtained for the multimodels devel-

oped with the three sets of feature selection criteria. In general, better results were achieved for the one

using the maximum distance. However, it can be seen how this criteria disfavors slightly the specificity.

The first criterion, the weighted distance, can be used in an alternative, as it achieves more balanced

results, although slightly worst than the ones from the maximum distance criterion. The average criterion

is the one that in general performed worst, favoring, however, the values of specificity.

In Table 6.16, the values of AUC obtained for the best multimodels developed for each feature selec-

tion and decision criterion are presented and in Table 6.17 the p-values resulting from each combination

of multimodels are presented.

Table 6.16: Mean and standard deviation of the AUC values obtained for the best multimodelsdeveloped.


Areas under the sensitivity and the (1) (2) (3)

specificity curves - sensitivity subset 1 0.77 ± 0.02 0.77 ± 0.01 0.77 ± 0.01

and specificity subset 3

Sensitivity and specificity at the (4) (5) (6)

intersection threshold - sensitivity 0.76 ± 0.02 0.76 ± 0.02 0.76 ± 0.02

subset 3 and specificity subset 3

Sensitivity and specificity close (7) (8) (9)

to the intersection threshold - sensitivity 0.75 ± 0.02 0.75 ± 0.02 0.75 ± 0.02

subset 3 and specificity subset 3

45

Table 6.17: P-values between the best multimodels developed.

(1) (2) (3) (4) (5) (6) (7) (8) (9)

(1) - 0.00 0.01 0.05 0.02 0.05 0.00 0.00 0.01

(2) - - 0.00 0.04 0.05 0.13 0.01 0.00 0.02

(3) - - - 0.04 0.01 0.28 0.00 0.00 0.01

(4) - - - - 0.00 0.00 0.05 0.02 0.15

(5) - - - - - 0.01 0.08 0.03 0.25

(6) - - - - - - 0.01 0.00 0.03

(7) - - - - - - - 0.00 0.00

(8) - - - - - - - - 0.00

(9) - - - - - - - - -

It can be seen in Table 6.17 how, mostly, the different multimodels developed present p-values

smaller then 0.05 among themselves, enhancing the statistical difference between the AUC values ob-

tained for each model, with a significance level of 5%.

The best model obtained was the one using the sensitivity subset 1 and specificity subset 3, with

the areas under the sensitivity and specificity curves as feature selection criteria and the maximum

distance as decision criterion, presented in Table 6.9, identified with the number 2 in Tables 6.16 and

6.17. This model performed better than the single models with extracted features in all measurements

but specificity, having achieved 0.77 in AUC, 0.71 in accuracy, 0.71 in sensitivity and 0.69 in specificity.

In Figure 6.1, the results obtained in terms of the different performance measures considered for the

best models obtained are presented.

Figure 6.1: AUC, accuracy, sensitivity and specificity obtained with the different approachesimplemented.

At last, in Table 6.18, a comparison between the results obtained for the best model developed in the

present work and the results obtained in previous ones is made.

46

Table 6.18: Performance comparison of different models.

Multimodel with [11] [25]

extracted features

AUC 0.77 ± 0.01 0.72 ± 0.04 0.74 ± 0.06

Accuracy 0.71 ± 0.02 0.71 ± 0.03 0.74 ± 0.07

Sensitivity 0.71 ± 0.03 0.68 ± 0.02 0.74 ± 0.16

Specificity 0.69 ± 0.03 0.73 ± 0.03 0.74 ± 0.09

It can be seen how compared to the work of [11], specially better results were obtained in terms of

AUC and sensitivity. In comparison to the results obtained in [25], it is shown how only in terms of AUC

an improvement was made. However, it should be noted how the standard deviation values presented

in the remaining measures are substantially higher in the work of [25]. Furthermore, it should be noted

how in the previous works only the discharge days were considered, while the present work proposes

an approach for a daily assessment of outcomes, where the modeling strategy considers the complete

evolution of a patient’s condition, since admission to discharge, making it a more suitable and useful

approach to deal with real-world applications.

6.1.5 Model Evaluation

In order to better perceive the performance of the best model obtained, sensitivity subset 1 and speci-

ficity subset 3 of Table 6.9 with the maximum distance as decision criterion, an additional performance

assessment was carried out considering the real and predicted class outcomes of different patients. This

model obtained an AUC of 0.77, an accuracy of 0.71, a specificity of 0.71 and a specificity of 0.69. In

Table 6.19, a list of the features that comprise the sensitivity subset 1 and the specificity subset 3 can

be found.

In an attempt to study the performance of this model for different situations, three patients admitted

for a short period of time, less than 5 days, were considered, together with three patients admitted for a

medium amount of time, between 6 and 15 days, and three patients admitted for a long period of time,

that is, superior to 16 days. In each group of patients, a readmitted patient and two not readmitted ones

were included.

47

Table 6.19: Features selected for the sensitivity subset 1 and the specificity subset 3.

Sensitivity Subset 1 Specificity Subset 3

Minimum Creatinine (0 - 1.3) value Minimum Creatinine (0 - 1.3) value

Time of maximum Lactic Acid (0.5 - 2) value Time of maximum Lactic Acid (0.5 - 2) value

Time of minimum Lactic Acid (0.5 - 2) value Time of minimum Lactic Acid (0.5 - 2) value

Last NBP Mean value Last NBP Mean value

Maximum NBP Mean value Time of maximum NBP Mean value

Time of maximum NBP Mean value Time of minimum NBP Mean value

Time of minimum NBP Mean value Minimum Platelets value

Maximum Platelets value Mean Platelets recorded values

Last Temperature value Last Temperature value

Maximum Temperature value Maximum Temperature value

Time of maximum Temperature value Time of maximum Temperature value

Time of minimum Temperature value Time of minimum Temperature value

Maximum Heart Rate value Last Heart Rate value

Last SpO2 value Maximum Heart Rate value

- Last SpO2 value

- First SpO2 value

The evolutions of the real and predicted outcomes of two not readmitted patients admitted for a short

period of time are presented in Figures 6.2 and 6.3 and the same evolutions for a readmitted patient is

presented in Figure 6.4.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5

Cla

ssific

atio

n

Number of Days since Admission in ICU

Not Readmitted Patient 30062

Real Value

Predicted Value

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4

Cla

ssific

atio

n



Real Value

Predicted Value

Figure 6.2: Real and predicted outcomes ofthe not readmitted patient with ID number 30062.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5

Cla

ssific

atio

n



Real Value

Predicted Value

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4

Cla

ssific

atio

n



Real Value

Predicted Value


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5

Cla

ssific

ation


Readmitted Patient 31651

Real Value

Predicted Value

Figure 6.4: Real and predicted outcomes ofthe readmitted patient with ID number 31651.

Figures 6.5 and 6.6 present the evolutions of the real and predicted outcomes of two not readmitted

48

patients admitted for a medium period of time and Figure 6.7 presents these evolutions for a readmitted

patient admitted for a medium period of time.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10 11 12 13

Cla

ssific

atio

n



Real Value

Predicted Value

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6

Cla

ssific

atio

n



Real Value

Predicted Value


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10 11 12 13

Cla

ssific

ation



Real Value

Predicted Value

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6

Cla

ssific

ation



Real Value

Predicted Value


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7

Cla

ssific

ation



Real Value

Predicted Value


At last, Figures 6.8 and 6.9 present the evolutions of real and predicted outcomes of two not readmit-

ted patients admitted for a long period of time and Figure 6.10 presents these evolutions for a readmitted

patient admitted for a long period of time.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

Cla

ssific

ation



Real Value

Predicted Value

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Cla

ssific

ation



Real Value

Predicted Value


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

Cla

ssific

ation



Real Value

Predicted Value

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Cla

ssific

ation



Real Value

Predicted Value


49

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Cla

ssific

ation



Real Value

Predicted Value


The previous figures show how for the different lengths of stay of not readmitted patients, the values

predicted by the model tend to decrease as the discharge day approaches, even if not in a linear way.

In the case of readmitted patients, although some oscillations can be spotted, in general the values tend

to stay close to one. Some decreases can also be spotted as the discharge day approaches, but not so

clear as the ones in not readmitted patients. Therefore, the model seems to present a relatively good

overall performance in terms of indicating the right moment to discharge a patient so that he does not

end up being readmitted.

6.2 Discharge Day Evaluation

In an attempt to obtain results comparable with previous works where patients were evaluated only

at the discharge day [11, 25], the previously described methodology was repeated, this time using only

samples obtained at discharge days in the testing sets.

6.2.1 Single Model with Prime Features

Initially, a single model was developed using the prime features presented in Table A.1.

Parameter Selection

To select the nc and m parameters, 5-fold cross-validation was used in the FS subset. The nc

parameter was varied between 2 and 10, along with the degree of fuzziness m, varied between 1.1

and 2, with a step of 0.1. Models were created and the AUC was used as performance measure. The

complete set of results obtained can be consulted in Tables B.1 and B.2 of Appendix B, being the best

combination of parameters obtained presented in Table 6.20.

Table 6.20: Best combination of parameters obtained with prime features after 5-fold cross-validation.

nc m AUC

3 1.4 0.66 ± 0.06

50

Feature Selection

To select the best subset of features, the SFS method was repeated 10 times, together with 5-fold

cross-validation. The AUC was used as performance criterion, being the best three subsets of features

obtained presented in Table 6.21. The list of features that integrates each subset can be consulted in

Table B.3 of Appendix B.

Table 6.21: Best subsets of features obtained by performing SFS with 5-fold cross-validation with primefeatures.



AUC 0.67 ± 0.07 0.67 ± 0.09 0.67 ± 0.10

Model Assessment

To perform MA, the three subsets of features presented in Table 6.21 together with the initial set

composed by 11 features were used to create models and evaluate their performance using different

criteria. Table 6.22 presents the results obtained, in terms of AUC, accuracy, sensitivity and specificity,

after performing 5 times 5-fold cross-validation.

Table 6.22: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA subset.


AUC 0.61 ± 0.01 0.67 ± 0.06 0.69 ± 0.06 0.68 ± 0.06

Accuracy 0.58 ± 0.05 0.62 ± 0.04 0.64 ± 0.03 0.64 ± 0.03Sensitivity 0.59 ± 0.02 0.61 ± 0.02 0.64 ± 0.01 0.62 ± 0.02

Specificity 0.58 ± 0.05 0.63 ± 0.05 0.63 ± 0.03 0.64 ± 0.04

6.2.2 Single Model with Extracted Features

In a second approach, single models were developed with the features obtained through feature ex-

traction, presented in Section 4.5.

Parameter Selection

Initially, models were created and evaluated using 5-fold cross-validation in the FS subset to select

the nc and m parameters. The two parameters were varied simultaneously, the nc between 2 and 10

and the m between 1.1 and 2, with a step of 0.1. The AUC was used as performance measure, being

the best combination of parameters achieved presented in Table 6.23. The results obtained for the

remaining combinations of parameters are available in Tables B.4 and B.5 of Appendix B.

51

Table 6.23: Best combination of parameters obtained with extracted features after 5-foldcross-validation.

nc m AUC

4 1.1 0.66 ± 0.06

Feature Selection

To select the best subset of features, SFS was performed 10 times with 5-fold cross-validation, using

the AUC as performance criterion. The three subsets that achieved the best results are presented in

Table 6.24 and the features that compose each subset are listed in Table B.6 of Appendix B.

Table 6.24: Best subsets of features obtained by performing SFS with 5-fold cross-validation withextracted features.



AUC 0.71 ± 0.08 0.70 ± 0.09 0.69 ± 0.10

Model Assessment

To conduct the MA, the three subsets of features presented in Table 6.24 together with the initial set

composed by 53 features, were evaluated in terms of AUC, accuracy, sensitivity and specificity. 5-fold

cross-validation was used, being the results obtained presented in Table 6.25.

Table 6.25: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset.


AUC 0.63 ± 0.08 0.66 ± 0.06 0.73 ± 0.05 0.67 ± 0.06

Accuracy 0.60 ± 0.04 0.65 ± 0.03 0.67 ± 0.05 0.65 ± 0.03

Sensitivity 0.56 ± 0.02 0.62 ± 0.01 0.69 ± 0.09 0.63 ± 0.02

Specificity 0.60 ± 0.05 0.65 ± 0.03 0.67 ± 0.06 0.65 ± 0.01

6.2.3 Multimodel with Extracted Features

At last, the extracted features presented in Section 4.5 were used to develop multimodels, following

the methodology and criteria presented in Section 5.3. As the initial features considered were the same

as the ones used in the single model with extracted features, the parameters nc and m were kept the

same, 4 and 1.1 respectively.

52

6.2.3.1 Areas Under Sensitivity and Specificity Curves

Feature Selection

To select the best subsets of features for each model, 5-fold cross-validation was used and the

first set of feature selection criteria presented in Section 5.3.1 was considered. The SFS process was

again carried out 10 times, being the best three subsets of features obtained for the models highlighting

sensitivity and specificity presented, respectively, in Tables 6.26 and 6.27. The list of features that

integrate each subset are available in Table B.7 of Appendix B.

Table 6.26: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the first set of feature selection criteria.



0.8∑1α=0 sα + 0.2

∑1α=0 spα 0.69 ± 0.01 0.69 ± 0.01 0.69 ± 0.01

Table 6.27: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the first set of feature selection criteria.



0.8∑1α=0 spα + 0.2

∑1α=0 sα 0.42 ± 0.01 0.42 ± 0.01 0.41 ± 0.01

Model Assessment

To conduct MA, 5-fold cross-validation was performed 5 times. The subsets of features obtained

for the models highlighting sensitivity and specificity, Tables 6.26 and 6.27, were combined and the

three decision criteria presented in Section 5.3.2 were used. The results obtained for the best three

combinations of features are presented in Table 6.28 and in Table B.10 of Appendix B can be found the

results obtained for the remaining combinations.

53

Table 6.28: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the first set of feature selection criteria.




Specificity 0.69 ± 0.03 0.63 ± 0.03 0.75 ± 0.03



Specificity 0.68 ± 0.03 0.62 ± 0.03 0.74 ± 0.03



Specificity 0.68 ± 0.04 0.63 ± 0.04 0.75 ± 0.04

6.2.3.2 Sensitivity and Specificity at the Intersection Threshold

Feature Selection

In an alternative approach, the second set of feature selection criteria presented in Section 5.3.1

was used to perform feature selection, together with 5-fold cross-validation. The SFS procedure was

performed 10 times and the best three subsets of features for each model were kept. Table 6.29 presents

the best subsets obtained for the model highlighting sensitivity and Table 6.30 the best subsets obtained

for the model highlighting specificity. Additionally, the features that compose each subset can be found

in in Table B.8 of Appendix B.

Table 6.29: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the second set of feature selection criteria.



0.7 sα + 0.3 spα 0.71 ± 0.15 0.71 ± 0.15 0.69 ± 0.14

Table 6.30: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the second set of feature selection criteria.



0.7 spα + 0.3 sα 0.67 ± 0.03 0.67 ± 0.06 0.66 ± 0.05

54

Model Assessment

For MA, 5-fold cross-validation was performed 5 times. The subsets of features obtained for the

models highlighting sensitivity and specificity, Tables 6.29 and 6.30, were combined and the three de-

cision criteria presented in Section 5.3.2 were used. Table 6.31 presents the results obtained for the

best three combinations of subsets. In Table B.11 of Appendix B, the results obtained for the remaining

combinations are available.

Table 6.31: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the second set of feature selection criteria.


AUC 0.69 ± 0.05 0.69 ± 0.05 0.68 ± 0.05

Sensitivity subset 2 Accuracy 0.66 ± 0.03 0.61 ± 0.03 0.72 ± 0.03and specificity subset 2 Sensitivity 0.62 ± 0.10 0.67 ± 0.08 0.50 ± 0.11

Specificity 0.66 ± 0.03 0.61 ± 0.03 0.73 ± 0.03



Specificity 0.66 ± 0.03 0.63 ± 0.03 0.70 ± 0.04



Specificity 0.65 ± 0.04 0.61 ± 0.04 0.70 ± 0.04

6.2.3.3 Sensitivity and Specificity Close to the Intersection Threshold

Feature Selection

At last, the third set of feature selection criteria presented in Section 5.3.1 was used to perform

feature selection, being repeated 10 times the SFS procedure with 5-fold cross-validation. In Table 6.32

the three best subsets of features obtained for the model highlighting sensitivity are presented and in

Table 6.33 the three best subsets of features obtained for the model highlighting specificity. In Table B.9

of Appendix B the lists of features that comprise each subset are available.

Table 6.32: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the third set of feature selection criteria.



0.7 spα + 0.3 sα 0.75 ± 0.06 0.74 ± 0.07 0.72 ± 0.05

55

Table 6.33: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the third set of feature selection criteria.



0.7 spα + 0.3 sα 0.69 ± 0.03 0.68 ± 0.04 0.66 ± 0.02

Model Assessment

To conduct MA, 5-fold cross-validation was performed 5 times. The subsets of features obtained

for the model highlighting sensitivity, Table 6.32, and the one highlighting specificity, Table 6.33, were

combined and the three decision criteria presented in Section 5.3.2 were used. The results obtained for

the best three combinations of subsets are presented in Table 6.34 and in Table B.12 of Appendix B the

results obtained for the remaining combinations can be found.

Table 6.34: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the third set of feature selection criteria.


AUC 0.68 ± 0.08 0.68 ± 0.06 0.68 ± 0.07

Sensitivity subset 1 Accuracy 0.66 ± 0.04 0.62 ± 0.03 0.71 ± 0.03and specificity subset 3 Sensitivity 0.59 ± 0.14 0.65 ± 0.12 0.52± 0.12

Specificity 0.66 ± 0.04 0.62 ± 0.03 0.72 ± 0.04



Specificity 0.67 ± 0.03 0.62 ± 0.04 0.72 ± 0.03



Specificity 0.67 ± 0.03 0.64 ± 0.03 0.72 ± 0.04

6.2.4 Results Comparison

Overall, two main conclusions can be drawn upon the evaluation of the models using test samples

obtained at discharge days. First, it is clear that worse results are obtained when comparing to the use

of the full hospitalization samples, and second, that worse results are also obtained in comparison to

previous studies [25, 11].

This can be due to several reasons. One is that the full dataset was used in the training steps. This

means big amounts of information concerning different stages of hospitalization were provided as exam-

ples, making it harder for models to classify the discharge days as accurately as other models trained

only with discharge samples. Another factor can be the diminished size of the readmitted patients’

56

class, of 105 patients. These patients were first divided into the FS and MA subsets and later in 5 folds

to perform cross-validation, resulting in groups of about 10 patients per folder. Such division influences

drastically the sensitivity, as slight changes in the number of correctly classified readmitted patients re-

sult in considerable changes in sensitivity, making it difficult to achieve models with even sensitivity and

specificity performances.

In terms of the single models developed, for both the prime and extracted features approaches can

be seen in Tables 6.22 and 6.25, respectively, how the use of subsets of features obtained through

feature selection resulted in better performances than the use of the initial sets. Also, it can be noted

how the increase in performance resulting from the use of extracted features was not as notorious as

the one in the previous method, using all samples in the testing set. Nonetheless, it can be noted how

the subset 2, obtained with the extracted features, presented results considerably better than the ones

obtained through prime features, with an AUC of 0.73, accuracy of 0.67, sensitivity of 0.69 and specificity

of 0.67.

In terms of the multimodels developed, no better results were obtained, nevertheless, some interest-

ing observations can be made. In terms of the sets of feature selection criteria used, a similar perfor-

mance ranking was obtained as previously. However, in this approach it was not obtained such a clear

distinction between the first and second sets of criteria. Nevertheless, again the third set of criteria, the

use of sensitivity and specificity close to the intersection thresholds was the one that performed worse.

In terms of the decision criterion, it can be noted, once again, how the one using the average disfa-

vors strongly the sensitivity. The weighted and the maximum distances criteria presented performances

relatively comparable, but, in most cases the use of the maximum distance resulted in more balanced

results in all performance measures.

In Figure 6.11, the results obtained in terms of AUC, accuracy, sensitivity and specificity for the best

models obtained in each approach are presented, where can be noted how the greatest improvement in

performance resulted from the use of extracted features. For the multimodel approach, the best model

considered was the one with the sensitivity and specificity subsets number 3 resulting from the second

set of feature selection criteria with the weighted distance decision criterion, presented in Table 6.31.

Figure 6.11: AUC, accuracy, sensitivity and specificity obtained with the different approachesimplemented.

57

58

Chapter 7

Conclusions

The present thesis was developed with the aim of predicting patients’ readmissions in ICUs, based

on patients’ demographics information and a set of physiological variables measured along the patients’

hospitalization.

The hypothesis proposed consisted of combining a multimodel approach with features extracted

from the time series of physiological variables. It was aimed to demonstrate how the use of extracted

features can be a viable option when dealing with time series of variables and how more knowledge

can be obtained through these features than from the current value of each physiological variable alone.

Besides, it was sought to explore the advantages of using multimodels instead of single ones and to

explore the results obtained by combining feature extraction with a multimodel approach. Additionally, a

new hypothesis was proposed, the use of information from the entire patient’s hospitalization instead of

solely from the discharge days, has it had been done in previous works [11, 25].

Considering initially the entire set of samples, satisfactory results were obtained. At first, with the use

of prime features in a single model configuration, increases in the results could be noted through the

use of feature selection alone, demonstrating the power of this procedure. With the addition of extracted

features, i.e. with the addition to each sample of features considering the full hospitalization period so

far, some improvements were also achieved, both when compared to the use of the entire initial set

of extracted features, before feature selection, as to the previous use of prime features. In this way, it

was demonstrated how considering information from as far as the admission moment is advantageous

and important in predicting patients’ readmissions in ICUs, contributing with more relevant information

than the use of the current values of each variable alone. Finally, multimodels were developed, being

used different feature selection criteria to obtain two models, one highlighting sensitivity and the other

highlighting specificity and different decision criteria to transform the outputs of each model into a final

one. It was demonstrated how the best set of feature selection criteria used to obtain each model was

the sum of the weighted areas under the sensitivity and specificity curves, followed by the use of the

sensitivity and specificity values at the intersection threshold and, at last, by the use of the sensitivity

and specificity values close to that same intersection threshold. In terms of the decision criteria used,

the maximum distance was the one that presented most often best results, although a slight disfavor of

59

specificity could be noted. The weighted distance criterion also presented considerably good results,

often more even in terms of sensitivity and specificity, which can be sometimes desirable. At last, the

average criterion was the one that presented worst results, showing a slight better performance in terms

of specificity, which can also be used to one’s favor in particular situations.

The best model obtained was the one using the sum of the weighted areas under the sensitivity

and specificity curves to perform feature selection and the maximum distance as decision criterion. This

model achieved an AUC of 0.77±0.01, an accuracy of 0.71±0.02, a sensitivity of 0.71±0.03 and a speci-

ficity of 0.69±0.03. Although the accuracy, sensitivity and specificity are below those presented in [25],

the proposed approach is particularly well suited to be implemented in a systems engineering design

to provide prognostic guidance for the care of the critically ill in a daily basis. Furthermore, significant

better results were obtained in terms of AUC and sensitivity compared to the ones obtained in [11].

It can be concluded that, hypothetically, a clinician could use this model in real life situations at every

stage of a patient’s hospitalization to assist the decision of discharge, in a way that the readmission risk

is minimized.

When assessing the performance of this model in patients with varying lengths of stay, it could be

observed how for not readmitted patients the outcomes of the model tended do decrease as the dis-

charge day approached and how for readmitted patients the predicted values tended to oscillate around

one, showing a much less pronounced decline, if any, as the discharge day approached.

Additionally, in an attempt to obtain results comparable with previous works, the same procedure

was carried out, using solely discharge day samples in the testing sets. This alteration resulted in worse

results, having the best model been obtained for the use of extracted features in a single model con-

figuration where the AUC achieved a value of 0.73, the accuracy of 0.67, the sensitivity of 0.69 and

the specificity of 0.67. Such outcome can be due to the use of the entire hospitalization samples in

the training sets, contrarily to what was done in previous approaches, which could have decreased the

performance of the model as the training examples provided included more information than the specific

one associated to discharge days. Additionally, the low amount of samples from readmitted patients can

have also influenced the results, leading often to much uneven values of sensitivity and specificity and

to elevated standard deviation values of sensitivity. Such is due to the fact that the sole alteration by one

patient in the total of positive classifications changes considerably the sensitivity values. Nevertheless,

the development of multimodels, even if presenting worst performances, was useful to confirm the best

feature selection and decision criteria, as a similar ranking to the previous one was obtained in terms of

performance for both the feature selection and the decision criteria.

Overall, it can be concluded that the methods implemented in this work were successful, considering

the usefulness of predicting the risk of readmission at any stage of a patient’s hospitalization. Addition-

ally, it was demonstrated how the use of extracted features and multimodels can help improve the results

in the problem of predicting patients’ readmissions in ICUs.

60

7.1 Limitations

Through the development of this thesis some limitations were found.

For one, the attribution of the readmitted label to any sample obtained more than 24 hours before

discharge can be argued against. The doubt on the validity of this assumption can result from the fact

that no real evidence confirms the hypothesis that a patient would in fact be readmitted case he/she was

discharged any time before the real discharge day. However, this assumption was made considering that

clinicians opt not to discharge patients before time mostly due to the risks of readmission, increase of

illness and death. Therefore, considering that samples obtained before the real discharge day are similar

to samples from not readmitted patients is, to a certain point, justified. Furthermore, this assumption

was made in an attempt to obtain a model suitable for real world applications where one cannot know in

advance when the discharge day is, making it a reasonable approach to deal with this situation.

Additionally, due to the complexity and large size of the database, not all steps of the work were

repeated a sufficient number of times, or in the best way, to totally mitigate the influence of the selected

train and test samples. In the case of parameter selection, where the AUC values obtained were often

extremely close for different parameters, using the AUC measure with 5-fold cross-validation might not

have been enough to find the best combinations of parameters. Also in feature selection, where the

process was carried out solely 10 times and the best three subsets were kept, there is a lack of guarantee

that the best subsets were, indeed, found. Taking into account the presented limitations and other

research suggestions further work is presented in the next section.

7.2 Future Work

Considering the work developed in this thesis, some considerations and suggestions can be made

towards future work, since not all possible paths were explored, mostly due to the computational com-

plexity of the problem.

In terms of parameter selection, some performance measures more adequate than the AUC with

cross-validation can be used to find the appropriate number of clusters. For example, several indexes

can be computed to assess the suitability of different numbers of clusters, as the Separation [53] or the

Xie-Beni [54] indexes where the optimal number of clusters should minimize the value of the index.

Also in regarding feature selection, a more consistent method is proposed in an alternative to the

repetition of the process ten times and consequent selection of the subsets that performed better in

terms of the feature selection criterion chosen. The feature section process could be repeated a greater

amount of times, somewhere in the order of one hundred, and the subsets chosen considering the

number of times they were selected. That is because, the final value obtained for the chosen criteria can

be influenced by the training and test sets used, not necessarily corresponding to the best subsets of

features. Furthermore, the method used (SFS) is easily trapped in local optimal solutions, being unable

to guarantee the selection of the optimal set of features. Using the number of repetitions of a given

subset instead would partially solve this problem, since the larger the number of times a given set is

61

selected the greater is the certainty associated with its good performance.

One other suggestion for future work is proposed, the removal of the BMI feature. Such is recom-

mended due to the great amount of missing values of height and weight in the MIMIC II database that

led to a considerable reduction in the number of patients selected for the present study. Besides, since

the BMI feature was not always selected in the feature selection process, it is believed that its removal

would strongly benefit the results, especially in the case where the low amount of readmitted patients

conditioned the obtained results.

Finally, considering the performance of the best model obtained, further developments are proposed.

It was shown how the predicted outputs of this model tend do decrease in a more pronounced way as

the discharge day approaches for not readmitted patients and how these values tend mostly to oscillate

around one for readmitted patients. Considering this, it is believed that relevant results could be obtained

by studying in detail the predicted outputs of this model in a way to more accurately understand for what

output, or sequence of outputs, a patient should indeed be discharged. Additionally, further studies

could be developed relative to the features used in this model.

62

Bibliography

[1] Andrew McAfee and Erik Brynjolfsson. Big data: the management revolution. Harvard business

review, (90):60–6, 2012.

[2] Soumen Chakrabarti, Earl Cox, Eibe Frank, Ralf Hartmut Guting, Jiawei Han, Xia Jiang, Micheline

Kamber, Sam S Lightstone, Thomas P Nadeau, Richard E Neapolitan, et al. Data Mining: Know It

All: Know It All. Morgan Kaufmann, 2008.

[3] Dongsong Zhang and Lina Zhou. Discovering golden nuggets: data mining in financial applica-

tion. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on,

34(4):513–522, 2004.

[4] Mary K Obenshain. Application of data mining techniques to healthcare data. Infection Control,

25(08):690–695, 2004.

[5] Xiang Li, Beth Plale, Nithya Vijayakumar, Rahul Ramachandran, Sara Graves, and Helen Conover.

Real-time storm detection and weather forecast activation through data mining and events process-

ing. Earth Science Informatics, 1(2):49–57, 2008.

[6] Michael JA Berry and Gordon S Linoff. Data mining techniques: for marketing, sales, and customer

relationship management. John Wiley & Sons, 2004.

[7] Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth. From data mining to knowledge

discovery in databases. AI magazine, 17(3):37, 1996.

[8] Rainu Kaushal, David Blumenthal, Eric G Poon, Ashish K Jha, Calvin Franz, Blackford Middleton,

John Glaser, Gilad Kuperman, Melissa Christino, Rushika Fernandopulle, et al. The costs of a

national health information network. Annals of Internal Medicine, 143(3):165–173, 2005.

[9] Robert Babuska. Fuzzy modeling for control. Kluwer Academic Publishers, 1998.

[10] Robert Tibshirani, Trevor Hastie, Balasubramanian Narasimhan, and Gilbert Chu. Diagnosis of

multiple cancer types by shrunken centroids of gene expression. Proceedings of the National

Academy of Sciences, 99(10):6567–6572, 2002.

[11] Andre S Fialho, Federico Cismondi, Susana M Vieira, Shane R Reti, Joao MC Sousa, and Stan N

Finkelstein. Data mining using clinical physiology at discharge to predict icu readmissions. Expert

Systems with Applications, 39(18):13158–13165, 2012.

63

[12] Donald B Chalfin, Stephen Trzeciak, Antonios Likourezos, Brigitte M Baumann, R Phillip Dellinger,

DELAY-ED study group, et al. Impact of delayed transfer of critically ill patients from the emergency

department to the intensive care unit*. Critical care medicine, 35(6):1477–1483, 2007.

[13] Malcolm Elliott, Linda Worrall-Carter, and Karen Page. Intensive care readmission: A contemporary

review of the literature. Intensive and Critical Care Nursing, 30(3):121–137, 2014.

[14] Nazima Allaudeen, Jeffrey L Schnipper, E John Orav, Robert M Wachter, and Arpana R Vidyarthi.

Inability of providers to predict unplanned readmissions. Journal of general internal medicine,

26(7):771–776, 2011.

[15] Andrew L Rosenberg and Charles Watts. Patients readmitted to icus*: a systematic review of risk

factors and outcomes. CHEST Journal, 118(2):492–502, 2000.

[16] Sujin Kim, Woojae Kim, and Rae Woong Park. A comparison of intensive care unit mortality

prediction models through the use of data mining techniques. Healthcare informatics research,

17(4):232–243, 2011.

[17] Alison J Campbell, Jonathan Alistair Cook, Gillian Adey, and Brian Cuthbertson. Predicting death

and readmission after intensive care discharge. 2008.

[18] William A Knaus, Elizabeth A Draper, Douglas P Wagner, and Jack E Zimmerman. Apache ii: a

severity of disease classification system. Critical care medicine, 13(10):818–829, 1985.

[19] Andrew L Rosenberg, Timothy P Hofer, Rodney A Hayward, Cathy Strachan, and Charles M Watts.

Who bounces back? physiologic and other predictors of intensive care unit readmission. Critical

care medicine, 29(3):511–518, 2001.

[20] Philipp GH Metnitz, Fabienne Fieux, Barbara Jordan, Thomas Lang, Rui Moreno, and Jean-Roger

Le Gall. Critically ill patients readmitted to intensive care units—lessons to learn? Intensive care

medicine, 29(2):241–248, 2003.

[21] Andre S Fialho, Uzay Kaymak, Federico Cismondi, Susana Margarida Vieira, Shane R Reti,

Joao MC Sousa, and Stan N Finkelstein. Predicting intensive care unit readmissions using prob-

abilistic fuzzy systems. In Fuzzy Systems (FUZZ), 2013 IEEE International Conference on, pages

1–7. IEEE, 2013.

[22] Henian Xia, Brian J Daley, Adam Petrie, and Xiaopeng Zhao. A neural network model for mortality

prediction in icu. Proc. Computing In Cardiology, 39:261–264, 2012.

[23] Yun Chen and Hui Yang. Heterogeneous postsurgical data analytics for predictive modeling of

mortality risks in intensive care units. In Engineering in Medicine and Biology Society (EMBC),

2014 36th Annual International Conference of the IEEE, pages 4310–4314. IEEE, 2014.

[24] Jose Miguel Mourinho Rodrigues. Data mining and modeling to predict the necessity of vasopres-

sors for sepsis patients. 2013.

64

[25] Marta PB Fernandes, Claudia F Silva, Susana M Vieira, and Joao Sousa. Multimodeling for the

prediction of patient readmissions in intensive care units. In Fuzzy Systems (FUZZ-IEEE), 2014

IEEE International Conference on, pages 1837–1842. IEEE, 2014.

[26] Federico Cismondi, Abigail L Horn, Andre S Fialho, Susana M Vieira, Shane R Reti, Joao MC

Sousa, and Stan Finkelstein. Multi-stage modeling using fuzzy multi-criteria feature selection

to improve survival prediction of icu septic shock patients. Expert Systems with Applications,

39(16):12332–12339, 2012.

[27] Nura Esfandiari, Mohammad Reza Babavalian, Amir-Masoud Eftekhari Moghadam, and

Vahid Kashani Tabar. Knowledge discovery in medicine: Current issue and future trend. Expert

Systems with Applications, 41(9):4434–4463, 2014.

[28] Kari Torkkola. Feature extraction by non parametric mutual information maximization. The Journal

of Machine Learning Research, 3:1415–1438, 2003.

[29] Isabelle Guyon. Feature extraction: foundations and applications, volume 207. Springer Science &

Business Media, 2006.

[30] Pierre A Devijver and Josef Kittler. Pattern recognition: A statistical approach, volume 761.

Prentice-Hall London, 1982.

[31] Isabelle Guyon and Andre Elisseeff. An introduction to variable and feature selection. The Journal

of Machine Learning Research, 3:1157–1182, 2003.

[32] Silvia Casado Yusta. Different metaheuristic strategies to solve the feature selection problem.

Pattern Recognition Letters, 30(5):525–534, 2009.

[33] Luıs F Mendonca, Susana M Vieira, and JMC Sousa. Decision tree search methods in fuzzy

modeling and classification. International Journal of Approximate Reasoning, 44(2):106–123, 2007.

[34] Ludmila I Kuncheva. Combining pattern classifiers: methods and algorithms. John Wiley & Sons,

2004.

[35] Ewout W Steyerberg, Andrew J Vickers, Nancy R Cook, Thomas Gerds, Mithat Gonen, Nancy

Obuchowski, Michael J Pencina, and Michael W Kattan. Assessing the performance of prediction

models: a framework for some traditional and novel measures. Epidemiology (Cambridge, Mass.),

21(1):128, 2010.

[36] Thomas A Lasko, Jui G Bhagwat, Kelly H Zou, and Lucila Ohno-Machado. The use of receiver oper-

ating characteristic curves in biomedical informatics. Journal of biomedical informatics, 38(5):404–

415, 2005.

[37] Daniel J Scott, Joon Lee, Ikaro Silva, Shinhyuk Park, George B Moody, Leo A Celi, and Roger G

Mark. Accessing the public mimic-ii intensive care relational database for clinical research. BMC

medical informatics and decision making, 13(1):9, 2013.

65

[38] Derek C Angus. Grappling with intensive care unit quality-does the readmission rate tell us any-

thing? Critical care medicine, 26(11):1779–1780, 1998.

[39] Anthony S Fauci et al. Harrison’s principles of internal medicine, volume 2. McGraw-Hill Medical

New York, 2008.

[40] Maite Garrouste-Orgeas, Gilles Troche, Elie Azoulay, Antoine Caubel, Arnaud de Lassence, Chris-

tine Cheval, Laurent Montesino, Marie Thuong, Francois Vincent, Yves Cohen, et al. Body mass

index. Intensive care medicine, 30(3):437–443, 2004.

[41] Federico Cismondi, Andre S Fialho, Susana M Vieira, Joao MC Sousa, Shane R Reti, Michael D

Howell, and Stan N Finkelstein. Computational intelligence methods for processing misaligned,

unevenly sampled time series containing missing data. In Computational Intelligence and Data

Mining (CIDM), 2011 IEEE Symposium on, pages 224–231. IEEE, 2011.

[42] Andries P Engelbrecht. Computational intelligence: an introduction. John Wiley & Sons, 2007.

[43] Robert Babuska. Fuzzy systems, modeling and identification. 2001.

[44] Joao MC Sousa and Uzay Kaymak. Fuzzy decision making in modeling and control, volume 27.

2002.

[45] Tomohiro Takagi and Michio Sugeno. Fuzzy identification of systems and its applications to model-

ing and control. Systems, Man and Cybernetics, IEEE Transactions on, (1):116–132, 1985.

[46] Uzay Kaymak and Magne Setnes. Extended fuzzy clustering algorithms. ERIM Report Series

Reference No. ERS-2001-51-LIS, 2000.

[47] James C Bezdek. Pattern recognition with fuzzy objective function algorithms. Kluwer Academic

Publishers, 1981.

[48] Joseph C Dunn. A fuzzy relative of the isodata process and its use in detecting compact well-

separated clusters. 1973.

[49] D. Gustafson and W. Kessel. Fuzzy clustering with a fuzzy covariance matrix. In Proc. IEEE CDC,

pages 761–766, 1979.

[50] Kenneth P Burnham and David R Anderson. Model selection and multimodel inference: a practical

information-theoretic approach. Springer Science & Business Media, 2002.

[51] Mohammed Chadli and Pierre Borne. Multiple models approach in automation: Takagi-Sugeno

fuzzy systems. John Wiley & Sons, 2012.

[52] Claudia Tebaldi and Reto Knutti. The use of the multi-model ensemble in probabilistic climate

projections. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engi-

neering Sciences, 365(1857):2053–2075, 2007.

66

[53] Amine M Bensaid, Lawrence O Hall, James C Bezdek, Laurence P Clarke, Martin L Silbiger, John A

Arrington, and Reed F Murtagh. Validity-guided (re) clustering with applications to image segmen-

tation. Fuzzy Systems, IEEE Transactions on, 4(2):112–123, 1996.

[54] Xuanli Lisa Xie and Gerardo Beni. A validity measure for fuzzy clustering. IEEE Transactions on

pattern analysis and machine intelligence, 13(8):841–847, 1991.

67

Appendix A

Daily Evaluation

Following are presented the extended results obtained for the different approaches implemented in

the readmissions dataset. Initially, in Section A.1, the results obtained for the single models with prime

features are presented. In Section A.2, the results obtained for the single models with extracted features

are then presented and in Section A.3 the results obtained for the multimodels with extracted features

are presented.

A.1 Single Model with Prime Features

Table A.1: Prime features acquired from the MIMIC II database.

# Feature

1 Age

2 Gender

3 Number of days since admission in the ICU

4 BMI

5 Creatinine (0 - 1.3)

6 Lactic Acid (0.5 - 2)

7 NBP Mean

8 Platelets

9 Temperature

10 Heart Rate

11 SpO2

A1

Table A.2: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with prime features, varying the number of clusters between 2 and 10

and the degree of fuzziness between 1.1 and 1.5.

Degree of Fuzziness

1.1 1.2 1.3 1.4 1.5

2 0.69 ± 0.03 0.69 ± 0.03 0.69 ± 0.02 0.68 ± 0.03 0.68 ± 0.03

3 0.69 ± 0.03 0.69 ± 0.03 0.69 ± 0.02 0.69 ± 0.03 0.69 ± 0.03

4 0.68 ± 0.03 0.69 ± 0.03 0.69 ± 0.03 0.69 ± 0.02 0.70 ± 0.02

Number 5 0.69 ± 0.03 0.68 ± 0.03 0.68 ± 0.03 0.69 ± 0.02 0.70 ± 0.02

of 6 0.68 ± 0.03 0.69 ± 0.03 0.68 ± 0.03 0.69 ± 0.03 0.69 ± 0.02

Clusters 7 0.68 ± 0.03 0.69 ± 0.03 0.68 ± 0.03 0.68 ± 0.02 0.70 ± 0.02

8 0.68 ± 0.02 0.69 ± 0.03 0.68 ± 0.02 0.68 ± 0.02 0.70 ± 0.02

9 0.68 ± 0.02 0.69 ± 0.03 0.68 ± 0.02 0.68 ± 0.03 0.70 ± 0.02

10 0.68 ± 0.03 0.69 ± 0.02 0.68 ± 0.03 0.68 ± 0.03 0.68 ± 0.02

Table A.3: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with prime features, varying the number of clusters between 2 and 10

and the degree of fuzziness between 1.6 and 2.

Degree of Fuzziness

1.6 1.7 1.8 1.9 2

2 0.68 ± 0.03 0.68 ± 0.03 0.68 ± 0.03 0.68 ± 0.03 0.68 ± 0.03

3 0.70 ± 0.03 0.70 ± 0.03 0.70 ± 0.03 0.69 ± 0.03 0.69 ± 0.03

4 0.71 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.03 0.69 ± 0.03

Number 5 0.70 ± 0.02 0.70 ± 0.03 0.70 ± 0.03 0.69 ± 0.02 0.69 ± 0.03

of 6 0.70 ± 0.02 0.70 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02

Clusters 7 0.70 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02

8 0.70 ± 0.03 0.70 ± 0.03 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02

9 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02

10 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.70 ± 0.03

Table A.4: Prime features obtained for each subset, after SFS with 5-fold cross-validation.

Subset Features

1 3, 4, 5, 7, 8, 9, 11

2 3, 5, 7, 8, 9, 10, 11

3 3, 5, 6, 7, 8, 9, 10, 11

A2

A.2 Single Model with Extracted Features

Table A.5: Extracted features acquired from the MIMIC II database.

# Feature # Feature

1 Age 28 Maximum Platelets value

2 Gender 29 Time of maximum Platelets value

3 Number of days since admission in the ICU 30 Minimum Platelets value

4 BMI 31 Time of minimum Platelets value

5 Last Creatinine (0 - 1.3) value 32 Mean Platelets recordedvalues

6 First Creatinine (0 - 1.3) value 33 Last Temperature value

7 Maximum Creatinine (0 - 1.3) value 34 First Temperature value

8 Time of maximum Creatinine (0 - 1.3) value 35 Maximum Temperature value

9 Minimum Creatinine (0 - 1.3) value 36 Time of maximum Temperature value

10 Time of minimum Creatinine (0 - 1.3) value 37 Minimum Temperature value

11 Mean Creatinine (0 - 1.3) values 38 Time of minimum Temperature value

12 Last Lactic Acid (0.5 - 2) value 39 Mean Temperature recorded values

13 First Lactic Acid (0.5 - 2) value 40 Last Heart Rate value

14 Maximum Lactic Acid (0.5 - 2) value 41 Firt Heart Rate value

15 Time of maximum Lactic Acid (0.5 - 2) value 42 Maximum Heart Rate value

16 Minimum Lactic Acid (0.5 - 2) value 43 Time of maximum Heart Rate value

17 Time of minimum Lactic Acid (0.5 - 2) value 44 Minimum Heart Rate value

18 First Lactic Acid (0.5 - 2) value 45 Time of minimum Heart Rate value

19 Last NBP Mean value 46 Mean Heart Rate values

20 First NBP Mean value 47 Last SpO2 value

21 Maximum NBP Mean value 48 First SpO2 value

22 Time of maximum NBP Mean value 49 Maximum SpO2 value

23 Minimum NBP Mean value 50 Time of maximum SpO2 value

24 Time of minimum NBP Mean value 51 Minimum SpO2 value

25 Mean NBP Mean values 52 Time of minimum SpO2 value

26 Last Platelets value 53 Mean SpO2 recorded values

27 First Platelets value

A3

Table A.6: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with extracted features, varying the number of clusters between 2

and 10 and the degree of fuzziness between 1.1 and 1.5.

Degree of Fuzziness

1.1 1.2 1.3 1.4 1.5

2 0.71 ± 0.03 0.72 ± 0.02 0.73 ± 0.02 0.71 ± 0.03 0.72 ± 0.02

3 0.71 ± 0.02 0.73 ± 0.02 0.73 ± 0.03 0.73 ± 0.03 0.74 ± 0.02

4 0.71 ± 0.03 0.72 ± 0.03 0.72 ± 0.03 0.71 ± 0.02 0.72 ± 0.03

Number 5 0.71 ± 0.03 0.71 ± 0.02 0.69 ± 0.02 0.71 ± 0.02 0.72 ± 0.03

of 6 0.69 ± 0.04 0.69 ± 0.03 0.71 ± 0.02 0.71 ± 0.01 0.70 ± 0.02

Culsters 7 0.70 ± 0.03 0.70 ± 0.02 0.71 ± 0.02 0.70 ± 0.03 0.71 ± 0.01

8 0.69 ± 0.02 0.70 ± 0.02 0.69 ± 0.01 0.69 ± 0.02 0.68 ± 0.03

9 0.69 ± 0.03 0.68 ± 0.03 0.68 ± 0.02 0.71 ± 0.02 0.69 ± 0.03

10 0.67 ± 0.01 0.69 ± 0.03 0.69 ± 0.03 0.68 ± 0.03 0.70 ± 0.02

Table A.7: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with extracted features, varying the number of clusters between 2

and 10 and the degree of fuzziness between 1.6 and 2.

Degree of Fuzziness

1.6 1.7 1.8 1.9 2

2 0.71 ± 0.03 0.72 ± 0.02 0.72 ± 0.02 0.72 ± 0.02 0.72 ± 0.02

3 0.73 ± 0.02 0.72 ± 0.03 0.72 ± 0.01 0.72 ± 0.02 0.72 ± 0.02

4 0.73 ± 0.02 0.73 ± 0.03 0.72 ± 0.02 0.73 ± 0.02 0.72 ± 0.02

Number 5 0.71 ± 0.02 0.72 ± 0.02 0.71 ± 0.03 0.73 ± 0.02 0.72 ± 0.01

of 6 0.70 ± 0.02 0.70 ± 0.02 0.71 ± 0.02 0.72 ± 0.01 0.72 ± 0.01

Clusters 7 0.70 ± 0.02 0.70 ± 0.02 0.70 ± 0.02 0.71 ± 0.02 0.70 ± 0.02

8 0.70 ± 0.02 0.71 ± 0.01 0.69 ± 0.03 0.70 ± 0.03 0.69 ± 0.02

9 0.68 ± 0.03 0.70 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.70 ± 0.02

10 0.69 ± 0.02 0.69 ± 0.03 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02

Table A.8: Extracted features obtained for each subset, after SFS with 5-fold cross-validation.

Subset Features

1 1, 3, 4, 6, 11, 12, 15, 17, 19, 22, 23, 24, 25, 32, 33, 35, 36, 38, 41, 42, 43, 44, 45, 47, 51

2 3, 6, 7, 11, 12, 15, 17, 19, 21, 22, 23, 24, 29, 33, 34, 35, 36, 37, 38, 40, 41, 42, 46, 47, 51, 52

3 3, 6, 11, 12, 15, 17, 19, 21, 22, 23, 24, 33, 35, 36, 38, 42, 47, 53

A4

A.3 Multimodel with Extracted Features

Table A.9: Extracted features obtained for each subset, by performing SFS with the first set of featureselection criteria, with 5-fold cross-validation.

Subset Features

Sensitivity1 9, 15, 17, 19, 21, 22, 24, 28, 33, 35, 36, 38, 42, 47

2 3, 9, 15, 17, 19, 22, 24, 33, 35, 42, 47, 51

3 13, 17, 19, 21, 24, 28, 33, 35, 36, 38, 42, 47, 51

Specificity1 9, 12, 15, 17, 19, 20, 21, 22, 24, 33, 35, 36, 38, 42, 47

2 9, 15, 17, 19, 21, 22, 24, 28, 33, 35, 36, 38, 42, 47

3 9, 15, 17, 19, 22, 24, 30, 32, 33, 35, 36, 38, 40, 42, 47, 48

Table A.10: Extracted features obtained for each subset, by performing SFS with the second set offeature selection criteria, with 5-fold cross-validation.

Subset Features

Sensitivity1 5, 8, 10, 11, 12, 15, 17, 19, 21, 22, 23, 29, 31, 33, 35, 38, 44, 47, 48, 49, 53

2 5, 15, 17, 19, 33, 35, 44, 47, 49, 50

3 11, 12, 17, 19, 28, 30, 33, 36, 42, 47, 49, 51

Specificity1 3, 12, 14, 15, 17, 20, 22, 31, 33, 39, 40, 47, 53

2 5, 7, 15, 17, 19, 20, 22, 27, 33, 34, 38, 47, 48, 50, 53

3 3, 4, 9, 12, 13, 15, 17, 19, 22, 29, 31, 32, 33, 34, 36, 38, 46, 47, 53

Table A.11: Extracted features obtained for each subset, by performing SFS with the third set of featureselection criteria, with 5-fold cross-validation.

Subset Features

Sensitivity1 5, 9, 15, 19, 20, 28, 33, 42, 44, 47, 48, 49, 50

2 3, 9, 12, 13, 14, 17, 19, 20, 27, 33, 35, 37, 47, 49, 50

3 3, 9, 12, 17, 24, 27, 33, 38, 43, 48, 49, 50

Specificity1 3, 8, 9, 11, 13, 15, 18, 21, 22, 25, 26, 29, 30, 33, 47, 49

2 3, 8, 9, 12, 13, 17, 18, 19, 22, 28, 33, 34, 37, 39, 47, 49

3 1, 3, 8, 12, 13, 14, 15, 18, 22, 26, 29, 30, 33, 47, 49

A5

Table A.12: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the first set of feature selection criteria.


AUC 0.76 ± 0.02 0.76 ± 0.03 0.76 ± 0.02

Sensitivity subset 1 Accuracy 0.69 ± 0.03 0.70 ± 0.03 0.68 ± 0.03


Specificity 0.70 ± 0.05 0.68 ± 0.05 0.72 ± 0.05

AUC 0.77 ± 0.02 0.76 ± 0.01 0.77 ± 0.01



Specificity 0.70 ± 0.03 0.69 ± 0.03 0.71 ± 0.03

AUC 0.77 ± 0.02 0.77 ± 0.01 0.77 ± 0.01



Specificity 0.71 ± 0.03 0.69 ± 0.03 0.73 ± 0.03

AUC 0.77 ± 0.02 0.76 ± 0.02 0.77 ± 0.02



Specificity 0.70 ± 0.04 0.67 ± 0.04 0.73 ± 0.04

AUC 0.77 ± 0.02 0.76 ± 0.02 0.77 ± 0.02



Specificity 0.69 ± 0.04 0.65 ± 0.04 0.70 ± 0.04

AUC 0.76 ± 0.02 0.76 ± 0.02 0.77 ± 0.02



Specificity 0.70 ± 0.03 0.67 ± 0.04 0.73 ± 0.03

AUC 0.77 ± 0.02 0.76 ± 0.02 0.77 ± 0.02



Specificity 0.70 ± 0.03 0.67 ± 0.03 0.73 ± 0.03

AUC 0.77 ± 0.03 0.76 ± 0.03 0.77 ± 0.03



Specificity 0.70 ± 0.03 0.67 ± 0.03 0.70 ± 0.03

AUC 0.77 ± 0.01 0.76 ± 0.01 0.77 ± 0.01



Specificity 0.70 ± 0.03 0.68 ± 0.03 0.73 ± 0.03

A6

Table A.13: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the second set of feature selection criteria.


AUC 0.76 ± 0.01 0.75 ± 0.02 0.76 ± 0.01



Specificity 0.69 ± 0.03 0.67 ± 0.03 0.72 ± 0.03

AUC 0.75 ± 0.02 0.75 ± 0.02 0.75 ± 0.02



Specificity 0.70 ± 0.03 0.67 ± 0.03 0.73 ± 0.03

AUC 0.76 ± 0.01 0.76 ± 0.01 0.76 ± 0.01



Specificity 0.71 ± 0.03 0.68 ± 0.03 0.73 ± 0.03

AUC 0.74 ± 0.02 0.74 ± 0.02 0.75 ± 0.02



Specificity 0.68 ± 0.04 0.64 ± 0.04 0.72 ± 0.03

AUC 0.74 ± 0.02 0.74 ± 0.02 0.74 ± 0.02



Specificity 0.69 ± 0.04 0.65 ± 0.04 0.72 ± 0.03

AUC 0.75 ± 0.02 0.75 ± 0.02 0.76 ± 0.02



Specificity 0.69 ± 0.04 0.65 ± 0.05 0.74 ± 0.04

AUC 0.75 ± 0.02 0.75 ± 0.02 0.76 ± 0.02



Specificity 0.69 ± 0.02 0.65 ± 0.03 0.73 ± 0.03

AUC 0.75 ± 0.02 0.74 ± 0.02 0.75 ± 0.02



Specificity 0.69 ± 0.04 0.65 ± 0.04 0.73 ± 0.04

AUC 0.76 ± 0.02 0.76 ± 0.02 0.76 ± 0.02



Specificity 0.70 ± 0.03 0.67 ± 0.03 0.73 ± 0.03

A7

Table A.14: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the third set of feature selection criteria.


AUC 0.73 ± 0.02 0.73 ± 0.02 0.74 ± 0.02



Specificity 0.68 ± 0.04 0.64 ± 0.04 0.73 ± 0.04

AUC 0.75 ± 0.02 0.75 ± 0.02 0.75 ± 0.02



Specificity 0.70 ± 0.03 0.66 ± 0.03 0.74 ± 0.03

AUC 0.73 ± 0.02 0.73 ± 0.02 0.74 ± 0.02



Specificity 0.68 ± 0.04 0.64 ± 0.04 0.73 ± 0.04

AUC 0.74 ± 0.02 0.73 ± 0.02 0.74 ± 0.02



Specificity 0.68 ± 0.03 0.64 ± 0.04 0.73 ± 0.03

AUC 0.74 ± 0.02 0.74 ± 0.02 0.74 ± 0.02



Specificity 0.70 ± 0.03 0.66 ± 0.03 0.73 ± 0.03

AUC 0.73 ± 0.02 0.73 ± 0.02 0.74 ± 0.02



Specificity 0.68 ± 0.04 0.64 ± 0.03 0.73 ± 0.04

AUC 0.75 ± 0.02 0.74 ± 0.02 0.75 ± 0.02



Specificity 0.69 ± 0.04 0.64 ± 0.04 0.74 ± 0.04

AUC 0.75 ± 0.02 0.75 ± 0.02 0.75 ± 0.02



Specificity 0.70 ± 0.03 0.66 ± 0.03 0.74 ± 0.04

AUC 0.74 ± 0.02 0.74 ± 0.02 0.75 ± 0.02



Specificity 0.68 ± 0.05 0.64 ± 0.05 0.73 ± 0.05

A8

Appendix B

Discharge Day Evaluation

In this section are presented the extended results obtained for the different approaches implemented,

using only the discharge day samples in the test set. In Section B.1 are presented the results obtained

for the single models with prime features, in Section B.2 the results obtained for the single models with

extracted features and in Section B.3 the results obtained for the multimodels with extracted features.

B.1 Single Model with Prime Features

Table B.1: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with extracted features, varying the number of clusters between 2


Degree of Fuzziness

1.1 1.2 1.3 1.4 1.5

2 0.63 ± 0.05 0.64 ± 0.06 0.64 ± 0.06 0.63 ± 0.07 0.62 ± 0.05

3 0.59 ± 0.07 0.62 ± 0.08 0.64 ± 0.07 0.66 ± 0.06 0.64 ± 0.06

4 0.61 ± 0.07 0.61 ± 0.08 0.65 ± 0.06 0.64 ± 0.07 0.65 ± 0.06

Number 5 0.63 ± 0.06 0.59 ± 0.08 0.59 ± 0.09 0.63 ± 0.05 0.64 ± 0.06

of 6 0.64 ± 0.08 0.61 ± 0.08 0.63 ± 0.07 0.62 ± 0.07 0.63 ± 0.06

Culsters 7 0.62 ± 0.07 0.60 ± 0.07 0.65 ± 0.07 0.60 ± 0.09 0.63 ± 0.06

8 0.63 ± 0.06 0.64 ± 0.08 0.61 ± 0.07 0.63 ± 0.06 0.62 ± 0.06

9 0.61 ± 0.08 0.62 ± 0.08 0.63 ± 0.08 0.61 ± 0.07 0.58 ± 0.06

10 0.61 ± 0.06 0.65 ± 0.06 0.61 ± 0.05 0.58 ± 0.08 0.61 ± 0.05

B1

Table B.2: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with prime features, varying the number of clusters between 2 and 10

and the degree of fuzziness between 1.6 and 2.

Degree of Fuzziness

1.6 1.7 1.8 1.9 2

2 0.61 ± 0.06 0.61 ± 0.06 0.61 ± 0.05 0.62 ± 0.04 0.62 ± 0.04

3 0.63 ± 0.05 0.62 ± 0.05 0.61 ± 0.04 0.60 ± 0.04 0.61 ± 0.05

4 0.63 ± 0.06 0.62 ± 0.04 0.62 ± 0.06 0.63 ± 0.06 0.62 ± 0.06

Number 5 0.63 ± 0.06 0.63 ± 0.06 0.63 ± 0.06 0.63 ± 0.07 0.63 ± 0.06

of 6 0.64 ± 0.06 0.63 ± 0.07 0.63 ± 0.07 0.62 ± 0.06 0.63 ± 0.06

Clusters 7 0.63 ± 0.06 0.63 ± 0.07 0.63 ± 0.07 0.64 ± 0.07 0.64 ± 0.07

8 0.64 ± 0.07 0.63 ± 0.07 0.63 ± 0.06 0.64 ± 0.07 0.64 ± 0.06

9 0.63 ± 0.07 0.63 ± 0.07 0.63 ± 0.06 0.63 ± 0.07 0.65 ± 0.07

10 0.63 ± 0.06 0.63 ± 0.07 0.63 ± 0.08 0.64 ± 0.06 0.63 ± 0.06

Table B.3: Prime features obtained for each subset, after SFS with 5-fold cross-validation.

Subset Features

1 3, 4, 5, 7, 8

2 3, 4, 5

3 3, 4, 5, 8

B.2 Single Model with Extracted Features



Degree of Fuzziness

1.1 1.2 1.3 1.4 1.5

2 0.61 ± 0.07 0.62 ± 0.08 0.61 ± 0.05 0.63 ± 0.07 0.62 ± 0.08

3 0.61 ± 0.10 0.64 ± 0.07 0.62 ± 0.08 0.64 ± 0.07 0.64 ± 0.06

4 0.66 ± 0.06 0.62 ± 0.07 0.64 ± 0.06 0.62 ± 0.06 0.61 ± 0.06

Number 5 0.64 ± 0.08 0.65 ± 0.04 0.63 ± 0.09 0.64 ± 0.04 0.61 ± 0.04

of 6 0.61 ± 0.11 0.58 ± 0.06 0.61 ± 0.10 0.59 ± 0.08 0.58 ± 0.07

Culsters 7 0.60 ± 0.09 0.63 ± 0.07 0.61 ± 0.07 0.59 ± 0.04 0.55 ± 0.05

8 0.58 ± 0.09 0.61 ± 0.05 0.62 ± 0.06 0.59 ± 0.06 0.57 ± 0.03

9 0.61 ± 0.10 0.60 ± 0.09 0.59 ± 0.05 0.56 ± 0.05 0.58 ± 0.08

10 0.60 ± 0.09 0.58 ± 0.08 0.59 ± 0.04 0.55 ± 0.03 0.57 ± 0.08

B2


and 10 and the degree of fuzziness between 1.6 and 2.

Degree of Fuzziness

1.6 1.7 1.8 1.9 2

2 0.63 ± 0.06 0.62 ± 0.07 0.63 ± 0.07 0.63 ± 0.07 0.63 ± 0.07

3 0.63 ± 0.06 0.62 ± 0.05 0.62 ± 0.06 0.61 ± 0.06 0.63 ± 0.06

4 0.61 ± 0.06 0.61 ± 0.06 0.61 ± 0.07 0.62 ± 0.05 0.62 ± 0.07

Number 5 0.60 ± 0.07 0.60 ± 0.08 0.61 ± 0.05 0.61 ± 0.06 0.61 ± 0.06

of 6 0.59 ± 0.06 0.61 ± 0.06 0.62 ± 0.06 0.60 ± 0.05 0.61 ± 0.07

Clusters 7 0.58 ± 0.06 0.59 ± 0.05 0.59 ± 0.05 0.61 ± 0.06 0.61 ± 0.08

8 0.58 ± 0.07 0.59 ± 0.05 0.58 ± 0.07 0.59 ± 0.06 0.57 ± 0.06

9 0.58 ± 0.04 0.59 ± 0.06 0.59 ± 0.04 0.60 ± 0.03 0.58 ± 0.03

10 0.58 ± 0.05 0.56 ± 0.04 0.58 ± 0.04 0.58 ± 0.05 0.57 ± 0.04

Table B.6: Extracted features obtained for each subset, after SFS with 5-fold cross-validation.

Subset Features

1 4, 6, 7, 9, 10, 12, 13, 14, 15, 18, 22, 23, 24, 26, 31, 36, 41, 45, 46, 51

2 3, 12, 22, 26, 51

3 3, 8, 23, 24, 49, 50

B.3 Multimodel with Extracted Features

Table B.7: Extracted features obtained for each subset, by performing SFS with the first set of featureselection criteria, with 5-fold cross-validation.

Subset Features

Sensitivity1 9, 11, 18, 21, 22, 23, 24, 28, 30, 31, 41, 45, 46, 48, 49, 50, 51, 52

2 14, 19, 25, 30, 31, 36, 40, 42, 43, 44, 50, 51

3 5, 10, 13, 15, 18, 21, 23, 28, 31, 41, 48, 51, 52

Specificity1 1, 8, 10, 15, 16, 17, 19, 22, 24, 25, 31, 33, 35, 36, 38, 42, 45, 46, 47, 49, 51

2 3, 7, 14, 16, 17, 19, 21, 22, 24, 32, 33, 36, 46, 47, 51, 52

3 1, 8, 15, 16, 17, 20, 21, 22, 29, 33, 35, 41, 42, 47, 48, 49, 51, 52

B3

Table B.8: Extracted features obtained for each subset, by performing SFS with the second set offeature selection criteria, with 5-fold cross-validation.

Subset Features

Sensitivity1 5, 8, 11, 12, 14, 20, 24, 30, 31, 40, 46, 48, 49, 51, 52

2 3, 5, 6, 7, 10, 12, 15, 16, 19, 23, 26, 31, 32, 36, 38, 43, 44, 45, 48, 49

3 3, 5, 6, 10, 18, 22, 23, 24, 25, 27, 30, 31, 41, 44, 49, 51

Specificity1 7, 10, 15, 20, 23, 24, 29, 31, 41, 44, 46, 49, 51, 53

2 10, 13, 18, 20, 22, 24, 26, 36, 46, 51

3 5, 6, 7, 10, 18, 26, 30, 31, 43, 49, 52

Table B.9: Extracted features obtained for each subset, by performing SFS with the third set of featureselection criteria, with 5-fold cross-validation.

Subset Features

Sensitivity1 3, 6, 9, 11, 13, 24, 31, 32, 38, 48, 49, 51, 52

2 5, 6, 9, 18, 20, 38, 49, 50

3 9, 10, 12, 13, 18, 26, 30, 37, 49, 50

Specificity1 3, 6, 7, 8, 9, 12, 13, 14, 15, 18, 21, 22, 23, 24, 25, 26, 31, 32, 38, 41, 45, 49, 50, 52

2 3, 10, 13, 14, 15, 18, 20, 22, 23, 24, 26, 27, 30, 31, 34, 36, 38, 41, 44, 49, 50, 52

3 5, 8, 11, 12, 13, 15, 24, 25, 26, 27, 31, 37, 41, 48, 49, 50

B4

Table B.10: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the first set of feature selection criteria.


AUC 0.67 ± 0.09 0.66 ± 0.09 0.67 ± 0.09



Specificity 0.69 ± 0.03 0.63 ± 0.03 0.75 ± 0.03

AUC 0.67 ± 0.09 0.67 ± 0.09 0.67 ± 0.09



Specificity 0.69 ± 0.04 0.64 ± 0.04 0.74 ± 0.03

AUC 0.65 ± 0.11 0.65 ± 0.10 0.65 ± 0.11



Specificity 0.69 ± 0.03 0.63 ± 0.04 0.75 ± 0.04

AUC 0.65 ± 0.09 0.64 ± 0.09 0.65 ± 0.08



Specificity 0.67 ± 0.04 0.61 ± 0.04 0.74 ± 0.05

AUC 0.66 ± 0.11 0.66 ± 0.12 0.67 ± 0.11



Specificity 0.68 ± 0.04 0.62 ± 0.04 0.74 ± 0.04

AUC 0.66 ± 0.08 0.65 ± 0.08 0.66 ± 0.08



Specificity 0.68 ± 0.03 0.62 ± 0.03 0.74 ± 0.03

AUC 0.67 ± 0.07 0.65 ± 0.08 0.68 ± 0.07



Specificity 0.68 ± 0.04 0.62 ± 0.04 0.75 ± 0.04

AUC 0.69 ± 0.08 0.69 ± 0.08 0.70 ± 0.08



Specificity 0.68 ± 0.04 0.63 ± 0.04 0.75 ± 0.04

AUC 0.67 ± 0.10 0.67 ± 0.11 0.68 ± 0.10



Specificity 0.68 ± 0.04 0.63 ± 0.04 0.73 ± 0.04

B5

Table B.11: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the second set of feature selection criteria.


AUC 0.67 ± 0.10 0.66 ± 0.04 0.66 ± 0.09



Specificity 0.65 ± 0.04 0.61 ± 0.04 0.71 ± 0.04

AUC 0.67 ± 0.09 0.67 ± 0.09 0.67 ± 0.08



Specificity 0.66 ± 0.04 0.61 ± 0.04 0.71 ± 0.05

AUC 0.68 ± 0.08 0.67 ± 0.08 0.68 ± 0.08



Specificity 0.63 ± 0.05 0.61 ± 0.05 0.70 ± 0.05

AUC 0.67 ± 0.08 0.67 ± 0.08 0.68 ± 0.08



Specificity 0.66 ± 0.05 0.61 ± 0.05 0.72 ± 0.06

AUC 0.69 ± 0.05 0.69 ± 0.05 0.68 ± 0.05



Specificity 0.66 ± 0.03 0.61 ± 0.03 0.73 ± 0.03

AUC 0.68 ± 0.08 0.67 ± 0.08 0.69 ± 0.08



Specificity 0.65 ± 0.04 0.62 ± 0.04 0.70 ± 0.04

AUC 0.68 ± 0.06 0.67 ± 0.06 0.68 ± 0.06



Specificity 0.66 ± 0.03 0.63 ± 0.03 0.70 ± 0.04

AUC 0.69 ± 0.06 0.68 ± 0.06 0.69 ± 0.06



Specificity 0.65 ± 0.05 0.62 ± 0.05 0.69 ± 0.05

AUC 0.70 ± 0.10 0.69 ± 0.10 0.71 ± 0.09



Specificity 0.65 ± 0.04 0.61 ± 0.04 0.70 ± 0.04

B6

Table B.12: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the third set of feature selection criteria.


AUC 0.67 ± 0.07 0.66 ± 0.04 0.68 ± 0.07



Specificity 0.67 ± 0.04 0.63 ± 0.04 0.72 ± 0.04

AUC 0.64 ± 0.12 0.64 ± 0.12 0.65 ± 0.12



Specificity 0.67 ± 0.04 0.62 ± 0.05 0.73 ± 0.04

AUC 0.68 ± 0.08 0.68 ± 0.06 0.68 ± 0.07



Specificity 0.66 ± 0.04 0.62 ± 0.03 0.72 ± 0.04

AUC 0.67 ± 0.08 0.66 ± 0.08 0.67 ± 0.07



Specificity 0.67 ± 0.03 0.62 ± 0.04 0.72 ± 0.03

AUC 0.65 ± 0.10 0.64 ± 0.09 0.65 ± 0.09



Specificity 0.66 ± 0.04 0.61 ± 0.04 0.71 ± 0.04

AUC 0.63 ± 0.09 0.63 ± 0.09 0.62 ± 0.09



Specificity 0.64 ± 0.04 0.59 ± 0.04 0.69 ± 0.05

AUC 0.66 ± 0.09 0.65 ± 0.09 0.67 ± 0.09



Specificity 0.67 ± 0.04 0.64 ± 0.04 0.71 ± 0.04

AUC 0.67 ± 0.06 0.67 ± 0.06 0.68 ± 0.06



Specificity 0.67 ± 0.03 0.64 ± 0.03 0.72 ± 0.04

AUC 0.67 ± 0.07 0.67 ± 0.07 0.67 ± 0.07



Specificity 0.64 ± 0.04 0.61 ± 0.04 0.70 ± 0.04

7

Documents

Feature Extraction for Modeling Patients’ Outcomes: an … · rais, como os valores maximos e m´ ´ınimos medidos ate ao momento da avaliac¸´ ao atual. Finalmente, a˜ abordagem