G E O S C I E N C E A U S T R A L I A

Performance of Machine Learning Methods and Their Combinations with Ordinary Kriging and Inverse Distance Squared

Jin Li, Andrew D. Heap, Anna Potter and James J. Daniell

APPLYING GEOSCIENCE TO AUSTRALIA’S MOST IMPORTANT CHALLENGES

Record

2011/07

Predicting Seabed Mud Content across the Australian Margin II

GeoCat # 71407

Predicting Seabed Mud Content across the Australian Margin II:

Performance of Machine Learning Methods and Their Combinations with Ordinary Kriging and

Inverse Distance Squared

Jin Li, Andrew D. Heap, Anna Potter and James J. Daniell

Geoscience Australia, GPO Box 378, Canberra, ACT 2601, Australia

Geoscience Australia Record 2011/07

Department of Resources, Energy and Tourism Minister for Resources and Energy: The Hon. Martin Ferguson, AM MP Secretary: Mr Drew Clarke Geoscience Australia Chief Executive Officer: Dr Chris Pigram

© Commonwealth of Australia (Geoscience Australia) 2011 With the exception of the Commonwealth Coat of Arms and where otherwise noted, all material in this publication is provided under a Creative Commons Attribution 3.0 Australia Licence (http://creativecommons.org/licenses/by/3.0/au/) Geoscience Australia has tried to make the information in this product as accurate as possible. However, it does not guarantee that the information is totally accurate or complete. Therefore, you should not solely rely on this information when making a commercial decision. ISSN 1448-2177 ISBN 978-1-921781-82-7 (hardcopy) ISBN 978-1-921781-81-0 (CD/DVD) ISBN 978-1-921781-80-3 (web) GeoCat # 71407 Bibliographic reference: Li, J., Heap, A. D., Potter, A. and Daniell, J. J., 2011. Predicting Seabed Mud Content across the Australian Margin II: the Performance of Machine Learning Methods and Their Combinations with Ordinary Kriging and Inverse Distance Squared. Geoscience Australia, Record 2011/07, 69 pp. Correspondence for feedback: Sales Centre Geoscience Australia GPO Box 378 Canberra ACT 2601 Australia sales@ga.gov.au

iii

Executive Summary Geoscience Australia provides spatial information about the physical and biological character of the seabed to support Australian marine zone management. Central to this approach is the prediction of Australia’s seabed biodiversity from spatially continuous data of physical seabed properties. However, information for these properties is usually collected at sparsely and unevenly distributed discrete locations, particularly in the deep ocean. Thus, methods for generating spatially continuous information from point samples become essential tools. In 2008, the performance of 14 statistical and mathematical methods for spatial interpolation was compared using samples of seabed mud content across the Australian Exclusive Economic Zone (AEEZ), which indicated that machine learning methods are generally among the most accurate methods.

In this study, we have further tested the performance of machine learning methods in combination with ordinary kriging (OK) and inverse distance squared (IDS). IDS was used as the control method. This study aims to identify the most accurate methods for spatial interpolation of seabed mud content in three regions (i.e., north, northeast and southwest) in AEEZ using samples extracted from Geoscience Australia’s Marine Samples Database (MARS). The performance of 18 methods (machine learning methods and their combinations with OK or IDS) was compared using a simulation experiment. We examined the effects of slope, search neighbourhood size and averaging predictions of the most accurate methods on the prediction accuracy. Finally the predictions of the most accurate methods were visually examined. Bathymetry, distance-to-coast, seabed slope, latitude and longitude were used as secondary variables. The accuracies of the methods were assessed using a 10-fold cross-validation. The effects of these factors on the prediction accuracy were compared based on the information extracted from 1,614 prediction datasets produced for this experiment.

The prediction accuracy changes with the methods, inclusion and exclusion of slope, search window size, model averaging and the study region. No single method performs best for all the tested scenarios. The combination of RF and OK (RFOK), the combination of RF and IDS (RFIDS) and RF are the three most accurate methods in the north and southwest regions; in the northeast region RFIDS and the combination of SVMlinear and IDS (SVMlinearIDS) are the most accurate methods and RFOK is slightly less accurate than the control (IDS). RKIDS and RFOK with slope are, on average, more accurate than the other methods based on the prediction accuracy and visual examination of prediction maps in all three regions when their search widow size is 12 and 7, respectively. Averaging the predictions of these two most accurate methods could be an alternative for spatial interpolation. The methods identified in this study reduce the prediction error by up to 19% (e.g., RFOK) and their predictions depict the transitional zones between geomorphic features in comparison with the control.

This study confirmed the effectiveness of combining machine learning methods with OK or IDS and produced an alternative source of methods for spatial interpolation. Procedures employed in this study for selecting the most accurate methods provide guidance for future studies. The outcomes of this study can be applied to modelling a range of physical properties to improve marine biodiversity prediction.

iv

Abbreviations AEEZ: Australian Exclusive Economic Zone IDS: inverse distance squared KSVM: support vector machine with Gaussian radial basis kernel KSVMIDS: the combination of KSVM and IDS KSVMOK: the combination of KSVM and ordinary kriging MAE: mean absolute error MARS: Marine Samples Database OK: ordinary kriging RF: random forest RFIDS: the combination of RF and IDS RFOK: the combination of RF and OK RMAE: relative MAE RMSE: root mean squared error RRMSE: relative RMSE RPART: regression tree (in rpart) RPARTOK: the combination of RPART and OK RPARTIDS: the combination of RPART and IDS SVM: support vector machine with a radial basis kernel SVMIDS: the combination of SVM and IDS SVMlinear: support vector machine with a linear kernel SVMlinearOK: the combination of SVMlinear and OK SVMlinearIDS: the combination of SVMlinear and IDS SVMOK: the combination of SVM and OK SVMpolynomial: support vector machine with a polynomial kernel SVMpolynomialIDS: the combination of SVMpolynomial and IDS SVMpolynomialOK: the combination of SVMpolynomial and OK

v

Table of Contents EXECUTIVE SUMMARY ..................................................................................................................III ABBREVIATIONS............................................................................................................................... IV LIST OF FIGURES .............................................................................................................................. VI LIST OF TABLES .............................................................................................................................VIII CHAPTER 1. INTRODUCTION.......................................................................................................... 9 CHAPTER 2. METHODS ................................................................................................................... 11

2.1. STUDY AREA AND DATASETS ....................................................................................................... 11 2.2. MACHINE LEARNING METHODS AND THE COMBINED METHODS................................................... 11

2.2.1. Methods ............................................................................................................................... 11 2.2.2. Model specification and parameter selection ..................................................................... 12

2.3. ASSESSMENT OF METHOD PERFORMANCE .................................................................................... 13 CHAPTER 3. RESULTS...................................................................................................................... 14

3.1. OPTIMAL METHODS IN EACH REGION ........................................................................................... 14 3.2. EFFECTS OF EXCLUSION OF SLOPE................................................................................................ 17 3.3. OPTIMAL SEARCH WINDOW SIZE OF THE MOST ACCURATE METHODS WITH SLOPE....................... 17 3.4. OPTIMAL SEARCH WINDOW SIZE OF METHODS WITHOUT SLOPE................................................... 24 3.5. EFFECTS OF AVERAGING THE PREDICTIONS OF THE MOST ACCURATE METHODS.......................... 30 3.6. VISUAL EXAMINATION ................................................................................................................. 31

3.6.1. With slope............................................................................................................................ 31 3.6.2. Without slope....................................................................................................................... 39

CHAPTER 4. DISCUSSION ............................................................................................................... 47 4.1. OPTIMAL MODELLING METHODS .................................................................................................. 47 4.2. WITH OR WITHOUT SLOPE? .......................................................................................................... 48 4.3. OPTIMAL SEARCH WINDOW SIZE .................................................................................................. 51 4.4. EFFECTS OF MODEL AVERAGING .................................................................................................. 52 4.5. VISUAL EXAMINATION OF THE PREDICTIONS OF THE METHODS WITH SLOPE................................ 52

4.5.1. North.................................................................................................................................... 52 4.5.2. Northeast ............................................................................................................................. 54 4.5.3. Southwest............................................................................................................................. 55

4.6. VISUAL EXAMINATION OF THE PREDICTIONS OF THE METHODS WITHOUT SLOPE ......................... 57 4.7. LIMITATIONS................................................................................................................................ 58

CHAPTER 5. CONCLUSIONS........................................................................................................... 59 ACKNOWLEDGEMENTS ................................................................................................................. 60 APPENDIX A. BASIC STATISTICAL SUMMARIES OF THE PREDICTIONS OF AND STATISTICS MEASURING THE PERFORMANCE OF EACH MODELLING METHOD. ... 64

vi

List of Figures FIGURE 1. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE

ERROR (RRMSE (%)) OF MODELLING METHODS FOR MUD CONTENT IN THE NORTH REGION. THE HORIZONTAL AND VERTICAL LINES (RED) INDICATE THE ACCURACY OF THE CONTROL (IDS) (LI ET AL., 2010)........................................................................................................................................14

FIGURE 2. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF MODELLING METHODS FOR MUD CONTENT IN THE NORTHEAST REGION. THE HORIZONTAL AND VERTICAL LINES (RED) INDICATE THE ACCURACY OF THE CONTROL (IDS) (LI ET AL., 2010)..............................................................................................................................15

FIGURE 3. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF MODELLING METHODS FOR MUD CONTENT IN THE SOUTHWEST REGION. THE HORIZONTAL AND VERTICAL LINES (RED) INDICATE THE ACCURACY OF THE CONTROL (IDS) (LI ET AL., 2010)..............................................................................................................................16

FIGURE 4. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFIDS (WITH SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE NORTH REGION. .............................................................................................18

FIGURE 5. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFOK (WITH SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE NORTH REGION. .............................................................................................19

FIGURE 6. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFIDS (WITH SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE NORTHEAST REGION. .....................................................................................20

FIGURE 7. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFOK (WITH SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE NORTHEAST REGION. .....................................................................................21

FIGURE 8. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFIDS (WITH SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE SOUTHWEST REGION......................................................................................22

FIGURE 9. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFOK (WITH SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE SOUTHWEST REGION......................................................................................23

FIGURE 10. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFIDS (WITHOUT SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE NORTH REGION. .............................................................................................24

FIGURE 11. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFOK (WITHOUT SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE NORTH REGION. .............................................................................................25

FIGURE 12. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFIDS (WITHOUT SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE NORTHEAST REGION. THE BEST SIZE IS 10-12 WITH RMAE: 35.27% AND RRMSE: 51.41%............................................................................................................................26

FIGURE 13. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFOK (WITHOUT SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE NORTHEAST REGION. .....................................................................................27

FIGURE 14. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFIDS (WITHOUT SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE SOUTHWEST REGION......................................................................................28

FIGURE 15. THE RELATIVE ABSOLUTE MEAN ERROR (RMAE (%)) AND RELATIVE ROOT MEAN SQUARE ERROR (RRMSE (%)) OF RFOK (WITHOUT SLOPE) FOR MUD CONTENT IN RELATION TO SEARCH WINDOW SIZE IN THE SOUTHWEST REGION......................................................................................29

FIGURE 16. THE PREDICTIONS OF THE TWO MOST ACCURATE METHODS WITH SLOPE AND OPTIMAL SEARCH WINDOW SIZE (RFIDS.12 AND RFOK.7) AND THE PREDICTIONS OF THEIR AVERAGING IN THE NORTH REGION.........................................................................................................................31

FIGURE 17. THE PREDICTIONS OF THE TWO MOST ACCURATE METHODS WITH SLOPE AND OPTIMAL SEARCH WINDOW SIZE (RFIDS.12 AND RFOK.7) AND THE PREDICTIONS OF THEIR AVERAGING IN THE NORTHEAST REGION.................................................................................................................33

FIGURE 18. THE PREDICTIONS OF THE TWO MOST ACCURATE METHODS WITH SLOPE AND OPTIMAL SEARCH WINDOW SIZE (RFIDS.12 AND RFOK.7) AND THE PREDICTIONS OF THEIR AVERAGING IN THE SOUTHWEST REGION. ...............................................................................................................36

vii

FIGURE 19. THE PREDICTIONS OF THE TWO MOST ACCURATE METHODS WITHOUT SLOPE FOR OPTIMAL SEARCH WINDOW SIZE (RFIDS.12 AND RFOK.7) AND THE PREDICTIONS OF THEIR AVERAGING IN THE NORTH REGION. ....................................................................................................................... 39

FIGURE 20. THE PREDICTIONS OF THE TWO MOST ACCURATE METHODS WITHOUT SLOPE FOR OPTIMAL SEARCH WINDOW SIZE (RFIDS.12 AND RFOK.7) AND PREDICTIONS OF THEIR AVERAGING IN THE NORTHEAST REGION. ...................................................................................................................... 41

FIGURE 21. THE PREDICTIONS OF THE TWO MOST ACCURATE METHODS WITHOUT SLOPE FOR OPTIMAL SEARCH WINDOW SIZE (RFIDS.12 AND RFOK.7) AND PREDICTIONS OF THEIR AVERAGING IN THE SOUTHWEST REGION....................................................................................................................... 44

FIGURE 22. SPATIAL PATTERN OF SLOPE IN THE NORTHEAST REGION. TO DISPLAY THE PATTERNS OF SLOPE IN THE MAJORITY AREA, VALUES OVER 30 WERE CONVERTED TO 31................................... 49

FIGURE 23. SPATIAL PATTERN OF SLOPE IN THE SOUTHWEST REGION. TO DISPLAY THE PATTERNS OF SLOPE IN THE MAJORITY AREA, VALUES OVER 30 WERE CONVERTED TO 31................................... 50

FIGURE 24. SPATIAL DISTRIBUTION OF GEOMORPHIC FEATURES IN THE NORTHEAST REGION (LI ET AL., 2010).............................................................................................................................................. 51

FIGURE 25. THE SPATIAL DISTRIBUTION OF GEOMORPHIC FEATURES (ABOVE) AND THE SPATIAL PATTERN OF BATHYMETRY (BOTTOM) IN THE NORTH REGION (LI ET AL., 2010). ........................................... 53

FIGURE 26. SPATIAL PATTERN OF BATHYMETRY IN THE NORTHEAST REGION (LI ET AL., 2010). ............ 55 FIGURE 27. SPATIAL DISTRIBUTION OF GEOMORPHIC FEATURES (LI ET AL., 2010). ................................ 56 FIGURE 28. SPATIAL PATTERN OF BATHYMETRY IN THE SOUTHWEST REGION (LI ET AL., 2010). ............ 57

viii

List of Tables

TABLE 1. EFFECTS OF SLOPE EXCLUSION OF THE PREDICTION ERROR OF RFOK AND RFIDS FOR A

SEARCH WINDOW SIZE OF 20. ..........................................................................................................17 TABLE 2. THE ACCURACY OF AVERAGING THE PREDICTIONS OF RFOK WITH SEARCH WINDOW SIZE OF 7

(RFOK.7) AND THE PREDICTIONS OF RFIDS WITH SEARCH WINDOW SIZE OF 12 (RFIDS.12) IN THE

THREE REGIONS...............................................................................................................................30 TABLE 3. THE ACCURACY OF AVERAGING THE PREDICTIONS OF RFIDS WITH A SEARCH WINDOW SIZE OF

20 SAMPLES AND THE PREDICTIONS OF SVMLINEARIDS WITH A SEARCH WINDOW SIZE OF 20

SAMPLES IN THE THREE REGIONS. ...................................................................................................30

9

Chapter 1. Introduction Geoscience Australia provides spatial information about the physical and biological character of the seabed for Australian marine management. Central to this approach is prediction of Australia’s seabed biodiversity from spatially continuous data of physical seabed properties. However, information for these properties is usually collected at sparsely- and unevenly- distributed discrete locations, particularly in the deep ocean. Thus, methods for generating spatially continuous information from point samples become essential tools. They can be classified into three groups: 1) non-geostatistical methods (e.g., inverse distance weighting), 2) geostatistical methods (e.g., ordinary kriging: OK) and 3) combined methods (e.g. regression kriging) (Li and Heap 2008). These methods are, however, often data- or even variable- specific and it is often difficult to select an appropriate method for any given dataset.

Machine learning methods, like random forest (RF) and support vector machine (SVM), have shown their robustness in data mining fields and other disciplines (Cutler et al., 2007; Diaz-Uriarte and de Andres, 2006; Drake et al., 2006; Marmion et al., 2009; Shan et al., 2006). However, RF has not been previously applied to the spatial interpolation of environmental variables (Li and Heap 2008) and SVM was only applied to rainfall data in one previous study (Gilardi, 2002). Improving the accuracy of these physical data for biodiversity prediction, by searching for the most robust spatial interpolation methods to predict physical seabed properties, is essential to better inform environmental management practices. In this regard, we compared the performance of 14 statistical and mathematical methods (37 sub-methods) for spatial interpolation, under various scenarios, using samples of seabed mud content across the Australian margin in 2008 (Li et al., 2010).

There are a number of factors that affect the prediction accuracy (Li and Heap, 2008). In the study by Li et al. (2010), a number of factors were tested and relevant suggestions have been made according to the findings.

• Sample stratification by geomorphic provinces was found to reduce the accuracy of the predictions in most cases and it suggested that geomorphic provinces should not be used as secondary information or as a factor to stratify the samples if a method uses bathymetry as its secondary variable.

• No threshold in sample density (or sample size) was found in relation to the accuracy of the predictions in any region, which suggested that the number of samples collected so far is still below the threshold (if one exists) and more samples should be collected to improve the predictions of future spatial interpolation.

• Given that only two levels of search window size were tested, it was recommend that additional levels should be tested to find an optimal size.

• Secondary information considered was limited to bathymetry, distance-to-coast, slope, latitude and longitude. Other correlated secondary variables should be identified and employed, and thus the artefacts associated with influence of latitude and longitude could be removed from the predictions.

• The combination of machine learning methods, like RF, with other methods for spatial interpolation was novel. There is great potential in applying machine learning methods in spatial interpolation. It should be one direction of future studies selecting statistical methods for spatial interpolation. Further studies are warranted for testing their performance in different scenarios.

10

• For spatial predictions, visual examination is recommended as an essential step to assessing the predictions of a method.

• To achieve an optimised spatial interpolation of environmental variables for AEEZ, further studies are needed.

The results of the previous study (Li et al., 2010) indicated that machine learning methods like RF and SVM are relatively robust in terms of relative mean absolute error (RMAE) and relative root mean squared error (RRMSE). Therefore, we conducted this study to further test the performance of these methods in combination with OK and inverse distance squared (IDS) using mud content data in three regions.

In this study, we aim to identify the most appropriate methods for spatial interpolation of seabed mud content for the continental AEEZ using samples extracted from Geoscience Austalia’s Marine Samples Database (MARS) database (www.ga.gov.au/oracle/mars). The performance of 18 statistical techniques for spatial interpolation is compared using a simulation experiment. The experiment examines:

• the effects of slope on the performance of the methods, • the effect of search neighbourhood size, • the accuracy of averaging the predictions of the most accurate methods, and • the prediction patterns of the most accurate methods visually based on their

prediction maps.

This study covers various aspects of the experiment, which are presented in five chapters. Chapter 2 contains a brief description of study methods including experimental design and data analysis. We analyse the simulation results, visually compare a few high performance methods and illustrate their applications in Chapter 3. Chapter 4 discusses the findings and their implications. Finally, in Chapter 5, we summarise our findings and provide recommendations for the application of the methods tested.

11

Chapter 2. Methods

2.1. Study area and datasets

This study is an extension of a previous simulation experiment by Li et al. (2010). It incorporates the findings and suggestions from the previous study. The study area and datasets including secondary information are the same as those used in the previous study. Datasets with 100% sample density in three regions (i.e., the north, northeast and southwest regions) were used without sample stratification based on the recommendations from the previous study. These datasets were then randomly divided into 10 datasets for cross validation for each of three regions, generating 30 datasets in total.

2.2. Machine learning methods and the combined methods

2.2.1. Methods

In this experiment, we firstly applied the following 18 methods to the datasets in the three regions, with search window size of 20 samples for IDS and OK:

1) RF; 2) the combination of RF and IDS (RFIDS); 3) the combination of RF and OK (RFOK); 4) regression tree in rpart (RPART); 5) the combination of RPART and IDS (RPARTIDS); 6) the combination of RPART and OK (RPARTOK); 7) support vector machine with a radial basis kernel (SVM); 8) the combination of SVM and IDS (SVMIDS); 9) the combination of SVM and OK (SVMOK); 10) support vector machine with linear kernel (SVMlinear); 11) the combination of SVMlinear and OK (SVMlinearOK); 12) the combination of SVMlinear and IDS (SVMlinearIDS); 13) support vector machine with polynomial kernel (SVMpolynomial); 14) the combination of SVMpolynomial and IDS (SVMpolynomialIDS); 15) the combination of SVMpolynomial and OK (SVMpolynomialOK); 16) support vector machine with Gaussian radial basis kernel (KSVM); 17) the combination of KSVM and ordinary kriging (KSVMOK); and 18) the combination of KSVM and IDS (KSVMIDS).

The methods above have been briefly described in previous studies (Li and Heap, 2008; Li et al., 2010). Please refer to these references for details.

Each method was applied to 30 datasets. SVMpolynomial failed to produce any results after >96 hours in both sun-proj1-common and sun-proj2-common servers for datasets of north and southwest regions. Consequently, SVMpolynomial and its combined methods are not reported in this study, leaving 15 methods for comparison. Additionally, 6 prediction datasets were not produced for KSVM because no predictive models were developed for the corresponding training datasets. In total, these applications generated 444 prediction datasets.

12

We then tested the effects of exclusion of slope related variables because they were the least important variables in terms of the results of RF:

1) RF without slope related variables; 2) RFOK without slope related variables; and 3) RFIDS without slope related variables.

These tests produced 90 prediction datasets.

We further tested the performance of the most accurate methods in relation to various search window size to identify optimal size for spatial interpolation, which produced 540 and 480 prediction datasets for methods with and without slope respectively. The selection of search window size was based on the prediction accuracies for the sizes being tested.

Finally, we tested whether averaging the predictions of the best performance methods could further improve the prediction accuracy. Model averaging (also termed as ensemble mean, ensemble forecasting, consensus method, average of forecasts, combining or combination forecasts etc., see Araújo and New (2007) and (Gregory et al., 2001) for more information) have been applied to a number of disciplines including ecology (Araújo and New, 2007; Ellison, 2004), economics (Gregory et al., 2001), biomedicine (Nilsson et al., 2000), climatology (Raftery et al., 2005) and hydrology (Goswami and O'Connor, 2007). In this study, we averaged the predictions of two best modelling methods in the north and southwest regions (RFOK and RFIDS) and also two best methods in the northeast regions (RFIDS and SVMlinearIDS) to produce the final predictions (60 prediction datasets); hence the term, “model averaging”, in this study. This method was argued to be one of the most accurate methods for averaging the predictions from different modelling methods (Marmion et al., 2009).

2.2.2. Model specification and parameter selection

The modelling procedure, including variogram modelling and secondary variables, were identical to those for RFOK in the previous study (Li et al., 2010). All modelling work was conducted in R (R Development Core Team, 2009).

RF was implemented using randomForest (Liaw and Wiener, 2002), and RPART was implemented using rpart (Therneau and Atkinson, 2009). Both KSVM and SVM are support vector machine methods but implemented using different packages, KSVM in kernlab (Karatzoglou et al., 2004) and SVM in e1071 (Dimitriadou et al., 2009). IDS and OK were implemented using gstat (Pebesma, 2004).

With respect to RF, to generate reliable predictions we specified the number of trees as 2,000. The number of variables randomly sampled as candidates at each split was assigned the number 6 according to the results of tuneRF (i.e., Tune randomForest) in randomForest.

To prune the regression tree generated by rpart with a complexity parameter of 0.001, a complexity parameter of 0.006 was specified. These parameters were selected on the basis of the plot of 10-fold cross-validation results.

In SVM, the gamma parameter for kernel and the cost of constrains violation were selected based on the mean squared error produced by 10-fold cross-validation. In the

13

north region, the best choice of cost was 1 and the optimal gamma was 4. In the northeast region, the cost was 4, and gamma 1. In the southwest region, the cost was 0.5 and gamma 1.

In KSVM, the sigma parameter was assigned 10 for the north and northeast regions and 5 for southwest region according to the results of 10-fold cross-validation.

2.3. Assessment of method performance

Ten-fold cross-validation was used to validate the predictions of each method. The procedures of the validation is detailed by Li et al. (2010). In total 1,614 prediction datasets were generated in this experiment, which formed a basis for the assessment of the performance of the methods tested. All derived summary statistics are attached in Appendix A for readers interested. RMAE and RRMSE (Li and Heap, 2008) were used to measure the prediction error.

14

Chapter 3. Results

3.1. Optimal methods in each region

The performance of the methods depends on the study regions. The performance of the methods showed that RFOK, RFIDS and RF were the most accurate in the north and southwest regions; RPARTOK also performed well in the southwest region; and RFIDS and SVMlinearIDS were the most accurate methods and RFOK was slightly less accurate than the control in the northeast region (Figs 1-3). All the remaining methods performed relatively poorly compared to the control (i.e., IDS). The accuracies of the various support machine learning methods varied considerably and also changed with regions. SVMlinear was the least accurate method in all three regions, but its combination with IDS performed relatively well and was one of the most accurate methods in the northeast region. The results indicate that RFIDS and RFOK are, on average, more accurate than other methods in all three regions.

Figure 1. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of modelling methods for mud content in the north region.

15

The horizontal and vertical lines (red) indicate the accuracy of the control (IDS) (Li et al., 2010).

Figure 2. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of modelling methods for mud content in the northeast region. The horizontal and vertical lines (red) indicate the accuracy of the control (IDS) (Li et al., 2010).

16

Figure 3. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of modelling methods for mud content in the southwest region. The horizontal and vertical lines (red) indicate the accuracy of the control (IDS) (Li et al., 2010).

17

3.2. Effects of exclusion of slope

The effects of the exclusion of slope on the prediction error of RFOK and RFIDS vary with the method and study area (Table 1). In the north and southwest regions, the prediction errors of both methods were reduced by the exclusion of slope related variables in terms of RMAE and RRMSE. In the northeast region, a reversed pattern was observed with an exception of RFOK in terms of RMAE. Overall, the results of these methods are relatively more accurate after excluding slope related variables.

Table 1. Effects of slope exclusion of the prediction error of RFOK and RFIDS for a

search window size of 20. Method3 Region Slope RMAE (%) RRMSE (%) RFOK n no 35.75 51.76 RFOK n yes 36.37 52.23 RFIDS n no 35.96 52.64 RFIDS n yes 36.28 52.73 RFOK ne no 36.99 53.17 RFOK ne yes 37.11 53.13 RFIDS ne no 35.79 52.01 RFIDS ne yes 35.71 51.77 RFOK sw no 17.87 28.68 RFOK sw yes 18.32 29.16 RFIDS sw no 18.02 29.79 RFIDS sw yes 18.52 30.13

3.3. Optimal search window size of the most accurate methods with slope

There is no single optimal search window size for all regions for the two most accurate methods (i.e., RFOK and RFIDS) with slope in terms of RMAE and RRMSE (Figs 4-9). In the north region, the accuracy of the RFIDS generally decreased as the search window size decreased, and the best search window size was 20 (RMAE: 36.28% and RRMSE: 52.73%), and then 12 (RMAE: 36.28% and RRMSE: 52.85%) (Fig. 4). For RFOK, the relationship between the accuracy and the search window size was not apparent, but the best sizes were 10 (RMAE: 36.16% and RRMSE: 52.23%) and 12 (RMAE: 36.19% and RRMSE: 52.14%) (Fig. 5).

18

Figure 4. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFIDS (with slope) for mud content in relation to search window size in the north region.

19

Figure 5. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFOK (with slope) for mud content in relation to search window size in the north region.

20

In the northeast region, there is little relationship between the prediction error and the search window size for RFIDS, but the best sizes were 12 (RMAE: 35.63% and RRMSE: 51.69%) and 5 (RMAE: 35.59% and RRMSE: 51.81%) (Fig. 6). The prediction error of RFOK increased linearly with the increasing search window size and the best sizes were 5 (RMAE: 36.15% and RRMSE: 52.37%), 6 (RMAE: 36.31% and RRMSE: 52.49%) and 7 (RMAE: 36.35% and RRMSE: 52.49%) (Fig. 7).

Figure 6. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFIDS (with slope) for mud content in relation to search window size in the northeast region.

21

Figure 7. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFOK (with slope) for mud content in relation to search window size in the northeast region.

22

In the southwest region, the prediction error of both RFIDS and RFOK shows a weak linear relationship with the search window size. For RFIDS the best sizes were 6 (RMAE: 18.36% and RRMSE: 29.98%), 5 (RMAE: 18.41% and RRMSE: 29.96%) and 7 (RMAE: 18.39% and RRMSE: 29.98%) (Fig. 8). For RFOK the best sizes were 6 (RMAE: 18.26% and RRMSE: 28.94%) and 7 (RMAE: 18.32% and RRMSE: 28.90%) (Fig. 9).

Overall, the optimal size is 12 for RFIDS and 7 for RFOK for all regions if a single search window size is required.

Figure 8. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFIDS (with slope) for mud content in relation to search window size in the southwest region.

23

Figure 9. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFOK (with slope) for mud content in relation to search window size in the southwest region.

24

3.4. Optimal search window size of methods without slope

Similar to the methods incorporating slope, there is also no single optimal search window size for two most accurate methods (i.e., RFOK and RFIDS) when slope related variables were excluded in the three regions in terms of RMAE and RRMASE (Figs 10-15). In the north region, the accuracy of the RFIDS generally decreased as the search window size decreased, and the best search window sizes were 20 (RMAE: 35.96% and RRMSE: 52.64%), and then 12 (RMAE: 35.99% and RRMSE: 52.76%) (Fig. 10). For RFOK, the relationship between the accuracy and the size is also largely linear, and the best size was 12 (RMAE: 35.61% and RRMSE: 51.70%) (Fig. 11).

Figure 10. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFIDS (without slope) for mud content in relation to search window size in the north region.

25

Figure 11. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFOK (without slope) for mud content in relation to search window size in the north region.

26

In the northeast region, the prediction error of RFIDS shows little relationship to the search window size and the prediction error of RFOK shows a linear relationship with the search window size (Figs 12-13). The best size was 10-12 (RMAE: 35.27% and RRMSE: 51.41%) for RFIDS; and 5 (RMAE: 35.75% and RRMSE: 51.97%) or 7 (RMAE: 35.83% and RRMSE: 51.93%) for RFOK, and with the least accurate size being 20 for both methods.

Figure 12. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFIDS (without slope) for mud content in relation to search window size in the northeast region. The best size is 10-12 with RMAE: 35.27% and RRMSE: 51.41%.

27

Figure 13. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFOK (without slope) for mud content in relation to search window size in the northeast region.

28

In the southwest region, the prediction error of both RFIDS and RFOK shows little relationship to the search window size (Figs 14-15). The best size was 7 (RMAE: 17.98% and RRMSE: 29.74%) for RFIDS; and the best sizes are 6 (RMAE: 17.76% and RRMSE: 28.47%) and 7 (RMAE: 17.80% and RRMSE: 28.40%) for RFOK.

Overall, the optimal size is 12 for RFIDS and 7 for RFOK for all regions if a single size is required.

Figure 14. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFIDS (without slope) for mud content in relation to search window size in the southwest region.

29

Figure 15. The relative absolute mean error (RMAE (%)) and relative root mean square error (RRMSE (%)) of RFOK (without slope) for mud content in relation to search window size in the southwest region.

30

3.5. Effects of averaging the predictions of the most accurate methods

Averaging the predictions of the two most accurate methods with slope (i.e., RFOK with search window size of 7 and RFIDS with search window size of 12) produced predictions with the accuracy changing with regions (Table 2). In the north region, averaging the predictions of two most accurate methods marginally improved the prediction accuracy. In the northeast region, averaging the predictions slightly reduced the prediction error in comparison with RFOK.7 but slightly increased the error in comparison with RFIDS.12, while in the southwest region, the reverse was observed.

Table 2. The accuracy of averaging the predictions of RFOK with search window size

of 7 (RFOK.7) and the predictions of RFIDS with search window size of 12

(RFIDS.12) in the three regions. Method Region RMAE RRMSE RFOK.7 n 36.16 52.47 RFIDS.12 n 36.28 52.85 RFOK.7 and RFIDS.12 n 36.08 52.44 RFOK.7 ne 36.35 52.49 RFIDS.12 ne 35.63 51.69 RFOK.7 and RFIDS.12 ne 35.83 51.85 RFOK.7 sw 18.32 28.9 RFIDS.12 sw 18.49 30.11 RFOK.7 and RFIDS.12 sw 18.34 29.33

Averaging the predictions of RFIDS with slope and the predictions of SVM with linear kernel (with slope) and IDS (RFIDS and SVMlinearIDS) also shows little improvement in the prediction accuracy (Table 3). In the north and southwest regions, averaging the predictions considerably reduced the prediction error in comparison with SVMlinearIDS, but slightly increased the error in comparison with RFIDS with an exception in north region in terms of RRMSE. In the northeast region, averaging the predictions marginally improved the prediction accuracy in comparison with both methods.

Table 3. The accuracy of averaging the predictions of RFIDS with a search window

size of 20 samples and the predictions of SVMlinearIDS with a search window size of

20 samples in the three regions. Method Region RMAE RRMSE RFIDS n 36.28 52.73 SVMlinearIDS n 38.13 54.58 RFIDS and SVMlinearIDS n 36.49 52.67 RFIDS ne 35.71 51.77 SVMlinearIDS ne 35.47 52.49 RFIDS and SVMlinearIDS ne 34.99 51.06 RFIDS sw 18.52 30.13 SVMlinearIDS sw 25.89 36.66 RFIDS and SVMlinearIDS sw 21.5 32.01

31

3.6. Visual examination

3.6.1. With slope

The spatial predictions of the two most accurate methods (i.e., RFIDS and RFOK) with slope and with their optimal search window size (12 and 7 respectively) are shown in Figs 16-18. In the north region, the spatial patterns are largely similar between these two methods (Fig. 16). Both methods captured the major spatial patterns and trends of mud content and both results displayed apparent vertical and horizontal banding patterns. The weak ‘bull’s eye’ patterns were evident for RFIDS in the north. The averages of these two methods produced a map similar to that of RFIDS.12, but the ‘bull’s eye’ patterns being less distinct.

Figure 16. The predictions of the two most accurate methods with slope and optimal search window size (RFIDS.12 and RFOK.7) and the predictions of their averaging in the north region.

RFIDS.12

32

Fig. 16. (cont.):

Averages of RFOK.7 and RFIDS.12

RFOK.7

33

In the northeast region, the spatial patterns are similar between RFIDS and RFOK (Fig. 17). Both methods captured the major spatial patterns and trends of seabed mud content, such as the prominent increase in mud content from the shallow shelf to the deep ocean basin of the Coral Sea. Also apparent in their predictions was the faint banding associated with the influence of latitude and longitude which were used as the secondary information. The averages of these two methods produced a similar map to RFIDS.12 and RFOK.7.

Figure 17. The predictions of the two most accurate methods with slope and optimal search window size (RFIDS.12 and RFOK.7) and the predictions of their averaging in the northeast region.

RFIDS.12

34

Fig. 17. (cont.):

RFOK.7

35

Fig. 17. (cont.):

Average of RFOK.7 and RFIDS.12

36

In the southwest region, the spatial patterns are also similar between RFIDS and RFOK (Fig. 18). Both methods depicted major spatial patterns and trends of seabed mud content, but weak ‘bull’s eye’ patterns were also evident for RFIDS. Linear tracks and sharp transitions were apparent in the predictions and so were banding patterns. The averages of these two methods produced a map similar to that of RFIDS.12, but the ‘bull’s eye’ patterns being less distinct.

Figure 18. The predictions of the two most accurate methods with slope and optimal search window size (RFIDS.12 and RFOK.7) and the predictions of their averaging in the southwest region.

RFIDS.12

37

Fig. 18. (cont.):

RFOK.7

38

Fig. 18. (cont.):

39

3.6.2. Without slope

The spatial predictions of the best two methods (i.e., RFIDS and RFOK) without slope show similar patterns to those when slope was included. The spatial patterns were largely similar between these two methods (Figs 19-21), and weak ‘bull’s eye’ patterns were evident for RFIDS in the north (Fig. 19) and southwest regions (Fig. 21). However, when slope was excluded, some details disappeared including the linear features in the southwest region and the effects of longitude and latitude became more prominent (in comparison with Figs 16-18). The averages of these two methods produced a map similar to that of RFIDS.12, but ‘bull’s eye’ patterns being less distinct in the north and southwest regions.

Figure 19. The predictions of the two most accurate methods without slope for optimal search window size (RFIDS.12 and RFOK.7) and the predictions of their averaging in the north region.

RFIDS.12.no.slope

RFOK.7.no.slope

40

Fig. 19. (cont.):

Average of RFOK.7.no.slope and RFIDS.12.no.slope

41

Figure 20. The predictions of the two most accurate methods without slope for optimal search window size (RFIDS.12 and RFOK.7) and predictions of their averaging in the northeast region.

RFIDS.12.no.slope

42

Fig. 20. (cont.):

RFOK.7.no.slope

43

Fig. 20. (cont.):

Average of RFOK.7.no.slope and RFIDS.12.no.slope

44

Figure 21. The predictions of the two most accurate methods without slope for optimal search window size (RFIDS.12 and RFOK.7) and predictions of their averaging in the southwest region.

RFIDS.12.no.slope

45

Fig. 21. (cont.):

RFOK.7.no.slope

46

Fig. 21. (cont.):

Average of RFOK.7.no.slope and RFIDS.12.no.slope

47

Chapter 4. Discussion

4.1. Optimal modelling methods

Overall, the performance of the statistical methods for spatial interpolation of mud content is region-specific and no single method is always superior to the other methods. This phenomenon has been observed in number of previous studies (Elith et al., 2006; Marmion et al., 2009; Olden and Jackson, 2002). It is consistent with the ‘no free lunch theorems’ in optimisation (Wolpert and Macready, 1997). This may be attributed to the differences in a number of aspects between the study regions, such as: 1) the physical processes that influence the formation and accumulation of mud sediment; 2) geomorphic features and bathymetry; and 3) coastline orientation and the consistency of this orientation, which may need to be confirmed in further studies. This suggests that optimal interpolation method changes with regions and efforts should be made to identify the optimal method for each individual region.

The prediction accuracies of the methods are higher in the southwest region than in other two regions. This difference is mainly due to the higher correlations between mud content and bathymetry, distance to coast and slope in the southwest region than in the other regions (Li et al., 2010). It may also relate to the changes in orientation of the coastline, which will be further discussed in section 4.5.

The relatively poor performance of SVMlinearOK is probably due to the poor variogram fitting (i.e., negative range in variogram modelling) for OK for some validation datasets in the north region, while the poor performance of KSVMOK may be due to the poor variogram fitting (i.e., the negative range in variogram modelling) for OK for some validation datasets in the north and northeast regions.

Generally, RFOK, RFIDS and the averages of their predictions are the best performing methods, with their accuracies being similar. This suggests that RF successfully predicted the general trend in each region, and OK and IDS made further contributions to the observed patterns at a local scale (although IDS performed even better than OK in the northeast region). The slightly superior performance of IDS to OK was also observed in the previous study (Li et al., 2010). Since all methods were compared under the same conditions in this study and the previous study, we conclude that the superior performance of RFOK and RFIDS is attributed to the methods themselves rather than to other factors. RF has been proven to be a robust method by the finding of Diaz-Uriarte and de Andres (2006), which supports the excellent performance of RFOK and RFIDS in this study.

The excellent performance of RFOK and RFIDS may be attributed to the following factors associated with the methods:

1. The most important variable is selected to split the samples at each node split, so RF implicitly performs variable selection (Okun and Priisalu, 2007); and consequently it may not be sensitive to correlated variables.

2. RF uses both bagging (bootstrap aggregation), a successful approach for combining unstable learners, and random variable selection for tree building; and therefore it yields an ensemble of low correlation trees. As a result, RF can achieve both low bias and low variance from averaging over a large

48

ensemble of low bias, high variance but low correlation trees (Breiman, 1996; Breiman et al., 1984; Diaz-Uriarte and de Andres, 2006).

3. Although decision trees, neural networks and support vector machines were able to deal with ‘poorly predictable data’ (Shan et al., 2006), they are outperformed by RFOK in both this study and the previous study (Li et al., 2010). This suggests that RFOK and RFIDS may be able to deal with ‘poorly predictable data’ because the data used in this study are sparsely and unevenly spatially distributed. This is also supported by findings in a previous study (Marmion et al., 2009).

4. Individual trees are unpruned so as to obtain low-bias trees(Diaz-Uriarte and de Andres, 2006).

5. RF can also deliver good predictive performance even when most predictive variables are noise (Diaz-Uriarte and de Andres, 2006).

6. Although the unpruned individual trees will result in overfitted models (Prasad et al., 2006), our results of RF based on unpruned trees suggest that the overfitting of individual trees may not be a big concern to RF.

7. RF can model complex interactions among predictive variables (Cutler et al., 2007; Diaz-Uriarte and de Andres, 2006; Okun and Priisalu, 2007).

8. The predictions of RF are more reasonable for extrapolation and more accurate than those of regression tree (RT), bagging trees (BT) and multivariate adaptive regression splines (MARS) (Prasad et al., 2006). Extrapolation occurred in spatial predictions of mud content across the three regions of the AEEZ.

9. RF does not overfit with respect to the source data because of the law of large numbers (Breiman, 2001; Maindonald and Xie, 2008; Okun and Priisalu, 2007).

10. It has been argued that the performance of RF is not much influenced by parameter choices (Diaz-Uriarte and de Andres, 2006; Liaw and Wiener, 2002; Okun and Priisalu, 2007).

11. Finally, the prediction residuals of RF are interpolated using OK or IDS and the interpolated values are added to the predictions of RF, which further reduces the prediction error as evidenced in this study.

4.2. With or without slope?

A prominent N-S trending linear feature and detailed small-scale patterns in the predicted spatial patterns of mud content in the northeast and NW-SE trending linear feature and detailed small-scale patterns in southwest regions by RFOK and RFIDS mainly reflect the influence of slope (Figs. 22 and 23). After excluding slope from the models, these features disappeared or became less distinct (Figs. 20 and 21). These features also reflect the transition between geomorphic features, especially in the northeast region (Fig. 24). However, the predictions show banding patterns with sharp transitions if slope is excluded. Consequently, the predicted patterns are less visually pleasant. Therefore, the inclusion of slope should be considered for producing the spatial predictions of mud content across the AEEZ if this visual appeal of the predictions is considered as an important factor, although slope is the least important variable and its inclusion may slightly reduce the prediction accuracy.

49

Figure 22. Spatial pattern of slope in the northeast region. To display the patterns of slope in the majority area, values over 30 were converted to 31.

50

Figure 23. Spatial pattern of slope in the southwest region. To display the patterns of slope in the majority area, values over 30 were converted to 31.

51

Figure 24. Spatial distribution of geomorphic features in the northeast region (Li et al., 2010).

4.3. Optimal search window size

Although guidelines exist for selecting search window size (Goovaerts, 1997; Isaaks and Srivastava, 1989; Li and Heap, 2008), little has been done to identify optimal search window size. This study found that the optimal search window size changes with method, region and the inclusion of slope. This is because in the north region the best method is RFOK.12 without slope (RMAE: 35.61% and RRMSE: 51.70%), in the northeast region the best method is the averages of SVMlinearIDS and RFIDS with slope (RMAE: 34.99% and RRMSE: 51.06%), and in the southwest region the best method is RFOK.6 without slope (RMAE: 17.76% and RRMSE: 28.47%). When compared to the control method (Li et al., 2010), the most accurate method reduces the prediction error by 3.9%, 3.2% and 19% in terms of RMAE and by 4.2%, 4.1%

52

and 16.5% in terms of RRMSE in the north, northeast and southwest regions respectively. These findings also suggest that there is no single optimal search window size for any method and for any given data or region. On average, the optimal search window size is 7 for RFOK and 12 for RFIDS based on the results of this study.

4.4. Effects of model averaging

In this study, we simply averaged the predictions of two modelling methods to produce the final predictions. The effect of model averaging changed with study region, and on average it marginally improved the prediction accuracy. This result is largely consistent with previous findings about the ensemble mean (Goswami and O'Connor, 2007; Hoeting et al., 1999; Marmion et al., 2009; Raftery et al., 2005).

Model averaging has been applied to a number of disciplines (Araújo and New, 2007; Ellison, 2004; Goswami and O'Connor, 2007; Hoeting et al., 1999), but it is novel to geostatistics. To our knowledge averaging the predictions of the combined method of machine learning method and IDS and OK has not been undertaken previously. Given the improved results of model averaging in other disciplines (Goswami and O'Connor, 2007; Hoeting et al., 1999; Marmion et al., 2009; Raftery et al., 2005) and the promising results from this study, tests on the effects of model averaging should be undertaken in future.

4.5. Visual examination of the predictions of the methods with slope

4.5.1. North

The effects of bathymetry as reflected by geomorphic features (Fig. 25) on the major patterns of the predictions of RFOK and of the averages of RFOK and RFIDS are apparent in the north region (Figs 16 and 19). These features include channels, basins and submarine canyons. Specifically, in the basins, mud content is predicted to be high relative to the surrounding seabed. This high mud content prediction is consistent with findings in previously published studies (van Andel and Veevers, 1967). The effects of geomorphic features were also discussed in detail by Li et al. (2010).

The spatial patterns (Fig. 16) depict apparent vertical and horizontal banding patterns, which are attributed to the predictions from RF and reflect the influence of longitude and latitude as secondary information although they are indirect variables. This indicates that in these areas mud content was more related to longitude and latitude than other secondary variables including bathymetry. This might also result from the coincidence of the spatial patterns of samples along lines of latitude and longitude. The apparent effect of geomorphic features on the predictions of RFOK is because geomorphic features were mainly based on bathymetric contours (Heap and Harris, 2008) and the bathymetry is the most important variable for RF in this study.

The accuracies of these methods in this region are adversely affected by its complex and contrasting coastal orientations. The western section of this study region has a northeast-southwest orientated coastline; the middle section has a northwest-southeast orientated coastline, while the eastern section has a roughly north-south orientated coastline. These differences in coastline orientation provided confusing information to

53

the model (i.e., RF), thus reduced predictive accuracy of the model. However, when this region is divided into these three sub-regions with relatively consistent coastal orientations, the banding patterns are considerably less distinct (as evidenced by our most recent unpublished results). This suggests that it is important to appropriately divide the study region into sub-regions with similar coastal orientation and it also emphasises the importance in providing right information to the model.

While relatively less distinct compared to IDS, ‘bull’s eye’ patterns are still evident for RFIDS in the north (Fig. 16). This is a common feature produced by IDW, as has been discussed in previous studies (e.g., Li et al., 2010).

Figure 25. The spatial distribution of geomorphic features (above) and the spatial pattern of bathymetry (bottom) in the north region (Li et al., 2010).

54

Fig. 25. (cont.):

4.5.2. Northeast

The muddy north-facing coastal embayment, the inner continental shelf and Capricorn Channel (Fig. 24) are accurately portrayed in the predictions of RFOK, RFIDS and their averages in the northeast region (Figs 17 and 20). Similarly, the relatively muddy seafloors of the Townsville and Queensland Troughs are also distinctively portrayed. Moreover, the sandy and gravelly seafloors of the outer continental shelf and Queensland and Marion Plateaus (Heap and Harris, 2008) are clearly depicted. These methods produced results that are broadly consistent with previously published work on the sedimentology of the northeast Australian margin (Belperio, 1983; Keene et al., 2008; Maxwell and Swinchatt, 1970).

The influence of bathymetry (Fig. 26) and geomorphic features (Fig. 24) on the predictions is apparent, especially in the deep-sea. The relatively strong influence of bathymetry and geomorphology at these deeper depths is mainly due to the sparse sample coverage compared to shelf areas. The predicted spatial patterns reflect the effect of geomorphic province 3 (i.e., the northwest-southeast orientated white strip in Fig. 24) through bathymetry (Fig. 26), and the predicted values are lower than those in the surrounding areas. The lower predicted values for geomorphic province 3 compared to surrounding areas are convincing given the shallower water depths in this area.

55

Figure 26. Spatial pattern of bathymetry in the northeast region (Li et al., 2010).

4.5.3. Southwest

The predicted spatial patterns of RFOK, RFIDS and their averages in the southwest region (Figs 18 and 21) mainly reflect the effects of geomorphology and bathymetry (Figs 27 and 28) because the patterns are similar to those identified by Heap and Harris (2008). On the shallow continental shelf, the predicted patterns of seabed mud content (i.e., low mud content) convincingly depict the relatively sandy nature of this geomorphic province and the accumulations of coarse sand and gravel around the shelf edge adjacent to Houtman-Abrolhos Reefs. In deeper water beyond the shelf, the influences of bathymetry and geomorphology are also apparent (Fig. 27). The influence is the greatest where the features are topographically prominent (e.g., the Perth Canyon).

The influence of bathymetry and geomorphology is expected because bathymetry was used as secondary information to derive the predictions. For submarine canyons, this may not necessarily be a problem as samples from the Perth Canyon indicate that at shallow depths (<1,000 m) the seafloor in the canyon is slightly more muddy than the

56

surrounding slope environments (Heap et al., 2008). The banding patterns visible in the predictions reflect the influence of longitude and latitude because they were also used as secondary information. As was the case for the north and northeast regions, these artefacts derived from secondary information are prominent due to the relatively poor sample coverage in the deep water areas of this margin.

Figure 27. Spatial distribution of geomorphic features (Li et al., 2010).

57

Figure 28. Spatial pattern of bathymetry in the southwest region (Li et al., 2010).

4.6. Visual examination of the predictions of the methods without slope

The predictions from the methods which exclude slope failed to pick up changes in mud content associated with transitions between geomorphic features and promoted the effects of latitude and longitude, although some specific linear features associated with including slope become less distinct or absent. In general, the spatial patterns are similar between the methods (i.e., RFIDS, RFOK and their averages), but the ‘bull’s eye’ patterns are still evident for RFIDS in the north and southwest regions. Therefore, the predictions of RFOK and the averaged method are more visually pleasant than that of RFIDS.

Visual examination is an essential step in accessing the performance of the prediction methods and is arguably as important as the error measurement. This is because two

58

methods with similar prediction error may produce different spatial patterns as evidenced in this study. The visual examination provides an alternative way to assess whether patterns were likely to be real by comparing with existing research findings and expert interpretation. Overall, RFOK and the averaged method including slope are favoured over those methods excluding slope.

4.7. Limitations

This study is an extension of the study by Li et al. (2010) so it shares similar limitations as discussed in the previous work. A further limitation is the influence of time span of samples collected. The samples used in this study (and the previous study) were collected over a long period of time as described by Li et al. (2010). The temporal change in seabed sediments has not been considered and consequently all data at each sample location were assumed unchanged over time in the study. This assumption is obviously questionable. Due to the limited sample availability we have to use this assumption, otherwise it is impossible to make any predictions because of the very limited sample size. This limitation is shared by all methods used, including the control. However, if possible, uncertainties resulted from this limitation should be taken into account when applying the findings of this study and the predictions of mud content. Despite of this limitation, the predictions achieved in this study are more accurate than those of the control method and of the methods compared in previous studies in environmental sciences (Li and Heap, 2008). Therefore, predictions with improved accuracy are produced in this study.

59

Chapter 5. Conclusions The most accurate method for interpolating mud content is region-dependent, but RFIDS, RFOK and their averages are among the most accurate methods for all regions. In addition, SVMlinearIDS performed very well in the northeast region. This study confirms the effectiveness of combining machine learning methods with commonly used spatial interpolation methods such as OK or IDS and has led to an alternative source of methods for the discipline of spatial statistics.

The optimal search window size is also region-dependent. If the best method with its optimal search window size for each region needs to be found, then similar experiments for each region will need to be run. If the objective is to select one size for all regions, then 12 is optimum for RFIDS and 7 for RFOK.

The exclusion of slope slightly increases the prediction accuracy in two regions, but the resulting predictions are less convincing according to existing knowledge of mud content compared to those with slope included.

Averaging the two best methods does not always improve the prediction accuracy. Given this finding and considering the computational time and visual examinations, model averaging is not recommended for predicting mud content in the AEEZ if only one method is adopted for the whole AEEZ, although it is a good alternative method.

To improve the prediction accuracy, it is recommended that the most accurate method should be identified by considering all relevant impact factors for each study region. These factors may include those factors recommended by Li et al. (2010). In addition, model diagnostics decisions suggested by Bivand et al. (2008) should also be incorporated.

The selection of the most robust method for any given region is a resource-intensive process and it may not be possible to test all possible combinations of influencing factors. The ‘best’ methods identified are therefore conditioned on the resources allocated. This really implies that the threshold of acceptable predictive error must be taken into account in selecting a prediction method.

RFOK, RFIDS or their averages are recommended for predicting mud content across the AEEZ. If a single method is required for predicting mud content across AEEZ, RFOK with slope and with a search window size 7 is recommended based on the prediction accuracy and visual examination. Averaging the predictions of RFOK and RFIDS may also be an alternative approach if prediction accuracy is the only criterion considered.

This study provides suggestions and guidelines for improving the spatial interpolations of marine environmental data in general, and results in more accurate mapping and characterisation of seabed across the AEEZ. This study has produced a more a more reliable and robust physical seabed dataset for the AEEZ that will assist in the development of management and conservation strategies of marine zones by government, industry and community.

60

Acknowledgements We thank Zhi Huang and Maggie Tran from Geoscience Australia for valuable comments on this record. Christina Baker, Shoaib Burq, Mark Webster, and Tanya Whiteway (all of Geoscience Australia) are appreciated for preparing datasets. Chris Lawson and Zhi Huang (all of Geoscience Australia) are thanked for providing bathymetry, slope and distance to coast data and producing several maps. Scott Nichol, David Ryan, and Frederic Saint-Cast (all of Geoscience Australia) are thanked for providing suggestions and comments on the experimental design. We also thank Roger Bivand (Norwegian School of Economics and Business Administration), Paul Hiemstra (University of Utrecht), Edzer Pebesma (University of Münster) and Michael Sumner (University of Tasmania) for their help in using various functions in gstat, maptools and sp packages in R. This record is published with permission of the Chief Executive Officer, Geoscience Australia.

61

References Araújo M.B., New M., 2007. Ensemble forecasting of species distribution. Trends in Ecology and Evolution 22: 42-47.

Belperio A.P., 1983. Terrigenous sedimentation in the central Great Barrier Reef lagoon: a model from the Burdekin region. BMR Journal of Australian Geology and Geophysics 8: 179-190.

Bivand R.S., Pebesma E.J., Gómez-Rubio V., 2008. Applied Spatial Data Analysis with R. Springer: New York.

Breiman L., 1996. Bagging predictors. Machine Learning 24: 123-140.

Breiman L., 2001. Random forests. Machine Learning 45: 5-32.

Breiman L., Friedman J.H., Olshen R.A., Stone C.J., 1984. Classification and regression trees. Wadsworth: Belmont.

Cutler D.R., Edwards T.C.J., Beard K.H., Cutler A., Hess K.T., Gibson J., Lawler J.J., 2007. Random forests for classification in ecology. Ecography 88: 2783-2792.

Diaz-Uriarte R., de Andres S.A., 2006. Gene selection and classification of microarray data using random forest. BMC Bioinformatics 7: 1-13.

Dimitriadou E., Hornik K., Leisch F., Meyer D., Weingessel A., 2009. e1071: Misc Functions of the Department of Statistics (e1071), TU Wien. R package version 1.5-19.

Drake J.M., Randin C., Guisan A., 2006. Modelling ecological niches with support vector machines. Journal of Applied Ecology 43: 424-432.

Elith J., Graham C.H., Anderson R.P., Dulik M., Ferrier S., Guisan A., Hijmans R.J., Huettmann F., Leathwick J.R., Lehmann A., Li J., Lohmann L.G., Loiselle B.A., Manion G., Moritz C., Nakamura M., Nakazawa Y., Overton J.M., Peterson A.T., Phillips S.J., Richardson K., Scachetti-Pereira R., Schapire R.E., Soberon J., Williams S., Wisz M.S., Zimmermann N.E., 2006. Novel methods improve prediction of species' distributions from occurrence data. Ecography 29: 129-151.

Ellison A.M., 2004. Bayesian inference in ecology. Ecology Letters 7: 509-520.

Gilardi N., 2002. Machine Learning for Spatial Data Analysis. 73.

Goovaerts P., 1997. Geostatistics for Natural Resources Evaluation. Oxford University Press: New York.

Goswami M., O'Connor K.M., 2007. Real-time flow forecasting in the absence of quantitative precipitation forecasts: A multi-model approach. Journal of Hydrology 334: 125-140.

Gregory A.W., Smith G.W., Yetman J., 2001. Testing for forecast consensus. Journal of Business and Economics Statistics 19.

62

Heap A.D., Edwards J., Fountain L., Spinnocia M., Hughes M., Mathews E., Griffin J., Borissova I., Blevin J., Mitchell C., Krassay A., 2008. Geomorphology, sedimentology and stratigraphy of submarine canyons on the SW Australian slope – post survey report. Geoscience Australia: Canberra; 138.

Heap A.D., Harris P.T., 2008. Geomorphology of the Australian margin and adjacent seafloor. Australian Journal of Earth Sciences 55: 555-585.

Hoeting J.A., Madigan D., Raftery A.E., Volinsky C.T., 1999. Bayesian model averaging: a tutorial. Statistical Science 14: 382-417.

Isaaks E.H., Srivastava R.M., 1989. Applied Geostatistics. Oxford University Press: New York.

Karatzoglou A., Smola A., Hornik K., Zeileis A., 2004. kernlab - An S4 package for kernel methods in R. Journal of Statistical Software 11: 1-20.

Keene J., Baker C., Tran M., Potter A., 2008. Sedimentology and geomorphology of the east marine region of Australia: a spatial analysis. Geoscience Australia: Canberra; 262.

Li J., Heap A., 2008. A Review of Spatial Interpolation Methods for Environmental Scientists. Geoscience Australia: Canberra; 137.

Li J., Potter A., Huang Z., Daniell J.J., Heap A., 2010. Predicting Seabed Mud Content across the Australian Margin: Comparison of Statistical and Mathematical Techniques Using a Simulation Experiment. Geoscience Australia: Canberra; 146.

Liaw A., Wiener M., 2002. Classification and regression by radomForest. R News 2: 18-22.

Maindonald J., Xie Y., 2008. Data Mining with R. http://cos.name/wp-content/uploads/2008/12/data-mining-with-r-by-john-maindonald.pdf.

Marmion M., Parviainen M., Luoto M., Heikkinen R.K., Thuiller W., 2009. Evaluation of consensus methods in predictive species distribution modelling. Diversity and Distributions 15: 59-69.

Maxwell W.G.H., Swinchatt J.P., 1970. Great Barrier Reef: regional variation in a terrigenous-carbonate province. Geological Society of America Bulletin 81: 691-724.

Nilsson J., Persson B., von Heijne G., 2000. Consensus prediction of membrane protein topology. FEBS Letters 486: 267-269.

Okun O., Priisalu H., 2007. Random forest for gene expression based cancer classification: overlooked issues. In Pattern Recognition and Image Analysis: Third Iberian Conference, IbPRIA 2007 Martí J., Benedí J.M., Mendonça A.M., Serrat J. (eds.); Lecture Notes in Computer Science: Girona, Spain; 4478: 4483-4490.

Olden J.D., Jackson D.A., 2002. A comparison of statistical approaches for modelling fish species distributions. Freshwater Biology 47: 1976-1995.

63

Pebesma E.J., 2004. Multivariable geostatistics in S: the gstat package. Computer & Geosciences 30: 683-691.

Prasad A.M., Iverson L.R., Liaw A., 2006. Newer classification and regression tree techniques: bagging and random forests for ecological prediction. Ecosystems 9: 181-199.

R Development Core Team, 2009. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna.

Raftery A.E., Gneiting T., Balabdaoui F., Polakowski M., 2005. Using Bayesian model averaging to calibrate forecast ensembles. Monthly Weather Review 133: 1155-1174.

Shan Y., Paull D., McKay R.I., 2006. Machine learning of poorly predictable ecological data. Ecological Modelling 195: 129-138.

Therneau T.M., Atkinson B., 2009. R port by Ripley B. rpart: Recursive Partitioning. R package version 3.1-43.

van Andel T.H., Veevers J.J., 1967. Morphology and Sediments of the Timor Sea. Bureau of Mineral Resources, Geology and Geophysics: Canberra.

Wolpert D., Macready W., 1997. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation 1: 67-82.

64

Appendix A

. Basic statistical sum

maries of the predictions of and statistics m

easuring the perform

ance of each modelling m

ethod. N

o. M

ethod R

egion Slope

Window

search

size

Minim

um

Mean

Maxim

um

MA

E

RM

SE

RM

AE

(%)

RR

MSE

(%)

1 K

SVM

n

yes 20

0 34.04

99.2 14.17

19.83 41.63

58.25 2

KSV

MID

S n

yes 20

0 34.04

99.2 13.67

19.05 40.16

55.96 3

RF

n no

20 0

34.04 99.2

12.3 17.67

36.13 51.91

4 R

F n

yes 20

0 34.04

99.2 12.58

17.89 36.96

52.56 5

RFID

S n

no 5

0 34.04

99.2 12.35

18.18 36.28

53.41 6

RFID

S n

yes 5

0 34.04

99.2 12.41

18.16 36.46

53.35 7

RFID

S n

no 6

0 34.04

99.2 12.33

18.12 36.22

53.23 8

RFID

S n

yes 6

0 34.04

99.2 12.41

18.13 36.46

53.26 9

RFID

S n

no 7

0 34.04

99.2 12.3

18.08 36.13

53.11 10

RFID

S n

yes 7

0 34.04

99.2 12.39

18.09 36.4

53.14 11

RFID

S n

no 8

0 34.04

99.2 12.28

18.03 36.08

52.97 12

RFID

S n

yes 8

0 34.04

99.2 12.37

18.05 36.34

53.03 13

RFID

S n

no 9

0 34.04

99.2 12.28

18 36.08

52.88 14

RFID

S n

yes 9

0 34.04

99.2 12.37

18.02 36.34

52.94 15

RFID

S n

no 10

0 34.04

99.2 12.27

17.99 36.05

52.85 16

RFID

S n

yes 10

0 34.04

99.2 12.36

18.01 36.31

52.91 17

RFID

S n

no 11

0 34.04

99.2 12.26

17.97 36.02

52.79 18

RFID

S n

yes 11

0 34.04

99.2 12.36

18 36.31

52.88 19

RFID

S n

no 12

0 34.04

99.2 12.25

17.96 35.99

52.76 20

RFID

S n

yes 12

0 34.04

99.2 12.35

17.99 36.28

52.85 21

RFID

S n

yes 15

0 34.04

99.2 12.36

17.98 36.31

52.82 22

RFID

S n

no 20

0 34.04

99.2 12.24

17.92 35.96

52.64 23

RFID

S n

yes 20

0 34.04

99.2 12.35

17.95 36.28

52.73 24

KSV

MO

K

n yes

20 0

34.51 99.2

14.03 19.32

40.65 55.98

65

No.

Method

Region

Slope W

indow

search size

Minim

um

Mean

Maxim

um

MA

E

RM

SE

RM

AE

(%)

RR

MSE

(%)

25 R

FO

K

n no

5 0

34.04 99.2

12.2 17.86

35.84 52.47

26 R

FO

K

n yes

5 0

34.04 99.2

12.34 17.95

36.25 52.73

27 R

FO

K

n no

6 0

34.04 99.2

12.19 17.78

35.81 52.23

28 R

FO

K

n yes

6 0

34.04 99.2

12.35 17.91

36.28 52.61

29 R

FO

K

n no

7 0

34.04 99.2

12.15 17.74

35.69 52.12

30 R

FO

K

n yes

7 0

34.04 99.2

12.31 17.86

36.16 52.47

31 R

FO

K

n no

8 0

34.04 99.2

12.15 17.69

35.69 51.97

32 R

FO

K

n yes

8 0

34.04 99.2

12.32 17.84

36.19 52.41

33 R

FO

K

n no

9 0

34.04 99.2

12.15 17.64

35.69 51.82

34 R

FO

K

n yes

9 0

34.04 99.2

12.32 17.78

36.19 52.23

35 R

FO

K

n no

10 0

34.04 99.2

12.15 17.64

35.69 51.82

36 R

FO

K

n yes

10 0

34.04 99.2

12.31 17.78

36.16 52.23

37 R

FO

K

n no

11 0

34.04 99.2

12.15 17.62

35.69 51.76

38 R

FO

K

n yes

11 0

34.04 99.2

12.33 17.78

36.22 52.23

39 R

FO

K

n no

12 0

34.04 99.2

12.12 17.6

35.61 51.7

40 R

FO

K

n yes

12 0

34.04 99.2

12.32 17.75

36.19 52.14

41 R

FO

K

n yes

15 0

34.04 99.2

12.35 17.8

36.28 52.29

42 R

FO

K

n no

20 0

34.04 99.2

12.17 17.62

35.75 51.76

43 R

FO

K

n yes

20 0

34.04 99.2

12.38 17.78

36.37 52.23

44 R

FO

K+

RF

IDS

n

yes na

0 34.04

99.2 12.28

17.85 36.08

52.44 45

RP

AR

TO

K

n yes

20 0

34.04 99.2

13.72 19.41

40.31 57.02

46 S

VM

OK

n

yes 20

0 34.68

99 13.5

19 38.93

54.79 47

SV

MlinearO

K

n yes

20 0

34.04 99.2

13.39 18.55

39.34 54.49

48 R

PA

RT

n

yes 20

0 34.04

99.2 15.33

20.67 45.04

60.72 49

RP

AR

TID

S

n yes

20 0

34.04 99.2

13.15 19.16

38.63 56.29

50 S

VM

n

yes 20

0 34.04

99.2 13.4

19.36 39.37

56.87 51

SV

MID

S

n yes

20 0

34.04 99.2

13.08 18.81

38.43 55.26

52 S

VM

linearIDS

n

yes 20

0 34.04

99.2 12.98

18.58 38.13

54.58 53

SV

MlinearID

S+

RF

IDS

n

yes na

0 34.04

99.2 12.42

17.93 36.49

52.67

66

No.

Method

Region

Slope W

indow

search size

Minim

um

Mean

Maxim

um

MA

E

RM

SE

RM

AE

(%)

RR

MSE

(%)

54 SV

Mlinear

n yes

20 0

34.04 99.2

18.74 24.5

55.05 71.97

55 K

SVM

ne

yes 20

0 25.09

99.49 11.5

16.69 45.83

66.52 56

KSV

MID

S ne

yes 20

0 25.09

99.49 10.23

14.77 40.77

58.87 57

RF

ne no

20 0

25.09 99.49

9.38 13.4

37.39 53.41

58 R

F ne

yes 20

0 25.09

99.49 9.44

13.41 37.62

53.45 59

RFID

S ne

no 5

0 25.09

99.49 8.87

12.94 35.35

51.57 60

RFID

S ne

yes 5

0 25.09

99.49 8.93

13 35.59

51.81 61

RFID

S ne

no 6

0 25.09

99.49 8.87

12.93 35.35

51.53 62

RFID

S ne

yes 6

0 25.09

99.49 8.94

12.99 35.63

51.77 63

RFID

S ne

no 7

0 25.09

99.49 8.86

12.92 35.31

51.49 64

RFID

S ne

yes 7

0 25.09

99.49 8.94

12.99 35.63

51.77 65

RFID

S ne

no 8

0 25.09

99.49 8.86

12.92 35.31

51.49 66

RFID

S ne

yes 8

0 25.09

99.49 8.94

12.99 35.63

51.77 67

RFID

S ne

no 9

0 25.09

99.49 8.86

12.91 35.31

51.45 68

RFID

S ne

yes 9

0 25.09

99.49 8.94

12.98 35.63

51.73 69

RFID

S ne

no 10

0 25.09

99.49 8.85

12.9 35.27

51.41 70

RFID

S ne

yes 10

0 25.09

99.49 8.94

12.98 35.63

51.73 71

RFID

S ne

no 11

0 25.09

99.49 8.85

12.9 35.27

51.41 72

RFID

S ne

yes 11

0 25.09

99.49 8.94

12.98 35.63

51.73 73

RFID

S ne

no 12

0 25.09

99.49 8.85

12.9 35.27

51.41 74

RFID

S ne

yes 12

0 25.09

99.49 8.94

12.97 35.63

51.69 75

RFID

S ne

yes 15

0 25.09

99.49 8.96

12.98 35.71

51.73 76

RFID

S ne

yes 20

0 25.09

99.49 8.96

12.99 35.71

51.77 77

RFID

S ne

no 20

0 25.09

99.49 8.98

13.05 35.79

52.01 78

KSV

MO

K

ne yes

20 0

25.09 99.49

11.06 15.67

44.08 62.46

79 R

FOK

ne

no 5

0 25.09

99.49 8.97

13.04 35.75

51.97 80

RFO

K

ne yes

5 0

25.09 99.49

9.07 13.14

36.15 52.37

81 R

FOK

ne

no 6

0 25.09

99.49 8.99

13.05 35.83

52.01 82

RFO

K

ne yes

6 0

25.09 99.49

9.11 13.17

36.31 52.49

67

No.

Method

Region

Slope W

indow

search size

Minim

um

Mean

Maxim

um

MA

E

RM

SE

RM

AE

(%)

RR

MSE

(%)

83 R

FO

K

ne no

7 0

25.09 99.49

8.99 13.03

35.83 51.93

84 R

FO

K

ne yes

7 0

25.09 99.49

9.12 13.17

36.35 52.49

85 R

FO

K

ne no

8 0

25.09 99.49

9.02 13.05

35.95 52.01

86 R

FO

K

ne yes

8 0

25.09 99.49

9.16 13.2

36.51 52.61

87 R

FO

K

ne no

9 0

25.09 99.49

9.06 13.08

36.11 52.13

88 R

FO

K

ne yes

9 0

25.09 99.49

9.2 13.23

36.67 52.73

89 R

FO

K

ne no

10 0

25.09 99.49

9.05 13.08

36.07 52.13

90 R

FO

K

ne yes

10 0

25.09 99.49

9.19 13.24

36.63 52.77

91 R

FO

K

ne no

11 0

25.09 99.49

9.07 13.09

36.15 52.17

92 R

FO

K

ne yes

11 0

25.09 99.49

9.23 13.26

36.79 52.85

93 R

FO

K

ne no

12 0

25.09 99.49

9.08 13.09

36.19 52.17

94 R

FO

K

ne yes

12 0

25.09 99.49

9.24 13.26

36.83 52.85

95 R

FO

K

ne yes

15 0

25.09 99.49

9.28 13.31

36.99 53.05

96 R

FO

K

ne no

20 0

25.09 99.49

9.28 13.34

36.99 53.17

97 R

FO

K

ne yes

20 0

25.09 99.49

9.31 13.33

37.11 53.13

98 R

FO

K+

RF

IDS

ne

yes na

0 25.09

99.49 8.99

13.01 35.83

51.85 99

RP

AR

TO

K

ne yes

20 0

25.09 99.49

10.96 15.31

43.68 61.02

100 S

VM

OK

ne

yes 20

0 25.09

99.49 11.1

15.68 44.24

62.5 101

SV

MlinearO

K

ne yes

20 0

25.09 99.49

10.07 14.21

40.14 56.64

102 R

PA

RT

ne

yes 20

0 25.09

99.49 12.08

16.39 48.15

65.32 103

RP

AR

TID

S

ne yes

20 0

25.09 99.49

9.46 13.78

37.7 54.92

104 S

VM

ne

yes 20

0 25.09

99.49 11.78

16.54 46.95

65.92 105

SV

MID

S

ne yes

20 0

25.09 99.49

9.97 14.53

39.74 57.91

106 S

VM

linearIDS

ne

yes 20

0 25.09

99.49 8.9

13.17 35.47

52.49 107

SV

MlinearID

S+

RF

IDS

ne

yes na

0 25.09

99.49 8.78

12.81 34.99

51.06 108

SV

Mlinear

ne yes

20 0

25.09 99.49

15.43 20.18

61.5 80.43

109 K

SV

M

sw

yes 20

0.01 46.23

98.25 15.18

20.23 32.84

43.76 110

KS

VM

IDS

sw

yes

20 0.01

46.23 98.25

12.85 17.51

27.8 37.88

111 R

F

sw

no 20

0.01 46.23

98.25 8.34

13.36 18.04

28.9

68

No.

Method

Region

Slope W

indow

search size

Minim

um

Mean

Maxim

um

MA

E

RM

SE

RM

AE

(%)

RR

MSE

(%)

112 R

F sw

yes

20 0.01

46.23 98.25

8.66 13.73

18.73 29.7

113 R

FIDS

sw

no 5

0.01 46.23

98.25 8.36

13.75 18.08

29.74 114

RFID

S sw

yes

5 0.01

46.23 98.25

8.51 13.85

18.41 29.96

115 R

FIDS

sw

no 6

0.01 46.23

98.25 8.32

13.75 18

29.74 116

RFID

S sw

yes

6 0.01

46.23 98.25

8.49 13.86

18.36 29.98

117 R

FIDS

sw

no 7

0.01 46.23

98.25 8.31

13.75 17.98

29.74 118

RFID

S sw

yes

7 0.01

46.23 98.25

8.5 13.86

18.39 29.98

119 R

FIDS

sw

no 8

0.01 46.23

98.25 8.32

13.76 18

29.76 120

RFID

S sw

yes

8 0.01

46.23 98.25

8.51 13.89

18.41 30.05

121 R

FIDS

sw

no 9

0.01 46.23

98.25 8.33

13.76 18.02

29.76 122

RFID

S sw

yes

9 0.01

46.23 98.25

8.53 13.89

18.45 30.05

123 R

FIDS

sw

no 10

0.01 46.23

98.25 8.34

13.78 18.04

29.81 124

RFID

S sw

yes

10 0.01

46.23 98.25

8.55 13.92

18.49 30.11

125 R

FIDS

sw

no 11

0.01 46.23

98.25 8.35

13.77 18.06

29.79 126

RFID

S sw

yes

11 0.01

46.23 98.25

8.55 13.92

18.49 30.11

127 R

FIDS

sw

no 12

0.01 46.23

98.25 8.35

13.77 18.06

29.79 128

RFID

S sw

yes

12 0.01

46.23 98.25

8.55 13.92

18.49 30.11

129 R

FIDS

sw

yes 15

0.01 46.23

98.25 8.55

13.92 18.49

30.11 130

RFID

S sw

no

20 0.01

46.23 98.25

8.33 13.77

18.02 29.79

131 R

FIDS

sw

yes 20

0.01 46.23

98.25 8.56

13.93 18.52

30.13 132

KSV

MO

K

sw

yes 20

0.01 46.23

98.25 14.22

19.29 30.76

41.73 133

RFO

K

sw

no 5

0.01 46.23

98.25 8.3

13.2 17.95

28.55 134

RFO

K

sw

yes 5

0.01 46.23

98.25 8.49

13.38 18.36

28.94 135

RFO

K

sw

no 6

0.01 46.23

98.25 8.21

13.16 17.76

28.47 136

RFO

K

sw

yes 6

0.01 46.23

98.25 8.44

13.38 18.26

28.94 137

RFO

K

sw

no 7

0.01 46.23

98.25 8.23

13.13 17.8

28.4 138

RFO

K

sw

yes 7

0.01 46.23

98.25 8.47

13.36 18.32

28.9 139

RFO

K

sw

no 8

0.01 46.23

98.25 8.24

13.2 17.82

28.55 140

RFO

K

sw

yes 8

0.01 46.23

98.25 8.5

13.45 18.39

29.09

69

No.

Method

Region

Slope W

indow

search size

Minim

um

Mean

Maxim

um

MA

E

RM

SE

RM

AE

(%)

RR

MSE

(%)

141 R

FO

K

sw

no 9

0.01 46.23

98.25 8.26

13.18 17.87

28.51 142

RF

OK

sw

yes

9 0.01

46.23 98.25

8.52 13.43

18.43 29.05

143 R

FO

K

sw

no 10

0.01 46.23

98.25 8.26

13.21 17.87

28.57 144

RF

OK

sw

yes

10 0.01

46.23 98.25

8.54 13.46

18.47 29.12

145 R

FO

K

sw

no 11

0.01 46.23

98.25 8.3

13.22 17.95

28.6 146

RF

OK

sw

yes

11 0.01

46.23 98.25

8.56 13.47

18.52 29.14

147 R

FO

K

sw

no 12

0.01 46.23

98.25 8.31

13.22 17.98

28.6 148

RF

OK

sw

yes

12 0.01

46.23 98.25

8.57 13.47

18.54 29.14

149 R

FO

K

sw

yes 15

0.01 46.23

98.25 8.56

13.53 18.52

29.27 150

RF

OK

sw

no

20 0.01

46.23 98.25

8.26 13.26

17.87 28.68

151 R

FO

K

sw

yes 20

0.01 46.23

98.25 8.53

13.54 18.45

29.29 152

RF

OK

+R

FID

S

sw

yes na

0.01 46.23

98.25 8.48

13.56 18.34

29.33 153

RP

AR

TO

K

sw

yes 20

0.01 46.23

98.25 9.48

15.44 20.51

33.4 154

SV

MO

K

sw

yes 20

0.01 46.23

98.25 12.49

17.71 27.02

38.31 155

SV

MlinearO

K

sw

yes 20

0.01 45.71

98.25 15.84

20.63 34.65

45.13 156

RP

AR

T

sw

yes 20

0.01 46.23

98.25 9.93

15.83 21.48

34.24 157

RP

AR

TID

S

sw

yes 20

0.01 46.23

98.25 9.23

15.96 19.97

34.52 158

SV

M

sw

yes 20

0.01 46.23

98.25 13.7

19.32 29.63

41.79 159

SV

MID

S

sw

yes 20

0.01 46.23

98.25 10.68

15.72 23.1

34 160

SV

MlinearID

S

sw

yes 20

0.01 46.23

98.25 11.97

16.95 25.89

36.66 161

SV

MlinearID

S+

RF

IDS

sw

yes

na 0.01

46.23 98.25

9.94 14.8

21.5 32.01

162 S

VM

linear sw

yes

20 0.01

46.23 98.25

18.78 24.76

40.62 53.56