Original Article

Structural Health Monitoring

1–20

� The Author(s) 2019

Article reuse guidelines:

sagepub.com/journals-permissions

DOI: 10.1177/1475921719830328

journals.sagepub.com/home/shm

Videoscope-based inspection ofturbofan engine blades usingconvolutional neural networks andimage processing

Yong-Ho Kim and Jung-Ryul Lee

AbstractA typical aircraft engine consists of fans, compressors, turbines, and so on, and each is made of multiple layers of blades.Discovering the site of damages among the large number of blades during aircraft engine maintenance is quite important.However, it is impossible to look directly into the engine unless it is disassembled. For this reason, optical equipmentsuch as a videoscope is used to visually inspect the blades of an engine through inspection holes. The videoscope inspec-tion method has some obvious drawbacks such as the long-time attention on microscopic video feed and high laborintensity. In this research, we developed a damage recognition algorithm using convolutional neural networks and someimage-processing techniques related to feature point extraction and matching in order to improve the videoscopeinspection method. The image-processing techniques were mainly used for the preprocessing of the videoscope images,from which a suspected damaged region is selected after the preprocessing. The suspected region is finally classified asdamaged or normal by the pre-trained convolutional neural networks. We trained the convolutional neural networks2000 times by using data from 380 images and calculated the classification accuracy using data from 40 images. Afterrepeating the above procedure 50 times with the data randomly divided into training and test groups, an average classifi-cation accuracy of 95.2% for each image and a damage detectability of 100% in video were obtained. For verification ofthe proposed approach, the convolutional neural network part was compared with the traditional neural network, andthe preprocessing was compared with the region proposal network of the faster region–based convolutional neural net-works. In addition, we developed a platform based on the developed damage recognition algorithm and conducted fieldtests with a videoscope for a real engine. The damage detection AI platform was successfully applied to the inspectionvideo probed in an in-service engine.

KeywordsTurbofan engine, blade inspection, convolutional neural networks, image processing, videoscope

Introduction

The most important issue in the structural health moni-toring (SHM) field is identifying damages to the struc-ture. Currently, the most popular method foridentifying damages is the vibration-based method. Thebasic idea behind the vibration-based method is thatmodal parameters such as frequencies, mode shapes,and modal damping are functions of the physical prop-erties of the structure.1,2 Therefore, damages can beidentified based on measured changes in the vibrationresponse. This method has been widely applied fromsimple beam shapes3,4 to complex structures such assteel bridges,5 wind turbines,6 and aircraft stabilizers.7

However, this popular method is still difficult to apply

to detect tiny damages in aircraft engine blades. Inaddition, many other advanced nondestructive andSHM techniques would not be able to completelyaccess the internal blades.

The internal structure of aircraft engine is very com-plicated. For example, the F404 turbofan engine used

Department of Aerospace Engineering, Korea Advanced Institute of

Science and Technology, Daejeon, Republic of Korea

Corresponding author:

Jung-Ryul Lee, Department of Aerospace Engineering, Korea Advanced

Institute of Science and Technology, 291 Daehak-ro, Yuseong-gu, Daejeon

34141, Republic of Korea.

Email: leejrr@kaist.ac.kr

in the T-50 Golden Eagle, and so on consists of a fan, acompressor, and a turbine. Meanwhile, the fan consistsof three layers of blade set, the compressor seven layersand the turbine two layers. Each blade set has at least40 blades, which implies that there are hundreds ofblades inside the engine. It is worthy to note that theblade is easily damaged by external objects. Therefore,aircraft engine and their blades should be examined reg-ularly and accurately to find micrometer-order impactdamages. In general, the aircraft engine is completelydisassembled before it is inspected because it is impossi-ble to see the inside of the aircraft engine with merehuman eyes while it is assembled. However, the disas-sembly inspection is a complicated process and takes avery long time. As a result, optical equipment such as avideoscope is currently employed to inspect aircraftengines without disassembling them.

In order to examine the blades of an aircraft engineby using a videoscope, an inspector must attach thevideoscope tubular probe tube into the engine throughthe inspection holes. During this process, it is quite dif-ficult to manipulate the probe tube to the desired posi-tion because the interior of the engine is highlycompacted and narrow. After the probe is fixed, theinspector rotates the blades manually at a very lowspeed and closely observes the blades one by one. Incase of the inspection of fan blades, the inspectordivides the blades into three image parts, a root part, amid-part, and a tip part, and examines each part once.Therefore, the inspector must fix the tube at the desiredposition three times and then carefully and slowlyrotate the blades three times for each blade layer. Toinspect the compressor blades, the blade is divided intotwo image parts, and the test is performed twice. Afterrepeating these tasks for a long time, the concentrationof the inspector is naturally reduced; thus, the inspec-tion accuracy may be lowered. In addition, depending

on the proficiency level of the inspector, the tests resultsmay vary because this inspection method requires highconcentration by the inspector for a long time.

A way to overcome such shortcomings is to developartificial intelligence to assist the inspector in inspectingthe blades, so we decided to introduce machine learninginto the study. There are several algorithms in machinelearning, among which convolutional neural network(CNN) is specially applied in computer vision. SinceKrizhevsky et al.8 drastically improved the perfor-mance of CNN, it has received tremendous attentionand used for various artificial intelligence applications.A study was conducted to detect crack damage usingsliding window techniques and traditional CNN byCha et al.,9 and furthermore, various types of structuraldamage were detected using faster region–based convo-lutional neural network (RCNN) by Cha et al.10,11 Inthis study, we also adopted CNN as the machine learn-ing algorithm. Basically, the role of CNN is to classifyblade images into two cases. The first case is for imageswithout blade damage, while the second case is forimages containing damaged parts on the blade surface.Therefore, several image-processing techniques areapplied together with CNN because the images fromthe videoscope may contain a large amount of unneces-sary information that does not help with the classifica-tion. Image-processing techniques for feature pointextraction and matching were mostly used in order toeliminate the unnecessary information from the video-scope images before applying the CNN.

Figure 1 shows the damage recognition algorithmdeveloped through this research. As mentioned above,image-processing techniques were used in the prepro-cessing step, while the CNN was designed and trainedto classify the result of the preprocessing. The result ofthe preprocessing only contains necessary informationto help with the classification as the preprocessor

Figure 1. Damage recognition algorithm.

2 Structural Health Monitoring 00(0)

proposes suspected damaged region. The suspecteddamaged region is resized to a size of 150 3 150. Thisspecific size was determined empirically considering thesize of the videoscope image and the results of the pre-processing. The final judgment on this suspected dam-aged region is the responsibility of the trained CNN. Inaddition, this algorithm gradually improves its perfor-mance with additional data and training. Furtherdetails regarding the overall artificial intelligence of thepreprocessor and the CNN will be discussed in subse-quent sections.

The approach in Figure 1 consists of two majorsteps, the preprocessing and CNN. In this study, twocomparative analyses were conducted to verify thevalidity of this approach. First, the preprocessing andregion proposal network (RPN) of the RCNN werecompared. The faster RCNN is widely known algo-rithm for object detection and uses RPN for regionproposals.11 However, the RPN did not work well inthis case due to the nature of notched blade. Next, per-formance comparison between the CNN and neuralnetwork (NN) was conducted. The presence of the pre-processing can raise questions about the need forCNN, but we can confirm that CNN is essential forthis approach by comparing classification performancewith NN.

Preprocessing

Suspected damaged region

The purpose of the preprocessing step is to select thesuspected damaged region in the videoscope image.

Although the damage size may be in micrometers, thevideoscope image contains too much informationabout other areas. Therefore, it is necessary to extractonly the suspected region from the image. The sus-pected region may or may not be really damaged. Ifdamage actually exists, that region will be selected as asuspected damaged region. However, if there is nodamage, a region most likely to be damaged will beselected as a suspected damaged region. Figure 2 showsthe overall process of selecting suspected damagedregions. The damages affected by foreign objects onblades of the fan and high-pressure compressor weretargeted in this article. This is the exactly same goal ofthe videoscope-based visual inspection in the real field.

In order to select the suspected region, the main ideaused in this article is to compare two images. One of thetwo images is of course the videoscope image and theother image is an ‘‘eigendamage’’ image. The eigendam-age image is obtained through principal componentanalysis (PCA) for which a more detailed algorithm canbe found in Paul and Sumam.12 Feature points are thenextracted from the two images, and the most similarportions are selected in the videoscope image by match-ing the feature points. Scale invariant feature transform(SIFT) is used to extract the feature points in bothimages, and K-dimensional (KD) tree and random sam-ple consensus (RANSAC) are used to match the featurepoints.13,14 We define the portion selected through thematching as a suspected damaged region.

In reality, the most important thing in this prepro-cessing step is the precomputation part. As can be seenin Figure 2, the precomputation part entails pre-

Figure 2. Preprocessing algorithm.

Kim and Lee 3

configuring the KD tree using the feature points extractedfrom the eigendamage image. This is done to obtainmatching results in real time. If the KD tree is not usedfor matching, a very large amount of computation isrequired each time to match the feature points. Therefore,in order to process the videoscope image in real time, it ispreferable to construct the KD tree in advance and inputthe real-time feature points into the tree to obtain thematching result. The time required for this preprocessingis about 0.2 s. Therefore, four to five frames per secondcan be calculated because the classification by CNN needsmerely 0.008 s. Detailed descriptions of the variousimage-processing techniques used in the preprocessingstep are given in the ‘‘Methodology’’ section.

Methodology

PCA. PCA was used to extract the eigendamage. Theeigendamage is representative of damage imagesbecause the eigendamage consists of common compo-nents of multiple damage images. In this study, 10damage images were used for PCA and it was con-firmed that the number of damage images was suffi-cient through the eigenface and face recognition case.15

The mathematical implication of PCA is reducingthe dimensionality of the data. To minimize the loss ofinformation contained in the original data, the varianceof the transformed data should be maximized.Equation (1) represents the problem of variance maxi-mization, where x denotes a vector of original data, udenotes a unit vector of the transformed axis, and ydenotes a vector of the transformed data. The barabove the variable indicates that it is an average value.The length of the unit vector should be 1, so equation(2) is needed

maxs =1

n

Xn

i = 1

yi � �yð Þ2 =1

n

Xn

i = 0

uxi � u�xð Þ2 ð1Þ

s:t: uuT = 1 ð2Þ

By introducing the Lagrange multiplier l, the aboveoptimization problem can be summarized as given inequation (3) that is a simple optimization problem with-out constraint, which can be solved using partial deriva-tives. The partial derivative equation is summarized inequation (4) and should equate to zero. Therefore, thefinal form of the equation is shown in equation (5),which is a simple eigenvalue problem where S denotes acovariance matrix of the original data. This means thatthe unit vector u can be obtained by computing the cov-ariance matrix and solving the eigenvalue problem forthat matrix, because u indicates the eigenvectors inequation (5)

maxL uð Þ=1

n

Xn

i = 0

uxi � u�xð Þ2 + l 1� uuT� �

ð3Þ

∂L uð Þ∂u

= 2u1

n

Xn

i = 0

xi � �xð ÞT xi � �xð Þ !

� 2lu ð4Þ

Su = lu ð5Þ

Some preliminary works need to be performed onthe PCA to images because an image is not a vector. Inthis research, damage images were converted to similarscales and directions. The images were converted togray scale images and the gray scale images rearrangedin vector form. It can be seen in Figure 3 that 10 dam-age images were converted to similar scales and direc-tions. These 10 images were also converted to gray scaleand rearranged in vector form. The results of the PCA

Figure 3. Ten damage images for principal component analysis.

4 Structural Health Monitoring 00(0)

are shown in Figure 4. They have been listed in theorder of magnitudes of the eigenvalues, with the resultimage having the largest eigenvalue at the top leftmostcorner. In this article, the result image with the largesteigenvalue is called the eigendamage image because itincludes much more common elements than the othernine images.

SIFT. In computer vision, the concept of feature pointsis considered for solving object recognition problems.For example, people can recognize objects because theycan naturally grasp the characteristics and features ofthe object. Discussions on how to describe the featuresof objects in images led to the selection of edges, cor-ners, blobs, and so on, as appropriate features for useand have been adopted so far, with the underlyingassumption that features of the same object extractedfrom different images will be the same.

In this research, SIFT was used to extract the scaleinvariant feature points from the engine blade images.16

When shooting engine blades, the distance between thelens of the probe of the videoscope and the bladechanges every time; therefore, it should be able to copewith the scale changes in the image. The feature pointsextracted by SIFT can help solve this problem. SIFTcreates a three-dimensional image by adding a scaleaxis to a two-dimensional image and then finds featurepoints in the three-dimensional image.16 Therefore,using the scale invariant feature points, an algorithmcan be constructed that is not affected by the scalechanges in the image.

Figure 5 shows the results of the SIFT process,denoted with red stars. The pictures on the left are ori-ginals, and each contains a notched damage at the lead-ing edge of the engine blade. The size of the damage is

about 1 mm in diameter. The pictures on the right arethe results, where the red points represent the featurepoints. Many feature points were extracted from thecorners and edges of the blade. In addition, featurepoints were abundantly extracted at the damaged edgeas expected, and thus they could be utilized for damagerecognition. The notched damage has sharp edge andcorners, so the edge and corners can be detected easilyby the SIFT. In addition, the SIFT is widely known asan algorithm that richly detects feature points com-pared to other algorithms such as SURF, FAST, andso on. Therefore, it is possible to extract sufficientlyscale invariant feature points located on the damage.The purpose of this preprocessing is to extract sus-pected damaged regions. In this research, scale invar-iant feature points extracted from the damaged regionswere used.

KD tree. After the extraction, the feature points of thetwo images should be matched. The easiest way tomatch the feature points is to compare the Euclideandistances of the feature vectors and to connect themalong the shortest distance. The shorter the Euclideandistance, the higher the similarity between the two fea-ture points. Each feature point has a 128-dimensionalfeature vector, so the 128 dimensional Euclidean dis-tances must be calculated each time, and their lengthcomparisons made. Thus, the method for calculatingand comparing all the Euclidean distances is inefficientand too slow to process in real time for the real-timeinspection.

For this reason, we introduced the KD tree as analgorithm to find a nearest neighbor. The KD tree is atype of data structure in which all nodes are KD points.Therefore, this algorithm consists of a part that

Figure 4. Ten result images from principal component analysis.

Kim and Lee 5

constructs a tree and a part that uses the constructedtree. In this article, we constructed the KD tree usingfeature vectors extracted from one image and thenentered feature vectors extracted from the other imagein the tree, and then matched the most similar featurepoints. The first image referred here corresponds to theeigendamage in Figure 2, while the other image corre-sponds to the videoscope image in Figure 2.

Figure 6 shows the matching of feature points usingthe KD tree. In this case, the tree was constructed usingfeature vectors extracted from the upper left image inFigure 6 and the other feature vectors extracted fromthe lower left image in Figure 6 entered into the con-structed tree. As shown in the picture on the right inFigure 6, matching results using the KD tree are quitecomplicated because all points are connected to similarones. Therefore, it is necessary to remove the outliersso that only the meaningful matches in these resultscan be considered. In the case of Figure 6, the meaning-ful matches are those that occur at the damaged part,as that is the only feature common to the two images.

RANSAC. In this research, RANSAC was used toremove outliers from the complex matching results

using the KD tree. RANSAC is a mathematical tech-nique used to estimate the parameters of a model froma randomly sampled data set. The total observed datacontain both inliers and outliers, so the RANSAC algo-rithm uses the random sampling and voting scheme toget the best fitting result. Therefore, as a first step, asample sub-dataset is randomly selected from the totalobserved data, and the parameters of a mathematicalmodel are computed. In the second step, this algorithmevaluates how well the computed model matches theremaining data.

In this article, a basic assumption used for the math-ematical modeling is that if feature points matchingoccurs between identical types of damage, the featurepoints on one side will all undergo the same geometrictransformation. Because the geometric transformationshould include scaling, translation, and rotation, it canbe expressed using a homogeneous coordinate system,as shown in equation (6). This geometric transforma-tion maps ai

! to bi

!, as shown in equation (7)

T =t11 t12 0

t21 t22 0

t31 t32 1

0@

1A ð6Þ

Figure 5. Scale invariant feature points extraction.

6 Structural Health Monitoring 00(0)

bi1 bi2 1ð Þ= ai1 ai2 1ð Þt11 t12 0

t21 t22 0

t31 t32 1

0@

1A ð7Þ

In this case, the error function can be defined as inequation (8). The error function can partially be differ-entiated into six parameters of the geometric transfor-mation so that the condition expressed by equation (9)can be derived. This means that it is possible to calcu-late the geometric transformation matrix that mini-mizes the error through the information obtained fromonly three matching pairs. Therefore, to remove theoutliers from the matching result using the RANSACalgorithm, three matching pairs should be selected ran-domly and the geometric transformation computedusing the three pairs. The number of matches that arecorrectly transformed is computed through the geo-metric transformation matrix

E =Xn

i = 1

bi

!� ai!T

��� ��� ð8Þ

∂E

∂tij

= 0 ð9Þ

Figure 7 shows the results after removing the outliersusing the RANSAC algorithm. Only the matches thatoccurred within the damaged part were retained, whileall other matches, being outliers, were removed. Inaddition, as shown by the right-hand images in Figure7, the RANSAC results are slightly different. This is

because the RANSAC algorithm works by generatingthree random numbers. The results were slightly differ-ent each time, but the outliers were efficiently removedin all cases. This property has the effect of making thewhole algorithm robust.

Figure 8 shows the results of applying the RANSACalgorithm to various cases. The algorithm works welleven if the size of damage was reduced. It works equallywell when using cropped images of the damaged part.Therefore, it is possible to determine the location of thedamaged part using this method, assuming that infor-mation regarding the part is known. However, as it isimpossible to know in advance whether the videoscopeimage contains a damaged part, the result obtainedfrom this method does not always mean that a part isdamaged. Considering the case where there is no dam-aged part in the videoscope image, the result obtainedusing this method was defined in this article as a sus-pected damaged part.

Accuracy of preprocessing

To check the accuracy of preprocessing, we prepared 30damaged blade images. This preprocessing alwaysextracts suspected region even if there is no damage inthe blade image; therefore, it is meaningless to prepareintact blade images. What we want to evaluate is howwell a damaged part matches with the eigendamage.The eigendamage and the 30 engine blade images weresubjected to the preprocessing, and the true-positive

Figure 6. Matching result using K-dimensional tree.

Kim and Lee 7

Figure 7. Outlier removal results by random sample consensus.

Figure 8. Results from the application of the random sample consensus algorithm to various cases.

8 Structural Health Monitoring 00(0)

and false-negative rates are shown in Table 1. In 26images, the damaged site was matched exactly, butthere were four wrong matches in the other four images.Therefore, for the single image, the true-positive rate is86.67% and the false-negative rate is 13.33%. However,it should be noted that we are processing a video clip.Figure 9 shows more detailed results. In Figure 9, it canbe confirmed that the damaged parts were connected toeach other in the eigendamage on the left side and thevideoscope image on the right side. The matchingoccurs at the damaged area regardless of the angle andscale of the videoscope image. The accuracy of 86.67%is not high, but the inspection video is composed of a

large number of single image. Therefore, damaged areacan be sufficiently detected with this accuracy. It is notpossible to find the exact damage of all the frames whilethe damage appears in the image and disappears.However, in the meantime, it can provide enough warn-ing to the inspector. For example, the probability thatthis preprocessing will not find damage in four consecu-tive frames is 0.0316%. Therefore, this preprocessingdoes not miss the damage in the actual inspection.Moreover, what is as important as the true-positive rateat the actual inspection site is to reduce the false-negative and false-positive rates. In this method, thefalse-negative rate and false-positive rate are almostzero because we have well-trained CNN which is ableto properly classify the suspected region. This was pos-sible because of combination of two-step detectionmethod with the proposed preprocessing and CNN.

CNN

Database construction

In order to recognize a desired object by learningthrough a CNN, it is important to build a learning

Table 1. Confusion table for results of preprocessing of a singleframe.

Damage matched Damage not matched

Damaged blade(30 images)

26 (TP = 86.67%) 4 (FN = 13.33%)

Intact blade(no images)

0 (FP = 0) 0 (TN = 0)

TP: true positive; FN: false negative; FP: false positive; TN: true negative.

Figure 9. Results of preprocessing.

Kim and Lee 9

database first. In this study, the database was con-structed as shown in Figure 10. We used the wholeimage of the engine blade as well as parts of it. Thereare several rectangles in Figure 10 that represent actualdata, cropped and resized to 150 pixels in height andwidth. There are four types of rectangles by color: thered rectangles represent the damaged blade edge, theblue rectangles denote the intact blade edge, the greenrectangles indicate the surface of the structure, whilethe purple rectangles represent the connection betweenthe blades and center shaft. In summary, the databaseconsists of one damaged class and three normal classes.

The database was constructed by subdividing thenormal class into 3 to reduce false-positive rates. In thepast, the three classes were integrated into one normalclass. This led to false positives such as recognizingnormal parts as damaged. In fact, the three classes areall normal, but possess different characteristics. Theintact blade edge class is characterized as a straight lineboundary because of the blade edge, while the surfaceclass is characterized as a gentle contrast value distribu-tion. In the connection class, a dark region betweentwo boundaries is a major feature. The three subdivi-sions of the normal class are selected as the regions

frequently extracted as suspected damaged regionsfrom the preprocessing phase.

The total number of data in the database was 420.As shown in Table 2, the damaged blade edge classconsists of 110 images. The intact blade edge class andthe surface class each contain 100 images and the con-nection class consists of 110 images. All the data usedwere color images of 150 pixels in height and width.Therefore, when the data were represented as an array,it had a size of 150 3 150 3 3, where the last numberis the red, green, and blue (RGB) values of the colorimage.

A total of 20 image data from each class are shownin Figures 11 to 14 that offers at a glance the effect ofconstructing a database by cropping the images to acertain size. The differences between the classes to beclassified through the CNN become clearer. Moreover,the database contained little unnecessary informationbecause only the regions of interest were used.Therefore, the classification accuracy is expected toimprove.

Training and validation test

At the beginning of this research, widely known CNNarchitecture such as AlexNet8 was used for reference.However, by extracting only suspected region of thevideoscope image through the preprocessing, a relativelysimple and accurate CNN structure was required.Therefore, various filter sizes were tested, and the currentstructure was selected as a result. The criterion for selec-tion was accuracy of learning. Although the CNN struc-ture alone does not determine the accuracy, the accuracyallowed us to determine the number of convolutionlayers and fully connected layers that are important.

Because the size of the input images was determinedas 150 3 150 3 3 pixels through the database buildingprocess outlined above, the CNN can be designedaccordingly. Table 3 shows the details of the CNNdesign. The CNN used in this research consists of twoconvolution layers and two fully connected layers, witha rectified linear unit (ReLU) used as an activationfunction. Max pooling was then applied with overlap-ping in order to minimize the loss of feature maps cre-ated by convolution layers, while dropout was used tocope with the overfitting problem.17 Finally, the designobtains the normalized output value using the softmaxfunction.

Subsequently, it is necessary to learn the designedCNN using the database. Learning a CNN is a type ofoptimization problem because it involves updating theparameters to minimize error. Stochastic gradient des-cent (SGD) method was employed as the optimizationalgorithm. The database was divided into a validationset that consists of 40 randomly selected images from

Figure 10. Examples of a database configuration.

Table 2. Number of data for each class.

Class #1Damagedblade edge

Class #2Intactblade edge

Class #3Structuralsurface

Class #4Bladeconnection

Number ofdata

110 100 100 110

10 Structural Health Monitoring 00(0)

Figure 11. Class #1: damaged blade edges.

Figure 12. Class #2: intact blade edges.

Kim and Lee 11

the database and a training set corresponding to therest of the database. In other words, the validation andtraining sets are independent of each other. In sum-mary, the designed CNN was learned with 380 images,while the performance was confirmed using 40 images.

The CNN was learned over 2000 times and Figure15 shows how CNN accuracy changes during learning.The initial learning rate was 0.0001 and after learninghas progressed 1000 times, this value becomes half. Asthe learning progresses more than 1600 times, the

Figure 14. Class #4: blade connections.

Figure 13. Class #3: surfaces.

12 Structural Health Monitoring 00(0)

accuracy improvement rate was slowed down andalmost saturated. GTX 1080 of NVIDIA was used toshorten learning time, which reduced the entire processto 40 s. The 2000 iterative learning was performed 50times with randomly divided data to obtain statisticallymeaningful results. The average classification accuracywas 95.2% for 50 validation tests. In Figure 16, forexample, only one of the 40 images was misclassified(the classification accuracy in this case was 97.5%).This false negative happens sometimes even for thesame image. This error arose from the classifier filingthe misclassified data into the surface class due to thetiny size of the damage. For the readers’ reference, thedatabase was obtained with a unit of pixels because thereal size of damages is classified information. The fieldinspector estimated that the damage size is about100 mm and explained that this is too tiny to be deter-mined as a damage in the real field. In addition, sincewe process five video images per second in the field, thepossibility to miss even this tiny damage becomes zerobecause the video showing the damage area has manyimage frames depending on the inspector’s speed rotat-ing the blades.

Comparative studies

Comparison with faster RCNN

The faster RCNN is an excellent example of effectivelysolving object detection problem, and the basis is theregion proposal. In the RCNN and fast RCNN, selec-tive search method was used for region proposals.18–20

In the faster RCNN, the region proposal stage was alsointegrated into the network, which improved detectionaccuracy and speed.11 The faster RCNN is a combina-tion of the RPN and fast RCNN. In the proposedapproach, the preprocessing can be considered as a roleof region proposal. Therefore, the RPN of fasterRCNN was compared with the preprocessing of theproposed method in this article. Details are given in fol-lowing sections.

Implementation details. To implement the faster RCNN,data labeling is required. Labeling means to represent atarget object to be detected with a bounding box. Thisbounding box should be set carefully because it is usedas the ground truth later. Figure 17 shows the labelingof the damage part by drawing bounding boxes at thedamaged blade. A total of 45 images were labeled in thesame manner as shown in Figure 17 and used for train-ing RPN.

The CNN structure used for faster RCNN is shownin Table 4. This CNN structure was designed consider-ing the minimum size of the damage region in thelabeled data. Two convolution layers and two fullyconnected layers were utilized and one max poolinglayer was inserted between convolution layers and fullyconnected layers. The four-step alternating trainingwas adopted in the same way as in Ren e al.11 and theregion proposals that overlap with the ground truthbounding boxes within the range of 0.3 or less wereused as negative training samples. The range of 0.6 ormore corresponded to positive training samples.

Table 3. Details of the convolutional neural networks.

Order Type Filter size The number of filters Stride Zero padding

1 Convolution 12 3 12 3 3 20 2 02 ReLU – – – -3 Max pooling 3 3 3 3 1 – 2 04 Convolution 9 3 9 3 20 96 2 05 ReLU – – – –6 Max pooling 3 3 3 3 1 – 2 07 Fully connected 6 3 6 3 96 128 1 08 ReLU – – – –9 Dropout – – – –10 Fully connected 1 3 1 3 128 4 1 011 Softmax – – – –

ReLU: rectified linear unit.

Figure 15. Accuracy change according to iterative learning.

Kim and Lee 13

Region proposals and detection results. After the four-stepalternating training, we attempted to detect the dam-aged parts using the trained faster RCNN. However, itwas confirmed that the region proposals were not cre-ated correctly in most of the damaged blade images.Figure 18 shows two representative results. The firstcase is that the region proposal does not adequatelycover the damaged part. The second case is that the

region proposals are too much in the wrong place.What we can infer from these results is that the RPNhas not been properly learned from data because thenotched blade or nick damage is not a region but aboundary. Therefore, it is thought that this methodfinding the region by applying nine anchors was notsuitable for this type of damage. However, this result isstill controversial because delamination was also a typeof boundary but it was correctly detected by the fasterRCNN.10 More optimized implementation of theRCNN is beyond the scope of this work.

Comparison with NNs

We have fully explained through the previous sectionsthat the preprocessing in the proposed method is effec-tive. However, the existence of this preprocessing lim-ited the role of CNN to a simple classification problem.Therefore, here we discuss the necessity of CNN basedon an experiment comparing the performance of CNNwith that of the traditional NN.

Implementation details. First, all the data used in CNNlearning and evaluation were converted into a vectorform and the new database was rebuilt. Since the image

Figure 16. Classification results.

Figure 17. Bounding boxes for damage parts of blade images.

14 Structural Health Monitoring 00(0)

was all 150 3 150 in pixel, the length of the vector is22,500. We can design hidden layers based on the sizeof input vector. In this study, we replaced two

convolution layers used in CNN with two fully con-nected layers, meaning to compare NN with CNN. Asa result, NN had a total of four fully connected layers,

Figure 18. Region proposals and detection results using the trained faster RCNN.

Table 4. Details of the faster RCNN structure implemented in this article.

Order Type Size The number of filters Stride Zero padding

1 Input layer 32 3 32 3 3 – – –2 Convolution 3 3 3 3 3 32 1 13 ReLU – – – –4 Convolution 3 3 3 3 32 32 1 15 ReLU – – –6 Max pooling 3 3 3 – 2 07 Fully connected 16 3 16 3 32 64 1 08 ReLU – – – –9 Fully connected 1 3 1 3 64 4 1 010 Softmax – – – –

RCNN: faster region–based convolutional neural network; ReLU: rectified linear unit.

Table 5. Details of the NN.

Order Type Size The number of filters Stride Zero padding

1 Input vector 22,500 3 1 – – –2 Fully connected 22,500 3 1 4096 1 03 ReLU – – – –4 Fully connected 4096 3 1 1024 1 05 ReLU – – – –6 Fully connected 1024 3 1 128 1 07 ReLU – – – –8 Fully connected 128 3 1 4 1 09 Softmax – – – –

NN: neural network; ReLU: rectified linear unit.

Kim and Lee 15

and ReLU was used as an activation function betweeneach fully connected layer. The detailed structure of theNN is also shown in Table 5. As the same manner ofthe previous CNN, 380 data were used to train NN,and the remaining 40 data were used to evaluate thetrained NN. A total of 40 pieces of data were arbitra-rily selected from the whole data and this process wasrepeated 50 times.

Classification results. Learning was repeated 2000 timesfor one dataset as same as the CNN. However, what isnoteworthy here is that unlike CNN, the NN did notshow any improvement by learning after 50 repetitions.By adjusting the learning rate, we could change the ten-dency slightly, but the saturation always occurred atthe final accuracy of 25%–26%. In addition, the classi-fication accuracy for the evaluation data was always25% exactly. Since we have four classes, this is exactlythe same level of accuracy as selecting one randomlyand means that the NN did not work well. This is in

stark contrast to the CNN that showed an average ofmore than 95% accuracy.

When CNN and NN are compared in general, thenumber of learning parameters of CNN is muchsmaller, and spatial information of image is much bet-ter preserved in CNN. In addition, CNN’s max poolinglayer effectively collects and enhances features fromimage, so the effect of the max pooling layer cannot beignored. Although CNN seems to be doing a simpletask of classifying the results of the preprocessing inthis method, it is actually a complex task for NN to doit instead of CNN. Therefore, CNN is essential in thisarticle’s approach.

Field validation test results

Fan blade inspection

Figure 19 shows the result from the fan blade inspec-tion in Stage II. Because there was no damage at thissite, it can be confirmed that damage was not detectedin the inspection result without any false positive. In

Figure 19. Inspection results for Stage II fan blades (root).

16 Structural Health Monitoring 00(0)

the inspection images, the black squares indicate theregions that the preprocessing stage suspected to bedamaged. The blade edges and connections were oftenselected as suspected damaged regions, but the CNNsuccessfully determined that these regions were freefrom damage. On the contrary, Figure 20 shows thatthere was actual damage that was correctly detected.As the blade rotates, the damage position changes. Thedamage was efficiently detected and marked by a redsquare despite the rotation of the blade and the verysmall size of the damage. If the CNN determines thatthere is a damage, the black squares turn red and pro-duce a notification on the monitor with an alarm soundfor the inspector.

High-pressure compressor blade inspection

Inspection was also conducted on the high-pressurecompressor blades as well as the fan blades. Figure 21shows the results of the inspection of the high-pressurecompressor blades in Stage I. In this case, no damage

was detected in the result because no actual damagewas present. The connection parts were mostly selectedas suspected damaged regions, but were all classified asnormal by the well-learned CNN. Conversely, Figure22 shows that a relatively large damage was detected.In this case, the blade was rotating in an upward direc-tion, thus the damage appears at the bottom of theimage and disappears upward. The damage was presentfor 46 frames and this method failed to detect the dam-age in eight frames. The two undetected frames werethe images where the damage begins to appear, and theother six frames were the images where the damage dis-appears. In all remaining frames, the damage was suc-cessfully detected. It should be noted that even thoughthe damage is detected in only one frame, the systemalarms successfully.

The field feasibility test introduced in this sectionshows that the manual rotation of the rotor can bemuch faster than the human-based inspection but theCNN can classify five video images per second. In addi-tion, even though the developed CNN makes

Figure 20. Inspection results for Stage II fan blades (mid).

Kim and Lee 17

Figure 21. Inspection results for Stage I high-pressure compressor blades (root).

Figure 22. Inspection results for Stage I high-pressure compressor blades (tip).

18 Structural Health Monitoring 00(0)

classification error for each image with less than 5%possibility as mentioned above, the video containsmany image frames per second and thus the CNN waseventually able to show 100% detectability of the dam-ages. More importantly, this algorithm includes a train-ing interface for upgrading the CNN, as presented inFigure 1 that makes it possible to input the currentlydetected damage image together with three normalimages from each class.

Conclusion

In this research, we developed an algorithm that canrecognize damages to engine blades. This algorithmconsists of preprocessing, CNN and GUI components.The preprocessor was developed using several image-processing techniques to extract suspected damagedregions from a videoscope image. The pre-trained CNNthen receives the suspected damaged regions and judgeswhether they are actually damaged or not. This algo-rithm was developed in software and verified using sev-eral field tests. In the field tests, engine fan blades andhigh-pressure compressor blades were inspected anddamages successfully detected.

Image-processing techniques related to featureextraction and matching were mostly used for the pre-processor. This is because the concept of feature pointsis very helpful in object recognition. In this study, weemployed the method of extracting the feature pointsfrom the eigendamage in advance and comparing themwith the feature points extracted from the videoscopeimage. This way, it is possible to find the regions mostsimilar to the eigendamage (obtained through PCA) inthe videoscope image.

The CNN designed in this study comprised two con-volution layers and two fully connected layers, and webuilt a database of 420 image data in order to learn it.The database consisted of four classes: three normalclasses and one damage class. Through 2000 iterativelearnings, a CNN with an average classification accu-racy of 95% or higher was developed. The role of theCNN is to determine whether a suspected damagedregion is actually damaged.

Damaged blade data are very rare in the real field.Nevertheless, the CNN combined with a preprocessorsuccessfully carried out a field validation test, even witha small data size. In addition, the GUI includes a func-tion for self-training when there is a new damage imagein the field application. Therefore, we confirm that thisalgorithm will improve existing videoscope inspectionmethods. This is equally the view of two inspectorsusing the proposed technology to work together. In thenear future, the technology will develop to a level

whereby this artificial intelligence inspector can con-duct inspection tests alone and unassisted.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest withrespect to the research, authorship, and/or publication of thisarticle.

Funding

The author(s) disclosed receipt of the following financialsupport for the research, authorship, and/or publication ofthis article: This work was supported by the Air ForceLogistics Command (AFLC) and by the Ministry of Trade,Industry, and Energy (MOTIE), Korea (program reference

number R0006462) supervised by the Korea institute forAdvancement of Technology (KIAT).

ORCID iD

Jung-Ryul Lee https://orcid.org/0000-0003-0742-4722

References

1. Doebling SW, Farrar CR and Prime MB. A summary

review of vibration-based damage identification methods.

Shock Vib Dig 1998; 30: 91–105.2. He K and Zhu WD. Structural damage detection using

changes in natural frequencies: theory and applications.

In: International conference on damage assessment of

structures, Oxford, 11–13 July 2011.

3. Manoach E, Warminski J, Kloda L, et al. Vibration

based method for damage detection in structures.

MATEC Web Conf 2016; 83: 05007.4. Zhang C, Cheng L, Qiu J, et al. Damage detection based

on sparse virtual element boundary measurement using

metal-core piezoelectric fiber. Struct Health Monit 2018;

17: 15–23.5. Li J, Deng J and Xie W. Damage detection with stream-

lined structural health monitoring data. Sensors 2015; 15:

8832–8851.6. Ou Y, Chatzi EN, Dertimanis VK, et al. Vibration-based

experimental damage detection of a small-scale wind tur-

bine blade. Struct Health Monit 2017; 16: 79–96.7. Sakaris CS, Sakellariou JS and Fassois SD. Random-

vibration-based damage detection and precise localiza-

tion on a lab-scale aircraft stabilizer structure via the

Generalized Functional Model Based Method. Struct

Health Monit 2017; 16: 594–610.8. Krizhevsky A, Sutskever I and Hinton GE. ImageNet

classification with deep convolutional neural networks.

In: Neural information processing systems, Lake Tahoe,

NV, 3–8 December 2012.9. Cha YJ, Choi W, Suh G, et al. Deep learning-based crack

damage detection using convolutional neural networks.

Comp Aided Civil Infrastruct Eng 2017; 32: 361–378.10. Cha YJ, Choi W, Suh G, et al. Autonomous structural

visual inspection using region-based deep learning for

Kim and Lee 19

detecting multiple damage types. Comp Aided Civil Infra-

struct Eng 2017; 33: 731–747.11. Ren S, He K, Girshick R, et al. Faster R-CNN: towards

real-time object detection with region proposal networks.In: Conference of computer vision and pattern recognition,2016, https://arxiv.org/abs/1506.01497

12. Paul LC and Sumam AA. Face recognition using princi-pal component analysis method. Int J Adv Res Comp Eng

Technol 2012; 1: 135–139.13. Anan CS and Hartley R. Optimised KD-trees for fast

image descriptor matching. In: Conference on computer

vision and pattern recognition, Anchorage, AK, 23–28June 2008.

14. Brown M and Lowe D. Recognising panoramas. In:International conference on computer vision, Nice, 13–16October 2003.

15. Turk MA and Pentland AP. Face recognition using eigen-faces. In: Conference on computer vision and pattern recog-

nition, Maui, HI, 3–6 June 1991.16. Lowe D. Distinctive image features from scale-invariant

keypoints. Int J Comp Vis 2004; 60: 91–110.17. Hinton GE, Srivastava N, Krizhevsky A, et al. Improv-

ing neural networks by preventing co-adaptation of fea-ture detectors, 2012, https://arxiv.org/abs/1207.0580

18. Girshick R, Donahue J, Darrell T, et al. Rich featurehierarchies for accurate and semantic segmentation. In:Conference on computer vision and pattern recognition,Columbus, OH, 23–28 June 2014.

19. Girshick R and Fast R-CNN. In: International conferenceon computer vision, Santiago, 7–13 December 2015.

20. Uijling J, van de Sande K, Gervers T, et al. Selective searchfor object recognition. Int J Comp Vis 2013; 104: 154–171.

20 Structural Health Monitoring 00(0)