13
Research Article Electric Shovel Teeth Missing Detection Method Based on Deep Learning Xiaobo Liu, 1 Xianglong Qi , 2 and Yiming Jiang 3 1 Shenyang Institute of Automation Chinese Academy of Sciences, Shenyang 110169, China 2 Liaoning Huading Technology Co.Ltd, Shenyang, Liaoning 110167, China 3 State Grid Liaoning Electric Power Company Limited Material Branch Company, Shenyang, China Correspondence should be addressed to Xianglong Qi; [email protected] Received 30 August 2021; Revised 17 September 2021; Accepted 29 September 2021; Published 22 November 2021 Academic Editor: Bai Yuan Ding Copyright © 2021 Xiaobo Liu et al. is is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Electric shovels are widely used in the mining industry to dig ore, and the teeth in shovels’ bucket can be lost due to the tremendous pressure exerted by ore materials during operation. When the teeth fall off and enter the crusher with other ore materials, serious damages to crusher gears and other equipment happen, which causes millions of economic loss, because it is made of high-manganese steel. us, it is urgent to develop an efficient and automatic algorithm for detecting broken teeth. However, existing methods for detecting broken teeth have little effect and most research studies depended on sensor skills, which will be disturbed by closed cavity in shovel and not stable in practice. In this paper, we present an intelligent computer vision system for monitoring teeth condition and detecting missing teeth. Since the pixel-level algorithm is carried out, the amount of calculation should be reduced to improve the superiority of the algorithm. To release computational pressure of subsequent work, salient detection based on deep learning is proposed for extracting the key frame images from video flow taken by the camera installed on the shovel including the teeth we intend to analyze. Additionally, in order to more efficiently monitor teeth condition and detect missing teeth, semantic segmentation based on deep learning is processed to get the relative position of the teeth in the image. Once semantic segmentation is done, floating images containing the shape of teeth are obtained. en, to detect missing teeth effectively, image registration is proposed. Finally, the result of image registration shows whether teeth are missing or not, and the system will immediately alert staff to check the shovel when teeth fall off. rough sufficient experiments, statistical result had demonstrated superiority of our presented model that serves more promising prospect in mining industry. 1. Introduction Shovel excavators are widely used as primary production equipment in surface mining for removing overburden and ore materials due to their flexibility and mobility [1]. ey digtheworkfrontandloadminingtrucksinathree-orfour- pass loading cycle. e shovel consists of three major as- semblies: lower frame, upper frame, and attachment bucket [2], which is responsible for digging up the payload to the trucks. e bucket tooth of the loader is an important component of the loader structure [3], as well as a vulnerable part;themountingpartofthebuckettoothisatthefrontend of the bucket in order to reduce friction [4]. In the process of shoveling loading, the bucket tooth is in contact with the rock material directly, so its working condition is very bad; it not only increases the wear amount of the bucket tooth but also causes the teeth of the bucket to break and fall off. After the tooth is broken, it will cause a series of troubles, such as increasing the frictional resistance of the shovel during the loading operation, accelerating the wear of the bucket, and affecting its service life. e tooth must be replaced in time if it breaks. According to statistics, there are currently 2,000 to 2,500 large-scale mining excavators in service in China, and the annual direct economic loss due to the failure of the tooth reaches approximately 30 million yuan. What is even worse is that the teeth made of high-manganese steel have much higher hardness than the ore material. If the hard tooth enters the crusher along with the ore material, serious Hindawi Computational Intelligence and Neuroscience Volume 2021, Article ID 6503029, 13 pages https://doi.org/10.1155/2021/6503029

ElectricShovelTeethMissingDetectionMethodBasedon DeepLearning

  • Upload
    others

  • View
    8

  • Download
    0

Embed Size (px)

Citation preview

Research ArticleElectric Shovel Teeth Missing Detection Method Based onDeep Learning

Xiaobo Liu1 Xianglong Qi 2 and Yiming Jiang3

1Shenyang Institute of Automation Chinese Academy of Sciences Shenyang 110169 China2Liaoning Huading Technology CoLtd Shenyang Liaoning 110167 China3State Grid Liaoning Electric Power Company Limited Material Branch Company Shenyang China

Correspondence should be addressed to Xianglong Qi 2010513stuneueducn

Received 30 August 2021 Revised 17 September 2021 Accepted 29 September 2021 Published 22 November 2021

Academic Editor Bai Yuan Ding

Copyright copy 2021 Xiaobo Liu et al +is is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Electric shovels are widely used in the mining industry to dig ore and the teeth in shovelsrsquo bucket can be lost due to thetremendous pressure exerted by ore materials during operation When the teeth fall off and enter the crusher with other orematerials serious damages to crusher gears and other equipment happen which causes millions of economic loss because it ismade of high-manganese steel +us it is urgent to develop an efficient and automatic algorithm for detecting broken teethHowever existing methods for detecting broken teeth have little effect and most research studies depended on sensor skills whichwill be disturbed by closed cavity in shovel and not stable in practice In this paper we present an intelligent computer visionsystem for monitoring teeth condition and detecting missing teeth Since the pixel-level algorithm is carried out the amount ofcalculation should be reduced to improve the superiority of the algorithm To release computational pressure of subsequent worksalient detection based on deep learning is proposed for extracting the key frame images from video flow taken by the camerainstalled on the shovel including the teeth we intend to analyze Additionally in order to more efficiently monitor teeth conditionand detect missing teeth semantic segmentation based on deep learning is processed to get the relative position of the teeth in theimage Once semantic segmentation is done floating images containing the shape of teeth are obtained +en to detect missingteeth effectively image registration is proposed Finally the result of image registration shows whether teeth are missing or notand the system will immediately alert staff to check the shovel when teeth fall off +rough sufficient experiments statistical resulthad demonstrated superiority of our presented model that serves more promising prospect in mining industry

1 Introduction

Shovel excavators are widely used as primary productionequipment in surface mining for removing overburden andore materials due to their flexibility and mobility [1] +eydig the work front and loadmining trucks in a three- or four-pass loading cycle +e shovel consists of three major as-semblies lower frame upper frame and attachment bucket[2] which is responsible for digging up the payload to thetrucks +e bucket tooth of the loader is an importantcomponent of the loader structure [3] as well as a vulnerablepart the mounting part of the bucket tooth is at the front endof the bucket in order to reduce friction [4] In the process ofshoveling loading the bucket tooth is in contact with the

rock material directly so its working condition is very bad itnot only increases the wear amount of the bucket tooth butalso causes the teeth of the bucket to break and fall off Afterthe tooth is broken it will cause a series of troubles such asincreasing the frictional resistance of the shovel during theloading operation accelerating the wear of the bucket andaffecting its service life +e tooth must be replaced in time ifit breaks According to statistics there are currently 2000 to2500 large-scale mining excavators in service in China andthe annual direct economic loss due to the failure of thetooth reaches approximately 30 million yuan What is evenworse is that the teeth made of high-manganese steel havemuch higher hardness than the ore material If the hardtooth enters the crusher along with the ore material serious

HindawiComputational Intelligence and NeuroscienceVolume 2021 Article ID 6503029 13 pageshttpsdoiorg10115520216503029

damage to the crusher gears and the production line isexpected At present it is impossible to completely avoid thefracture of the tooth although many scholars have carriedout a lot of research work from the aspects of the toothmechanical analysis [5] the optimum replacement time forshovel teeth [6] the tooth material [7] and the heattreatment method [8]

In this case machine vision technology combined withan automatic detection algorithm can monitor the workingstate of the tooth in real-time When the tooth is falling off itcan provide accurate warning information and instruct thestaff to deal with it in time which has important practicalsignificance and economic value for the production of themine Many scholars and enterprises have done a lot ofresearch work on the detection of the shovel teeth breakingand falling off Xiujuan Luo proposed a detection systemsolution which uses a laser range finder to scan and re-construct the three-dimensional model of the tooth in theworking state and then compare it with the original model todetermine whether there is any tooth drop in 2004 [9] Toimprove the calculation speed it uses the parallel computingmethod to collect and process the data synchronously so thatthe whole detection system has a better real-time perfor-mance +is method is less affected by changes in illumi-nation and environment and has a good detection effectcompared with machine vision detection methods He Liproposed a novel object detection algorithm with thecombination of appearance structural and shape features in2011 [10] It follows the paradigm of object detection byparts which first uses detectors developed for individualparts of an object and then imposes structural constraintsamong the parts for the detection of the entire object +eyfurther incorporate a priori shape information about theobject parts for their detection to improve the performanceof the object detector For validation of this object detectionalgorithm they apply it to the detection of the tooth line of amining shovel under various lighting conditions +e ex-perimental results demonstrate that this system can improvethe detection performance significantly when part shapeinformation is used Ser Nam Lim presented a vision systemfor monitoring tooth condition and detecting missing toothfor a mining shovel in 2016 [11] First it extracts the imagesample of the bucket during the movement of the shovel anduses this sample information to locate the approximateposition of the tooth Second it uses the frame differencemethod and the optical flowmethod to correct the result andthen uses the template matching and the tooth line fittingmethod to accurately locate the tooth target Finally itcombines the grayscale features of the tooth image to de-termine if the tooth is falling off +e outstanding perfor-mance and high reliability of the proposed system have beendemonstrated by experiments on video sequences collectedfrom an iron ore mining site and a two-month trial of aninstalled unit on a production line Abdallah Almarsquoaitahutilized Sprouts WSN platform to monitor the steel shovelteeth on shoveling equipment used in oil sands miningoperations in Ft McMurray Alberta Canada [12] +eSprouts platform is utilized to monitor the viability of theshovel teeth on the shovel bucketrsquos teeth adapter Besides if a

shovel-tooth becomes detached Sprouts is used to estimateits location Hannington focuses on a method to prevent theteeth from ending up in the crusher based on an infrareddetection system that reads the bucket profile every time ithauls the ore onto the truck in 2008 At the same timeanother detection system available on the market has beenstudied and a probable preventive maintenance program hasbeen suggested [13]

With the rapid development of deep learning skills inmultiple areas [14ndash17] in this paper a novel electric shovelteethmissing detectionmethod which includes the followingsignificant steps is proposed to monitor teeth condition anddetect missing teeth At first salient object detection basedon deep learning is presented for extracting crucial frames invideo flow efficiently and releasing computational pressureof the terminal end Additionally the working environmentof shovel teeth is relatively special and the messy ore willproduce a lot of noise in the frame images Semantic seg-mentation is proposed to get the relative position of teeth ina noisy background Finally to detect the broken teeth animage registration approach is introduced With the pro-posed procedure without human being affected the electricshovel teeth missing detection method is ensured auto-matically and results in a high-accuracy rate Our proposedprocedure for missing tooth detection serves a fully auto-matic quality which provides great significance in miningindustry If this work is supported by human resources itshould increase dangerous risks especially in low visibleenvironments Moreover depending on real-time humansupervised detection the accuracy cannot be ensured andwill result in great costs from economic perspectives

Our main contributions are summarized as follows

(1) A salient object detection based on deep learning isproposed for deleting useless frame images and re-leasing computational pressure of subsequent workBecause salient detection is completed at the edgethe efficiency of the pixel-level algorithm isimproved

(2) Semantic segmentation can find the relative positionof teeth in the frame images and solve the noiseproblem generated by the working environment ofthe electric shovel Compared with the previousmethod based on machine vision the experimentalresult of semantic segmentation is more effective

(3) Image registration converts teeth into rectangularstructures by processing the results of the imageregistration experiment the proposed strategy iscapable of handling the system uncertainties causedby the working environment of electric shovel teeth+e final results show that the electric shovel teethdetection algorithm works with high accuracy inactual production

2 Methodology

+e proposed method is an intelligent computer visionsystem which consists of salient object detection imagesegmentation and image registration Generally the

2 Computational Intelligence and Neuroscience

number of shovel teeth (M) is fixed M 5 +is invariantvalue can be taken as an indicator of the detection of missingteeth and key frame extraction By setting the number ofdetected teeth as S we adapt the following formula to de-termine whether the teeth are missing or not

S 0

0lt SltM

S M

⎧⎪⎪⎨

⎪⎪⎩(1)

When S 0 there is no tooth in the image when0lt SltM the teeth are broken when S M it is in a normalstate

21 Salient Object Detection In the mining process dozensof mining machines are working at the same time To moreefficiently detect whether the tooth is missing on the shovelin each machine a camera is installed on each shovel tomonitor the status of the teeth +e position of the camera issignificant for the detected system +e best position shouldallow the mounted camera to be stable during operation asshown in Figure 1 In the process of monitoring the state ofteeth the frame images are extracted from the video frameobtained by the camera for edge calculation If every frameimage in the video stream is sent to the server terminal thecomputation will be very large which will reduce thecomputational efficiency and require highly on the servicefacilities

To alleviate the pressure of the data processing centerand ensure the cost of facilities we proposed a strategy basedon salient object detection to obtain more significant framesin video flow and delete unnecessary frames +e purpose ofthis paper to detect whether the tooth on the shovel ismissing or not and give a timely alarm when the tooth ismissing +erefore when there is no tooth in the frameimage we can identify that this frame image is an unnec-essary frame image +e function of salient object detectionis to find the region of interest (ROI) and obviously in thispaper our ROI is the teeth +e idea of salient object de-tection in deep learning depends on pixel values in featuremaps located in deeper layers of neural networks In generalpixels with great values indicate the salient section of theforeground we explore

However traditional salient object detection based ondeep learning has many problems

(i) +e training is divided into several stages and thesteps are relatively complicated fine-tuning net-work + training SVM+ training border regression

(ii) +e training is time-consuming and takes up a lot ofdisk space 5000 images produce hundreds ofgigabytes of feature files

(iii) Computing slowly it takes 47s for VGG-16 toprocess an image by using GPU

Low computational efficiency and a large amount ofcomputation are not what we are willing to see in edgecalculation Based on these shortcomings of traditionalsalient object detection and our purpose to delete

unnecessary frame images a neural network of dichotomy isproposed retaining frame images labeled with teeth anddeleting frame images with no teeth +e network structureis shown in Figure 2

+e model contains three simple convolutional layersand two full connection layers By calculating the parametersin this network it can be found that the complexity of thenetwork structure is lower than that of AlexNet +e lowcomplexity network structure determines the efficiency ofedge computation and improves the efficiency of the wholesystem under the premise of realizing salient detection

22 Image Segmentation After the salient object detectionthe key frame images are obtained To facilitate the subse-quent image registration to detect the shovel teeth imagesegmentation is carried out to extract the information ofteeth In the process of salient object detection it is noticedthat the position of teeth is relatively skewed and the size ofthe teeth is too small +erefore the common convolutionalneural networks are not able to complete high-accuracyimage segmentation because of inability to obtain infor-mation about teeth In this case the DeepLabV3+ model isadopted to realize accurate acquisition of teeth informationby using dilated convolution and finally to process imageregistration to realize missing teeth detection To guaranteethe scientific nature of the experiment the dilation rate ischanged to zero as a common convolutional neural networkfor the contrast test By contrasting the result of the normalconvolution neural network and the result of the dilatedconvolution neural network it shows that the dilatedconvolution is more effective

DeepLabV3+ is an efficient model for image segmen-tation It extends DeepLabV3+ optimization results with asimple and efficient decoder module especially along thetarget boundaries Besides in the encoder-decoder structurethe resolution of the extracted encoder features can becontrolled arbitrarily using dilated convolution to com-promise accuracy and running time Atrous Spatial PyramidPooling (ASPP) module contains four dilated convolutionallayers and one global pooling layer In the experiment of thispaper more efficient image segmentation is achieved byadjusting the dilation rate of the convolution kernel in ASPPDecoder carries out 1times 1 convolution to reduce channelsbefore the fusion of low-level information +en theupsampling of the encoderrsquos results is quadrupled andcombined with the previous convolution and 3 times 3 convo-lution is carried out Finally the final result is obtained byquadrupling the upsampling +e specific network structureis shown in Figure 3

To obtain more pixel information under the permissionof no adding parameters it contains dilated convolutionallayers in DeepLabV3+ However there is the gridding effect[18] in the dilated convolution since the convolution kernelsare not continuous in deep convolutional layers In this casethere are always some pixels not involved in the calculationMoreover it is difficult to realize the segmentation of smallobjects in the dilated convolution with a large dilation rateSince the relative size of teeth in this paper is small in the

Computational Intelligence and Neuroscience 3

image the dilated convolution with a small dilation rate isproposed to ensure the accurate segmentation of the teethTo ensure that all pixels are involved in the calculation thedilation rates in the deep convolutional layers satisfy thefollowing formula

Mi max M(i1) minus 2r(i1)M(i1) minus 2 Mi1 minus ri( 1113857 ri1113858 1113859 (2)

where Mi is the maximum dilation rate of the ith layer and ri

denotes the dilation rate of the ith layer with Mn rnIn the actual work of detection the relative position and

size of the teeth in the image recorded by the camera areconstantly changing which is a challenging difficultyDeepLabV3+ adopts a combination of dilated convolutioncascade and parallel with multiple dilation rates to effectively

(a) (b)

Figure 1 Illustration of our proposed solution on automatic electric shovel teeth detection

Input 64643

646432 323232 323232 161632 1616328864

114096

Convolution

ConvMax-pooling Pooling PoolingConv

Fully connected

Output

Figure 2 +e structure of classification network for salient detection

DeepLab-v3+Encoder

1times1 Conv

1times1 Conv

1times1 Conv

3times3 Conv

3times3 Convrate = 1

3times3 Convrate = 2

3times3 Convrate = 3

ImagePooling

low-levelfeatures

Upsampleby 4

Upsampleby 4

Decoder

Concat

Figure 3 DeepLabV3+ structure

4 Computational Intelligence and Neuroscience

extract multiscale semantic information As a result nomatter where the relative position of the teeth and no matterwhat the size of teeth the model processes image seg-mentation efficiently Also by combining image-level in-formation the ability of ASPP to extract global semanticinformation is improved to further improve the effect

23 Image Registration After the image segmentation bi-nary images containing only teeth are obtained as shown inFigure 4(a) In this section the image registration is pro-posed to detect whether the teeth are missing or not +eimage in which the number of teeth (M) is zero is deleted inthe salient object detection +erefore in the process ofimage registration the number of teeth (M) ranges from oneto five

In the process of image registration the reference imageis set up firstly In this paper the images obtained after imagesegmentation are highly similar except the relative positionof teeth is different Also the relative position of each tooth isthe same +erefore one relatively clear image obtainedfrom image segmentation with intact teeth can be used as thereference image +e final reference image shows five nearlyidentical rectangular structures at the bottom of the binaryimage as shown in Figure 4(b)

However in the segmented images the relative positionof teeth is different and the pixelsrsquo information of the imagesis also very different +e reference image and the process ofimage registration are shown in Figure 4 As shown inFigure 5 the teeth in registered images are at the bottom ofthe image and have obvious rectangular structures Whenthere is no tooth missing the registered image contains fiverectangular structures When the teeth are missing thenumber of the rectangular structure is less than fiveMoreover the registered image is a binary image which isvery beneficial to some processing methods such as edgedetection +erefore we detect the rectangles in the regis-tered image and calculate the number of rectangles to de-termine whether the teeth are missing or not

3 Experiments and Discussion

+e experiments that we conducted are divided into threemain sections (1) salient object detection (2) image seg-mentation and (3) image registration

+e first sets of experiments are conducted to deleteimages without teeth to reduce the amount of computationand the second sets of experiments are conducted to find theregion of interest (ROI) and the location of teeth in theimages Moreover it should be noticed that multiple dilationrates are proposed to extract multisemantic informationand the dilated convolution neural network is contracted toa normal convolution to demonstrate that the DeepLabV3+model works more effectively +ird image registration isconducted to detect whether the teeth are missing or not bycounting the number of rectangles

31 Datasets and Hardware +e datasets in this paper arefrom frame images of the video stream taken by the camera

on the shovel Frame images are extracted from the videostream consisting of 500 with and 500 without teeth Afterlabeling these images twenty percent of the images are usedas a validation set to test the training model and the othereighty percent of the images are used as the training set totrain the classification model +e training and inference ofour proposed model are run with two NVIDIA GTX1080 Ti11GB GPUs 32G RAM and Intel i7-7700K CPU with 8cores 420GHz Especially we did not adopt any pre-processing of noise management for our dataset in order topreserve original details and avoid possible artifacts fortraining and inference process

32 Salient Object Detection Studies To reduce the com-putation of the terminal server salient object detection isproposed to delete the images without teeth To solve theshortcomings of traditional salient object detection a di-chotomous neural network is adapted to delete the imagelabeled no teeth

321 Preprocessing +is set of experiments aims to use edgecomputing to alleviate the pressure of the terminal server+erefore to complete the edge computing faster and reducethe calculation amount the images are resized into64times64times 3

322 Parameters +e networkrsquos detailed connectivity andkernel configuration are illustrated in Figure 2 And all theconvolutional kernelsrsquo sizes are the same as the size of 3times 3+e pooling layer of every convolutional layer is Max-Pooling the kernel size is 2times 2 and the step size is 2 +enumber of this neural network is 233472 which is less thanhalf of the AlexNet Adam optimizer is adapted with alearning rate of 00001 +e batch size is set as 32 and thenumber of iterations is set as 8000

323 Training Process Normally too many iterations maycause the model to fit the noise and unnecessary features ofthe training sets and then overfitting occurs In the case ofoverfitting themodel is of high complexity and does not workto the data except the training set To avoid excessive com-putation the loss function of the validation dataset and theaccuracy of the validation dataset are adapted as the evalu-ation metrics for model selection When the accuracy of thevalidation dataset reaches a high level and the accuracy is notimproved after 10 epochs also the loss function is not de-clined the iteration is terminating in advance+e final modelis obtained In the training process the accuracy of thetraining dataset the accuracy of the validation dataset and theloss function of the validation dataset are shown in Figure 6

As shown in Figure 6 the accuracy of the validationdataset and the training dataset reaches the global maximumwithin fifty epochs and the loss function of the validationdataset does not decrease in the subsequent epochs Asshown in Figure 6 the accuracy of training and validationfluctuates periodically since some groups of sampled imagescontain lots of noise as a result of complicated wild working

Computational Intelligence and Neuroscience 5

environments which leads to poor performances of se-mantic segmentation +erefore to prevent the trainingmodel from overfitting the number of iterations is changed

into 1600 making the epochs to 64 reducing the trainingtime and calculation amount of the training model +etraining result is shown in Figure 7

(a) (b)

Figure 4 Segmented images

e reference image

e image registration process

Registered images

Segmented images

Figure 5 +e image registration process

20

15

10

05

00

09

10

08

07

06

05

04

valu

e

valu

e

0 50 100 150 200 250 300epoch

0 50 100 150 200 250 300epoch

train accval accval loss

train accval acc

Figure 6 Training metrics

6 Computational Intelligence and Neuroscience

4 Results

As can be seen from Figure 7 in the 56th epoch the accuracyof the training dataset and the accuracy and the loss functionof the validation dataset reach the optimal level +e ac-curacy of the training dataset is 9375 the accuracy of thevalidation dataset is 9687 and the loss function of thevalidation is 00965 +erefore ldquoteeth-no teeth-model-1400rdquogenerated by the model trained to the 56th epoch is adaptedto detect whether the teeth are missing or not

One video stream of a working shovel is incepted allframe images are extracted from it and the frame images areput into the trained model for prediction If the predictionresult of the image is ldquoteethrdquo this image is kept for the nextpart of the experiments Otherwise this image is deleted Inthe prediction experiment of the frame images of the cap-tured video stream the prediction results are all correctindicating that the model predicts effectively In the threeconvolutional layers of the network the feature maps ofimages with teeth and without teeth are shown in the fol-lowing figures It is through learning these features that ourmodel can correctly predict whether the images containteeth or not and then complete the salient object detection

41 Image Segmentation To detect whether the teeth aremissing or not the relative position of the teeth should befound at first +e DeepLabV3+ model is adapted forprocessing semantic segmentation to find the location of theteeth+e dilation rate is adjusted several times in our modelto avoid missing information in the convolution process+en to verify the advantages of dilated convolution thedilation rate in the network is changed to zero for a com-parison experiment

411 Preprocess First to facilitate the later training processthe frame images are resized as 512times 512+en the region ofinterest of the images is annotated through the ldquolabelme rdquo an

image labeling tool Images with teeth are labeled with thelocation of teeth and the images without teeth do not needany operation After the image labeling is completed toensure the rationality of our model the image is processedby methods such as rotation scaling and cropping and theprocessed image is added to the data as new images +elabeled image is shown in Figure 8

412 Parameters +e details of the DeepLabV3+ are shownin Figure 3 In this section the structure of themodel will notbe changed but the dilation rates are changed to prevent theinformation missing from the dilated convolution To obtainthe information of each pixel by dilated convolution thedilation rates are changed to 0 1 2 and 3 Later to prove thesuperiority of the dilated convolution for image segmen-tation the dilated rates are set as 0 0 0 and 0 for com-parison experiments

413 Training +e colorful images are transformed intograyscale images for training reducing the color differencenoise brought by the day and night +e initial learning rateis set as 001 and the learning rate will be adjusted dy-namically [19]

414 Evaluation Metrics

(i) Pixel accuracy (PA) means the ratio of correctlyclassified pixel points in all pixel points In thispaper there are only two kinds of classes of the pixelteeth and background P10 represents the totalnumber of pixels in the background predicted to bein the teeth P11 represents true positive P10 meansfalse positives P01 means false negative and P00represents true negative

PA P11 + P00

P11 + P10 + P01 + P00 (3)

train accval acc

20

15

10

05

00

valu

e10

09

08

07

06

05

04

valu

e

100 20 30 40 50 60epoch

100 20 30 40 50 60epoch

train accval accval loss

Figure 7 Training metrics (less iterations)

Computational Intelligence and Neuroscience 7

(ii) Mean intersection over union (MIOU) is the stan-dard metric of semantic segmentation It calculatesthe ratio of the intersection and union of two sets thereal value and the predicted value Calculating theintersection over union for each class and averaging itthen the MIOU is obtained as follows

MIOU 12

P11

P11 + P10 + P01+

P00

P01 + P00 + P101113888 1113889 (4)

415 Result After about 1000 epochs of training a rela-tively efficient semantic segmentation model for teeth isobtained +e variation trend of the loss function pixelaccuracy rate and MIOU in the training process is shown inFigure 9

As can be seen from Figure 9 after 1000 times oftraining all the indicators have stabilized However to avoidoverfitting caused by too many iterations the training resultof the 512th epoch is adapted as our model for semanticsegmentation It can be seen from Figure 9 that when theiterations reach the 512th epoch the loss function in oneepoch is stable at about 25 the loss function in one iterationis stable at about 03 PA is about 095 and MIOU is about075 +us the model of the 512th epoch is a very efficienttraining result +e model is adapted to process dozens ofimages that are not in the training dataset to carry outsemantic segmentation and find out the teeth in these im-ages Later verifying the predicted results manually findsthat the predicted results of the teeth are accurate Moreoverthe training dataset consists of grayscale images in the ex-periment and the images of day and night could be accu-rately segmented out the teeth +e segmented results areshown in Figure 10

416 Comparative Experiments A comparative experimentis conducted to verify the validity of the dilation convolu-tion +e dilated rates of the dilation convolution layers arechanged to zero and a new model is carried out By ob-serving the evaluation metrics of the model the comparativeexperimental results are obtained as shown in Figure 11

As shown in Figure 11 the results of the dilation con-volution are better +e model is stable at the 500th epochbut the comparative experiment will not converge until the1000th epoch Moreover the experimental results of thedilation convolution are relatively stable while the results ofthe other show great differences after each epoch +ereforethis set of comparative experiments effectively illustrates theexcellent features of fast converge and stable experimentalresults of dilation convolution in teeth semanticsegmentation

42 Image Registration After the image segmentation thebinary images with teeth are obtained +e number of teethin these binary images is considered as the metric to dis-charge whether the teeth are missing or not Since the shapeof the tooth is similar to a rectangle the number of rect-angles is calculated in the image after registration If thenumber of teeth is less than five the teeth are consideredmissing otherwise it is intact

First of all the reference image is a need in imageregistration An image with relatively clear and intact teeth isselected as the reference image from the results of imageregistration Because of the camerarsquos angle of view the teethare always centered in the images and only move up ordown +e registration process is not complicated +eobjects of the image registration experiments are all thebinary images with teeth obtained from image segmentationin previous experiments After the image registration the

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(a)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(b)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(c)

Figure 8 Labeled images

8 Computational Intelligence and Neuroscience

number of rectangles at the bottom of the images is cal-culated Part of the experimental results is shown inFigure 12

According to the number of rectangular structures in theregistration experimental results there are some defects injudging whether the teeth are missing or not +e rectan-gular structure is the smallest rectangle that covers each

connected domain in the image When multiple teeth areconnected due to noise as shown in Figure 13(a) themethod of counting the number of rectangles fails Besideswhen the teeth are broken not fall off as shown inFigure 13(b) the broken teeth will also have a rectangularstructure after image registration Due to noise there may bemore rectangular structures than the teeth in the result ofimage registration as shown in Figure 13(c) As a result thismethod does not work

+erefore another method is proposed which adaptedthe area ratio (α) of the teeth in a registered image to thereference image as the evaluation metric As noise will affectthe area to more accurately judge whether the teeth arebroken or fall off the area of teeth in the registered image iscalculated where the teeth are located in the reference imageIn the image registration due to noise registration ab-normalities will occur As shown in Figure 14 the ratio ofteeth area in the registered image to the reference image istoo small or too large

Without considering simultaneous falling off or fractureof multiple teeth the following threshold values for judgingwhether teeth fall off or fracture can be obtained

(i) αle 06 registration anomaly(ii) 06lt αle 08 a tooth falls off at least(iii) 08lt αle 09 a tooth fractures(iv) 09lt αle 11 teeth intact(v) 11lt α registration anomaly

0900

0700

0500

0300

0100

traintotal_loss_iter

0000 2000 k 4000 k 6000 k 8000 k

(a)

traintotal_loss_epoch

0000 2000 4000 6000 8000 1000 k

800

700

600

500

400

300

200

(b)

valAcc_class

0000 2000 4000 6000 8000 1000 k

0950

0850

0750

0650

0550

0450

(c)

valmloU

0000 2000 4000 6000 8000 1000 k

0800

0900

0700

0600

0500

0400

0300

(d)

Figure 9 Evaluation metrics (a) loss function in one iteration (b) loss function in epoch (c) PA (d) MIOU

Semantic segmentation

Semantic segmentation

Semantic segmentation

Figure 10 +e segmented images

Computational Intelligence and Neuroscience 9

To verify the effectiveness of the image registration weconducted two groups of test experiments and observe thetest results +e dataset of the two groups of experiments

consists of the following one is 70 positive sampleswithout teeth missing and 35 negative samples withmissing teeth and the other one is 65 positive samples and

10

09

08

07

06

05

04

0 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(a)

10

09

08

07

06

05

040 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(b)

Figure 11 Evaluation metrics (a) dilation convolution (b) comparative experiment

Segmented images

Registration Mark teeth

Marked images

5 meansintact

alarm

Registered images

Figure 12 +e registration result

(a) (b) (c)

Figure 13 Special circumstances (a) connected teeth (b) broken teeth (c) more teeth

10 Computational Intelligence and Neuroscience

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

damage to the crusher gears and the production line isexpected At present it is impossible to completely avoid thefracture of the tooth although many scholars have carriedout a lot of research work from the aspects of the toothmechanical analysis [5] the optimum replacement time forshovel teeth [6] the tooth material [7] and the heattreatment method [8]

In this case machine vision technology combined withan automatic detection algorithm can monitor the workingstate of the tooth in real-time When the tooth is falling off itcan provide accurate warning information and instruct thestaff to deal with it in time which has important practicalsignificance and economic value for the production of themine Many scholars and enterprises have done a lot ofresearch work on the detection of the shovel teeth breakingand falling off Xiujuan Luo proposed a detection systemsolution which uses a laser range finder to scan and re-construct the three-dimensional model of the tooth in theworking state and then compare it with the original model todetermine whether there is any tooth drop in 2004 [9] Toimprove the calculation speed it uses the parallel computingmethod to collect and process the data synchronously so thatthe whole detection system has a better real-time perfor-mance +is method is less affected by changes in illumi-nation and environment and has a good detection effectcompared with machine vision detection methods He Liproposed a novel object detection algorithm with thecombination of appearance structural and shape features in2011 [10] It follows the paradigm of object detection byparts which first uses detectors developed for individualparts of an object and then imposes structural constraintsamong the parts for the detection of the entire object +eyfurther incorporate a priori shape information about theobject parts for their detection to improve the performanceof the object detector For validation of this object detectionalgorithm they apply it to the detection of the tooth line of amining shovel under various lighting conditions +e ex-perimental results demonstrate that this system can improvethe detection performance significantly when part shapeinformation is used Ser Nam Lim presented a vision systemfor monitoring tooth condition and detecting missing toothfor a mining shovel in 2016 [11] First it extracts the imagesample of the bucket during the movement of the shovel anduses this sample information to locate the approximateposition of the tooth Second it uses the frame differencemethod and the optical flowmethod to correct the result andthen uses the template matching and the tooth line fittingmethod to accurately locate the tooth target Finally itcombines the grayscale features of the tooth image to de-termine if the tooth is falling off +e outstanding perfor-mance and high reliability of the proposed system have beendemonstrated by experiments on video sequences collectedfrom an iron ore mining site and a two-month trial of aninstalled unit on a production line Abdallah Almarsquoaitahutilized Sprouts WSN platform to monitor the steel shovelteeth on shoveling equipment used in oil sands miningoperations in Ft McMurray Alberta Canada [12] +eSprouts platform is utilized to monitor the viability of theshovel teeth on the shovel bucketrsquos teeth adapter Besides if a

shovel-tooth becomes detached Sprouts is used to estimateits location Hannington focuses on a method to prevent theteeth from ending up in the crusher based on an infrareddetection system that reads the bucket profile every time ithauls the ore onto the truck in 2008 At the same timeanother detection system available on the market has beenstudied and a probable preventive maintenance program hasbeen suggested [13]

With the rapid development of deep learning skills inmultiple areas [14ndash17] in this paper a novel electric shovelteethmissing detectionmethod which includes the followingsignificant steps is proposed to monitor teeth condition anddetect missing teeth At first salient object detection basedon deep learning is presented for extracting crucial frames invideo flow efficiently and releasing computational pressureof the terminal end Additionally the working environmentof shovel teeth is relatively special and the messy ore willproduce a lot of noise in the frame images Semantic seg-mentation is proposed to get the relative position of teeth ina noisy background Finally to detect the broken teeth animage registration approach is introduced With the pro-posed procedure without human being affected the electricshovel teeth missing detection method is ensured auto-matically and results in a high-accuracy rate Our proposedprocedure for missing tooth detection serves a fully auto-matic quality which provides great significance in miningindustry If this work is supported by human resources itshould increase dangerous risks especially in low visibleenvironments Moreover depending on real-time humansupervised detection the accuracy cannot be ensured andwill result in great costs from economic perspectives

Our main contributions are summarized as follows

(1) A salient object detection based on deep learning isproposed for deleting useless frame images and re-leasing computational pressure of subsequent workBecause salient detection is completed at the edgethe efficiency of the pixel-level algorithm isimproved

(2) Semantic segmentation can find the relative positionof teeth in the frame images and solve the noiseproblem generated by the working environment ofthe electric shovel Compared with the previousmethod based on machine vision the experimentalresult of semantic segmentation is more effective

(3) Image registration converts teeth into rectangularstructures by processing the results of the imageregistration experiment the proposed strategy iscapable of handling the system uncertainties causedby the working environment of electric shovel teeth+e final results show that the electric shovel teethdetection algorithm works with high accuracy inactual production

2 Methodology

+e proposed method is an intelligent computer visionsystem which consists of salient object detection imagesegmentation and image registration Generally the

2 Computational Intelligence and Neuroscience

number of shovel teeth (M) is fixed M 5 +is invariantvalue can be taken as an indicator of the detection of missingteeth and key frame extraction By setting the number ofdetected teeth as S we adapt the following formula to de-termine whether the teeth are missing or not

S 0

0lt SltM

S M

⎧⎪⎪⎨

⎪⎪⎩(1)

When S 0 there is no tooth in the image when0lt SltM the teeth are broken when S M it is in a normalstate

21 Salient Object Detection In the mining process dozensof mining machines are working at the same time To moreefficiently detect whether the tooth is missing on the shovelin each machine a camera is installed on each shovel tomonitor the status of the teeth +e position of the camera issignificant for the detected system +e best position shouldallow the mounted camera to be stable during operation asshown in Figure 1 In the process of monitoring the state ofteeth the frame images are extracted from the video frameobtained by the camera for edge calculation If every frameimage in the video stream is sent to the server terminal thecomputation will be very large which will reduce thecomputational efficiency and require highly on the servicefacilities

To alleviate the pressure of the data processing centerand ensure the cost of facilities we proposed a strategy basedon salient object detection to obtain more significant framesin video flow and delete unnecessary frames +e purpose ofthis paper to detect whether the tooth on the shovel ismissing or not and give a timely alarm when the tooth ismissing +erefore when there is no tooth in the frameimage we can identify that this frame image is an unnec-essary frame image +e function of salient object detectionis to find the region of interest (ROI) and obviously in thispaper our ROI is the teeth +e idea of salient object de-tection in deep learning depends on pixel values in featuremaps located in deeper layers of neural networks In generalpixels with great values indicate the salient section of theforeground we explore

However traditional salient object detection based ondeep learning has many problems

(i) +e training is divided into several stages and thesteps are relatively complicated fine-tuning net-work + training SVM+ training border regression

(ii) +e training is time-consuming and takes up a lot ofdisk space 5000 images produce hundreds ofgigabytes of feature files

(iii) Computing slowly it takes 47s for VGG-16 toprocess an image by using GPU

Low computational efficiency and a large amount ofcomputation are not what we are willing to see in edgecalculation Based on these shortcomings of traditionalsalient object detection and our purpose to delete

unnecessary frame images a neural network of dichotomy isproposed retaining frame images labeled with teeth anddeleting frame images with no teeth +e network structureis shown in Figure 2

+e model contains three simple convolutional layersand two full connection layers By calculating the parametersin this network it can be found that the complexity of thenetwork structure is lower than that of AlexNet +e lowcomplexity network structure determines the efficiency ofedge computation and improves the efficiency of the wholesystem under the premise of realizing salient detection

22 Image Segmentation After the salient object detectionthe key frame images are obtained To facilitate the subse-quent image registration to detect the shovel teeth imagesegmentation is carried out to extract the information ofteeth In the process of salient object detection it is noticedthat the position of teeth is relatively skewed and the size ofthe teeth is too small +erefore the common convolutionalneural networks are not able to complete high-accuracyimage segmentation because of inability to obtain infor-mation about teeth In this case the DeepLabV3+ model isadopted to realize accurate acquisition of teeth informationby using dilated convolution and finally to process imageregistration to realize missing teeth detection To guaranteethe scientific nature of the experiment the dilation rate ischanged to zero as a common convolutional neural networkfor the contrast test By contrasting the result of the normalconvolution neural network and the result of the dilatedconvolution neural network it shows that the dilatedconvolution is more effective

DeepLabV3+ is an efficient model for image segmen-tation It extends DeepLabV3+ optimization results with asimple and efficient decoder module especially along thetarget boundaries Besides in the encoder-decoder structurethe resolution of the extracted encoder features can becontrolled arbitrarily using dilated convolution to com-promise accuracy and running time Atrous Spatial PyramidPooling (ASPP) module contains four dilated convolutionallayers and one global pooling layer In the experiment of thispaper more efficient image segmentation is achieved byadjusting the dilation rate of the convolution kernel in ASPPDecoder carries out 1times 1 convolution to reduce channelsbefore the fusion of low-level information +en theupsampling of the encoderrsquos results is quadrupled andcombined with the previous convolution and 3 times 3 convo-lution is carried out Finally the final result is obtained byquadrupling the upsampling +e specific network structureis shown in Figure 3

To obtain more pixel information under the permissionof no adding parameters it contains dilated convolutionallayers in DeepLabV3+ However there is the gridding effect[18] in the dilated convolution since the convolution kernelsare not continuous in deep convolutional layers In this casethere are always some pixels not involved in the calculationMoreover it is difficult to realize the segmentation of smallobjects in the dilated convolution with a large dilation rateSince the relative size of teeth in this paper is small in the

Computational Intelligence and Neuroscience 3

image the dilated convolution with a small dilation rate isproposed to ensure the accurate segmentation of the teethTo ensure that all pixels are involved in the calculation thedilation rates in the deep convolutional layers satisfy thefollowing formula

Mi max M(i1) minus 2r(i1)M(i1) minus 2 Mi1 minus ri( 1113857 ri1113858 1113859 (2)

where Mi is the maximum dilation rate of the ith layer and ri

denotes the dilation rate of the ith layer with Mn rnIn the actual work of detection the relative position and

size of the teeth in the image recorded by the camera areconstantly changing which is a challenging difficultyDeepLabV3+ adopts a combination of dilated convolutioncascade and parallel with multiple dilation rates to effectively

(a) (b)

Figure 1 Illustration of our proposed solution on automatic electric shovel teeth detection

Input 64643

646432 323232 323232 161632 1616328864

114096

Convolution

ConvMax-pooling Pooling PoolingConv

Fully connected

Output

Figure 2 +e structure of classification network for salient detection

DeepLab-v3+Encoder

1times1 Conv

1times1 Conv

1times1 Conv

3times3 Conv

3times3 Convrate = 1

3times3 Convrate = 2

3times3 Convrate = 3

ImagePooling

low-levelfeatures

Upsampleby 4

Upsampleby 4

Decoder

Concat

Figure 3 DeepLabV3+ structure

4 Computational Intelligence and Neuroscience

extract multiscale semantic information As a result nomatter where the relative position of the teeth and no matterwhat the size of teeth the model processes image seg-mentation efficiently Also by combining image-level in-formation the ability of ASPP to extract global semanticinformation is improved to further improve the effect

23 Image Registration After the image segmentation bi-nary images containing only teeth are obtained as shown inFigure 4(a) In this section the image registration is pro-posed to detect whether the teeth are missing or not +eimage in which the number of teeth (M) is zero is deleted inthe salient object detection +erefore in the process ofimage registration the number of teeth (M) ranges from oneto five

In the process of image registration the reference imageis set up firstly In this paper the images obtained after imagesegmentation are highly similar except the relative positionof teeth is different Also the relative position of each tooth isthe same +erefore one relatively clear image obtainedfrom image segmentation with intact teeth can be used as thereference image +e final reference image shows five nearlyidentical rectangular structures at the bottom of the binaryimage as shown in Figure 4(b)

However in the segmented images the relative positionof teeth is different and the pixelsrsquo information of the imagesis also very different +e reference image and the process ofimage registration are shown in Figure 4 As shown inFigure 5 the teeth in registered images are at the bottom ofthe image and have obvious rectangular structures Whenthere is no tooth missing the registered image contains fiverectangular structures When the teeth are missing thenumber of the rectangular structure is less than fiveMoreover the registered image is a binary image which isvery beneficial to some processing methods such as edgedetection +erefore we detect the rectangles in the regis-tered image and calculate the number of rectangles to de-termine whether the teeth are missing or not

3 Experiments and Discussion

+e experiments that we conducted are divided into threemain sections (1) salient object detection (2) image seg-mentation and (3) image registration

+e first sets of experiments are conducted to deleteimages without teeth to reduce the amount of computationand the second sets of experiments are conducted to find theregion of interest (ROI) and the location of teeth in theimages Moreover it should be noticed that multiple dilationrates are proposed to extract multisemantic informationand the dilated convolution neural network is contracted toa normal convolution to demonstrate that the DeepLabV3+model works more effectively +ird image registration isconducted to detect whether the teeth are missing or not bycounting the number of rectangles

31 Datasets and Hardware +e datasets in this paper arefrom frame images of the video stream taken by the camera

on the shovel Frame images are extracted from the videostream consisting of 500 with and 500 without teeth Afterlabeling these images twenty percent of the images are usedas a validation set to test the training model and the othereighty percent of the images are used as the training set totrain the classification model +e training and inference ofour proposed model are run with two NVIDIA GTX1080 Ti11GB GPUs 32G RAM and Intel i7-7700K CPU with 8cores 420GHz Especially we did not adopt any pre-processing of noise management for our dataset in order topreserve original details and avoid possible artifacts fortraining and inference process

32 Salient Object Detection Studies To reduce the com-putation of the terminal server salient object detection isproposed to delete the images without teeth To solve theshortcomings of traditional salient object detection a di-chotomous neural network is adapted to delete the imagelabeled no teeth

321 Preprocessing +is set of experiments aims to use edgecomputing to alleviate the pressure of the terminal server+erefore to complete the edge computing faster and reducethe calculation amount the images are resized into64times64times 3

322 Parameters +e networkrsquos detailed connectivity andkernel configuration are illustrated in Figure 2 And all theconvolutional kernelsrsquo sizes are the same as the size of 3times 3+e pooling layer of every convolutional layer is Max-Pooling the kernel size is 2times 2 and the step size is 2 +enumber of this neural network is 233472 which is less thanhalf of the AlexNet Adam optimizer is adapted with alearning rate of 00001 +e batch size is set as 32 and thenumber of iterations is set as 8000

323 Training Process Normally too many iterations maycause the model to fit the noise and unnecessary features ofthe training sets and then overfitting occurs In the case ofoverfitting themodel is of high complexity and does not workto the data except the training set To avoid excessive com-putation the loss function of the validation dataset and theaccuracy of the validation dataset are adapted as the evalu-ation metrics for model selection When the accuracy of thevalidation dataset reaches a high level and the accuracy is notimproved after 10 epochs also the loss function is not de-clined the iteration is terminating in advance+e final modelis obtained In the training process the accuracy of thetraining dataset the accuracy of the validation dataset and theloss function of the validation dataset are shown in Figure 6

As shown in Figure 6 the accuracy of the validationdataset and the training dataset reaches the global maximumwithin fifty epochs and the loss function of the validationdataset does not decrease in the subsequent epochs Asshown in Figure 6 the accuracy of training and validationfluctuates periodically since some groups of sampled imagescontain lots of noise as a result of complicated wild working

Computational Intelligence and Neuroscience 5

environments which leads to poor performances of se-mantic segmentation +erefore to prevent the trainingmodel from overfitting the number of iterations is changed

into 1600 making the epochs to 64 reducing the trainingtime and calculation amount of the training model +etraining result is shown in Figure 7

(a) (b)

Figure 4 Segmented images

e reference image

e image registration process

Registered images

Segmented images

Figure 5 +e image registration process

20

15

10

05

00

09

10

08

07

06

05

04

valu

e

valu

e

0 50 100 150 200 250 300epoch

0 50 100 150 200 250 300epoch

train accval accval loss

train accval acc

Figure 6 Training metrics

6 Computational Intelligence and Neuroscience

4 Results

As can be seen from Figure 7 in the 56th epoch the accuracyof the training dataset and the accuracy and the loss functionof the validation dataset reach the optimal level +e ac-curacy of the training dataset is 9375 the accuracy of thevalidation dataset is 9687 and the loss function of thevalidation is 00965 +erefore ldquoteeth-no teeth-model-1400rdquogenerated by the model trained to the 56th epoch is adaptedto detect whether the teeth are missing or not

One video stream of a working shovel is incepted allframe images are extracted from it and the frame images areput into the trained model for prediction If the predictionresult of the image is ldquoteethrdquo this image is kept for the nextpart of the experiments Otherwise this image is deleted Inthe prediction experiment of the frame images of the cap-tured video stream the prediction results are all correctindicating that the model predicts effectively In the threeconvolutional layers of the network the feature maps ofimages with teeth and without teeth are shown in the fol-lowing figures It is through learning these features that ourmodel can correctly predict whether the images containteeth or not and then complete the salient object detection

41 Image Segmentation To detect whether the teeth aremissing or not the relative position of the teeth should befound at first +e DeepLabV3+ model is adapted forprocessing semantic segmentation to find the location of theteeth+e dilation rate is adjusted several times in our modelto avoid missing information in the convolution process+en to verify the advantages of dilated convolution thedilation rate in the network is changed to zero for a com-parison experiment

411 Preprocess First to facilitate the later training processthe frame images are resized as 512times 512+en the region ofinterest of the images is annotated through the ldquolabelme rdquo an

image labeling tool Images with teeth are labeled with thelocation of teeth and the images without teeth do not needany operation After the image labeling is completed toensure the rationality of our model the image is processedby methods such as rotation scaling and cropping and theprocessed image is added to the data as new images +elabeled image is shown in Figure 8

412 Parameters +e details of the DeepLabV3+ are shownin Figure 3 In this section the structure of themodel will notbe changed but the dilation rates are changed to prevent theinformation missing from the dilated convolution To obtainthe information of each pixel by dilated convolution thedilation rates are changed to 0 1 2 and 3 Later to prove thesuperiority of the dilated convolution for image segmen-tation the dilated rates are set as 0 0 0 and 0 for com-parison experiments

413 Training +e colorful images are transformed intograyscale images for training reducing the color differencenoise brought by the day and night +e initial learning rateis set as 001 and the learning rate will be adjusted dy-namically [19]

414 Evaluation Metrics

(i) Pixel accuracy (PA) means the ratio of correctlyclassified pixel points in all pixel points In thispaper there are only two kinds of classes of the pixelteeth and background P10 represents the totalnumber of pixels in the background predicted to bein the teeth P11 represents true positive P10 meansfalse positives P01 means false negative and P00represents true negative

PA P11 + P00

P11 + P10 + P01 + P00 (3)

train accval acc

20

15

10

05

00

valu

e10

09

08

07

06

05

04

valu

e

100 20 30 40 50 60epoch

100 20 30 40 50 60epoch

train accval accval loss

Figure 7 Training metrics (less iterations)

Computational Intelligence and Neuroscience 7

(ii) Mean intersection over union (MIOU) is the stan-dard metric of semantic segmentation It calculatesthe ratio of the intersection and union of two sets thereal value and the predicted value Calculating theintersection over union for each class and averaging itthen the MIOU is obtained as follows

MIOU 12

P11

P11 + P10 + P01+

P00

P01 + P00 + P101113888 1113889 (4)

415 Result After about 1000 epochs of training a rela-tively efficient semantic segmentation model for teeth isobtained +e variation trend of the loss function pixelaccuracy rate and MIOU in the training process is shown inFigure 9

As can be seen from Figure 9 after 1000 times oftraining all the indicators have stabilized However to avoidoverfitting caused by too many iterations the training resultof the 512th epoch is adapted as our model for semanticsegmentation It can be seen from Figure 9 that when theiterations reach the 512th epoch the loss function in oneepoch is stable at about 25 the loss function in one iterationis stable at about 03 PA is about 095 and MIOU is about075 +us the model of the 512th epoch is a very efficienttraining result +e model is adapted to process dozens ofimages that are not in the training dataset to carry outsemantic segmentation and find out the teeth in these im-ages Later verifying the predicted results manually findsthat the predicted results of the teeth are accurate Moreoverthe training dataset consists of grayscale images in the ex-periment and the images of day and night could be accu-rately segmented out the teeth +e segmented results areshown in Figure 10

416 Comparative Experiments A comparative experimentis conducted to verify the validity of the dilation convolu-tion +e dilated rates of the dilation convolution layers arechanged to zero and a new model is carried out By ob-serving the evaluation metrics of the model the comparativeexperimental results are obtained as shown in Figure 11

As shown in Figure 11 the results of the dilation con-volution are better +e model is stable at the 500th epochbut the comparative experiment will not converge until the1000th epoch Moreover the experimental results of thedilation convolution are relatively stable while the results ofthe other show great differences after each epoch +ereforethis set of comparative experiments effectively illustrates theexcellent features of fast converge and stable experimentalresults of dilation convolution in teeth semanticsegmentation

42 Image Registration After the image segmentation thebinary images with teeth are obtained +e number of teethin these binary images is considered as the metric to dis-charge whether the teeth are missing or not Since the shapeof the tooth is similar to a rectangle the number of rect-angles is calculated in the image after registration If thenumber of teeth is less than five the teeth are consideredmissing otherwise it is intact

First of all the reference image is a need in imageregistration An image with relatively clear and intact teeth isselected as the reference image from the results of imageregistration Because of the camerarsquos angle of view the teethare always centered in the images and only move up ordown +e registration process is not complicated +eobjects of the image registration experiments are all thebinary images with teeth obtained from image segmentationin previous experiments After the image registration the

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(a)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(b)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(c)

Figure 8 Labeled images

8 Computational Intelligence and Neuroscience

number of rectangles at the bottom of the images is cal-culated Part of the experimental results is shown inFigure 12

According to the number of rectangular structures in theregistration experimental results there are some defects injudging whether the teeth are missing or not +e rectan-gular structure is the smallest rectangle that covers each

connected domain in the image When multiple teeth areconnected due to noise as shown in Figure 13(a) themethod of counting the number of rectangles fails Besideswhen the teeth are broken not fall off as shown inFigure 13(b) the broken teeth will also have a rectangularstructure after image registration Due to noise there may bemore rectangular structures than the teeth in the result ofimage registration as shown in Figure 13(c) As a result thismethod does not work

+erefore another method is proposed which adaptedthe area ratio (α) of the teeth in a registered image to thereference image as the evaluation metric As noise will affectthe area to more accurately judge whether the teeth arebroken or fall off the area of teeth in the registered image iscalculated where the teeth are located in the reference imageIn the image registration due to noise registration ab-normalities will occur As shown in Figure 14 the ratio ofteeth area in the registered image to the reference image istoo small or too large

Without considering simultaneous falling off or fractureof multiple teeth the following threshold values for judgingwhether teeth fall off or fracture can be obtained

(i) αle 06 registration anomaly(ii) 06lt αle 08 a tooth falls off at least(iii) 08lt αle 09 a tooth fractures(iv) 09lt αle 11 teeth intact(v) 11lt α registration anomaly

0900

0700

0500

0300

0100

traintotal_loss_iter

0000 2000 k 4000 k 6000 k 8000 k

(a)

traintotal_loss_epoch

0000 2000 4000 6000 8000 1000 k

800

700

600

500

400

300

200

(b)

valAcc_class

0000 2000 4000 6000 8000 1000 k

0950

0850

0750

0650

0550

0450

(c)

valmloU

0000 2000 4000 6000 8000 1000 k

0800

0900

0700

0600

0500

0400

0300

(d)

Figure 9 Evaluation metrics (a) loss function in one iteration (b) loss function in epoch (c) PA (d) MIOU

Semantic segmentation

Semantic segmentation

Semantic segmentation

Figure 10 +e segmented images

Computational Intelligence and Neuroscience 9

To verify the effectiveness of the image registration weconducted two groups of test experiments and observe thetest results +e dataset of the two groups of experiments

consists of the following one is 70 positive sampleswithout teeth missing and 35 negative samples withmissing teeth and the other one is 65 positive samples and

10

09

08

07

06

05

04

0 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(a)

10

09

08

07

06

05

040 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(b)

Figure 11 Evaluation metrics (a) dilation convolution (b) comparative experiment

Segmented images

Registration Mark teeth

Marked images

5 meansintact

alarm

Registered images

Figure 12 +e registration result

(a) (b) (c)

Figure 13 Special circumstances (a) connected teeth (b) broken teeth (c) more teeth

10 Computational Intelligence and Neuroscience

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

number of shovel teeth (M) is fixed M 5 +is invariantvalue can be taken as an indicator of the detection of missingteeth and key frame extraction By setting the number ofdetected teeth as S we adapt the following formula to de-termine whether the teeth are missing or not

S 0

0lt SltM

S M

⎧⎪⎪⎨

⎪⎪⎩(1)

When S 0 there is no tooth in the image when0lt SltM the teeth are broken when S M it is in a normalstate

21 Salient Object Detection In the mining process dozensof mining machines are working at the same time To moreefficiently detect whether the tooth is missing on the shovelin each machine a camera is installed on each shovel tomonitor the status of the teeth +e position of the camera issignificant for the detected system +e best position shouldallow the mounted camera to be stable during operation asshown in Figure 1 In the process of monitoring the state ofteeth the frame images are extracted from the video frameobtained by the camera for edge calculation If every frameimage in the video stream is sent to the server terminal thecomputation will be very large which will reduce thecomputational efficiency and require highly on the servicefacilities

To alleviate the pressure of the data processing centerand ensure the cost of facilities we proposed a strategy basedon salient object detection to obtain more significant framesin video flow and delete unnecessary frames +e purpose ofthis paper to detect whether the tooth on the shovel ismissing or not and give a timely alarm when the tooth ismissing +erefore when there is no tooth in the frameimage we can identify that this frame image is an unnec-essary frame image +e function of salient object detectionis to find the region of interest (ROI) and obviously in thispaper our ROI is the teeth +e idea of salient object de-tection in deep learning depends on pixel values in featuremaps located in deeper layers of neural networks In generalpixels with great values indicate the salient section of theforeground we explore

However traditional salient object detection based ondeep learning has many problems

(i) +e training is divided into several stages and thesteps are relatively complicated fine-tuning net-work + training SVM+ training border regression

(ii) +e training is time-consuming and takes up a lot ofdisk space 5000 images produce hundreds ofgigabytes of feature files

(iii) Computing slowly it takes 47s for VGG-16 toprocess an image by using GPU

Low computational efficiency and a large amount ofcomputation are not what we are willing to see in edgecalculation Based on these shortcomings of traditionalsalient object detection and our purpose to delete

unnecessary frame images a neural network of dichotomy isproposed retaining frame images labeled with teeth anddeleting frame images with no teeth +e network structureis shown in Figure 2

+e model contains three simple convolutional layersand two full connection layers By calculating the parametersin this network it can be found that the complexity of thenetwork structure is lower than that of AlexNet +e lowcomplexity network structure determines the efficiency ofedge computation and improves the efficiency of the wholesystem under the premise of realizing salient detection

22 Image Segmentation After the salient object detectionthe key frame images are obtained To facilitate the subse-quent image registration to detect the shovel teeth imagesegmentation is carried out to extract the information ofteeth In the process of salient object detection it is noticedthat the position of teeth is relatively skewed and the size ofthe teeth is too small +erefore the common convolutionalneural networks are not able to complete high-accuracyimage segmentation because of inability to obtain infor-mation about teeth In this case the DeepLabV3+ model isadopted to realize accurate acquisition of teeth informationby using dilated convolution and finally to process imageregistration to realize missing teeth detection To guaranteethe scientific nature of the experiment the dilation rate ischanged to zero as a common convolutional neural networkfor the contrast test By contrasting the result of the normalconvolution neural network and the result of the dilatedconvolution neural network it shows that the dilatedconvolution is more effective

DeepLabV3+ is an efficient model for image segmen-tation It extends DeepLabV3+ optimization results with asimple and efficient decoder module especially along thetarget boundaries Besides in the encoder-decoder structurethe resolution of the extracted encoder features can becontrolled arbitrarily using dilated convolution to com-promise accuracy and running time Atrous Spatial PyramidPooling (ASPP) module contains four dilated convolutionallayers and one global pooling layer In the experiment of thispaper more efficient image segmentation is achieved byadjusting the dilation rate of the convolution kernel in ASPPDecoder carries out 1times 1 convolution to reduce channelsbefore the fusion of low-level information +en theupsampling of the encoderrsquos results is quadrupled andcombined with the previous convolution and 3 times 3 convo-lution is carried out Finally the final result is obtained byquadrupling the upsampling +e specific network structureis shown in Figure 3

To obtain more pixel information under the permissionof no adding parameters it contains dilated convolutionallayers in DeepLabV3+ However there is the gridding effect[18] in the dilated convolution since the convolution kernelsare not continuous in deep convolutional layers In this casethere are always some pixels not involved in the calculationMoreover it is difficult to realize the segmentation of smallobjects in the dilated convolution with a large dilation rateSince the relative size of teeth in this paper is small in the

Computational Intelligence and Neuroscience 3

image the dilated convolution with a small dilation rate isproposed to ensure the accurate segmentation of the teethTo ensure that all pixels are involved in the calculation thedilation rates in the deep convolutional layers satisfy thefollowing formula

Mi max M(i1) minus 2r(i1)M(i1) minus 2 Mi1 minus ri( 1113857 ri1113858 1113859 (2)

where Mi is the maximum dilation rate of the ith layer and ri

denotes the dilation rate of the ith layer with Mn rnIn the actual work of detection the relative position and

size of the teeth in the image recorded by the camera areconstantly changing which is a challenging difficultyDeepLabV3+ adopts a combination of dilated convolutioncascade and parallel with multiple dilation rates to effectively

(a) (b)

Figure 1 Illustration of our proposed solution on automatic electric shovel teeth detection

Input 64643

646432 323232 323232 161632 1616328864

114096

Convolution

ConvMax-pooling Pooling PoolingConv

Fully connected

Output

Figure 2 +e structure of classification network for salient detection

DeepLab-v3+Encoder

1times1 Conv

1times1 Conv

1times1 Conv

3times3 Conv

3times3 Convrate = 1

3times3 Convrate = 2

3times3 Convrate = 3

ImagePooling

low-levelfeatures

Upsampleby 4

Upsampleby 4

Decoder

Concat

Figure 3 DeepLabV3+ structure

4 Computational Intelligence and Neuroscience

extract multiscale semantic information As a result nomatter where the relative position of the teeth and no matterwhat the size of teeth the model processes image seg-mentation efficiently Also by combining image-level in-formation the ability of ASPP to extract global semanticinformation is improved to further improve the effect

23 Image Registration After the image segmentation bi-nary images containing only teeth are obtained as shown inFigure 4(a) In this section the image registration is pro-posed to detect whether the teeth are missing or not +eimage in which the number of teeth (M) is zero is deleted inthe salient object detection +erefore in the process ofimage registration the number of teeth (M) ranges from oneto five

In the process of image registration the reference imageis set up firstly In this paper the images obtained after imagesegmentation are highly similar except the relative positionof teeth is different Also the relative position of each tooth isthe same +erefore one relatively clear image obtainedfrom image segmentation with intact teeth can be used as thereference image +e final reference image shows five nearlyidentical rectangular structures at the bottom of the binaryimage as shown in Figure 4(b)

However in the segmented images the relative positionof teeth is different and the pixelsrsquo information of the imagesis also very different +e reference image and the process ofimage registration are shown in Figure 4 As shown inFigure 5 the teeth in registered images are at the bottom ofthe image and have obvious rectangular structures Whenthere is no tooth missing the registered image contains fiverectangular structures When the teeth are missing thenumber of the rectangular structure is less than fiveMoreover the registered image is a binary image which isvery beneficial to some processing methods such as edgedetection +erefore we detect the rectangles in the regis-tered image and calculate the number of rectangles to de-termine whether the teeth are missing or not

3 Experiments and Discussion

+e experiments that we conducted are divided into threemain sections (1) salient object detection (2) image seg-mentation and (3) image registration

+e first sets of experiments are conducted to deleteimages without teeth to reduce the amount of computationand the second sets of experiments are conducted to find theregion of interest (ROI) and the location of teeth in theimages Moreover it should be noticed that multiple dilationrates are proposed to extract multisemantic informationand the dilated convolution neural network is contracted toa normal convolution to demonstrate that the DeepLabV3+model works more effectively +ird image registration isconducted to detect whether the teeth are missing or not bycounting the number of rectangles

31 Datasets and Hardware +e datasets in this paper arefrom frame images of the video stream taken by the camera

on the shovel Frame images are extracted from the videostream consisting of 500 with and 500 without teeth Afterlabeling these images twenty percent of the images are usedas a validation set to test the training model and the othereighty percent of the images are used as the training set totrain the classification model +e training and inference ofour proposed model are run with two NVIDIA GTX1080 Ti11GB GPUs 32G RAM and Intel i7-7700K CPU with 8cores 420GHz Especially we did not adopt any pre-processing of noise management for our dataset in order topreserve original details and avoid possible artifacts fortraining and inference process

32 Salient Object Detection Studies To reduce the com-putation of the terminal server salient object detection isproposed to delete the images without teeth To solve theshortcomings of traditional salient object detection a di-chotomous neural network is adapted to delete the imagelabeled no teeth

321 Preprocessing +is set of experiments aims to use edgecomputing to alleviate the pressure of the terminal server+erefore to complete the edge computing faster and reducethe calculation amount the images are resized into64times64times 3

322 Parameters +e networkrsquos detailed connectivity andkernel configuration are illustrated in Figure 2 And all theconvolutional kernelsrsquo sizes are the same as the size of 3times 3+e pooling layer of every convolutional layer is Max-Pooling the kernel size is 2times 2 and the step size is 2 +enumber of this neural network is 233472 which is less thanhalf of the AlexNet Adam optimizer is adapted with alearning rate of 00001 +e batch size is set as 32 and thenumber of iterations is set as 8000

323 Training Process Normally too many iterations maycause the model to fit the noise and unnecessary features ofthe training sets and then overfitting occurs In the case ofoverfitting themodel is of high complexity and does not workto the data except the training set To avoid excessive com-putation the loss function of the validation dataset and theaccuracy of the validation dataset are adapted as the evalu-ation metrics for model selection When the accuracy of thevalidation dataset reaches a high level and the accuracy is notimproved after 10 epochs also the loss function is not de-clined the iteration is terminating in advance+e final modelis obtained In the training process the accuracy of thetraining dataset the accuracy of the validation dataset and theloss function of the validation dataset are shown in Figure 6

As shown in Figure 6 the accuracy of the validationdataset and the training dataset reaches the global maximumwithin fifty epochs and the loss function of the validationdataset does not decrease in the subsequent epochs Asshown in Figure 6 the accuracy of training and validationfluctuates periodically since some groups of sampled imagescontain lots of noise as a result of complicated wild working

Computational Intelligence and Neuroscience 5

environments which leads to poor performances of se-mantic segmentation +erefore to prevent the trainingmodel from overfitting the number of iterations is changed

into 1600 making the epochs to 64 reducing the trainingtime and calculation amount of the training model +etraining result is shown in Figure 7

(a) (b)

Figure 4 Segmented images

e reference image

e image registration process

Registered images

Segmented images

Figure 5 +e image registration process

20

15

10

05

00

09

10

08

07

06

05

04

valu

e

valu

e

0 50 100 150 200 250 300epoch

0 50 100 150 200 250 300epoch

train accval accval loss

train accval acc

Figure 6 Training metrics

6 Computational Intelligence and Neuroscience

4 Results

As can be seen from Figure 7 in the 56th epoch the accuracyof the training dataset and the accuracy and the loss functionof the validation dataset reach the optimal level +e ac-curacy of the training dataset is 9375 the accuracy of thevalidation dataset is 9687 and the loss function of thevalidation is 00965 +erefore ldquoteeth-no teeth-model-1400rdquogenerated by the model trained to the 56th epoch is adaptedto detect whether the teeth are missing or not

One video stream of a working shovel is incepted allframe images are extracted from it and the frame images areput into the trained model for prediction If the predictionresult of the image is ldquoteethrdquo this image is kept for the nextpart of the experiments Otherwise this image is deleted Inthe prediction experiment of the frame images of the cap-tured video stream the prediction results are all correctindicating that the model predicts effectively In the threeconvolutional layers of the network the feature maps ofimages with teeth and without teeth are shown in the fol-lowing figures It is through learning these features that ourmodel can correctly predict whether the images containteeth or not and then complete the salient object detection

41 Image Segmentation To detect whether the teeth aremissing or not the relative position of the teeth should befound at first +e DeepLabV3+ model is adapted forprocessing semantic segmentation to find the location of theteeth+e dilation rate is adjusted several times in our modelto avoid missing information in the convolution process+en to verify the advantages of dilated convolution thedilation rate in the network is changed to zero for a com-parison experiment

411 Preprocess First to facilitate the later training processthe frame images are resized as 512times 512+en the region ofinterest of the images is annotated through the ldquolabelme rdquo an

image labeling tool Images with teeth are labeled with thelocation of teeth and the images without teeth do not needany operation After the image labeling is completed toensure the rationality of our model the image is processedby methods such as rotation scaling and cropping and theprocessed image is added to the data as new images +elabeled image is shown in Figure 8

412 Parameters +e details of the DeepLabV3+ are shownin Figure 3 In this section the structure of themodel will notbe changed but the dilation rates are changed to prevent theinformation missing from the dilated convolution To obtainthe information of each pixel by dilated convolution thedilation rates are changed to 0 1 2 and 3 Later to prove thesuperiority of the dilated convolution for image segmen-tation the dilated rates are set as 0 0 0 and 0 for com-parison experiments

413 Training +e colorful images are transformed intograyscale images for training reducing the color differencenoise brought by the day and night +e initial learning rateis set as 001 and the learning rate will be adjusted dy-namically [19]

414 Evaluation Metrics

(i) Pixel accuracy (PA) means the ratio of correctlyclassified pixel points in all pixel points In thispaper there are only two kinds of classes of the pixelteeth and background P10 represents the totalnumber of pixels in the background predicted to bein the teeth P11 represents true positive P10 meansfalse positives P01 means false negative and P00represents true negative

PA P11 + P00

P11 + P10 + P01 + P00 (3)

train accval acc

20

15

10

05

00

valu

e10

09

08

07

06

05

04

valu

e

100 20 30 40 50 60epoch

100 20 30 40 50 60epoch

train accval accval loss

Figure 7 Training metrics (less iterations)

Computational Intelligence and Neuroscience 7

(ii) Mean intersection over union (MIOU) is the stan-dard metric of semantic segmentation It calculatesthe ratio of the intersection and union of two sets thereal value and the predicted value Calculating theintersection over union for each class and averaging itthen the MIOU is obtained as follows

MIOU 12

P11

P11 + P10 + P01+

P00

P01 + P00 + P101113888 1113889 (4)

415 Result After about 1000 epochs of training a rela-tively efficient semantic segmentation model for teeth isobtained +e variation trend of the loss function pixelaccuracy rate and MIOU in the training process is shown inFigure 9

As can be seen from Figure 9 after 1000 times oftraining all the indicators have stabilized However to avoidoverfitting caused by too many iterations the training resultof the 512th epoch is adapted as our model for semanticsegmentation It can be seen from Figure 9 that when theiterations reach the 512th epoch the loss function in oneepoch is stable at about 25 the loss function in one iterationis stable at about 03 PA is about 095 and MIOU is about075 +us the model of the 512th epoch is a very efficienttraining result +e model is adapted to process dozens ofimages that are not in the training dataset to carry outsemantic segmentation and find out the teeth in these im-ages Later verifying the predicted results manually findsthat the predicted results of the teeth are accurate Moreoverthe training dataset consists of grayscale images in the ex-periment and the images of day and night could be accu-rately segmented out the teeth +e segmented results areshown in Figure 10

416 Comparative Experiments A comparative experimentis conducted to verify the validity of the dilation convolu-tion +e dilated rates of the dilation convolution layers arechanged to zero and a new model is carried out By ob-serving the evaluation metrics of the model the comparativeexperimental results are obtained as shown in Figure 11

As shown in Figure 11 the results of the dilation con-volution are better +e model is stable at the 500th epochbut the comparative experiment will not converge until the1000th epoch Moreover the experimental results of thedilation convolution are relatively stable while the results ofthe other show great differences after each epoch +ereforethis set of comparative experiments effectively illustrates theexcellent features of fast converge and stable experimentalresults of dilation convolution in teeth semanticsegmentation

42 Image Registration After the image segmentation thebinary images with teeth are obtained +e number of teethin these binary images is considered as the metric to dis-charge whether the teeth are missing or not Since the shapeof the tooth is similar to a rectangle the number of rect-angles is calculated in the image after registration If thenumber of teeth is less than five the teeth are consideredmissing otherwise it is intact

First of all the reference image is a need in imageregistration An image with relatively clear and intact teeth isselected as the reference image from the results of imageregistration Because of the camerarsquos angle of view the teethare always centered in the images and only move up ordown +e registration process is not complicated +eobjects of the image registration experiments are all thebinary images with teeth obtained from image segmentationin previous experiments After the image registration the

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(a)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(b)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(c)

Figure 8 Labeled images

8 Computational Intelligence and Neuroscience

number of rectangles at the bottom of the images is cal-culated Part of the experimental results is shown inFigure 12

According to the number of rectangular structures in theregistration experimental results there are some defects injudging whether the teeth are missing or not +e rectan-gular structure is the smallest rectangle that covers each

connected domain in the image When multiple teeth areconnected due to noise as shown in Figure 13(a) themethod of counting the number of rectangles fails Besideswhen the teeth are broken not fall off as shown inFigure 13(b) the broken teeth will also have a rectangularstructure after image registration Due to noise there may bemore rectangular structures than the teeth in the result ofimage registration as shown in Figure 13(c) As a result thismethod does not work

+erefore another method is proposed which adaptedthe area ratio (α) of the teeth in a registered image to thereference image as the evaluation metric As noise will affectthe area to more accurately judge whether the teeth arebroken or fall off the area of teeth in the registered image iscalculated where the teeth are located in the reference imageIn the image registration due to noise registration ab-normalities will occur As shown in Figure 14 the ratio ofteeth area in the registered image to the reference image istoo small or too large

Without considering simultaneous falling off or fractureof multiple teeth the following threshold values for judgingwhether teeth fall off or fracture can be obtained

(i) αle 06 registration anomaly(ii) 06lt αle 08 a tooth falls off at least(iii) 08lt αle 09 a tooth fractures(iv) 09lt αle 11 teeth intact(v) 11lt α registration anomaly

0900

0700

0500

0300

0100

traintotal_loss_iter

0000 2000 k 4000 k 6000 k 8000 k

(a)

traintotal_loss_epoch

0000 2000 4000 6000 8000 1000 k

800

700

600

500

400

300

200

(b)

valAcc_class

0000 2000 4000 6000 8000 1000 k

0950

0850

0750

0650

0550

0450

(c)

valmloU

0000 2000 4000 6000 8000 1000 k

0800

0900

0700

0600

0500

0400

0300

(d)

Figure 9 Evaluation metrics (a) loss function in one iteration (b) loss function in epoch (c) PA (d) MIOU

Semantic segmentation

Semantic segmentation

Semantic segmentation

Figure 10 +e segmented images

Computational Intelligence and Neuroscience 9

To verify the effectiveness of the image registration weconducted two groups of test experiments and observe thetest results +e dataset of the two groups of experiments

consists of the following one is 70 positive sampleswithout teeth missing and 35 negative samples withmissing teeth and the other one is 65 positive samples and

10

09

08

07

06

05

04

0 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(a)

10

09

08

07

06

05

040 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(b)

Figure 11 Evaluation metrics (a) dilation convolution (b) comparative experiment

Segmented images

Registration Mark teeth

Marked images

5 meansintact

alarm

Registered images

Figure 12 +e registration result

(a) (b) (c)

Figure 13 Special circumstances (a) connected teeth (b) broken teeth (c) more teeth

10 Computational Intelligence and Neuroscience

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

image the dilated convolution with a small dilation rate isproposed to ensure the accurate segmentation of the teethTo ensure that all pixels are involved in the calculation thedilation rates in the deep convolutional layers satisfy thefollowing formula

Mi max M(i1) minus 2r(i1)M(i1) minus 2 Mi1 minus ri( 1113857 ri1113858 1113859 (2)

where Mi is the maximum dilation rate of the ith layer and ri

denotes the dilation rate of the ith layer with Mn rnIn the actual work of detection the relative position and

size of the teeth in the image recorded by the camera areconstantly changing which is a challenging difficultyDeepLabV3+ adopts a combination of dilated convolutioncascade and parallel with multiple dilation rates to effectively

(a) (b)

Figure 1 Illustration of our proposed solution on automatic electric shovel teeth detection

Input 64643

646432 323232 323232 161632 1616328864

114096

Convolution

ConvMax-pooling Pooling PoolingConv

Fully connected

Output

Figure 2 +e structure of classification network for salient detection

DeepLab-v3+Encoder

1times1 Conv

1times1 Conv

1times1 Conv

3times3 Conv

3times3 Convrate = 1

3times3 Convrate = 2

3times3 Convrate = 3

ImagePooling

low-levelfeatures

Upsampleby 4

Upsampleby 4

Decoder

Concat

Figure 3 DeepLabV3+ structure

4 Computational Intelligence and Neuroscience

extract multiscale semantic information As a result nomatter where the relative position of the teeth and no matterwhat the size of teeth the model processes image seg-mentation efficiently Also by combining image-level in-formation the ability of ASPP to extract global semanticinformation is improved to further improve the effect

23 Image Registration After the image segmentation bi-nary images containing only teeth are obtained as shown inFigure 4(a) In this section the image registration is pro-posed to detect whether the teeth are missing or not +eimage in which the number of teeth (M) is zero is deleted inthe salient object detection +erefore in the process ofimage registration the number of teeth (M) ranges from oneto five

In the process of image registration the reference imageis set up firstly In this paper the images obtained after imagesegmentation are highly similar except the relative positionof teeth is different Also the relative position of each tooth isthe same +erefore one relatively clear image obtainedfrom image segmentation with intact teeth can be used as thereference image +e final reference image shows five nearlyidentical rectangular structures at the bottom of the binaryimage as shown in Figure 4(b)

However in the segmented images the relative positionof teeth is different and the pixelsrsquo information of the imagesis also very different +e reference image and the process ofimage registration are shown in Figure 4 As shown inFigure 5 the teeth in registered images are at the bottom ofthe image and have obvious rectangular structures Whenthere is no tooth missing the registered image contains fiverectangular structures When the teeth are missing thenumber of the rectangular structure is less than fiveMoreover the registered image is a binary image which isvery beneficial to some processing methods such as edgedetection +erefore we detect the rectangles in the regis-tered image and calculate the number of rectangles to de-termine whether the teeth are missing or not

3 Experiments and Discussion

+e experiments that we conducted are divided into threemain sections (1) salient object detection (2) image seg-mentation and (3) image registration

+e first sets of experiments are conducted to deleteimages without teeth to reduce the amount of computationand the second sets of experiments are conducted to find theregion of interest (ROI) and the location of teeth in theimages Moreover it should be noticed that multiple dilationrates are proposed to extract multisemantic informationand the dilated convolution neural network is contracted toa normal convolution to demonstrate that the DeepLabV3+model works more effectively +ird image registration isconducted to detect whether the teeth are missing or not bycounting the number of rectangles

31 Datasets and Hardware +e datasets in this paper arefrom frame images of the video stream taken by the camera

on the shovel Frame images are extracted from the videostream consisting of 500 with and 500 without teeth Afterlabeling these images twenty percent of the images are usedas a validation set to test the training model and the othereighty percent of the images are used as the training set totrain the classification model +e training and inference ofour proposed model are run with two NVIDIA GTX1080 Ti11GB GPUs 32G RAM and Intel i7-7700K CPU with 8cores 420GHz Especially we did not adopt any pre-processing of noise management for our dataset in order topreserve original details and avoid possible artifacts fortraining and inference process

32 Salient Object Detection Studies To reduce the com-putation of the terminal server salient object detection isproposed to delete the images without teeth To solve theshortcomings of traditional salient object detection a di-chotomous neural network is adapted to delete the imagelabeled no teeth

321 Preprocessing +is set of experiments aims to use edgecomputing to alleviate the pressure of the terminal server+erefore to complete the edge computing faster and reducethe calculation amount the images are resized into64times64times 3

322 Parameters +e networkrsquos detailed connectivity andkernel configuration are illustrated in Figure 2 And all theconvolutional kernelsrsquo sizes are the same as the size of 3times 3+e pooling layer of every convolutional layer is Max-Pooling the kernel size is 2times 2 and the step size is 2 +enumber of this neural network is 233472 which is less thanhalf of the AlexNet Adam optimizer is adapted with alearning rate of 00001 +e batch size is set as 32 and thenumber of iterations is set as 8000

323 Training Process Normally too many iterations maycause the model to fit the noise and unnecessary features ofthe training sets and then overfitting occurs In the case ofoverfitting themodel is of high complexity and does not workto the data except the training set To avoid excessive com-putation the loss function of the validation dataset and theaccuracy of the validation dataset are adapted as the evalu-ation metrics for model selection When the accuracy of thevalidation dataset reaches a high level and the accuracy is notimproved after 10 epochs also the loss function is not de-clined the iteration is terminating in advance+e final modelis obtained In the training process the accuracy of thetraining dataset the accuracy of the validation dataset and theloss function of the validation dataset are shown in Figure 6

As shown in Figure 6 the accuracy of the validationdataset and the training dataset reaches the global maximumwithin fifty epochs and the loss function of the validationdataset does not decrease in the subsequent epochs Asshown in Figure 6 the accuracy of training and validationfluctuates periodically since some groups of sampled imagescontain lots of noise as a result of complicated wild working

Computational Intelligence and Neuroscience 5

environments which leads to poor performances of se-mantic segmentation +erefore to prevent the trainingmodel from overfitting the number of iterations is changed

into 1600 making the epochs to 64 reducing the trainingtime and calculation amount of the training model +etraining result is shown in Figure 7

(a) (b)

Figure 4 Segmented images

e reference image

e image registration process

Registered images

Segmented images

Figure 5 +e image registration process

20

15

10

05

00

09

10

08

07

06

05

04

valu

e

valu

e

0 50 100 150 200 250 300epoch

0 50 100 150 200 250 300epoch

train accval accval loss

train accval acc

Figure 6 Training metrics

6 Computational Intelligence and Neuroscience

4 Results

As can be seen from Figure 7 in the 56th epoch the accuracyof the training dataset and the accuracy and the loss functionof the validation dataset reach the optimal level +e ac-curacy of the training dataset is 9375 the accuracy of thevalidation dataset is 9687 and the loss function of thevalidation is 00965 +erefore ldquoteeth-no teeth-model-1400rdquogenerated by the model trained to the 56th epoch is adaptedto detect whether the teeth are missing or not

One video stream of a working shovel is incepted allframe images are extracted from it and the frame images areput into the trained model for prediction If the predictionresult of the image is ldquoteethrdquo this image is kept for the nextpart of the experiments Otherwise this image is deleted Inthe prediction experiment of the frame images of the cap-tured video stream the prediction results are all correctindicating that the model predicts effectively In the threeconvolutional layers of the network the feature maps ofimages with teeth and without teeth are shown in the fol-lowing figures It is through learning these features that ourmodel can correctly predict whether the images containteeth or not and then complete the salient object detection

41 Image Segmentation To detect whether the teeth aremissing or not the relative position of the teeth should befound at first +e DeepLabV3+ model is adapted forprocessing semantic segmentation to find the location of theteeth+e dilation rate is adjusted several times in our modelto avoid missing information in the convolution process+en to verify the advantages of dilated convolution thedilation rate in the network is changed to zero for a com-parison experiment

411 Preprocess First to facilitate the later training processthe frame images are resized as 512times 512+en the region ofinterest of the images is annotated through the ldquolabelme rdquo an

image labeling tool Images with teeth are labeled with thelocation of teeth and the images without teeth do not needany operation After the image labeling is completed toensure the rationality of our model the image is processedby methods such as rotation scaling and cropping and theprocessed image is added to the data as new images +elabeled image is shown in Figure 8

412 Parameters +e details of the DeepLabV3+ are shownin Figure 3 In this section the structure of themodel will notbe changed but the dilation rates are changed to prevent theinformation missing from the dilated convolution To obtainthe information of each pixel by dilated convolution thedilation rates are changed to 0 1 2 and 3 Later to prove thesuperiority of the dilated convolution for image segmen-tation the dilated rates are set as 0 0 0 and 0 for com-parison experiments

413 Training +e colorful images are transformed intograyscale images for training reducing the color differencenoise brought by the day and night +e initial learning rateis set as 001 and the learning rate will be adjusted dy-namically [19]

414 Evaluation Metrics

(i) Pixel accuracy (PA) means the ratio of correctlyclassified pixel points in all pixel points In thispaper there are only two kinds of classes of the pixelteeth and background P10 represents the totalnumber of pixels in the background predicted to bein the teeth P11 represents true positive P10 meansfalse positives P01 means false negative and P00represents true negative

PA P11 + P00

P11 + P10 + P01 + P00 (3)

train accval acc

20

15

10

05

00

valu

e10

09

08

07

06

05

04

valu

e

100 20 30 40 50 60epoch

100 20 30 40 50 60epoch

train accval accval loss

Figure 7 Training metrics (less iterations)

Computational Intelligence and Neuroscience 7

(ii) Mean intersection over union (MIOU) is the stan-dard metric of semantic segmentation It calculatesthe ratio of the intersection and union of two sets thereal value and the predicted value Calculating theintersection over union for each class and averaging itthen the MIOU is obtained as follows

MIOU 12

P11

P11 + P10 + P01+

P00

P01 + P00 + P101113888 1113889 (4)

415 Result After about 1000 epochs of training a rela-tively efficient semantic segmentation model for teeth isobtained +e variation trend of the loss function pixelaccuracy rate and MIOU in the training process is shown inFigure 9

As can be seen from Figure 9 after 1000 times oftraining all the indicators have stabilized However to avoidoverfitting caused by too many iterations the training resultof the 512th epoch is adapted as our model for semanticsegmentation It can be seen from Figure 9 that when theiterations reach the 512th epoch the loss function in oneepoch is stable at about 25 the loss function in one iterationis stable at about 03 PA is about 095 and MIOU is about075 +us the model of the 512th epoch is a very efficienttraining result +e model is adapted to process dozens ofimages that are not in the training dataset to carry outsemantic segmentation and find out the teeth in these im-ages Later verifying the predicted results manually findsthat the predicted results of the teeth are accurate Moreoverthe training dataset consists of grayscale images in the ex-periment and the images of day and night could be accu-rately segmented out the teeth +e segmented results areshown in Figure 10

416 Comparative Experiments A comparative experimentis conducted to verify the validity of the dilation convolu-tion +e dilated rates of the dilation convolution layers arechanged to zero and a new model is carried out By ob-serving the evaluation metrics of the model the comparativeexperimental results are obtained as shown in Figure 11

As shown in Figure 11 the results of the dilation con-volution are better +e model is stable at the 500th epochbut the comparative experiment will not converge until the1000th epoch Moreover the experimental results of thedilation convolution are relatively stable while the results ofthe other show great differences after each epoch +ereforethis set of comparative experiments effectively illustrates theexcellent features of fast converge and stable experimentalresults of dilation convolution in teeth semanticsegmentation

42 Image Registration After the image segmentation thebinary images with teeth are obtained +e number of teethin these binary images is considered as the metric to dis-charge whether the teeth are missing or not Since the shapeof the tooth is similar to a rectangle the number of rect-angles is calculated in the image after registration If thenumber of teeth is less than five the teeth are consideredmissing otherwise it is intact

First of all the reference image is a need in imageregistration An image with relatively clear and intact teeth isselected as the reference image from the results of imageregistration Because of the camerarsquos angle of view the teethare always centered in the images and only move up ordown +e registration process is not complicated +eobjects of the image registration experiments are all thebinary images with teeth obtained from image segmentationin previous experiments After the image registration the

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(a)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(b)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(c)

Figure 8 Labeled images

8 Computational Intelligence and Neuroscience

number of rectangles at the bottom of the images is cal-culated Part of the experimental results is shown inFigure 12

According to the number of rectangular structures in theregistration experimental results there are some defects injudging whether the teeth are missing or not +e rectan-gular structure is the smallest rectangle that covers each

connected domain in the image When multiple teeth areconnected due to noise as shown in Figure 13(a) themethod of counting the number of rectangles fails Besideswhen the teeth are broken not fall off as shown inFigure 13(b) the broken teeth will also have a rectangularstructure after image registration Due to noise there may bemore rectangular structures than the teeth in the result ofimage registration as shown in Figure 13(c) As a result thismethod does not work

+erefore another method is proposed which adaptedthe area ratio (α) of the teeth in a registered image to thereference image as the evaluation metric As noise will affectthe area to more accurately judge whether the teeth arebroken or fall off the area of teeth in the registered image iscalculated where the teeth are located in the reference imageIn the image registration due to noise registration ab-normalities will occur As shown in Figure 14 the ratio ofteeth area in the registered image to the reference image istoo small or too large

Without considering simultaneous falling off or fractureof multiple teeth the following threshold values for judgingwhether teeth fall off or fracture can be obtained

(i) αle 06 registration anomaly(ii) 06lt αle 08 a tooth falls off at least(iii) 08lt αle 09 a tooth fractures(iv) 09lt αle 11 teeth intact(v) 11lt α registration anomaly

0900

0700

0500

0300

0100

traintotal_loss_iter

0000 2000 k 4000 k 6000 k 8000 k

(a)

traintotal_loss_epoch

0000 2000 4000 6000 8000 1000 k

800

700

600

500

400

300

200

(b)

valAcc_class

0000 2000 4000 6000 8000 1000 k

0950

0850

0750

0650

0550

0450

(c)

valmloU

0000 2000 4000 6000 8000 1000 k

0800

0900

0700

0600

0500

0400

0300

(d)

Figure 9 Evaluation metrics (a) loss function in one iteration (b) loss function in epoch (c) PA (d) MIOU

Semantic segmentation

Semantic segmentation

Semantic segmentation

Figure 10 +e segmented images

Computational Intelligence and Neuroscience 9

To verify the effectiveness of the image registration weconducted two groups of test experiments and observe thetest results +e dataset of the two groups of experiments

consists of the following one is 70 positive sampleswithout teeth missing and 35 negative samples withmissing teeth and the other one is 65 positive samples and

10

09

08

07

06

05

04

0 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(a)

10

09

08

07

06

05

040 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(b)

Figure 11 Evaluation metrics (a) dilation convolution (b) comparative experiment

Segmented images

Registration Mark teeth

Marked images

5 meansintact

alarm

Registered images

Figure 12 +e registration result

(a) (b) (c)

Figure 13 Special circumstances (a) connected teeth (b) broken teeth (c) more teeth

10 Computational Intelligence and Neuroscience

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

extract multiscale semantic information As a result nomatter where the relative position of the teeth and no matterwhat the size of teeth the model processes image seg-mentation efficiently Also by combining image-level in-formation the ability of ASPP to extract global semanticinformation is improved to further improve the effect

23 Image Registration After the image segmentation bi-nary images containing only teeth are obtained as shown inFigure 4(a) In this section the image registration is pro-posed to detect whether the teeth are missing or not +eimage in which the number of teeth (M) is zero is deleted inthe salient object detection +erefore in the process ofimage registration the number of teeth (M) ranges from oneto five

In the process of image registration the reference imageis set up firstly In this paper the images obtained after imagesegmentation are highly similar except the relative positionof teeth is different Also the relative position of each tooth isthe same +erefore one relatively clear image obtainedfrom image segmentation with intact teeth can be used as thereference image +e final reference image shows five nearlyidentical rectangular structures at the bottom of the binaryimage as shown in Figure 4(b)

However in the segmented images the relative positionof teeth is different and the pixelsrsquo information of the imagesis also very different +e reference image and the process ofimage registration are shown in Figure 4 As shown inFigure 5 the teeth in registered images are at the bottom ofthe image and have obvious rectangular structures Whenthere is no tooth missing the registered image contains fiverectangular structures When the teeth are missing thenumber of the rectangular structure is less than fiveMoreover the registered image is a binary image which isvery beneficial to some processing methods such as edgedetection +erefore we detect the rectangles in the regis-tered image and calculate the number of rectangles to de-termine whether the teeth are missing or not

3 Experiments and Discussion

+e experiments that we conducted are divided into threemain sections (1) salient object detection (2) image seg-mentation and (3) image registration

+e first sets of experiments are conducted to deleteimages without teeth to reduce the amount of computationand the second sets of experiments are conducted to find theregion of interest (ROI) and the location of teeth in theimages Moreover it should be noticed that multiple dilationrates are proposed to extract multisemantic informationand the dilated convolution neural network is contracted toa normal convolution to demonstrate that the DeepLabV3+model works more effectively +ird image registration isconducted to detect whether the teeth are missing or not bycounting the number of rectangles

31 Datasets and Hardware +e datasets in this paper arefrom frame images of the video stream taken by the camera

on the shovel Frame images are extracted from the videostream consisting of 500 with and 500 without teeth Afterlabeling these images twenty percent of the images are usedas a validation set to test the training model and the othereighty percent of the images are used as the training set totrain the classification model +e training and inference ofour proposed model are run with two NVIDIA GTX1080 Ti11GB GPUs 32G RAM and Intel i7-7700K CPU with 8cores 420GHz Especially we did not adopt any pre-processing of noise management for our dataset in order topreserve original details and avoid possible artifacts fortraining and inference process

32 Salient Object Detection Studies To reduce the com-putation of the terminal server salient object detection isproposed to delete the images without teeth To solve theshortcomings of traditional salient object detection a di-chotomous neural network is adapted to delete the imagelabeled no teeth

321 Preprocessing +is set of experiments aims to use edgecomputing to alleviate the pressure of the terminal server+erefore to complete the edge computing faster and reducethe calculation amount the images are resized into64times64times 3

322 Parameters +e networkrsquos detailed connectivity andkernel configuration are illustrated in Figure 2 And all theconvolutional kernelsrsquo sizes are the same as the size of 3times 3+e pooling layer of every convolutional layer is Max-Pooling the kernel size is 2times 2 and the step size is 2 +enumber of this neural network is 233472 which is less thanhalf of the AlexNet Adam optimizer is adapted with alearning rate of 00001 +e batch size is set as 32 and thenumber of iterations is set as 8000

323 Training Process Normally too many iterations maycause the model to fit the noise and unnecessary features ofthe training sets and then overfitting occurs In the case ofoverfitting themodel is of high complexity and does not workto the data except the training set To avoid excessive com-putation the loss function of the validation dataset and theaccuracy of the validation dataset are adapted as the evalu-ation metrics for model selection When the accuracy of thevalidation dataset reaches a high level and the accuracy is notimproved after 10 epochs also the loss function is not de-clined the iteration is terminating in advance+e final modelis obtained In the training process the accuracy of thetraining dataset the accuracy of the validation dataset and theloss function of the validation dataset are shown in Figure 6

As shown in Figure 6 the accuracy of the validationdataset and the training dataset reaches the global maximumwithin fifty epochs and the loss function of the validationdataset does not decrease in the subsequent epochs Asshown in Figure 6 the accuracy of training and validationfluctuates periodically since some groups of sampled imagescontain lots of noise as a result of complicated wild working

Computational Intelligence and Neuroscience 5

environments which leads to poor performances of se-mantic segmentation +erefore to prevent the trainingmodel from overfitting the number of iterations is changed

into 1600 making the epochs to 64 reducing the trainingtime and calculation amount of the training model +etraining result is shown in Figure 7

(a) (b)

Figure 4 Segmented images

e reference image

e image registration process

Registered images

Segmented images

Figure 5 +e image registration process

20

15

10

05

00

09

10

08

07

06

05

04

valu

e

valu

e

0 50 100 150 200 250 300epoch

0 50 100 150 200 250 300epoch

train accval accval loss

train accval acc

Figure 6 Training metrics

6 Computational Intelligence and Neuroscience

4 Results

As can be seen from Figure 7 in the 56th epoch the accuracyof the training dataset and the accuracy and the loss functionof the validation dataset reach the optimal level +e ac-curacy of the training dataset is 9375 the accuracy of thevalidation dataset is 9687 and the loss function of thevalidation is 00965 +erefore ldquoteeth-no teeth-model-1400rdquogenerated by the model trained to the 56th epoch is adaptedto detect whether the teeth are missing or not

One video stream of a working shovel is incepted allframe images are extracted from it and the frame images areput into the trained model for prediction If the predictionresult of the image is ldquoteethrdquo this image is kept for the nextpart of the experiments Otherwise this image is deleted Inthe prediction experiment of the frame images of the cap-tured video stream the prediction results are all correctindicating that the model predicts effectively In the threeconvolutional layers of the network the feature maps ofimages with teeth and without teeth are shown in the fol-lowing figures It is through learning these features that ourmodel can correctly predict whether the images containteeth or not and then complete the salient object detection

41 Image Segmentation To detect whether the teeth aremissing or not the relative position of the teeth should befound at first +e DeepLabV3+ model is adapted forprocessing semantic segmentation to find the location of theteeth+e dilation rate is adjusted several times in our modelto avoid missing information in the convolution process+en to verify the advantages of dilated convolution thedilation rate in the network is changed to zero for a com-parison experiment

411 Preprocess First to facilitate the later training processthe frame images are resized as 512times 512+en the region ofinterest of the images is annotated through the ldquolabelme rdquo an

image labeling tool Images with teeth are labeled with thelocation of teeth and the images without teeth do not needany operation After the image labeling is completed toensure the rationality of our model the image is processedby methods such as rotation scaling and cropping and theprocessed image is added to the data as new images +elabeled image is shown in Figure 8

412 Parameters +e details of the DeepLabV3+ are shownin Figure 3 In this section the structure of themodel will notbe changed but the dilation rates are changed to prevent theinformation missing from the dilated convolution To obtainthe information of each pixel by dilated convolution thedilation rates are changed to 0 1 2 and 3 Later to prove thesuperiority of the dilated convolution for image segmen-tation the dilated rates are set as 0 0 0 and 0 for com-parison experiments

413 Training +e colorful images are transformed intograyscale images for training reducing the color differencenoise brought by the day and night +e initial learning rateis set as 001 and the learning rate will be adjusted dy-namically [19]

414 Evaluation Metrics

(i) Pixel accuracy (PA) means the ratio of correctlyclassified pixel points in all pixel points In thispaper there are only two kinds of classes of the pixelteeth and background P10 represents the totalnumber of pixels in the background predicted to bein the teeth P11 represents true positive P10 meansfalse positives P01 means false negative and P00represents true negative

PA P11 + P00

P11 + P10 + P01 + P00 (3)

train accval acc

20

15

10

05

00

valu

e10

09

08

07

06

05

04

valu

e

100 20 30 40 50 60epoch

100 20 30 40 50 60epoch

train accval accval loss

Figure 7 Training metrics (less iterations)

Computational Intelligence and Neuroscience 7

(ii) Mean intersection over union (MIOU) is the stan-dard metric of semantic segmentation It calculatesthe ratio of the intersection and union of two sets thereal value and the predicted value Calculating theintersection over union for each class and averaging itthen the MIOU is obtained as follows

MIOU 12

P11

P11 + P10 + P01+

P00

P01 + P00 + P101113888 1113889 (4)

415 Result After about 1000 epochs of training a rela-tively efficient semantic segmentation model for teeth isobtained +e variation trend of the loss function pixelaccuracy rate and MIOU in the training process is shown inFigure 9

As can be seen from Figure 9 after 1000 times oftraining all the indicators have stabilized However to avoidoverfitting caused by too many iterations the training resultof the 512th epoch is adapted as our model for semanticsegmentation It can be seen from Figure 9 that when theiterations reach the 512th epoch the loss function in oneepoch is stable at about 25 the loss function in one iterationis stable at about 03 PA is about 095 and MIOU is about075 +us the model of the 512th epoch is a very efficienttraining result +e model is adapted to process dozens ofimages that are not in the training dataset to carry outsemantic segmentation and find out the teeth in these im-ages Later verifying the predicted results manually findsthat the predicted results of the teeth are accurate Moreoverthe training dataset consists of grayscale images in the ex-periment and the images of day and night could be accu-rately segmented out the teeth +e segmented results areshown in Figure 10

416 Comparative Experiments A comparative experimentis conducted to verify the validity of the dilation convolu-tion +e dilated rates of the dilation convolution layers arechanged to zero and a new model is carried out By ob-serving the evaluation metrics of the model the comparativeexperimental results are obtained as shown in Figure 11

As shown in Figure 11 the results of the dilation con-volution are better +e model is stable at the 500th epochbut the comparative experiment will not converge until the1000th epoch Moreover the experimental results of thedilation convolution are relatively stable while the results ofthe other show great differences after each epoch +ereforethis set of comparative experiments effectively illustrates theexcellent features of fast converge and stable experimentalresults of dilation convolution in teeth semanticsegmentation

42 Image Registration After the image segmentation thebinary images with teeth are obtained +e number of teethin these binary images is considered as the metric to dis-charge whether the teeth are missing or not Since the shapeof the tooth is similar to a rectangle the number of rect-angles is calculated in the image after registration If thenumber of teeth is less than five the teeth are consideredmissing otherwise it is intact

First of all the reference image is a need in imageregistration An image with relatively clear and intact teeth isselected as the reference image from the results of imageregistration Because of the camerarsquos angle of view the teethare always centered in the images and only move up ordown +e registration process is not complicated +eobjects of the image registration experiments are all thebinary images with teeth obtained from image segmentationin previous experiments After the image registration the

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(a)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(b)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(c)

Figure 8 Labeled images

8 Computational Intelligence and Neuroscience

number of rectangles at the bottom of the images is cal-culated Part of the experimental results is shown inFigure 12

According to the number of rectangular structures in theregistration experimental results there are some defects injudging whether the teeth are missing or not +e rectan-gular structure is the smallest rectangle that covers each

connected domain in the image When multiple teeth areconnected due to noise as shown in Figure 13(a) themethod of counting the number of rectangles fails Besideswhen the teeth are broken not fall off as shown inFigure 13(b) the broken teeth will also have a rectangularstructure after image registration Due to noise there may bemore rectangular structures than the teeth in the result ofimage registration as shown in Figure 13(c) As a result thismethod does not work

+erefore another method is proposed which adaptedthe area ratio (α) of the teeth in a registered image to thereference image as the evaluation metric As noise will affectthe area to more accurately judge whether the teeth arebroken or fall off the area of teeth in the registered image iscalculated where the teeth are located in the reference imageIn the image registration due to noise registration ab-normalities will occur As shown in Figure 14 the ratio ofteeth area in the registered image to the reference image istoo small or too large

Without considering simultaneous falling off or fractureof multiple teeth the following threshold values for judgingwhether teeth fall off or fracture can be obtained

(i) αle 06 registration anomaly(ii) 06lt αle 08 a tooth falls off at least(iii) 08lt αle 09 a tooth fractures(iv) 09lt αle 11 teeth intact(v) 11lt α registration anomaly

0900

0700

0500

0300

0100

traintotal_loss_iter

0000 2000 k 4000 k 6000 k 8000 k

(a)

traintotal_loss_epoch

0000 2000 4000 6000 8000 1000 k

800

700

600

500

400

300

200

(b)

valAcc_class

0000 2000 4000 6000 8000 1000 k

0950

0850

0750

0650

0550

0450

(c)

valmloU

0000 2000 4000 6000 8000 1000 k

0800

0900

0700

0600

0500

0400

0300

(d)

Figure 9 Evaluation metrics (a) loss function in one iteration (b) loss function in epoch (c) PA (d) MIOU

Semantic segmentation

Semantic segmentation

Semantic segmentation

Figure 10 +e segmented images

Computational Intelligence and Neuroscience 9

To verify the effectiveness of the image registration weconducted two groups of test experiments and observe thetest results +e dataset of the two groups of experiments

consists of the following one is 70 positive sampleswithout teeth missing and 35 negative samples withmissing teeth and the other one is 65 positive samples and

10

09

08

07

06

05

04

0 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(a)

10

09

08

07

06

05

040 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(b)

Figure 11 Evaluation metrics (a) dilation convolution (b) comparative experiment

Segmented images

Registration Mark teeth

Marked images

5 meansintact

alarm

Registered images

Figure 12 +e registration result

(a) (b) (c)

Figure 13 Special circumstances (a) connected teeth (b) broken teeth (c) more teeth

10 Computational Intelligence and Neuroscience

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

environments which leads to poor performances of se-mantic segmentation +erefore to prevent the trainingmodel from overfitting the number of iterations is changed

into 1600 making the epochs to 64 reducing the trainingtime and calculation amount of the training model +etraining result is shown in Figure 7

(a) (b)

Figure 4 Segmented images

e reference image

e image registration process

Registered images

Segmented images

Figure 5 +e image registration process

20

15

10

05

00

09

10

08

07

06

05

04

valu

e

valu

e

0 50 100 150 200 250 300epoch

0 50 100 150 200 250 300epoch

train accval accval loss

train accval acc

Figure 6 Training metrics

6 Computational Intelligence and Neuroscience

4 Results

As can be seen from Figure 7 in the 56th epoch the accuracyof the training dataset and the accuracy and the loss functionof the validation dataset reach the optimal level +e ac-curacy of the training dataset is 9375 the accuracy of thevalidation dataset is 9687 and the loss function of thevalidation is 00965 +erefore ldquoteeth-no teeth-model-1400rdquogenerated by the model trained to the 56th epoch is adaptedto detect whether the teeth are missing or not

One video stream of a working shovel is incepted allframe images are extracted from it and the frame images areput into the trained model for prediction If the predictionresult of the image is ldquoteethrdquo this image is kept for the nextpart of the experiments Otherwise this image is deleted Inthe prediction experiment of the frame images of the cap-tured video stream the prediction results are all correctindicating that the model predicts effectively In the threeconvolutional layers of the network the feature maps ofimages with teeth and without teeth are shown in the fol-lowing figures It is through learning these features that ourmodel can correctly predict whether the images containteeth or not and then complete the salient object detection

41 Image Segmentation To detect whether the teeth aremissing or not the relative position of the teeth should befound at first +e DeepLabV3+ model is adapted forprocessing semantic segmentation to find the location of theteeth+e dilation rate is adjusted several times in our modelto avoid missing information in the convolution process+en to verify the advantages of dilated convolution thedilation rate in the network is changed to zero for a com-parison experiment

411 Preprocess First to facilitate the later training processthe frame images are resized as 512times 512+en the region ofinterest of the images is annotated through the ldquolabelme rdquo an

image labeling tool Images with teeth are labeled with thelocation of teeth and the images without teeth do not needany operation After the image labeling is completed toensure the rationality of our model the image is processedby methods such as rotation scaling and cropping and theprocessed image is added to the data as new images +elabeled image is shown in Figure 8

412 Parameters +e details of the DeepLabV3+ are shownin Figure 3 In this section the structure of themodel will notbe changed but the dilation rates are changed to prevent theinformation missing from the dilated convolution To obtainthe information of each pixel by dilated convolution thedilation rates are changed to 0 1 2 and 3 Later to prove thesuperiority of the dilated convolution for image segmen-tation the dilated rates are set as 0 0 0 and 0 for com-parison experiments

413 Training +e colorful images are transformed intograyscale images for training reducing the color differencenoise brought by the day and night +e initial learning rateis set as 001 and the learning rate will be adjusted dy-namically [19]

414 Evaluation Metrics

(i) Pixel accuracy (PA) means the ratio of correctlyclassified pixel points in all pixel points In thispaper there are only two kinds of classes of the pixelteeth and background P10 represents the totalnumber of pixels in the background predicted to bein the teeth P11 represents true positive P10 meansfalse positives P01 means false negative and P00represents true negative

PA P11 + P00

P11 + P10 + P01 + P00 (3)

train accval acc

20

15

10

05

00

valu

e10

09

08

07

06

05

04

valu

e

100 20 30 40 50 60epoch

100 20 30 40 50 60epoch

train accval accval loss

Figure 7 Training metrics (less iterations)

Computational Intelligence and Neuroscience 7

(ii) Mean intersection over union (MIOU) is the stan-dard metric of semantic segmentation It calculatesthe ratio of the intersection and union of two sets thereal value and the predicted value Calculating theintersection over union for each class and averaging itthen the MIOU is obtained as follows

MIOU 12

P11

P11 + P10 + P01+

P00

P01 + P00 + P101113888 1113889 (4)

415 Result After about 1000 epochs of training a rela-tively efficient semantic segmentation model for teeth isobtained +e variation trend of the loss function pixelaccuracy rate and MIOU in the training process is shown inFigure 9

As can be seen from Figure 9 after 1000 times oftraining all the indicators have stabilized However to avoidoverfitting caused by too many iterations the training resultof the 512th epoch is adapted as our model for semanticsegmentation It can be seen from Figure 9 that when theiterations reach the 512th epoch the loss function in oneepoch is stable at about 25 the loss function in one iterationis stable at about 03 PA is about 095 and MIOU is about075 +us the model of the 512th epoch is a very efficienttraining result +e model is adapted to process dozens ofimages that are not in the training dataset to carry outsemantic segmentation and find out the teeth in these im-ages Later verifying the predicted results manually findsthat the predicted results of the teeth are accurate Moreoverthe training dataset consists of grayscale images in the ex-periment and the images of day and night could be accu-rately segmented out the teeth +e segmented results areshown in Figure 10

416 Comparative Experiments A comparative experimentis conducted to verify the validity of the dilation convolu-tion +e dilated rates of the dilation convolution layers arechanged to zero and a new model is carried out By ob-serving the evaluation metrics of the model the comparativeexperimental results are obtained as shown in Figure 11

As shown in Figure 11 the results of the dilation con-volution are better +e model is stable at the 500th epochbut the comparative experiment will not converge until the1000th epoch Moreover the experimental results of thedilation convolution are relatively stable while the results ofthe other show great differences after each epoch +ereforethis set of comparative experiments effectively illustrates theexcellent features of fast converge and stable experimentalresults of dilation convolution in teeth semanticsegmentation

42 Image Registration After the image segmentation thebinary images with teeth are obtained +e number of teethin these binary images is considered as the metric to dis-charge whether the teeth are missing or not Since the shapeof the tooth is similar to a rectangle the number of rect-angles is calculated in the image after registration If thenumber of teeth is less than five the teeth are consideredmissing otherwise it is intact

First of all the reference image is a need in imageregistration An image with relatively clear and intact teeth isselected as the reference image from the results of imageregistration Because of the camerarsquos angle of view the teethare always centered in the images and only move up ordown +e registration process is not complicated +eobjects of the image registration experiments are all thebinary images with teeth obtained from image segmentationin previous experiments After the image registration the

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(a)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(b)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(c)

Figure 8 Labeled images

8 Computational Intelligence and Neuroscience

number of rectangles at the bottom of the images is cal-culated Part of the experimental results is shown inFigure 12

According to the number of rectangular structures in theregistration experimental results there are some defects injudging whether the teeth are missing or not +e rectan-gular structure is the smallest rectangle that covers each

connected domain in the image When multiple teeth areconnected due to noise as shown in Figure 13(a) themethod of counting the number of rectangles fails Besideswhen the teeth are broken not fall off as shown inFigure 13(b) the broken teeth will also have a rectangularstructure after image registration Due to noise there may bemore rectangular structures than the teeth in the result ofimage registration as shown in Figure 13(c) As a result thismethod does not work

+erefore another method is proposed which adaptedthe area ratio (α) of the teeth in a registered image to thereference image as the evaluation metric As noise will affectthe area to more accurately judge whether the teeth arebroken or fall off the area of teeth in the registered image iscalculated where the teeth are located in the reference imageIn the image registration due to noise registration ab-normalities will occur As shown in Figure 14 the ratio ofteeth area in the registered image to the reference image istoo small or too large

Without considering simultaneous falling off or fractureof multiple teeth the following threshold values for judgingwhether teeth fall off or fracture can be obtained

(i) αle 06 registration anomaly(ii) 06lt αle 08 a tooth falls off at least(iii) 08lt αle 09 a tooth fractures(iv) 09lt αle 11 teeth intact(v) 11lt α registration anomaly

0900

0700

0500

0300

0100

traintotal_loss_iter

0000 2000 k 4000 k 6000 k 8000 k

(a)

traintotal_loss_epoch

0000 2000 4000 6000 8000 1000 k

800

700

600

500

400

300

200

(b)

valAcc_class

0000 2000 4000 6000 8000 1000 k

0950

0850

0750

0650

0550

0450

(c)

valmloU

0000 2000 4000 6000 8000 1000 k

0800

0900

0700

0600

0500

0400

0300

(d)

Figure 9 Evaluation metrics (a) loss function in one iteration (b) loss function in epoch (c) PA (d) MIOU

Semantic segmentation

Semantic segmentation

Semantic segmentation

Figure 10 +e segmented images

Computational Intelligence and Neuroscience 9

To verify the effectiveness of the image registration weconducted two groups of test experiments and observe thetest results +e dataset of the two groups of experiments

consists of the following one is 70 positive sampleswithout teeth missing and 35 negative samples withmissing teeth and the other one is 65 positive samples and

10

09

08

07

06

05

04

0 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(a)

10

09

08

07

06

05

040 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(b)

Figure 11 Evaluation metrics (a) dilation convolution (b) comparative experiment

Segmented images

Registration Mark teeth

Marked images

5 meansintact

alarm

Registered images

Figure 12 +e registration result

(a) (b) (c)

Figure 13 Special circumstances (a) connected teeth (b) broken teeth (c) more teeth

10 Computational Intelligence and Neuroscience

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

4 Results

As can be seen from Figure 7 in the 56th epoch the accuracyof the training dataset and the accuracy and the loss functionof the validation dataset reach the optimal level +e ac-curacy of the training dataset is 9375 the accuracy of thevalidation dataset is 9687 and the loss function of thevalidation is 00965 +erefore ldquoteeth-no teeth-model-1400rdquogenerated by the model trained to the 56th epoch is adaptedto detect whether the teeth are missing or not

One video stream of a working shovel is incepted allframe images are extracted from it and the frame images areput into the trained model for prediction If the predictionresult of the image is ldquoteethrdquo this image is kept for the nextpart of the experiments Otherwise this image is deleted Inthe prediction experiment of the frame images of the cap-tured video stream the prediction results are all correctindicating that the model predicts effectively In the threeconvolutional layers of the network the feature maps ofimages with teeth and without teeth are shown in the fol-lowing figures It is through learning these features that ourmodel can correctly predict whether the images containteeth or not and then complete the salient object detection

41 Image Segmentation To detect whether the teeth aremissing or not the relative position of the teeth should befound at first +e DeepLabV3+ model is adapted forprocessing semantic segmentation to find the location of theteeth+e dilation rate is adjusted several times in our modelto avoid missing information in the convolution process+en to verify the advantages of dilated convolution thedilation rate in the network is changed to zero for a com-parison experiment

411 Preprocess First to facilitate the later training processthe frame images are resized as 512times 512+en the region ofinterest of the images is annotated through the ldquolabelme rdquo an

image labeling tool Images with teeth are labeled with thelocation of teeth and the images without teeth do not needany operation After the image labeling is completed toensure the rationality of our model the image is processedby methods such as rotation scaling and cropping and theprocessed image is added to the data as new images +elabeled image is shown in Figure 8

412 Parameters +e details of the DeepLabV3+ are shownin Figure 3 In this section the structure of themodel will notbe changed but the dilation rates are changed to prevent theinformation missing from the dilated convolution To obtainthe information of each pixel by dilated convolution thedilation rates are changed to 0 1 2 and 3 Later to prove thesuperiority of the dilated convolution for image segmen-tation the dilated rates are set as 0 0 0 and 0 for com-parison experiments

413 Training +e colorful images are transformed intograyscale images for training reducing the color differencenoise brought by the day and night +e initial learning rateis set as 001 and the learning rate will be adjusted dy-namically [19]

414 Evaluation Metrics

(i) Pixel accuracy (PA) means the ratio of correctlyclassified pixel points in all pixel points In thispaper there are only two kinds of classes of the pixelteeth and background P10 represents the totalnumber of pixels in the background predicted to bein the teeth P11 represents true positive P10 meansfalse positives P01 means false negative and P00represents true negative

PA P11 + P00

P11 + P10 + P01 + P00 (3)

train accval acc

20

15

10

05

00

valu

e10

09

08

07

06

05

04

valu

e

100 20 30 40 50 60epoch

100 20 30 40 50 60epoch

train accval accval loss

Figure 7 Training metrics (less iterations)

Computational Intelligence and Neuroscience 7

(ii) Mean intersection over union (MIOU) is the stan-dard metric of semantic segmentation It calculatesthe ratio of the intersection and union of two sets thereal value and the predicted value Calculating theintersection over union for each class and averaging itthen the MIOU is obtained as follows

MIOU 12

P11

P11 + P10 + P01+

P00

P01 + P00 + P101113888 1113889 (4)

415 Result After about 1000 epochs of training a rela-tively efficient semantic segmentation model for teeth isobtained +e variation trend of the loss function pixelaccuracy rate and MIOU in the training process is shown inFigure 9

As can be seen from Figure 9 after 1000 times oftraining all the indicators have stabilized However to avoidoverfitting caused by too many iterations the training resultof the 512th epoch is adapted as our model for semanticsegmentation It can be seen from Figure 9 that when theiterations reach the 512th epoch the loss function in oneepoch is stable at about 25 the loss function in one iterationis stable at about 03 PA is about 095 and MIOU is about075 +us the model of the 512th epoch is a very efficienttraining result +e model is adapted to process dozens ofimages that are not in the training dataset to carry outsemantic segmentation and find out the teeth in these im-ages Later verifying the predicted results manually findsthat the predicted results of the teeth are accurate Moreoverthe training dataset consists of grayscale images in the ex-periment and the images of day and night could be accu-rately segmented out the teeth +e segmented results areshown in Figure 10

416 Comparative Experiments A comparative experimentis conducted to verify the validity of the dilation convolu-tion +e dilated rates of the dilation convolution layers arechanged to zero and a new model is carried out By ob-serving the evaluation metrics of the model the comparativeexperimental results are obtained as shown in Figure 11

As shown in Figure 11 the results of the dilation con-volution are better +e model is stable at the 500th epochbut the comparative experiment will not converge until the1000th epoch Moreover the experimental results of thedilation convolution are relatively stable while the results ofthe other show great differences after each epoch +ereforethis set of comparative experiments effectively illustrates theexcellent features of fast converge and stable experimentalresults of dilation convolution in teeth semanticsegmentation

42 Image Registration After the image segmentation thebinary images with teeth are obtained +e number of teethin these binary images is considered as the metric to dis-charge whether the teeth are missing or not Since the shapeof the tooth is similar to a rectangle the number of rect-angles is calculated in the image after registration If thenumber of teeth is less than five the teeth are consideredmissing otherwise it is intact

First of all the reference image is a need in imageregistration An image with relatively clear and intact teeth isselected as the reference image from the results of imageregistration Because of the camerarsquos angle of view the teethare always centered in the images and only move up ordown +e registration process is not complicated +eobjects of the image registration experiments are all thebinary images with teeth obtained from image segmentationin previous experiments After the image registration the

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(a)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(b)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(c)

Figure 8 Labeled images

8 Computational Intelligence and Neuroscience

number of rectangles at the bottom of the images is cal-culated Part of the experimental results is shown inFigure 12

According to the number of rectangular structures in theregistration experimental results there are some defects injudging whether the teeth are missing or not +e rectan-gular structure is the smallest rectangle that covers each

connected domain in the image When multiple teeth areconnected due to noise as shown in Figure 13(a) themethod of counting the number of rectangles fails Besideswhen the teeth are broken not fall off as shown inFigure 13(b) the broken teeth will also have a rectangularstructure after image registration Due to noise there may bemore rectangular structures than the teeth in the result ofimage registration as shown in Figure 13(c) As a result thismethod does not work

+erefore another method is proposed which adaptedthe area ratio (α) of the teeth in a registered image to thereference image as the evaluation metric As noise will affectthe area to more accurately judge whether the teeth arebroken or fall off the area of teeth in the registered image iscalculated where the teeth are located in the reference imageIn the image registration due to noise registration ab-normalities will occur As shown in Figure 14 the ratio ofteeth area in the registered image to the reference image istoo small or too large

Without considering simultaneous falling off or fractureof multiple teeth the following threshold values for judgingwhether teeth fall off or fracture can be obtained

(i) αle 06 registration anomaly(ii) 06lt αle 08 a tooth falls off at least(iii) 08lt αle 09 a tooth fractures(iv) 09lt αle 11 teeth intact(v) 11lt α registration anomaly

0900

0700

0500

0300

0100

traintotal_loss_iter

0000 2000 k 4000 k 6000 k 8000 k

(a)

traintotal_loss_epoch

0000 2000 4000 6000 8000 1000 k

800

700

600

500

400

300

200

(b)

valAcc_class

0000 2000 4000 6000 8000 1000 k

0950

0850

0750

0650

0550

0450

(c)

valmloU

0000 2000 4000 6000 8000 1000 k

0800

0900

0700

0600

0500

0400

0300

(d)

Figure 9 Evaluation metrics (a) loss function in one iteration (b) loss function in epoch (c) PA (d) MIOU

Semantic segmentation

Semantic segmentation

Semantic segmentation

Figure 10 +e segmented images

Computational Intelligence and Neuroscience 9

To verify the effectiveness of the image registration weconducted two groups of test experiments and observe thetest results +e dataset of the two groups of experiments

consists of the following one is 70 positive sampleswithout teeth missing and 35 negative samples withmissing teeth and the other one is 65 positive samples and

10

09

08

07

06

05

04

0 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(a)

10

09

08

07

06

05

040 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(b)

Figure 11 Evaluation metrics (a) dilation convolution (b) comparative experiment

Segmented images

Registration Mark teeth

Marked images

5 meansintact

alarm

Registered images

Figure 12 +e registration result

(a) (b) (c)

Figure 13 Special circumstances (a) connected teeth (b) broken teeth (c) more teeth

10 Computational Intelligence and Neuroscience

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

(ii) Mean intersection over union (MIOU) is the stan-dard metric of semantic segmentation It calculatesthe ratio of the intersection and union of two sets thereal value and the predicted value Calculating theintersection over union for each class and averaging itthen the MIOU is obtained as follows

MIOU 12

P11

P11 + P10 + P01+

P00

P01 + P00 + P101113888 1113889 (4)

415 Result After about 1000 epochs of training a rela-tively efficient semantic segmentation model for teeth isobtained +e variation trend of the loss function pixelaccuracy rate and MIOU in the training process is shown inFigure 9

As can be seen from Figure 9 after 1000 times oftraining all the indicators have stabilized However to avoidoverfitting caused by too many iterations the training resultof the 512th epoch is adapted as our model for semanticsegmentation It can be seen from Figure 9 that when theiterations reach the 512th epoch the loss function in oneepoch is stable at about 25 the loss function in one iterationis stable at about 03 PA is about 095 and MIOU is about075 +us the model of the 512th epoch is a very efficienttraining result +e model is adapted to process dozens ofimages that are not in the training dataset to carry outsemantic segmentation and find out the teeth in these im-ages Later verifying the predicted results manually findsthat the predicted results of the teeth are accurate Moreoverthe training dataset consists of grayscale images in the ex-periment and the images of day and night could be accu-rately segmented out the teeth +e segmented results areshown in Figure 10

416 Comparative Experiments A comparative experimentis conducted to verify the validity of the dilation convolu-tion +e dilated rates of the dilation convolution layers arechanged to zero and a new model is carried out By ob-serving the evaluation metrics of the model the comparativeexperimental results are obtained as shown in Figure 11

As shown in Figure 11 the results of the dilation con-volution are better +e model is stable at the 500th epochbut the comparative experiment will not converge until the1000th epoch Moreover the experimental results of thedilation convolution are relatively stable while the results ofthe other show great differences after each epoch +ereforethis set of comparative experiments effectively illustrates theexcellent features of fast converge and stable experimentalresults of dilation convolution in teeth semanticsegmentation

42 Image Registration After the image segmentation thebinary images with teeth are obtained +e number of teethin these binary images is considered as the metric to dis-charge whether the teeth are missing or not Since the shapeof the tooth is similar to a rectangle the number of rect-angles is calculated in the image after registration If thenumber of teeth is less than five the teeth are consideredmissing otherwise it is intact

First of all the reference image is a need in imageregistration An image with relatively clear and intact teeth isselected as the reference image from the results of imageregistration Because of the camerarsquos angle of view the teethare always centered in the images and only move up ordown +e registration process is not complicated +eobjects of the image registration experiments are all thebinary images with teeth obtained from image segmentationin previous experiments After the image registration the

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(a)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(b)

0

100

200

300

400

500

0

100

200

300

400

500

0 200 400

0 200 400

(c)

Figure 8 Labeled images

8 Computational Intelligence and Neuroscience

number of rectangles at the bottom of the images is cal-culated Part of the experimental results is shown inFigure 12

According to the number of rectangular structures in theregistration experimental results there are some defects injudging whether the teeth are missing or not +e rectan-gular structure is the smallest rectangle that covers each

connected domain in the image When multiple teeth areconnected due to noise as shown in Figure 13(a) themethod of counting the number of rectangles fails Besideswhen the teeth are broken not fall off as shown inFigure 13(b) the broken teeth will also have a rectangularstructure after image registration Due to noise there may bemore rectangular structures than the teeth in the result ofimage registration as shown in Figure 13(c) As a result thismethod does not work

+erefore another method is proposed which adaptedthe area ratio (α) of the teeth in a registered image to thereference image as the evaluation metric As noise will affectthe area to more accurately judge whether the teeth arebroken or fall off the area of teeth in the registered image iscalculated where the teeth are located in the reference imageIn the image registration due to noise registration ab-normalities will occur As shown in Figure 14 the ratio ofteeth area in the registered image to the reference image istoo small or too large

Without considering simultaneous falling off or fractureof multiple teeth the following threshold values for judgingwhether teeth fall off or fracture can be obtained

(i) αle 06 registration anomaly(ii) 06lt αle 08 a tooth falls off at least(iii) 08lt αle 09 a tooth fractures(iv) 09lt αle 11 teeth intact(v) 11lt α registration anomaly

0900

0700

0500

0300

0100

traintotal_loss_iter

0000 2000 k 4000 k 6000 k 8000 k

(a)

traintotal_loss_epoch

0000 2000 4000 6000 8000 1000 k

800

700

600

500

400

300

200

(b)

valAcc_class

0000 2000 4000 6000 8000 1000 k

0950

0850

0750

0650

0550

0450

(c)

valmloU

0000 2000 4000 6000 8000 1000 k

0800

0900

0700

0600

0500

0400

0300

(d)

Figure 9 Evaluation metrics (a) loss function in one iteration (b) loss function in epoch (c) PA (d) MIOU

Semantic segmentation

Semantic segmentation

Semantic segmentation

Figure 10 +e segmented images

Computational Intelligence and Neuroscience 9

To verify the effectiveness of the image registration weconducted two groups of test experiments and observe thetest results +e dataset of the two groups of experiments

consists of the following one is 70 positive sampleswithout teeth missing and 35 negative samples withmissing teeth and the other one is 65 positive samples and

10

09

08

07

06

05

04

0 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(a)

10

09

08

07

06

05

040 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(b)

Figure 11 Evaluation metrics (a) dilation convolution (b) comparative experiment

Segmented images

Registration Mark teeth

Marked images

5 meansintact

alarm

Registered images

Figure 12 +e registration result

(a) (b) (c)

Figure 13 Special circumstances (a) connected teeth (b) broken teeth (c) more teeth

10 Computational Intelligence and Neuroscience

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

number of rectangles at the bottom of the images is cal-culated Part of the experimental results is shown inFigure 12

According to the number of rectangular structures in theregistration experimental results there are some defects injudging whether the teeth are missing or not +e rectan-gular structure is the smallest rectangle that covers each

connected domain in the image When multiple teeth areconnected due to noise as shown in Figure 13(a) themethod of counting the number of rectangles fails Besideswhen the teeth are broken not fall off as shown inFigure 13(b) the broken teeth will also have a rectangularstructure after image registration Due to noise there may bemore rectangular structures than the teeth in the result ofimage registration as shown in Figure 13(c) As a result thismethod does not work

+erefore another method is proposed which adaptedthe area ratio (α) of the teeth in a registered image to thereference image as the evaluation metric As noise will affectthe area to more accurately judge whether the teeth arebroken or fall off the area of teeth in the registered image iscalculated where the teeth are located in the reference imageIn the image registration due to noise registration ab-normalities will occur As shown in Figure 14 the ratio ofteeth area in the registered image to the reference image istoo small or too large

Without considering simultaneous falling off or fractureof multiple teeth the following threshold values for judgingwhether teeth fall off or fracture can be obtained

(i) αle 06 registration anomaly(ii) 06lt αle 08 a tooth falls off at least(iii) 08lt αle 09 a tooth fractures(iv) 09lt αle 11 teeth intact(v) 11lt α registration anomaly

0900

0700

0500

0300

0100

traintotal_loss_iter

0000 2000 k 4000 k 6000 k 8000 k

(a)

traintotal_loss_epoch

0000 2000 4000 6000 8000 1000 k

800

700

600

500

400

300

200

(b)

valAcc_class

0000 2000 4000 6000 8000 1000 k

0950

0850

0750

0650

0550

0450

(c)

valmloU

0000 2000 4000 6000 8000 1000 k

0800

0900

0700

0600

0500

0400

0300

(d)

Figure 9 Evaluation metrics (a) loss function in one iteration (b) loss function in epoch (c) PA (d) MIOU

Semantic segmentation

Semantic segmentation

Semantic segmentation

Figure 10 +e segmented images

Computational Intelligence and Neuroscience 9

To verify the effectiveness of the image registration weconducted two groups of test experiments and observe thetest results +e dataset of the two groups of experiments

consists of the following one is 70 positive sampleswithout teeth missing and 35 negative samples withmissing teeth and the other one is 65 positive samples and

10

09

08

07

06

05

04

0 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(a)

10

09

08

07

06

05

040 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(b)

Figure 11 Evaluation metrics (a) dilation convolution (b) comparative experiment

Segmented images

Registration Mark teeth

Marked images

5 meansintact

alarm

Registered images

Figure 12 +e registration result

(a) (b) (c)

Figure 13 Special circumstances (a) connected teeth (b) broken teeth (c) more teeth

10 Computational Intelligence and Neuroscience

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

To verify the effectiveness of the image registration weconducted two groups of test experiments and observe thetest results +e dataset of the two groups of experiments

consists of the following one is 70 positive sampleswithout teeth missing and 35 negative samples withmissing teeth and the other one is 65 positive samples and

10

09

08

07

06

05

04

0 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(a)

10

09

08

07

06

05

040 250 500 750 1000 1250 1500 1750 2000

Acc-classMlou

(b)

Figure 11 Evaluation metrics (a) dilation convolution (b) comparative experiment

Segmented images

Registration Mark teeth

Marked images

5 meansintact

alarm

Registered images

Figure 12 +e registration result

(a) (b) (c)

Figure 13 Special circumstances (a) connected teeth (b) broken teeth (c) more teeth

10 Computational Intelligence and Neuroscience

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

20 negative samples +e test experiment results areshown in Table 1

As can be seen from Table 1 all negative samples can bedetected accurately and the detection accuracy is more than95However there are still some images without broken teethdetected as broken teeth images and there is still a false alarmrate but it can still meet themissing detection function of teeth

By observing the images of false positives the spatialposition of the shovel is the culprit When the shovel is faraway from the camera the relative size in the image is smallOn the other hand when the shovel is close to the camerathe relative size is large Adapting multiple images is an

efficient solution to handle this +ese images arenamedrdquoDenominatorrdquo According to the differences in thespatial position of the shovel seven reference images areselected as shown in Figure 15

After the image registration because rigid registration isadapted in this paper the starting point of registration can befound by taking the pixel feature As shown in Figure 16 thered line represents the starting point of image registration andobtains the ordinate of the red line(β0) From Figure 17 eachdenominator image has a registration starting point which isβ1 β2 β7 While βi minus 30lt β0 le β(i + 1) minus 30 the ith

denominator image is adapted in area ratio

(a) (b)

Figure 14 Registration abnormalities (a) registered anomaly (b) excessive noise

Table 1 Test result

Positive samples Negative samples True positive True negative False positive70 35 68 35 265 20 64 20 1

(a) (b) (c) (d) (e) (f ) (g)

Figure 15 Denominator images

Segmented image Registered image

Registration

(521)β0

Figure 16 +e coordinates of segmented images

Computational Intelligence and Neuroscience 11

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

To show the experimental result more clearly imageswith an area ratio less than 06 or large than 11 are deleted asabnormal registered images +e detection results can bedivided into two categories

(i) 06lt αle threshold means abnormal teeth(ii) Thresholdlt αle 11 means intact teeth

Meanwhile the detection experiment is transformedinto a two-classifier and the selection of threshold is crucial+e ROC curve is adapted to obtain the best threshold

After setting a range of threshold values a classificationtable (confusion matrix) is built for each threshold valueEach table produces a point in the ROC curve [20] Table 2shows how the performances of these points are calculated in

Registered denominator images

Denominator images

(00) (200)

β1(260)

β2(320) (380)

β3 β4(440)

β5(500) (560)

β6 β7

Figure 17 +e coordinates of denominator images

Table 2 Confusion matrix

ClassifiedActual

Abnormal teeth Intact teethαle threshold True positive False positiveαgt threshold False negative True negative

1

09

08

07

06

05

04

1

09

08

07

06

05

04

03

02

01

0

TPR

TPR

0 01 02 03 04 05 06 07 08 09 1

(0090899)

FPR

FPR0 002 004 006 008 01 012 014 016 018 02

Figure 18 +e ROC curve

12 Computational Intelligence and Neuroscience

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13

this study +ese thresholds separate the dataset into twosubsets the error and no-error sets As shown in the fol-lowing formula the coordinates of the points on the ROCcurve are obtained from the confusion matrix

TPR TP

TP + FN

FPR FN

TP + FN

(5)

+e closest point to point (0 1) is the best threshold ofthis classifier [21] As shown in Figure 18 the coordinate ofthe closest point to point (0 1) is (009 099) and thethreshold of this point is 0899 After selecting the corre-sponding denominator image the threshold value at thistime is close to the ideal value of 09 which shows the logicalcorrectness of the experiment +e experiment in this paperhas also been proved to have high working efficiency when itis applied in the mine of Anshan Iron and Steel MiningCompany

5 Conclusion

In this paper a detection method based on deep learning isproposed for the detection of missing teeth of mine electricshovel +is method effectively solves the instability of thetraditional method can monitor the state of the shovel teethin real-time and can detect the teeth falling off or breakingin time It has great practical significance for mineproduction

Data Availability

+e raw data supporting the conclusions of this article willbe made available by the authors without undue reservation

Conflicts of Interest

+e authors declare that they have no conflicts of interestregarding this work

References

[1] S Frimpong and Y Hu ldquoParametric simulation of shovel-oilsands interactions during excavationrdquo International Journalof Surface Mining Reclamation and Environment vol 18no 3 pp 205ndash219 2004

[2] R Lee ldquoVieselmeyerrdquo Loader Attachment US5669750A1997

[3] C Tian J Zhang T Nie and P University ldquoSimulation onstructure strength of bucket teeth in mine loaderrdquo Journal ofPingxiang University 2016

[4] X Yu Y Chen and G Xie ldquo+e simulation analysis ofdifferent tooth numbers of loader bucket based on edemrdquo IOPConference Series Materials Science and Engineering vol 423Article ID 012055 2018

[5] S Park Y Choi and H S Park ldquoSimulation of shovel-truckhaulage systems in open-pit mines by considering breakdownof trucks and crusher capacityrdquo Tunnelling and UndergroundSpace Technology vol 24 no 1 2014

[6] H Schunnesson HMwagalanyi andU Kumar ldquoShovel teethmonitoring and maintenance in open-pit mines hakanschunnesson hannington nwagalanyi and uday kumarrdquoCondition Monitoring and Diagnostic Engineering Manage-ment vol 14 no 1 pp 3ndash10 2011

[7] D Sembakutti A Ardian M Kumral and A P SasmitoldquoOptimizing replacement time for mining shovel teeth usingreliability analysis and Markov chain Monte Carlo simula-tionrdquo International Journal of Quality and Reliability Man-agement vol 35 no 6 2017

[8] C C Chang J G Yang L Qi and C P Chou ldquoOptimizationof heat treatment parameters with the taguchi method for thea7050 aluminum alloyrdquo Advanced Materials Researchvol 139ndash141 pp 157ndash162 2010

[9] X Luo and H Zhang ldquoMissing tooth detection with laserrange sensingrdquo in Proceedings of the World Congress on In-telligent Control and Automation Hangzhou China June2004

[10] Li He H Wang and H Zhang ldquoObject detection by partsusing appearance structural and shape featuresrdquo in Pro-ceedings of the International Conference on Mechatronics andAutomation Beijing China August 2011

[11] S N Lim J Soares and N Zhou ldquoTooth guard a visionsystem for detecting missing tooth in rope mine shovelrdquo inApplications of Computer Vision 2016

[12] ArsquoA Abdallah and H S Hassanein ldquoUtilizing sprouts WSNplatform for equipment detection and localization in harshenvironmentsrdquo in Proceedings of the 39th Annual IEEEConference on Local Computer Networks WorkshopsEdmonton Canada September 2014

[13] R Mcclanahan T L Briscoe and R A Van Raden ldquoWearassembly for an excavating bucketrdquo WO 2008051966 A32008

[14] H Farhat G E Sakr and R Kilany ldquoDeep learning appli-cations in pulmonary medical imaging recent updates andinsights on COVID-19rdquo Machine Vision and Applicationsvol 31 no 6 pp 53ndash42 2020

[15] A M Ozbayoglu M U Gudelek and O B Sezer ldquoDeeplearning for financial applications a surveyrdquo Applied SoftComputing vol 93 Article ID 106384 2020

[16] Z Deng B Wang Y Xu T Xu C Liu and Z Zhu ldquoMulti-scale convolutional neural network with time-cognition formulti-step short-term load forecastingrdquo IEEE Access vol 7pp 88058ndash88071 2019

[17] Z Deng C Liu and Z Zhu ldquoInter-hours rolling scheduling ofbehind-the-meter storage operating systems using electricityprice forecasting based on deep convolutional neural net-workrdquo International Journal of Electrical Power amp EnergySystems vol 125 Article ID 106499 2021

[18] P Wang P Chen Ye Yuan et al ldquoUnderstanding convo-lution for semantic segmentationrdquo in Proceedings of the 2018IEEE Winter Conference on Applications of Computer Vision(WACV) Lake Tahoe NV USA March 2018

[19] L W Chan and F Fallside ldquoAn adaptive training algorithmfor back propagation networksrdquo Computer Speech Languagevol 2 no 3-4 pp 205ndash218 1987

[20] R Shatnawi W Li J Swain and T Newman ldquoFindingsoftware metrics threshold values using roc curvesrdquo Journal ofSoftware Maintenance and Evolution Research and Practicevol 22 no 1 pp 1ndash16 2010

[21] T Fawcett ldquoAn introduction to roc analysisrdquo Pattern Rec-ognition Letters vol 27 no 8 pp 861ndash874 2005

Computational Intelligence and Neuroscience 13