1

Automated Rivet Inspection System for Aging Aircrafts

Unsang Park, Lalita Udpa, George C. Stockman

Computer Science and Engineering

Michigan State University

2

Contents

Nondestructive Inspection (NDI)Magneto-optic Imager in NDIMotion-based Filtering (MBF)Real-time implementation of MBFAutomated rivet inspection systemRivet detectionRivet classificationResults and conclusionsFuture work

3

Nondestructive Inspection for Aircrafts

Detect subsurface defects

Crack

Rivet

Seam

4

Nondestructive Inspection for Aircrafts

Increase service life of airplane

Prevent disasters

Aloha Airlines B-737-200 lost part of its front fuselage during a flight in Hawaii, 1985

5

Magneto-optic Imager (MOI)

CCD Camera

Induction Sheet

MO Sensor

Sample

Bias Coil

Light SourcePolarizer Analyzer

Eddy current excitationMagneto-optic sensingImaging

6

Magneto-optic Imager (cont.)

Produce real-time analog images of inspected part

Images both surface breaking and subsurface

cracks

Easy to interpret with minimal training

Applicable both on conducting samples as well as

composites by tagging with ferromagnetic

particles

7

Sample MOI Images

Crack along seam Crack between two rivets,

Radial crack on a rivet

Corrosion dome

8

Drawback of MOI images

MOI image contains serpentine pattern

noises due to the magnetic domain walls

in magneto-optic sensor

Rivet

Signals due to domain walls

9

Motion-based Filtering (MBF)

Additive Frame Subtraction

Moving direction of objects

Moving direction of MOI

In-5 In-4 In-3 In-2 In-1 In

D1D2D3D4D5

Sn

),(),(),(

)},({),(1

yxIyxIyxD

yxDMAXyxS

nini

iwi

n

10

Motion-based Filtering (cont.)

Preprocessing RGB to Gray

Additive Frame Subtraction

Threshold

Median Filter

Stretch

Post processing

Input Image

Output Image

11

MB filtered images

FilteredOriginal

12

MB filtered images (cont.)

FilteredOriginal

13

Real-time implementation of MBF

Experimental setup for proof of concept

Record to VHS

Collects MOI image Data

Record to movie file

Play on a Video player

Play on a PC monitor

Frame grabber

Web cameranoise

14

Real-time implementation of MBF (cont.)

Data transfer rate4.6 Mbytes /sec ( 320240 pixels 16 bit 30 fps )

Data are down sampled as the input images are dropped while an image is processed

Diagram of real-time Motion-based Filtering

Sensing

Displaying

Imaging

RGB to Gray Subtract

ThresholdMedian FilteringStretch

Max

MOI Real-time MBF

Max

15

Real-time implementation of MBF (cont.)

Optimizing MBF algorithm in C++

RGB to Gray: 20 ~ 23 ms

Additive Frame Subtraction: 1~2 ms/image

Threshold: 1 ~ 2 ms

Median Filter: 200 ~ 250 ms

Stretch: 1 ~ 2 ms

Image capture: 20 ms

Output Image

16

Real-time implementation of MBF (cont.)

RGB to Gray conversion

Additive frame subtractionMAX(I1-I3,I2-I3) MAX(I1,I2) - I3

Median filter

Normal Table Lookup Table build time

320*240

16 bit image

20~23 ms 1~2 ms 5~6 ms (1M bytes)

5 by 5 7 by 7

MATLAB 50 ms 88 ms

Modified Quick Sort (C++) 200 ~ 250 ms 400 ~ 450 ms

Moving Median with Sorting (C++) 100 ~ 150 ms 150 ~ 200 ms

Moving Median with Histogram (C++)

20 ~ 25 ms 20 ~ 40 ms

17

Real-time implementation of MBF (cont.)

Execution time of MBF algorithm in C++

Before optimizationIntel 2GHz, C++

After optimizationIntel 2GHz, C++

Capture 20 ms 20 ms

RGB to gray 20 ~ 23 ms 1 ~ 2 ms

Subtraction (x10)Max (x10)Threshold

1 ~ 2 ms (x10)1 ~ 2 ms (x10)

1~2 ms

1 ~ 2 ms (x10)1 ~ 2 ms

Median Filter (3x3) 200 ~ 250 ms 20 ~ 30 ms

Stretch 1 ~ 2 ms 1 ~ 2 ms

Total 262 ~ 337 ms 53 ~ 76 ms

18

Drawbacks of current MOI inspection

No measure for quantitative interpretation Data interpretation is subjective

Manual inspection by human operator (more than 10 hours per airplane)

Expensive labor cost Error due to fatigue

19

Automated MOI inspection system

PRI Research and Development Corporation (PRI) Developing and improving magneto-optic imager (MOI)

Michigan State University, ECE department Image processing algorithm for filtering and classification

Boeing Phantom Works Self-guided, suction cup robot – crawls over airplane skin

20

Automated MOI inspection (cont.)

Currently focusing on radial cracks on rivets

Quantification of defects in MOI imagesImplementing real-time rivet inspection algorithm

Motion-based Filtering

Rivet detection

Rivet classification

21

Rivet detection

Hough transformation-based method Circular Hough transformation

Morphological operation-based method

Obtain center and radius, c1, r1

Erode rivet with a circle of radius r1

Segment out each rivet

Area(rivet) = 0

Obtain center and radius, c2, r2

yes, r1 r1-1

no

22

Rivet detection (cont.)

Original

Hough transformation

Morphological operation

23

Rivet classification

Two-pass Hough transformation1st pass – Rivet detection

2nd pass – Blob detection

Original image Filtered image - After 1st pass

After 2nd Pass

good bad

24

Rivet classification (cont.)

Bayesian classifier Feature selection

df

rcycxd yx

max

)()(

1

22

Hough transformation Morphological op.Original image

25

Off-line test

Training 10 normal, 10 defective rivet images

Obtain mean and variance of feature f1

Testing 222 rivet images including 66 defective rivet images

Two-pass Hough Morph. - BayesHough - Bayes

26

Experimental Results

Accuracies of three algorithms

Inspection

algorithm

Rivet detection

Rivet classificationFalse negative False positive

Two-pass Hough 1/242 0/242 90% (200/222)

Hough-Bayes 1/242 0/242 96% (214/222)

Morph.-Bayes 1/242 1/242 99% (220/222)

27

Conclusions

MB filtered image is optimal in image processing for automated rivet inspectionMorphological operation-based rivet detection is superior to Hough-based rivet detection both for execution time and accuracyBayesian classifier is superior to Hough-based classifierRadial crack detection on rivets showed 99% accuracy in off-line test

28

Future work

Implement MBF and rivet inspection algorithms on the Digital Signal Processing (DSP) board

Improve robustness of the algorithms with the feedback from field test

Develop MOI inspection algorithms for other types of defects in aircrafts