Procedures for FOD Detection System Performance

AssessmentsRadar-Based and Dual Sensor

Systems

Beth WoodworthEdwin HerricksUniversity of Illinois

Introduction

• I will be talking today about two performance assessment efforts.

• The initial effort began in 2004 and was completed with assessment of an installed system on a runway at PVD in 2008.

• The second began in 2007 and was completed in 2009.

Sensors Evaluated

• Qinetiq TarsierTM

– Radar based system– Installed at PVD

• Xsight FODetectTM

– Hybrid system– Installed at BOS

Testing of QinetiQ TarsierTM at PVD• 94Ghz radar• Detects FOD at ranges

up to 1 km• 2 sensor units placed on

towers scanned Runway 5/23 at PVD

• Primary performance criterion was a standard -20dBm2 target at a range of 1km

Testing of FODetectTM at BOS

• 74GHz radar and video camera with image processing

• Sensors replaced edge lights• Partial installation at BOS• Sensors covered 550 ft on

Runway 15R• 10 surface detection units (SDUs)

– 5 SDUs on each side of the runway

• Each sensor covered approximately 75ft x 200ft

• Primary performance criterion detection of 0.8 in (2 cm) object

General Assessment Protocol

• Testing scheduled over approximately 12-month period to capture varied weather conditions– PVD – June 2007- March 2008– BOS – June 2008 - March 2009

• Targets selected to challenge each of the systems and included

• Different sizes• Different shapes• Different materials• Different hazard potentials

Detecting a Target• Radar

– Reflectivity – energy returned to sensor– “Complex Scatterers” – most FOD has a variety of

edges which have different likelihoods of being detected.

• Electro-Optical– Background– Item surface conditions

• Color, Flat/Glossy

– Illumination – • Lighting and shadow

Overall Goals of Testing• Calibration – Standard Target Testing

– Uses items of known detection characteristics– Provides information on system reliability and robustness

• Performance– Group of FOD items with different hazard potentials – Repeatedly placed at known locations in defined positions

• Blind– Random placement and actual and simulated FOD items– This test most represents typical detection needs at an

airport– Challenges system to detect a diverse and unpredictable

set of items.

General Methods• Safety –

– Because assessments were conducted at operational airports, safety considerations were paramount.

– Personnel were trained and briefed regularly by airport operations personnel.

– Although runways were closed, safety areas were identified and drills held so that all assessment personnel were aware of their environment.

– Rigorous accounting was made of all experimental FOD taken on the airport.

General Methods• FOD log – Over 1000 items placed and retrieved!

– all items were written in a log before proceeding to the runway– after return and often during the testing the log was checked again the

items in the box

• FOD box – items put in boxes with dividers in order to be easily and quickly counted

• Marking – Used for calibration and performance testing– Small, nearly invisible UV marks were placed on the runway– UV lights used to find markings– Locations were surveyed using a differential GPS accurate to

millimeters in the X/Y plane– Surveyed locations were compared with locations provided for each

detection by each technology

General Methods• Radar

– Does NOT depend on illumination– All assessment campaigns at PVD were performed

during normal 12 pm to 5 am runway closures• Electro-Optical

– Did depend on illumination. – Testing at BOS was on Runway 15R which saw

limited use allowing testing at different times.– Tests were performed at BOS during the day, at

night, and at dawn and dusk to capture a range of lighting conditions.

Calibration – Standard Target Testing• GOAL = confirm detection performance using objects with

defined detection characteristics• Targets selected based on technology

– Metal cylinders 3in X 4.5in 1.25in with a nominal reflectivity of 0dBm2 (Large)

– Metal cylinders 2in X 2.5in with a nominal reflectivity of -10dBm2 (Medium)

– Metal cylinders 1.5in x 1.25in with a nominal reflectivity of -20dBm2 (Small)

– Metal 2in spheres– Colored PVC cylinders 1.5in X 1.25in colored grey, white

and black

Metal cylinders 3in X 4.5in 1.25in

0dBm2 (Large)

Metal cylinders 2in X 2.5in

-10dBm2 (Medium)

Metal cylinders 1.5in x 1.25in

-20dBm2 (Small)

Metal spheres 2in Colored PVC cylinders 1.5in X 1.25in

Calibration Targets

Calibration- Standard Target Testing for TarsierTM

• 4 types of metal targets • 6 transects along full length

of the runway • Transects 2, 3, and 4 within

1km of both radars• Transects 1, 5, and 6 used

for system assessments

Transect

Distance from (m)

Distance from (ft)

Distance from S

Radar 2 (m)

Distance from S

Radar 2 (ft)

1 1675 5494 555 18212 1268 4160 220 7233 864 2834 356 11684 502 1647 708 23225 204 668 1140 37386 443 1454 1547 5075

Calibration for FODetectTM

• Metal cylinders 1.5in x 1.25in tested the radar portion of the sensor

• 3 colors of PVC cylinders (white, grey, black) and small metal cylinder

• Targets placed at varying distances from sensor in a test rectangle (approx. 150ft x 70ft)

• Targets placed to test the system and a single sensor

• Standard target array included 1 metal, 1 white, 1 grey, and 1 black cylinder

• Groups of targets oriented differently

Performance Testing• Items not selected based on sensor type as

with calibration testing• Used examples of common items found at

airports• Selection of items based on frequency of

occurrence and hazard potential• Calibration items also used during

performance testing• Items placed in know positions at specific

angles

Table of Standard FOD Items Used in Performance Assessments

FOD Itema Expected Hazard Frequency of Occurrence

1. Small Piece of Concrete High Common2. Standard Lug Nut From Service Vehicle

High Common

3. Roller Bearing High Common4. Chunk of Rubber Low Common5. Mechanics Wrench High Common6. Fuel Cap High Common7. Cotter Key Moderate Common8. Plastic Bottle/Bottle Cap Low Common9. Strapping Material Moderate Common10. Expansion Joint Material

Low Common

11. Construction Material–Galvanized Nails or Sheetrock Screws

Moderate Based on Construction Activity

12. Runway Infrastructure Part–Piece of Runway Light or Signage

High Uncommon

13. Small Fasteners Moderate Common14. Metal Strip High Uncommon15. Fiberglass Door Moderate Common16. Asphalt Chunk High Common

This table contains the items selected by CEAT

Performance Testing Methods at PVD• 6 arrays related to standard

target transects• In array 5x5 grid established,

approximately 25ft between items

• 23 targets total per array (no items set on runway lights)

• Position of items varied between campaigns (random selection of location achieved in test planning); items in the same position in all arrays in any single campaign.

• All items rotated 45 degrees after each scan to cover 8 cardinal points of the compass

• Line established midway between the centerline and the edge line

• 5 identical performance items placed at equal distances on this line across the length of test rectangle

• Items rotated so detections at 0, 45, and 90 degrees relative center line recorded

Performance Testing Methods at BOS

Blind Testing• Used actual FOD items

collected from runways• Range of sizes, materials,

colors• Over 100 items in collection• 30 items randomly selected

for each campaign• Placement locations chosen

at random from a grid• Items dropped or tossed so

the orientation was random• Up to 10 items used at one

time.

Blind Testing Grid at PVD

Typical Grouping of FOD Blind Items

Blind Testing Grid of Locations at BOS

Inclement Weather Testing• Objective to evaluate sensor performance in

variable weather conditions• Opportunistic based on long term plan, but

scheduled adjusted to coincide with storm events.

• Access to the runways in snow was impossible so assessments performed just after snow emergency conditions

Inclement Weather Assessment - Rain

• PVD - TarsierTM radar system• October 2007 • Calibration targets were deployed as a rain squall moved

across the runway

• BOS – FODetectTM hybrid system• March 2008 • One light rain event at dusk

Inclement Weather Assessment - Snow

• PVD – TarsierTM radar system– Assessments after 2 snow events

• Jan. 24, 2008 – flurries but no accumulation• Feb. 12, 2008 – snow changing to sleet

– Time for placement and retrieval were limited due to needs to clear the runway

Inclement Weather Assessment - Snow

• BOS – FODetectTM hybrid system– Winter Storm Jan. 7-12 –

• Weather Conditions: – rain, freezing rain, freezing drizzle, mist, ice pellets, fog, snow– Totaling 2.1in of wet precipitation

• Testing took place on Jan 8, 9, and 13, 2009– Testing on Jan. 29 and 30 after snow events as well

Summary

• 2 systems – radar and hybrid• Assessments performed at PVD and BOS• Assessments took place in variable weather conditions• General protocol – calibration, performance, blind –

consistent for all FOD detection system types• Actual procedures were adapted to each senor and adjusted

for runway availability • Targets were selected based on

– the performance claims of manufacturers – related to sensor characteristics and performance

parameters