View
214
Download
2
Category
Tags:
Preview:
Citation preview
Procedures for FOD Detection System Performance
AssessmentsRadar-Based and Dual Sensor
Systems
Beth WoodworthEdwin HerricksUniversity of Illinois
Introduction
• I will be talking today about two performance assessment efforts.
• The initial effort began in 2004 and was completed with assessment of an installed system on a runway at PVD in 2008.
• The second began in 2007 and was completed in 2009.
Sensors Evaluated
• Qinetiq TarsierTM
– Radar based system– Installed at PVD
• Xsight FODetectTM
– Hybrid system– Installed at BOS
Testing of QinetiQ TarsierTM at PVD• 94Ghz radar• Detects FOD at ranges
up to 1 km• 2 sensor units placed on
towers scanned Runway 5/23 at PVD
• Primary performance criterion was a standard -20dBm2 target at a range of 1km
Testing of FODetectTM at BOS
• 74GHz radar and video camera with image processing
• Sensors replaced edge lights• Partial installation at BOS• Sensors covered 550 ft on
Runway 15R• 10 surface detection units (SDUs)
– 5 SDUs on each side of the runway
• Each sensor covered approximately 75ft x 200ft
• Primary performance criterion detection of 0.8 in (2 cm) object
General Assessment Protocol
• Testing scheduled over approximately 12-month period to capture varied weather conditions– PVD – June 2007- March 2008– BOS – June 2008 - March 2009
• Targets selected to challenge each of the systems and included
• Different sizes• Different shapes• Different materials• Different hazard potentials
Detecting a Target• Radar
– Reflectivity – energy returned to sensor– “Complex Scatterers” – most FOD has a variety of
edges which have different likelihoods of being detected.
• Electro-Optical– Background– Item surface conditions
• Color, Flat/Glossy
– Illumination – • Lighting and shadow
Overall Goals of Testing• Calibration – Standard Target Testing
– Uses items of known detection characteristics– Provides information on system reliability and robustness
• Performance– Group of FOD items with different hazard potentials – Repeatedly placed at known locations in defined positions
• Blind– Random placement and actual and simulated FOD items– This test most represents typical detection needs at an
airport– Challenges system to detect a diverse and unpredictable
set of items.
General Methods• Safety –
– Because assessments were conducted at operational airports, safety considerations were paramount.
– Personnel were trained and briefed regularly by airport operations personnel.
– Although runways were closed, safety areas were identified and drills held so that all assessment personnel were aware of their environment.
– Rigorous accounting was made of all experimental FOD taken on the airport.
General Methods• FOD log – Over 1000 items placed and retrieved!
– all items were written in a log before proceeding to the runway– after return and often during the testing the log was checked again the
items in the box
• FOD box – items put in boxes with dividers in order to be easily and quickly counted
• Marking – Used for calibration and performance testing– Small, nearly invisible UV marks were placed on the runway– UV lights used to find markings– Locations were surveyed using a differential GPS accurate to
millimeters in the X/Y plane– Surveyed locations were compared with locations provided for each
detection by each technology
General Methods• Radar
– Does NOT depend on illumination– All assessment campaigns at PVD were performed
during normal 12 pm to 5 am runway closures• Electro-Optical
– Did depend on illumination. – Testing at BOS was on Runway 15R which saw
limited use allowing testing at different times.– Tests were performed at BOS during the day, at
night, and at dawn and dusk to capture a range of lighting conditions.
Calibration – Standard Target Testing• GOAL = confirm detection performance using objects with
defined detection characteristics• Targets selected based on technology
– Metal cylinders 3in X 4.5in 1.25in with a nominal reflectivity of 0dBm2 (Large)
– Metal cylinders 2in X 2.5in with a nominal reflectivity of -10dBm2 (Medium)
– Metal cylinders 1.5in x 1.25in with a nominal reflectivity of -20dBm2 (Small)
– Metal 2in spheres– Colored PVC cylinders 1.5in X 1.25in colored grey, white
and black
Metal cylinders 3in X 4.5in 1.25in
0dBm2 (Large)
Metal cylinders 2in X 2.5in
-10dBm2 (Medium)
Metal cylinders 1.5in x 1.25in
-20dBm2 (Small)
Metal spheres 2in Colored PVC cylinders 1.5in X 1.25in
Calibration Targets
Calibration- Standard Target Testing for TarsierTM
• 4 types of metal targets • 6 transects along full length
of the runway • Transects 2, 3, and 4 within
1km of both radars• Transects 1, 5, and 6 used
for system assessments
Transect
Distance from (m)
Distance from (ft)
Distance from S
Radar 2 (m)
Distance from S
Radar 2 (ft)
1 1675 5494 555 18212 1268 4160 220 7233 864 2834 356 11684 502 1647 708 23225 204 668 1140 37386 443 1454 1547 5075
Calibration for FODetectTM
• Metal cylinders 1.5in x 1.25in tested the radar portion of the sensor
• 3 colors of PVC cylinders (white, grey, black) and small metal cylinder
• Targets placed at varying distances from sensor in a test rectangle (approx. 150ft x 70ft)
• Targets placed to test the system and a single sensor
• Standard target array included 1 metal, 1 white, 1 grey, and 1 black cylinder
• Groups of targets oriented differently
Performance Testing• Items not selected based on sensor type as
with calibration testing• Used examples of common items found at
airports• Selection of items based on frequency of
occurrence and hazard potential• Calibration items also used during
performance testing• Items placed in know positions at specific
angles
Table of Standard FOD Items Used in Performance Assessments
FOD Itema Expected Hazard Frequency of Occurrence
1. Small Piece of Concrete High Common2. Standard Lug Nut From Service Vehicle
High Common
3. Roller Bearing High Common4. Chunk of Rubber Low Common5. Mechanics Wrench High Common6. Fuel Cap High Common7. Cotter Key Moderate Common8. Plastic Bottle/Bottle Cap Low Common9. Strapping Material Moderate Common10. Expansion Joint Material
Low Common
11. Construction Material–Galvanized Nails or Sheetrock Screws
Moderate Based on Construction Activity
12. Runway Infrastructure Part–Piece of Runway Light or Signage
High Uncommon
13. Small Fasteners Moderate Common14. Metal Strip High Uncommon15. Fiberglass Door Moderate Common16. Asphalt Chunk High Common
This table contains the items selected by CEAT
Performance Testing Methods at PVD• 6 arrays related to standard
target transects• In array 5x5 grid established,
approximately 25ft between items
• 23 targets total per array (no items set on runway lights)
• Position of items varied between campaigns (random selection of location achieved in test planning); items in the same position in all arrays in any single campaign.
• All items rotated 45 degrees after each scan to cover 8 cardinal points of the compass
• Line established midway between the centerline and the edge line
• 5 identical performance items placed at equal distances on this line across the length of test rectangle
• Items rotated so detections at 0, 45, and 90 degrees relative center line recorded
Performance Testing Methods at BOS
Blind Testing• Used actual FOD items
collected from runways• Range of sizes, materials,
colors• Over 100 items in collection• 30 items randomly selected
for each campaign• Placement locations chosen
at random from a grid• Items dropped or tossed so
the orientation was random• Up to 10 items used at one
time.
Blind Testing Grid at PVD
Typical Grouping of FOD Blind Items
Blind Testing Grid of Locations at BOS
Inclement Weather Testing• Objective to evaluate sensor performance in
variable weather conditions• Opportunistic based on long term plan, but
scheduled adjusted to coincide with storm events.
• Access to the runways in snow was impossible so assessments performed just after snow emergency conditions
Inclement Weather Assessment - Rain
• PVD - TarsierTM radar system• October 2007 • Calibration targets were deployed as a rain squall moved
across the runway
• BOS – FODetectTM hybrid system• March 2008 • One light rain event at dusk
Inclement Weather Assessment - Snow
• PVD – TarsierTM radar system– Assessments after 2 snow events
• Jan. 24, 2008 – flurries but no accumulation• Feb. 12, 2008 – snow changing to sleet
– Time for placement and retrieval were limited due to needs to clear the runway
Inclement Weather Assessment - Snow
• BOS – FODetectTM hybrid system– Winter Storm Jan. 7-12 –
• Weather Conditions: – rain, freezing rain, freezing drizzle, mist, ice pellets, fog, snow– Totaling 2.1in of wet precipitation
• Testing took place on Jan 8, 9, and 13, 2009– Testing on Jan. 29 and 30 after snow events as well
Summary
• 2 systems – radar and hybrid• Assessments performed at PVD and BOS• Assessments took place in variable weather conditions• General protocol – calibration, performance, blind –
consistent for all FOD detection system types• Actual procedures were adapted to each senor and adjusted
for runway availability • Targets were selected based on
– the performance claims of manufacturers – related to sensor characteristics and performance
parameters
Recommended