3D Imaging and Data Fusion in Robotics · 2004-09-16 · • Image Fusion on Dual Energy X-Ray Images • Entropy-Based Multilevel Segmentation • Computer Simulation of Luggage

Electrical & Computer Engineering Department

The University of Tennessee, Knoxville334 Ferris Hall, Knoxville, TN, 37996-2100, USA 1

Imaging,Robotics, &IntelligentSystemsLaboratory

Imaging,Robotics, &IntelligentSystemsLaboratory

DOE-URPR Project

3D Imaging and Data Fusion in Robotics

Mongi Abidi

Professor

Director of IRIS

Associate Department Head

Electrical and Computer Engineering

University of Tennessee, USA

http://imaging.utk.edu



UTENN - IRIS Laboratory

Prof. M. A. Abidi(Group Leader)

Prof. A. KoschanProf. J. PaikDr. A. GribokDr. B. R. Abidi

Dr. D. Page

PARTICIPANTS

$21,000,000+ since 87

Sponsors: DOE, DOD, NSF, TACOM, NASA, TVA, L-Martin, TI, Radian, Eastman Chemical

Company, National Automotive Center

FUNDING

• 30,000 sq. feet of Lab space• Workstations: Sun, SGI, PC

•Sensors: Vision, Range, etc...

FACILITIES

Faculty: 70+ man-years of Exper.in Image Processing & Robotics

Staff of 30 (17 Ph.D./MS/UG Students)

300 Journal/Conference Papers

8 Textbooks/Reference Volumes

ACCOMPLISHMENTS

UT- IRISLab

UT- IRISLab



Robot System Communication

and control

High-level Reasoning

Trajectory Planning

Motion Control

End-effector Control

Sensor Activation

Data Acquisition

Vision

Range

Sound

Proximity

Force/Torque

Touch

Interface Functions for Autonomous Robotic Inspection and Manipulation



IRIS Academic Program

Senior Level: 405 Digital Signal Processing and Filter Design451 Microprocessors in Computer Engineering471 Introduction to Pattern Recognition472 Introduction to Digital Image Processing494 Special Topic in Image Processing and Robotics

Master’s Level: 503 Modern Transform Methods504 Random Process Theory for Engineers505 Digital Signal Processing I506 Digital Signal Processing II571 Pattern Recognition572 Digital Image Processing573 Vision Sensing for Robotics & Automation I574 Vision Sensing for Robotics & Automation II

Ph.D. Level: 671 Image Processing and Robotics I672 Image Processing and Robotics II673 Image Processing and Robotics III

Senior Level: 405 Digital Signal Processing and Filter Design451 Microprocessors in Computer Engineering471 Introduction to Pattern Recognition472 Introduction to Digital Image Processing494 Special Topic in Image Processing and Robotics

Master’s Level: 503 Modern Transform Methods504 Random Process Theory for Engineers505 Digital Signal Processing I506 Digital Signal Processing II571 Pattern Recognition572 Digital Image Processing573 Vision Sensing for Robotics & Automation I574 Vision Sensing for Robotics & Automation II

Ph.D. Level: 671 Image Processing and Robotics I672 Image Processing and Robotics II673 Image Processing and Robotics III



Sensory Data Acquisition

IRIS Range Sensing Platform: Perceptron-based 3D Laser finder

High-resolution Color Camera

(Kodak 3000x2000 pls)

Point/Line Scanning Laser Range System

assembled at IRIS lab.



From IMAGES to 3D MODELS

3D models containing information such as visual, geometrical, thermal, radiometric, etc.

INPUT OUTPUTRange imagesDigital elevation maps 3D models



Functional Diagram of Data Fusion SystemFeaturesSensors

TransformationFunctions

FusionFunctional

f1

f2

fi

fn

T11

T21

Ti1

Tn1

F[Tij(fk)]

FusionTechniques

• Heuristic

• Logic OR/AND• Averaging• Least Square• Geometric• Regularization• Bayesian• Possibility (Fuzzy)

T11(f1)

T21(f2)

Ti1(fi)

Tn1(fn)

FUSION



Core Technologies at the IRIS Lab.

3 - DATA VISUALIZATION

• Visualizing Reality

• Hardware, Object, View

Dependent Display

2 - SCENE DESCRIPTION

• Scene Segmentation• Multi - resolution Data Analysis• Data Reduction• Data Enhancement• Object Modeling

1 - SCENE BUILDING

• Data Acquisition and Sensor Characterization

• Sensor Placement• Data Registration• Data Fusion

Multi-Modal Model

Multi-Resolution,Multi-Modal,

Object Based Model

Multi-Modal Model



Univ. of Michigan

Univ. of Alaska

Clemson Univ.

Univ. of Iowa

Other Gov. Agencies: DOD (TACOM, Army NVL), DOE, FAA, etc.

ARMY/TACOM - UTENN-IRIS

Thrust Area 1: Intelligent Vehicle Dynamics & Control

UT IRISL A B O R A T O R Y

Thrust Area 3: High Performance Structures & Materials

Thrust Area 5: Integrated System Design & Simulation

Oakland Univ.

Wayne State Univ.

Univ. of Wisconsin

IndustryPartners

Thrust Area 2: Synthetic & Virtual Environments

Thrust Area 4: Advanced & Hybrid Powertrains

http://arc.engin.umich.edu/



Tasks

Flow diagram illustrating the five primary tasks that comprise the proposed research.

Flow diagram illustrating the five primary tasks that comprise the proposed research.

Platform Outfitting

Data Acquisition

Data ProcessingAnd Registration

Data FusionAnd Integration

Data Reduction

1

2

3

4

5Complete Scene Model Suitable For NADS

Preliminary LRI Database(no texture or attributes)

Dense Scene Database(too complex for real time)

http://www.nads-sc.uiowa.edu/



Multi-Sensor UTENN-IRIS Mobile Mapper

Vehicle

Control Room

Sensors

Communications and Sensors

Mast



Full 3D model recovered by integrating 4 views: 624,265 triangles

Riegl Laser Range Scanner:

• 9100x1100 image size (Max)•1-2 in standard deviation•Range, RGB color, Intensity

33--D Reconstruction from Real Range DataD Reconstruction from Real Range Data

• 70 (50 US + 20 Overseas) Faculty, Staff, and Students

• $3.5M of Externally Funded Research Per Year

• 40,000 square feet of Office and Laboratory Space

• Research AreasWarfighter Security & Physical SecurityNuclear SecurityAirport SecurityHarbor Security

• Major Funding AgenciesUS ArmyNational Nuclear Security AdministrationTSA – Department of Homeland SecurityOffice of Naval Research

ID CARD

1

UTK IRIS: Imaging, Robotics, and Intelligent Systems

2

3

4

Warfighter Security / Physical SecurityUS Army & Office of Naval Research

Nuclear SecurityNational Nuclear Security Administration

Airport SecurityTSA – Department of Homeland Security

Harbor SecurityOffice of Naval Research

IRIS Research Themes

5

• 3D Reconstruction Based on Stereo Images• Modeling Objects and Scenes from Multi-modal Imagery• Creating Wide-View Representations of Real-World Environments• Multiresolution Analysis of Irregular Meshes with Attributes• Reverse Engineering of Mechanical Parts• 3D Object Reconstruction Using Boolean Operations on Supershapes• Segmentation Based on Perceptive Metrics• Digitization of Real World Environments Using Vehicle-borne Scanning• Image Acquisition and 3D Modeling on Microscopic Scales

• Weapon Detection System for Under Vehicle Inspection• A Module-Based Generic Architecture for Mobile Robots• A Full-Information Sampling Strategy• Hardware/Software Development of 1D Laser Range Scanner• Simulation and Segmentation of Thermal IR Images• Informational Approach to Shape Description• Fusion of Visual and Thermal Face Recognition• A Combinatorial Face Detection Technique using Color• Co-registration of Thermal and Visual Image Pairs• 3D Face Recognition

Warfighter Security / Physical Security

On-Going Research Projects

Harbor SecurityAirport Security

Nuclear Security

• Image Fusion on Dual Energy X-Ray Images• Entropy-Based Multilevel Segmentation• Computer Simulation of Luggage• PTZ Camera Driven from Wide-Angle Camera• Multi-Object Motion Pattern Classification for Surveillance• Automatic Target Acquisition/Tracking Using Color and Shape• Illumination Invariant Video Tracking and Surveillance

• Data Acquisition and Transfer for Remote Video Surveillance• Hybrid 3D Modeling of Large Landscapes from Satellite Maps• Sensor Planning for Surveillance and Modeling• Multi Perspective Mosaicking of Harbor Vessels• Wireless Video and Control with PTZ Camera• Implementation of 3D Maps on Internet with Customizable Interface• Multiple Sensor Synchronization • Radar Ranging and Tracking

6

• Developing Shape Correlation methods for registering multiple rangeviews.

• Automatic alignment of color and rangdata.

• Rendering andmanipulation otextured mode

• Developing Shape Correlation methods for registering multiple rangeviews.

• Automatic alignment of color and rangdata.

• Rendering andmanipulation otextured mode

ans multi-modal dainput.

• Efficient 3-D surface model building, texturmapping and rendering.

multi-modal dainput.

• Efficient 3-D surface model building, texturmapping and rendering.

3D Scans of partsAligne

Registration

Path Determination from Multiple

Sensors

Path Determination from Multiple

Sensors

Geometry and Texture

Information

Geometry and Texture

Information

ctcoet

•SlafoAgstFl(FDtiocoet

Detection for Against Flow atExit Lane Simulation

intruder is calculated tocontrol PTZ camera for capturing froimages

• Real-time target initialization for color tracking

i

dy

dx

iy

thch

),( ji

ix

j

CamerahandoverCamera

handoverTarge

initializaTarge

initializa

Tracking uTracking u

resulting from different

lighting and orientation of the object.

e images and also the original color

AccomplishmentsSurvey of various color constancy an

llumination invariant tracking algorithmIllumination invariant Object

R iti i H hi Filt i

Example of a Macbeths chart under different lighting conditions

images.

Performance

visualized con• Application---

setup for humface scanning

minal-based control window

- from stereo images

video

Reconstructed Face

LASER SCANNED VIDEO INPUT

CALIBRATION PARAMETERS

LSC

VI

CAMERA CALIBRATION

3D Face Recognition System Diagram

Detectability

1

2

3

4

5

6

7

8

9

Ave

rage

Ran

k

Graphical User Interface for threat detection

Phase correlation and image segmentation

Results from Normal Vector Voting algorithshowing crease detection for a disc brake

ReconstructedModel

Positive Curvature

rface Registrati

● Real tivisualizon large scale models

● Simpliftion

● Frequencycontent assessme● Feature

detectiN i

solution levels

vel 3 Level 4

Initial noisy scanned mod

ange imageAccomplishments

•Manage real data ty(range images) creafrom scanners•Constructapproximating anddetail surfaces at ealevel of representatwith polynomial surfaces

Accomplishments•Manage real data ty(range images) creafrom scanners•Constructapproximating anddetail surfaces at ealevel of representatiwith polynomial surfaces

l

surface

at

level

Approximating surface

Control mesh

Detail coefficients

Image defines a control mesh to

The distance to the surface is

Wenewconpothadef

Input 3-D triangulated model

A distributor cap component

4 stroke engine

tic converterRepreswith m

e software to software to

inspectio• Develop

software interfacefor rangedata processinfor bettevisualizaon and hilevel processin

under vehscenes

Examplesrange da

nder vehicle 3D ange data cquisition & rocessing

eluction

Textured 3DModel

gnionh

he cart

Internet to transmit time video streams under remote PTZ controls

• Power and weathermanagement: Deveand deploy portableweather-hardened modules to carry allsystems component

Client1

ClientN

INTERNET

eless Surveillance System Architecture Rapid Prototype at IRIS Lab in

slice 4 slice 10 slice 21 slice 24 slice 41 slice 45 slice 50

A stack of 2images

results frothe one sc

The part is scanned under the microscope

The stack is processed to obtain a range image

Documents

3D Imaging and Data Fusion in Robotics · 2004-09-16 · • Image Fusion on Dual Energy X-Ray Images • Entropy-Based Multilevel Segmentation • Computer Simulation of Luggage