Overview of Research Thrust R3Image and Data Information Management

Overview of Research Thrust R3Image and Data Information Management

Diverse Problems – Similar SolutionsDiverse Problems – Similar Solutions

James ModestinoRPI

Dave KaeliNU

James ModestinoRPI

Dave KaeliNU

R3 Fundamental Research Topics R3a. Data Compression

R3b. Multimedia Networking

R3c. Scalable Computing

R3d. Image Databases

R3 Fundamental Research Topics R3a. Data Compression

R3b. Multimedia Networking

R3c. Scalable Computing

R3d. Image Databases

! Acquisition, transmission, storage, processing, retrieval, dissemination, display and visualization of sensor data

! Development of a distributed collaborative research environment! Provide remote Web access to CenSSIS resources

ScopeScope

SubsurfaceSensing and

Modeling

Validation Testbeds

Environmental-CivilApplications

Bio-MedicalApplications

Image and DataInformation

Management

Physics-BasedSignal Processing andImage Understanding

I–PLUS

R1R1R2R2

R3R3L1L1

L2L2

L3L3

Context and Scope of R3 ResearchContext and Scope of R3 Research

EngineeredSystemEngineeredSystem

EnablingTechnologiesEnablingTechnologies

FundamentalScienceFundamentalScience

Barrier 5Barrier 5

CenSSIS Barriers Addressed in R3CenSSIS Barriers Addressed in R3

Lack of Robust, Physics-Based Recognition and Sensor Fusion TechniquesLack of Robust, Physics-Based Recognition and Sensor Fusion Techniques

Barrier 6Barrier 6 Lack of Rapid Processing and Management of Large Image DatabasesLack of Rapid Processing and Management of Large Image Databases

Lack of Optimal End-to-End Sensor DesignMethodsLack of Optimal End-to-End Sensor DesignMethods

Barrier 7Barrier 7 Lack of Validated, Integrated Processing andComputational ToolsLack of Validated, Integrated Processing andComputational Tools

Barrier 3Barrier 3

Focussed on Attacking Unique CenSSISRequirements for Data Processing and Management

Focussed on Attacking Unique CenSSISRequirements for Data Processing and Management

Goal: Develop information management concepts, tools and infrastructure to support basic CenSSIS scientific mission

Goal: Develop information management concepts, tools and infrastructure to support basic CenSSIS scientific mission

! Distribution

! Transport

! Collaboration

! Web Access

! Distribution

! Transport

! Collaboration

! Web Access

! Representation

! Compression

! Transmission

! Storage

! Representation

! Compression

! Transmission

! Storage

! Computing

! Storage

! Scalability

! Solutionware

! Computing

! Storage

! Scalability

! Solutionware

! Archiving

! Image Formatting

! Image Browsing

! Data Migration

! Archiving

! Image Formatting

! Image Browsing

! Data Migration

Components of R3 Research ThrustComponents of R3 Research Thrust

R3Image and Data

InformationManagement

R3Image and Data

InformationManagement

R3cScalable

Computation

R3cScalable

Computation

R3bMultimedia Networks

R3bMultimedia Networks

R3aData

Compression

R3aData

Compression

R3dImage

Databases

R3dImage

Databases

R3a/bData Compression/

Multimedia Networks

R3a/bData Compression/

Multimedia Networks

R3c/d Scalable Computation/

Image Databases

R3c/d Scalable Computation/

Image Databases

James ModestinoLeader, RPI

Shiv Kalyanaraman, RPIMasoud Salehi, NUJohn Woods, RPIRichard Radke*, RPI

*New Hire-Cost Match

David KaeliLeader, NU

Steve Lerner, WHOIMiriam Leeser, NUWaleed Meleis, NUEric Miller, NU

The R3 Research TeamThe R3 Research Team

Motivation for R3a Data Compression ResearchMotivation for R3a Data Compression Research

" CenSSIS applications will result in massivelylarge data sets.

" Will stress existing processing/storage transmission capabilities.

" Flexible/efficient multiresolution compressionschemes are required.

" General-purpose pixel-based techniques will beinadequate.

" Object-oriented, application-specific approachespromise 1-2 orders of magnitude improvement.

" Reasonable performance/complexity will require exploitation of underlying physics.

Remote HyperspectralSensing of Coastal Regions

UPRM Data Set

Remote HyperspectralSensing of Coastal Regions

UPRM Data Set

MultipleWavelengths

MultipleWavelengths

MultipleImagesMultipleImages

SensorData

SensorData

(MSD)

Wide BandProbe

Narrow BandDetectors

backgroundagua1agua-2agua4basuracultivo1cultivo2cultivo-recortadomontanascultivo4cutlivo-costarenazona-urbanaagua-5cultivo6agua6techo-almacencultivo-corto-Nagua-costa-Nvegetacionrio

Number ofImages/

Wavelength103

Number ofImages/

Wavelength103

Image SizeIn Pixels/View

1K x 1K

Image SizeIn Pixels/View

1K x 1K

Number ofBits/Pixels

16

Number ofBits/Pixels

16

Total SizeOf Data Set20 TBytes

Total SizeOf Data Set20 TBytes

Number ofWavelengths

103

Number ofWavelengths

103

! Transmission Time 56 hr @ 100 Mb/s! Compression Reduces to < 30 min! Utilize 3-D Volumetric Compression! Exploit Correlation Across Wavelengths! Layered Progressive Scheme Desirable

! Transmission Time 56 hr @ 100 Mb/s! Compression Reduces to < 30 min! Utilize 3-D Volumetric Compression! Exploit Correlation Across Wavelengths! Layered Progressive Scheme Desirable

Typical Multi-Spectral Discrimination (MSD) ApplicationTypical Multi-Spectral Discrimination (MSD) Application

Airborne Visible Infrared Imaging Spectrometer (AVIRIS) ApplicationAirborne Visible Infrared Imaging Spectrometer (AVIRIS) Application

! AVIRIS-Visible IR Imager! Multiple Bands

! 224 Bands! 380nm-2500nm

! Individual Images! 397 x 268 pixels! 16 bits/pixel

! Reasonably Large Data Sets! 47.6 Mbytes/SetRemote Hyperspectral

Sensing of Coastal Regions

UPRM Data Set (Band 110)

Multiresolution Approach Based on SPIHTMultiresolution Approach Based on SPIHT

" Multiresolution Wavelet Transform." Exploits Spatial Correlation Properties

of Wavelet Transform." Partitions Data into Spatial Orientation

Tree Sets. " Efficient Strategy for Encoding the

Sorting Information." Efficient Coding of Wavelet Subbands." Embedded Code Providing Lossy-

Lossless Progressive Coding/Transport. " Current Benchmark for Efficient, Low

Complexity Image Encoder." Extensible to 3-D MSD Data Sets Using

3-D Spatial Orientation Trees.2-D SPIHT Frame

Dyadic 3-LevelWavelet SubbandDecomposition

Dyadic 3-LevelWavelet SubbandDecomposition

Comparison of 3-D SPIHT vs. JPEG for AVIRIS ImagesComparison of 3-D SPIHT vs. JPEG for AVIRIS Images

JPEG:0.15 bpp, 24.5 dB

3-D SPIHT:0.15 bpp,29.5 dB

0.5 bpp, 28.2 dB

0.5bpp, 33.4 dB

1.0 bpp, 30.2 dB

1.0 bpp, 35.5 dB

Comparison of 3-D SPIHT vs. JPEG for AVIRIS ImagesComparison of 3-D SPIHT vs. JPEG for AVIRIS Images

JPEG:0.15 bpp, 24.5 dB

3-D SPIHT:0.15 bpp,29.5 dB

0.5 bpp, 28.2 dB

0.5bpp, 33.4 dB

1.0 bpp, 30.2 dB

1.0 bpp, 35.5 dB

Comparison of 3-D SPIHT vs. JPEG for AVIRIS ImagesComparison of 3-D SPIHT vs. JPEG for AVIRIS Images

JPEG:0.15 bpp, 24.5 dB

3-D SPIHT:0.15 bpp,29.5 dB

0.5 bpp, 28.2 dB

0.5bpp, 33.4 dB

1.0 bpp, 30.2 dB

1.0 bpp, 35.5 dB

Comparison of 3-D SPIHT vs. JPEG for AVIRIS ImagesComparison of 3-D SPIHT vs. JPEG for AVIRIS Images

JPEG:0.15 bpp, 24.5 dB

3-D SPIHT:0.15 bpp,29.5 dB

0.5 bpp, 28.2 dB

0.5bpp, 33.4 dB

1.0 bpp, 30.2 dB

1.0 bpp, 35.5 dB

3-D SPIHT vs. JPEG for AVIRIS Images3-D SPIHT vs. JPEG for AVIRIS Images

! Efficient Use of Interband! Information ! Superior R-D Performance

! Approx. 5 dB PSNR! At Least 5x Compress.

! Subjectively Better! No Block Artifacts! Lower Complexity! Embedded/Progressive! Coding! Acquiring and Testing! Other MSD Data Sets! Investigate Other 3-D! Transforms! Use JPEG2000 as Baseline

Good Initial Step in Developing EfficientCompression Schemes for MSD Applications

Good Initial Step in Developing EfficientCompression Schemes for MSD Applications

Focused orPulsed Probe

Focused orGated Detector

(LPM) (MSD)

Wide BandProbe

Narrow BandDetectors

Photomosaic of the Curved Human Retina

RPI Data Set

Photomosaic of the Curved Human Retina

RPI Data Set

MultipleOverlapping

MultipleOverlapping

Temporal Sequence of

Multi-SpectralImages

Temporal Sequence of

Multi-SpectralImages Sensor

DataSensor

Data

A: Retinal Surface (visible channel)B: Subsurface (near infrared)

A: Retinal Surface (visible channel)B: Subsurface (near infrared)

! Real-Time Transmission Requires 720Mb/s ! Compression Reduces to < 6 Mb/s! Transmission Time 6.6 hr @ 100 Mb/s! Utilize Mosaic-Aided Coding Approach ! Physics-Motivated Region-Based Coding! Layered Transport Scheme Desirable

! Real-Time Transmission Requires 720Mb/s ! Compression Reduces to < 6 Mb/s! Transmission Time 6.6 hr @ 100 Mb/s! Utilize Mosaic-Aided Coding Approach ! Physics-Motivated Region-Based Coding! Layered Transport Scheme Desirable

Image SizeIn Pixels1K x 1K

Image SizeIn Pixels1K x 1K

Number ofBits/Pixels

12

Number ofBits/Pixels

12

Total SizeOf Data Set

300-600 GBytes

Total SizeOf Data Set

300-600 GBytes

Length of Procedure

1-2 hrs

Length of Procedure

1-2 hrs

Number ofChannels

2

Number ofChannels

2Frame Rate

30/secFrame Rate

30/sec

Typical LPM/MSD ApplicationTypical LPM/MSD Application

Visible LightSequence

Near IRSequence

Each Frame ofBoth Sequences

Aligned/RegisteredWith Mosaic andWith Each Other

Registration ProcessCan be Used to

Derive Motion Vectorsfor MC PredictionOnly Prediction

Residuals Encoded

Physics-Based Mosaic-Aided Video CodingPhysics-Based Mosaic-Aided Video Coding

Rapid Frame-Frame Motion

PSNR COMPRESSION RATIOS (dB) New Method Baseline

Original Image

52.6 13:1 4:1

47.5 26:1 8:1

42.9 106:1 32:1

38.5 1580:1 128:1

34.00 11000:1 512:1

Decoded Image by New Method after

1580:1 Compression

Decoded Image by New Method after

106:1 Compression

Decoded Image by New Method after

11000:1 Compression

Compression Results for Equal PSNRCompression Results for Equal PSNR

Other R3a Compression WorkOther R3a Compression Work

! Object-Based Compression Schemes! Identify Important Objects in Image! Differential ROI Coding Relative to Background! Matched to Recognition-Based Processing

! Joint Source-Channel Coding (JSCC) Image/VideoTransport Approaches! Joint Design of Source and Channel Coders! Optimum Source/Channel Coding Tradeoffs! Efficient End-to-End Transport of Images and Video

Results Reported in a Number of Reports,Papers and Poster Presentations

Results Reported in a Number of Reports,Papers and Poster Presentations

R3b Multimedia NetworkingR3b Multimedia Networking

CenSSISCenSSIS

backgroundagua1agua-2agua4basuracultivo1cultivo2cultivo-recortadomontanascultivo4cutlivo-costarenazona-urbanaagua-5cultivo6agua6techo-almacencultivo-corto-Nagua-costa-Nvegetacionrio

ClustersClusters CollaborationCollaboration

ConferencingConferencing

Web AccessWeb Access

DatabasesDatabases

Multimedia NetworkInternet 2

Multimedia NetworkInternet 2

TestbedsTestbeds

" Research Issues" Implementation

Internet 2Core Network

Internet 2Core Network

Edge RouterWorkstation

(RPI)Workstation

(NU)Edge Router

" Core Network Typically Overengineered" Bottlenecks at Ingress/Egress Nodes" Keep Core Network Routers Simple" Maintain Single-Class, Best-Effort Delivery Service " Provisioning of QoS Provided Outside Core Network

" Edge-to-Edge (EE) Overlay Architectures" Application Layer Framing (ALF) Using JSCC

" In Either Case, QoS Provisioning Must be ApplicationAware

EEALF

QoS Provisioning for Multimedia TransportQoS Provisioning for Multimedia Transport

155 Mbps

Single Service

CenSSIS

CenSSIS

Internet 2

H.323Conference

Server(RPI)

CenSSIS

CenSSISH.323Clients

H.323ClientsBU

UPRMNU

Evolving CenSSIS Collaborative Work EnvironmentEvolving CenSSIS Collaborative Work Environment

Will Evolve From H.323 Videoconferencing Model with Extensions

Will Evolve From H.323 Videoconferencing Model with Extensions

OtherCollaborators

CenSSIS

H.323 IP Telephony

Standard

RPI

Status of Collaborative Work EnvironmentStatus of Collaborative Work Environment

! Internet 2 Connectivity Between RPI and NU.

! Remote Access to Compressed Image Database at NU Underway.

! Several Collaborative Applications Under Study:! Delivery of CenSSIS-Related Courses! High-Quality Video Conferencing! Remote Access to Testbeds

! Incorporate Multimedia Networking ResearchResults.

R3c/d Research ProjectsR3c/d Research ProjectsR3c. Parallel Computing

• Beowulf Cluster Development (C. Shaffer and D. Kaeli) • Profile Guidance and Characterization Tools of Data Parallel Applications (D. Kaeli and Mercury Computer)• Parallel Programming Tools Targeting CenSSIS Applications (W.Meleis, D. Kaeli, D. Brooks, C. Rappaport, M. El-Shenawee and ISI)

R3c. Reconfigurable Embedded Computing• Applying Reconfigurable Hardware to Segmentation for MultispectralImagery (M. Leeser and Los Alamos) • An Investigation into an FPGA Implementation of Backprojection for Rapid Topographic Imaging (E. Miller, M. Leeser and Mercury )

R3d. Image Databases and Metadata• Content Searchable Image Databases (R. Norum, D. Kaeli, E. Miller, S. Lerner and J. Howland)

How can CenSSIS applications scale up to exploit supercomputers to overcome processing barriers?

! Provide a pathway for researchers to exploit NSF Partnerships for Advanced Computational Infrastructure (PACI) resources

! Construct Beowulf-class clusters to provide low-cost, scalable processing (BioMed 8-node cluster and NSF MRI 32-node cluster)

! Utilize BU Mariner SGI clusters! Develop parallel programming toolsets around MPI,

MATLAB and Java standards

How do we parallelize CenSSIS applications?

! CenSSIS applications possess unique processing characteristics

! We have developed a methodology specifically tailored to the data-dominated nature of CenSSIS applications

! We consider parallelism at three grains:! Coarse-grained (potentially application-level)! Loop-grained! Data parallelism

Main0/2502

Multdibobj2/25

Tfqmr1/1604

3/35

Mult11/1602Multdibso

4/59

Mult4/1599

Multdib50/680

Diflinit0/852

Multdibos3/27

Multfaflone4m523/6

Multfaflone4359/5

Multdifl706/0

Snuin7121/84

Epsumag290/94

Epsued42/637Suin7

29/227

0/0

0/0

0/0

18/0

16/0Function

Self /Children Runtime

FunctionSelf /Children Runtime

Number of Calls

0/0 0/0

1

1018

1018

1018

1018

14.4 M

14.4 M14.4 M

14.4 M

11

11

273

273

273

273

273

273

273

54K2.4M

0.6M

Soil

Air

Mine

Steepest Descent Fast Multi-pole Method (SDFMM) Program Control FlowSteepest Descent Fast Multi-pole Method (SDFMM) Program Control Flow

Soil

Air

Mine

Program run time versus number of processors

0500

10001500200025003000350040004500

0 5 10 15 20 25 30

Number of processors

Run

time

(s)

The w hole run timeThe unparallelized part

Fine-grained and coarse-grainedparallelization using MPI on a 32-node clusterFine-grained and coarse-grainedparallelization using MPI on a 32-node cluster

Speedup from parallelization of main loop

05

101520253035

0 5 10 15 20 25 30 35

Number of Processors

Sp

eed

up

CenSSIS Parallel Processing

Status! Obtained linear speedup on SDFMM ! Developed a complete methodology for

assessing control and data parallelism! Completing construction of next generation

gigabit-based cluster

Year 2 Goals! Fully exploit availability of clusters and PACI! resources! Parallelize FDTD, FDFD and SAMM

algorithms

Image ID Image typeLPM099 HyperspectralMVT002 ECGMSD321 Diffusive waveMVT061 CAT scan

y = C(x, z) + n

Database (Oracle8i)

Metadata

* TBs of RAID storage

* Replication and distribution

* Security

*Image format standards

! The data about the data! Contextual browsing

How do we enable researchers to share their image/sensor data?How do we enable researchers to share their image/sensor data?

Sensor Manufacturer Calibration date

Serial number

Compression algorithm

Software Version

Related publications

HLLs supported

CenSSIS Image DatabasesCenSSIS Image Databases

Example of XML labelling

<?xml version="1.0" encoding="UTF-8" ?>

<embryo>

<feature label="1" xPos1 ="50" yPos1="50"xPos2="65" yPos2="65"> Nucleus </feature>

<feature label="2" xPos1="75" yPos1="75" xPos2="85" yPos2="85"> <fragmentation>none</fragmentation> </feature>

<feature label="3" xPos1="100" yPos1="10" xPos2="110" yPos2="15"> <description>mouse oocyte</description> </feature>

<feature label="4" xPos1="25" yPos1="25" xPos2="35" yPos2="35">Cell Membrane </feature>

</embryo>

nucleus

12

34

12

34

mouse oocyte

CenSSIS Image DatabasesCenSSIS Image Databases

Status! Submission, query and update capabilities

online! Includes both images and image metadata! Basic authentication for website

implemented! Construction in progress of the CenSSIS

database/fileserver cluster (Sun)

Year 2 Goals! Implement secure web authentication (SSL)! XML image feature tagging! XML query capabilities

R3 Interaction with IndustryR3 Interaction with Industry

! Mercury Computer! Porting CenSSIS applications to RACE++! Profile-guided compilation! Back projection acceleration with FPGAs

! Integrated Sensors! RTExpress - parallel MATLAB

! MathWorks! Object-oriented MATLAB

! Future partners! EMC - storage array! Clinomics - drug interaction analysis using

content-based image indexing

! Mercury Computer! Porting CenSSIS applications to RACE++! Profile-guided compilation! Back projection acceleration with FPGAs

! Integrated Sensors! RTExpress - parallel MATLAB

! MathWorks! Object-oriented MATLAB

! Future partners! EMC - storage array! Clinomics - drug interaction analysis using

content-based image indexing

Relationships with Other Research ThrustsRelationships with Other Research Thrusts

R1SDFMM, FTFT, FTDT,

SAMM Models Multi-sensor instruments

BioBED, MedBED, SeaBED, SoilBEDMotivating applications +

experimental data

R3Parallel computation

Image databasesSoftware toolsCompressionNetworking

Visualization

L1

L2

R2Inverse ECG solutionsSub-retinal mapping

Diffuse optical tomographyHyperspectral imaging

What are our plans for Year 2?What are our plans for Year 2?

! Physics-Based Image and Data Compression! 3-D Volumetric Compression of MSD Data! Mosaic-Aided Compression of Retinal Sequences! Incorporate Region-of-Interest (ROI) Coding

! Multimedia Networking! QoS Provisioning on Internet 2! Development of Collaborative Work Environment

! Parallelization Processing! Tools for a broad set of R1 modeling methods including

FDTD, FTFD and SAMM! Utilization of RTExpress and parallel MATLAB

! Image Databases! Content-based XML query capability of image data! Multi-dimensional database indexing and visualization

! Physics-Based Image and Data Compression! 3-D Volumetric Compression of MSD Data! Mosaic-Aided Compression of Retinal Sequences! Incorporate Region-of-Interest (ROI) Coding

! Multimedia Networking! QoS Provisioning on Internet 2! Development of Collaborative Work Environment

! Parallelization Processing! Tools for a broad set of R1 modeling methods including

FDTD, FTFD and SAMM! Utilization of RTExpress and parallel MATLAB

! Image Databases! Content-based XML query capability of image data! Multi-dimensional database indexing and visualization