25
High-Performance Computing High-Performance Computing for for Processing Earth Observation Processing Earth Observation Data Data By By Dr Ashok Kaushal Dr Ashok Kaushal Senior Divisional Director Senior Divisional Director Enterprise Geospatial & Defense Solutions Enterprise Geospatial & Defense Solutions Rolta India Limited Rolta India Limited [email protected] Innovative Technologies for Insightful Impact Innovative Technologies for Insightful Impact

Innovative Technologies for Insightful Impact

  • Upload
    lyle

  • View
    56

  • Download
    3

Embed Size (px)

DESCRIPTION

High-Performance Computing for Processing Earth Observation Data By Dr Ashok Kaushal Senior Divisional Director Enterprise Geospatial & Defense Solutions Rolta India Limited [email protected]. Innovative Technologies for Insightful Impact. Agenda. Trends Needs Process Automation - PowerPoint PPT Presentation

Citation preview

Page 1: Innovative Technologies for Insightful Impact

High-Performance Computing High-Performance Computing forfor

Processing Earth Observation Processing Earth Observation DataData

ByBy

Dr Ashok KaushalDr Ashok KaushalSenior Divisional DirectorSenior Divisional Director

Enterprise Geospatial & Defense SolutionsEnterprise Geospatial & Defense SolutionsRolta India LimitedRolta India Limited

[email protected] Innovative Technologies for Insightful ImpactInnovative Technologies for Insightful Impact

Page 2: Innovative Technologies for Insightful Impact

Agenda • Trends

• Needs

• Process Automation

• GeoImaging Accelerators/

GXL

• Job Processing Systems/ JPS

• Conclusions

Page 3: Innovative Technologies for Insightful Impact

Trends • 230 EO [versus 107 in last decade] satellites

projected over next decade for use of satellite imagery– Emerging markets expected to account for 75 satellites

- four-fold increase over last decade

– 41 Nations [currently 26] to have own satellites

• Commercial sale of EO data expected to double– Commercial EO data from satellites expect CAGR of

15% over next 10 years, reaching $4 billion by 2019

– Optical data will represent 79% of overall sales

– Number of high resolution satellites offering commercial data are expected to double from currently 24

‘Satellites to be Built & Launched by 2019, World Market Survey’, Euroconsult

Page 4: Innovative Technologies for Insightful Impact

Trends• Exponential increase of volumes of satellite EO

data • Increasing value of EO data with applications in

– Agriculture, Environment, Urban Development, Disaster Management, Surveillance and others

• Increasing value of up-to-date info – RapidEye, GeoEye, Digital Globe, IRS/ Cartosat

• Significant growth of awareness in EO data – Google Earth, Microsoft Bing Maps, Bhuvan

• Increasing importance of collaboration and sharing of current data/information for Situational Awareness

Page 5: Innovative Technologies for Insightful Impact

Needs • Satellite Programming

• Timely Data Acquisition

• Process Automation

• Data Pre-Processing

• Data Management

• Data Dissemination

• Information Sharing

• Geo Collaboration

Page 6: Innovative Technologies for Insightful Impact

For Production

Move from this

To This

Process Automation

Page 7: Innovative Technologies for Insightful Impact

Incoming raw

imageExtract raw Image

to native format

Collect GCP Using Master Image

Refine collected GCP

Compute MathModel

OrthorectifyRaw Image

Load Image toOracle Database

OracleDatabase

Process Automation

Page 8: Innovative Technologies for Insightful Impact

Geoimaging Accelerators are

automated workflows

created from linking together

of any number of pluggable

image processing functions

Geoimaging Accelerator (GXL)

Data Ingest Process 1 Process 2 Process 3 DeliveryData Ingest Process 1 Process 2 Process 3 DeliveryData Ingest Process 1 Process 2 Process 3Data Ingest Process 1 Process 2 Process 3 Delivery

Page 9: Innovative Technologies for Insightful Impact

Objectives:•Need for large volume image data processing

• to reduce image pre-processing bottlenecks•Demand for greater automation & less user

interaction• to save money on operator time

•Workflows that can scale across multiple processors

• to add capacity as and when needed•Plug & Play architecture

• to add new components or functions to expand•Cost Effective Solution to Remain Competitive

• to run 24/7 with zero or little operator intervention

Geoimaging Accelerator (GXL)

Page 10: Innovative Technologies for Insightful Impact

• Distributed processing• Two levels

• Basic• Automated CPU

• Accelerated • Multi-core CPU• Optimized GPU

• Ortho / Ortho XL• Satellite & Airphoto

• PanSharp / PanSharp XL• Mosaic/ Mosaic XL

Geoimaging Accelerator (GXL)

IngestIngest

Job Processing System (JPS)

Job Processing System (JPS)

GXLGXL OutputOutput

Page 11: Innovative Technologies for Insightful Impact

Orthorectification GXLOrthorectification GXL Ortho Product

Ortho Product

WorldView-1 Level 1b e.g.

WorldView-1 Level 1b e.g.

RPC Model Calculation

RPC Model Calculation OrthorectificationOrthorectification

DEMDEM

Ortho Product

Ortho Product

UltraCam X Imagery e.g.

UltraCam X Imagery e.g. Orthorectification GXLOrthorectification GXL

AP Model Calculation

AP Model Calculation OrthorectificationOrthorectification

DEMDEM

Ortho / Ortho XL

Airphoto / Airphoto XL

Geoimaging Accelerator (GXL)

Page 12: Innovative Technologies for Insightful Impact

Mosaic GXLMosaic GXL

DEM Extraction GXLDEM Extraction GXL

OrthophotosOrthophotos Mosaic ProductMosaic Product

Stereo PairStereo Pair Raster DEMRaster DEM

Colour-Balance

Colour-Balance

Cutline Selection

Cutline Selection MosaicMosaic

Epipolar Rotation

Epipolar Rotation

DEM Extraction

DEM Extraction GeocodingGeocoding

PanSharp GXLPanSharp GXLPan and MS ImageryPan and MS Imagery PanSharp ProductPanSharp Product

Pan SharpeningPan Sharpening

Geoimaging Accelerator (GXL)

Page 13: Innovative Technologies for Insightful Impact

Accelerated GXL?• A hardware-based, GPU enabled, high-

performance image processing system • Design to process large volumes

– 40 times faster than desktop product – 2-4 TB per day for desk-side system– 10 TB + for rack mounted system

Orthorectify & Mosaic India in a Day!

Geoimaging Accelerator (GXL)

Page 14: Innovative Technologies for Insightful Impact

C++ SDKGPU / HW

C++ SDKGPU / HW

Interface LayerInterface Layer

Processing LayerProcessing Layer

Architecture LayerArchitecture LayerFormats:BIL, TIFF,

etc.

Formats:BIL, TIFF,

etc.

Algorithms: Pansharp, Ortho, etc.

Algorithms: Pansharp, Ortho, etc.

PPFsPPFs

Workflows: GXL

Workflows: GXL

Bindings:Python,

Java

Bindings:Python,

Java

Layer:Layer: Component:Component: Integration:Integration:

Job Processing System Data / Imagery Level

Data / Imagery Level

Operations / Systems Level

Operations / Systems Level

HW / Architecture Level

HW / Architecture Level

Arc

hit

ectu

re

Geoimaging Accelerator (GXL)

Page 15: Innovative Technologies for Insightful Impact

• Flexible orthorectification:– Support for several sensors (SPOT, QB, Ikonos, WV, …) – Optional radiometric calibration of SPOT images– Optional GCP collection from multiple reference data types

• Flexible mosaicking:– Mosaics from mixed-resolution raw scenes– Optional tie point collection and refinement– Various types of color balancing– Various tiling schemes

• High quality:– Sub-pixel accuracy of GCPs and orthoimages– Nicely color-balanced mosaics

Hig

hlig

hts

Geoimaging Accelerator (GXL)

Page 16: Innovative Technologies for Insightful Impact

Product Type

Dataset Resolution [m]

Volume [TB/Day]

Area [km2/day]

SPOT5

- Level 1A 2.5 meter 8U Pan 2.5 2.0013.7

Million e.g. Europe:

10.1M km2

IKONOS

- Geo Ortho Kit16U Pan Ikonos 1.0 2.94

1.62 Million

e.g. Mongolia:1.56M km2

WorldView-1 and

QuickBird Level 1B 16U Pan 0.5 3.26448k

e.g. Sweden:450k km2

QuickBird- OrthoReady- 4 channel PS

16U Multispectral 0.6 3.52174k

e.g. Florida:170k km2

QuickBird

- Level 1B16U Multispectral 2.4 4.57

3.62 Millione.g. India:3.17M km2

Pro

cessin

g M

etr

ics

Geoimaging Accelerator (GXL)

Page 17: Innovative Technologies for Insightful Impact

Product Type

Dataset MB/Sec GB/Min TB/Day

SPOT5

- Level 1A 2.5 meter 8U Pan 24.23 1.42 2.00

IKONOS

- Geo Ortho Kit 16U Pan Ikonos 35.67 2.09 2.94

WorldView-1 and

QuickBird Level 1B 16U Pan 39.59 2.32 3.26

QuickBird- OrthoReady- 4 channel PS

16U Multispectral 42.67 2.50 3.52

QuickBird

- Level 1B 16U Multispectral 55.47 3.25 4.57

Pro

cessin

g

Th

rou

gh

pu

tGeoimaging Accelerator (GXL)

Page 18: Innovative Technologies for Insightful Impact

Performance /Day GB 1 - 5TB 5 - 10TB

Batch Processing

20 Orthos per day10GB Project Scale

GXL Deskside Accelerated

2000 Orthos per day1TB Project Scale

GXL RackAccelerated

5000 Orthos per dayPlus

100 Image Mosaic per day5TB Project Scale100 Orthos per day

50GB Project Scale

10

200

500

Cost $1,000

GXL Basic

Geoimaging Accelerator (GXL)

Page 19: Innovative Technologies for Insightful Impact

• Environmental• Carbon sequestration• Biomass estimation

• Agricultural• Crop yield• Crop forecasting

• Aerospace & Defense• Border monitoring• Disaster management

• Data Supply• Product delivery• Archive re-processing

GeoImaging Accelerator

Ap

plicati

on

s

Page 20: Innovative Technologies for Insightful Impact

Job Processing System• Distributed Processing System

– Run multiple jobs concurrently on multiple servers

Job

Job

Job

Job

Job

Job Job

JPS Processing Server

JPSDatabase

ComputerComputer Computer Computer

JPS Processing Server

JPS Processing Server

JPS Processing Server

Page 21: Innovative Technologies for Insightful Impact

• Job:

– An entry in the JPS-DB

– A Process started and monitored by a Processing Server

• Processing Server

– Daemon managing jobs

JPS-DB

Processing ServerJob

Job

Job Processing System

Page 22: Innovative Technologies for Insightful Impact

22

• Distributed Cloud Computing (Autonomous Nodes)

• Automatic Load Balancing• Simple Web Interface• Threefold Value:

1. Automation = Increased Throughput (Revenue)2. Job Tracking = Improved QA (Operational

Costs)3. Multi-Platform, Multi-Language =

SustainabilityJPS-DB

GXL1

GXL2 Job

Job

Job

Other Nodes

Job

Job

Job

Job Processing System

Page 23: Innovative Technologies for Insightful Impact

Job Processing System

Page 24: Innovative Technologies for Insightful Impact

• Effective use of voluminous satellite imagery from numerous high-resolution satellites desires automated pre-processing using HPC

• Distributed processing using multi-core CPU and GPU with CUDA and Open MP provides an ideal platform for faster turn-around-time during pre-processing of geoimaging

Conclusions

Page 25: Innovative Technologies for Insightful Impact

Thank you !Thank you !