Virtual Reality (VR360): Production Framework for the Combination of High Dynamic Range Imaging (HDRI) and Spherical Panorama towards Lean Approach

Researcher: Zi Siang See

Production Framework for the Combination of HDRI

and Spherical Panorama towards Lean Approach

Email (University Tunku Abdul Rahman) [email protected]

Email (Industrial collaboration): [email protected]

University Tunku Abdul Rahman

Faculty of Creative Industries

Centre for Immersive Technology and Creativity

Department of Multimedia Design and Animation

Centre of Immersive Technology and Creativity

Department of Multimedia Design and Animation

Research Talk: Zi Siang See

Virtual Reality 360 (VR360)

collaborators from


Research Presentation

1) Introduction (VR360)

a. Aim, Objectives, Research Problem

b. Literature Review and Related Work

2) Research Design and Method

3) Data Collection and Analysis

4) Discussion and Conclusion

collaborators from


Professional Background:

Industrial involvements since 2003/04


Industrial Specialization: Creative and Advertising Imaging

Research Area: High Dynamic Range Imaging (HDRI)

Spherical Panorama

Augmented Reality (AR)

Creative and Advertising Processes



Work: McDonald’s Malaysia

Creative: NagaDDB

Operation: Quality Ingredient Story

Creative Imaging: Reina Imaging

Researcher: Zi Siang SeeDigital Imaging (DI) image manipulation for advertising usage, high technical finishing | Presenter: Zi Siang See

Researcher: Zi Siang SeeDigital Imaging (D() for Advertising Visuals| Presenter: Zi Siang See

Researcher: Zi Siang SeeAdvertising visual | Presenter: Zi Siang See

Researcher: Zi Siang SeeDigital Imaging (D() for Advertising Visuals| Presenter: Zi Siang See


Researcher: Zi Siang SeeObject Virtual Reality (Object VR) | Presenter: Zi Siang See

Researcher: Zi Siang SeeVirtual Reality 360 (VR360) | Presenter: Zi Siang See

Virtual Reality 360 (VR360)


Introduction

Multimedia (Combination of Elements)

• Text

• Image

• Sound

• Video

• Animation Industries

• Creative and advertising

• Corporate communications visuals

• Architectural imaging

• Cultural heritage preservation

• Scientific achieve

Research - Virtual Reality 360 (VR360)

1) High Dynamic Range Imaging (HDRI)

2) Spherical Panorama


Introduction: HDRI & Spherical Panorama

Obstacles and Issues

• High Dynamic Range Imaging (HDRI)

• Spherical Panorama

Towards Lean Approach

• Inaccurate reproduction of location-based subjects in image

• Architectural context as case study subject of location-based content for data collection and analysis

Proposed Production Framework

• To have real-world likelihood photographic outcome reproduced by HDRI and spherical panorama with reduced imaging variables caused by various obstacles and issues.

production processes


Figure: combined luminance (EV) range being covered by HDRI outcome.

Darker images

(luminance) Brighter images

(luminance)

Introduction: HDRI

• High Dynamic Range Imaging (HDRI) in photography technique has

a goal to increase the dynamic range resulting an improved

luminance reproduced in shadow and highlight areas (Reinhard et

al, 2010; Debevec and Malik, 1997; Fairchild, 2007), usually by

combining multiple exposures consist of darker and brighter images.


• Possible major obstacles and issues of HDRI that involves multiple

exposures can include ghosting effect and misalignment issue due

to inconsistent acquisitions (Reinhard et al, 2010)

Figure: HDRI with visual abnormalities (by Reina).

Introduction: HDRI

Multiple exposures for HDRISingle exposure (LDR)

Visual abnormality


Introduction: HDRI

• ‘The HDR Photographic Survey’ by Fairchild (2007) has

demonstrated HDRI reproduction may require massive amount of

multiple exposures, however may speculate to have production

wastage or technical redundancy.

(1) (2) (3)

(4) (5) (6)

(7) (8) (9)

(a) (b)

Massive amount of multiple

exposures, may have

wastage or redundancy.

For example, some image

sequences have exceed the

need of constructing HDRI.

Figure: example taken from ‘The HDR Photographic Survey’ by Fairchild (2007)


Figure: Multirow production process to photographically create a spherical imagery

Processing (Stitching)

Multirow setup, to have

multiple photographs

covering 360 degrees.

Spherical

panorama

rendering

Introduction: Spherical Panorama

• Spherical panorama photography imaging technique aims to acquire

wider panoramic visibility in image usually unable to be acquired

from the conventional single angle photograph image (Benosman

and Kang, 2001; Brown and Lowe, 2006; Felinto et al, 2012), this

can be arranged by combining multiple angle images.


• Microsoft launched a feature called the ‘Street Side’ in Bing Map on

November 2010 in competition to Google Map with ‘Street View’

featuring spherical panorama interactivity, however numerous

content published have shown high amount of visible imaging

errors and insufficient of dynamic range.


Figure: (a) “Street View” feature in Google Map. (b) “Street Side” feature in Microsoft Bing Map.

(a) (b) Visual abnormalityVisual abnormality


• Major difficulty for spherical panorama reproduction may occur, this

can include limited dynamic range reproduced and parallax error

(Brown and Lowe, 2006; Felinto et al, 2012)

Figure: panoramic imagery with visual abnormality (by Reina).


Visual abnormality

Limited dynamic range

Parallax error




a. Research Aim, Objectives






Research Design and Method

Technical VariablesUnknown Variables

Review of Literature and Related Work


Data collection and analysis with experiment samplings

HDRI Spherical Panorama

Study: Reduction of Imaging

Variables (Obstacles, Issues)



Optimization towards Lean Approach

HDRI and Spherical Panorama

Propose Production Framework:

Combination HDRI and Spherical Panorama

Discussion and Conclusion

1 2

3


Research Design and Method:

Lean Thinking:

What and how is the production framework of lean approach in the

context for the combination of HDRI and spherical panorama?

• Value: Value of an outcome (or a product) specified by target user

• Value Stream: Value-creating activities to be remained and waste to be reduced

• Flow: Flow of production processes for possible rearrangement

• Pull: Any new value to be refined according to new requirements (continuously)

• Perfection: Effort of continuous improvement and study

According to Lean Thinking (Womack and Jones, 1996)



High Dynamic Range Imaging (HDRI)

HDRI Technical Configurations• Observe and study the obstacles

and issues.

• Study: The process of multiple

exposures for HDRI (Debevec and

Malik, 1997; Reinhard et al, 2010;

Fairchild, 2007; Larson, 1997).

• Quantify the dynamic range for

observation (Reinhard et al, 2010;

Imatest, 2012; Gardner, 2012).

• Exploration attempts to verify the

potential dynamic range can be

extended by digital negative – RAW

(Hill, 2010; Gakken Co, 2008).

• Produce samplings of HDRI during

the data collection and analysis

process for location-based images

using architectural context as case

study subject.

• Industrial convention DSLR system

(D3x)

• Quantify and verify the latitude of

the projected dynamic range covered

by the RAW image file of D3X at

ISO100

• Observe usable dynamic range

observed (pixel value reading)

-2EV (expanded) 0EV (neutral) +2EV (expanded)



Spherical Panorama

Spherical Panorama

Technical Configurations• Observe and study the obstacles

and issues.

• Considerations about the spherical

panorama photographic production

process (Felinto et al, 2012; Guan et

al, 2009; Hoshi and Kato, 2006;

Andrews, 2003).

• Study: The multirow configuration

for spherical panorama production

processes that is to acquire and

combine multiple angle images

(Brown and Lowe, 2006; Chen, 1995;

Gawthrop, 2007).

• Produce samplings of spherical

panorama during the data collection

and analysis process for location-

based images using architectural

context as case study subject.

• Industrial convention DSLR system

(D3x)

• 16mm Fish-eye, Manfrotto 303SPH

for multirow setup.

Multirow setup, to

have multiple angle

image sequence

covering 360 degrees.

1 x Zenith Angle

6 x Horizontal

Angle(s)

1 x Nadir Angle


• Related studies (Felinto et al 2012, Gawthrop 2007, Brown and

Lowe 2006; Schulz, 2012) have shown that photographic

combination for HDRI and spherical panorama can be possible,

however conditionally tolerating with various obstacles and issues

as reported in previous literature and work.

Spherical Panorama Multirow HDRI requires multiple exposures

-4, -3, -2, -1, 0, +1, +2, +3, +4 (EV)

VR360: HDRI & Spherical Panorama(Existing Production Processes, according related studies)










Data Collection and Analysis Progress















1


Data Collection & Analysis:


Measuring Dynamic Range of Instrument Used

• Quantify and verify the latitude of the projected dynamic range

covered by the digital negative of instrument used (D3X at ISO100).

• Different RAW processors tested (native, extended dynamic range).

pix

el

valu

e

Sequence of exposure(s)

RAW Processor: C1Pro RAW Processor: ACR RAW Processor: ViewNX

thesis_research_sim/dynamic_range_observation

thesis_research_sim/dynamic_range_observation




Digital Negative

(RAW)Observations • The study implies that extending

high dynamic range is possible with

digital negative image by using

single acquisition as shown in the

initial result of measuring dynamic

range with instruments used.

• Similar to the idea brought up by

Debevec and Malik (1997) about

extending dynamic range can be

possible from conventional single

film processing - multiple scanning.

• The finding in the study about

native EV of dynamic range

reproduction was observed to have

similar approximated usable

exposure values with the RAW

produced by ISO100 according to the

test by Rehm (2009) with the same

camera equipment tested using

calibrated Stouffer Step Wedge.

• Extending Dynamic Range from

Digital Negative (RAW), observation.

• (-2 EV and +2 EV, resulting an

increment of 8.5EV to 12.5EV)

Extended Dynamic Range

Data taken from RAW Processor: C1Pro

• 8.5EV, contrast ratio 362:1

• 12.5EV, contrast ratio 5793:1 2 f-stopContrast ratio =


High Dynamic Range Imaging (HDRI) with single acquisition

Figure: HDRI by retrieving extended high dynamic range from digital negative image

Blended result,

HDRI expanded from single

source of RAW.

Proposed Method and Apparatus:

monotonic mask

from RGB

channel

monotonic mask

from RGB

channel

(inverted)

masking on the

extended (-2EV) layer

on top of the neutral

(0EV) layer

masking on the

extended (+2EV) layer

on top of the neutral

(0EV) layer

RAW processing, EV parameter specified -2EV (expanded) 0EV (neutral) +2EV (expanded)

Apply Image: 50% opacity

on top of each layer.

(can be any sequence)

Example of single exposure required for this process

Global or local HDRI

reproduction. For

local HDRI, blurred

mask can be applied

(optional)

Global or local HDRI

reproduction. For

local HDRI, blurred

mask can be applied

(optional)


High Dynamic Range Imaging (HDRI) with single acquisition


Blended result,

HDRI expanded from single

source of RAW.

Proposed Method and Apparatus:

Single

Acquisition

(RAW)

Process: Extending

Dynamic Range

Extended

+2EV

Original

0EV

Extended

-2EV

HDRI from

single source


(simplified production process illustration)


(a) Conventional LDR Photo (b) Proposed HDRI from single acquisition (c) HDRI from Multiple Exposures

-2EV +2EV0EV

Proposed HDRI

using single acquisition

(HDRI extended from RAW)



HDRI (Multiple exposures)

Sampling(s)


(a) Conventional LDR Photo

(b) Proposed HDRI produced

from single acquisition

Can have moderate extended

dynamic range reproduced, least

production processes and free

from imaging errors (zero visual

abnormality).

(c) HDRI reproduced from

multiple exposures technique

Observations: HDRI

Visual abnormalities


-2EV +2EV0EV

(a) LDR (b) HDRI, single

acquisition

(c) HDRI, multiple

exposures

-2EV +2EV0EV

Proposed HDRI produced



Proposed HDRI produced





(a) LDR (b) HDRI, single

acquisition

(c) HDRI, multiple

exposures

Sampling(s) Sampling(s)


Ghosting effect /issue of HDRI

reproduction from source of

multiple exposures

HDRI reproduction from source of single

acquisition where dynamic range can be

extended from Digital Negative (RAW)

(achieve result of reduced imaging errors

while needing least production processes

and time)

Moving objects, plants, weather

Observation: HDRI

(Obstacles and Issues)


Outcome: HDRI with single acquisition using digital negative

• Reduced production processes and imaging variables.

• Avoided visual abnormalities (issues from multiple exposures).

• Imagery features can be preserved moderately in high contrast scene.


Discussion: HDRI

Single

Acquisition

(RAW)

Process: Extending

Dynamic Range

Extended

+2EV

Original

0EV

Extended

-2EV

HDRI from

single source

HDRI that requires single acquisition

Proposed Production Process DesignExisting Process

HDRI that requires multiple exposures

-4, -3, -2, -1, 0, +1, +2, +3, +4 (EV)

















2


multiple angles of multirow configuration,

covering horizontal, zenith and nadir angles.

(However having nadir angle difficulty)Data Collection & Analysis:

Spherical Panorama Sampling(s)


Parallax error

Nadir angle difficulty

(unwanted imagery content )

Observations: Spherical Panorama (cubic projection)

Insufficient dynamic range

Sampling(s)


(a) Spherical Panorama

(nadir angle difficulty resulting

visual abnormality is a mistake

that requires rectification)

(b) Spherical Panorama

(post-processing correction)

Observations: Spherical Panorama (rectification required)

Visual abnormalities after post processing correction (manual image manipulation)

Sampling(s)


• Need to have higher luminance (dynamic range) in multiple angles.

• Suggested to avoid parallax error & ghosting effect (acquisition)

Critical consideration

• Nadir angle difficulty

(if rectification required?)

• Consistency of white balance and exposures in multiple angles.

(a) (b)

Post

Processing

Correction

(Manual

Correction

Required)

Consistency white balance

Is critical for the process

Observation: Spherical Panorama

(Obstacles and Issues)


Outcome: Spherical Panorama Multirow Configuration Redesign

• Redesign that aims to reduce mistake that requires rectification

• Redesign that can reproduce the nadir (bottom) angle adequately

from the original source of the location-based subject in image.


Discussion: Spherical Panorama

Multirow with reduced obstacles and issues

Proposed Production Process DesignExisting Process

Multirow with various obstacles and issues

















3


Total images required = 72 images (simulated scenario)• 8 angles required (image sequence)

• 9 exposures required for each angle

Spherical Panorama configuration covering the

immersive 360 degrees.

• 8 angles required (image sequence)

HDRI that requires multiple exposures

• 9 exposures required for each angle

-4, -3, -2, -1, 0, +1, +2, +3, +4 (EV)

Combination of HDRI and Spherical Panorama

(Existing Production Process, according related studies)

Figure: combination of HDRI and spherical panorama, existing production processes.


Total images required = 11 images (sampling scenario)• 11 angles required (image sequence)

• 1 exposure (RAW) required for each angle

Figure: Proposed production framework towards lean approach for the combination of HDRI and spherical panorama according to

previous studies and related works.

HDRI that requires single source acquisition

• 1 physical exposure required for each angle

(Extended dynamic range -2EV and +2EV can be

reproduced from RAW processing)

Spherical panorama configuration covering

the immersive 360 degrees.

• 11 angles required (image sequence)

Single

Acquisition

(RAW)

Process: Extending

Dynamic Range

Extended

+2EV

Original

0EV

Extended

-2EV

HDRI from

single source

Combination of HDRI and Spherical Panorama

(Proposed Production Framework towards Lean Approach)


Production Framework: HDRI and Spherical Panorama

multiple angles facilitated by HDRI required

for producing the spherical panorama

(zero visual abnormality)

Successful sampling(s)


Observations: HDRI and Spherical Panorama(a) LDR photo image (b) HDRI from single source acquisition (c) HDRI from multiple exposures

Moderate extended high dynamic range,

reduced overall production processes.








HDRI ghosting effect

using multiple

exposures method.

HDRI reproduction

from single acquisition,

digital negative (RAW)

visual abnormality is

eliminated or avoided

(for multiple angles

required for spherical

panorama)

Observations: HDRI and Spherical Panorama








Multiple angles of multirow configuration

(from the proposed production framework)

with reduced error or mistakes that requires

rectifications, including nadir angle reproduction.


Nadir Angle:

Reproduction from the original

location-based scene, minimal

or no post-production

correction required

Sampling(s)








Inconsistent HDRI local adaptation for different angles, for spherical panorama

(multiple exposures: 9 image sequences for each angle)

Can produce consistent multiple angles of HDRI, for spherical panorama

(single acquisition: extended high dynamic range from RAW)


Sampling(s)

Sampling(s)










Elements from Lean Approach HDRI Spherical Panorama

Waste: mistakes which requires

rectification

Obstacle avoided

(Ghosting, Misalignment)

Obstacle minimized,

minimal correction

Waste: excessive productions objects or

volume that remaindered as goods pile up

Single acquisition,

reduced storage and timeIrrelevant

Waste: processing steps which are actually

not needed

Single acquisition

(instead of multiple)

New multi angles

configuration

Waste: redundant human movements of

production processes or transportationLess activity

Faster production

process

Waste: waiting time for an processed

materials to be used in the next process

which has not been delivered in time

Reduced to single

acquisition (instead of

multiple exposures)

Single acquisition only

from HDRI for each

angle

Waste: Production outcome which do not

meet the needs according to the value

defined by the end user or customer

Moderate extended

dynamic range limited to

capability of DNG/RAW

Avoided nadir error,

geometrically

accurate/reproduced

Discussion: HDRI and Spherical Panorama

Improvements made towards Lean ApproachTable: Improvements with lean approach for the combination of HDRI and spherical panorama

Updated reference: Lean Lexicon (The Lean Enterprise Institution, 2008) and Lean Thinking (Womack and Jones, 1996)


Field Implementation: Corporate communications image for advertising visual (SP Setia, 2013)

Field Implementation


Discover MINIEnter at your own risk of losing control


Conclusion

Proposed Production Framework

Before optimization

The proposed production framework towards lean approach has

shown that the outcome of spherical panorama facilitated with

HDRI can be reproduced towards improved photographic

reproduction with least visual abnormalities. This is possible with

reduced production processes, wastes, obstacles and issues.


• Zi Siang See, Adrian David Cheok. (2014) Virtual Reality 360 Interactive Panorama Reproduction Obstacles and

Issues. Virtual Reality, Springer [indexed by SCOPUS]

• Zi Siang See, Khairul Hazrin Hashim, Kamarulzaman Ab Aziz, Adrian David Cheok (work-in-progress). Spherical

Panorama and High Dynamic Range Imaging: Lean Reproduction Approach. [Journal submission /review in

progress]

• Zi Siang See, Khairul Hazrin Hashim, Harold Thwaites, Lee Xia Sheng, Ooi Wooi Har. (2012). Retrieving

Extended High Dynamic Range From Digital Negative Image - An Experiment On Architectural Photo Imaging.

World Academy of Science, Engineering and Technology, Issue 68 August 2012, pp. 2158-2166. [indexed by

EBSCO]

• Zi Siang See, Lee Xia Sheng. (2011). Non-bias Architectural Image Archive using High Dynamic Range

Approach. 2nd International Conference on Research and Innovation in Information Systems – 2011 (ICRIIS’11)

[CD Proceedings], Kuala Lumpur, Malaysia, 23-24 November. [indexed by SCOPUS, IEEE Xplorer]

• Zi Siang See. (2011). Non-bias Scientific Imaging Archive for Urban Morphology Research. Exchanging Views on

How to Sustain Urban and Rural Environments in Aspects of Technology, Social and Ecology in Green Innovation

Research Group (GIRG) [CD-ROM], Universiti Technologi Malaysia (UTM), Malaysia.

Publications & Collaborations


Recommendations and Suggestions

for Future Research

• Explore production framework adaptation with new technologies available.

• Collaboration with academic and field researchers.

Figure: Mobile device (Android) featuring spherical panorama functionality. Technical areas of HDRI and spherical panorama are

currently the industrial interests for digital equipment developers and multimedia content producers.


Prototype (program name): See Through 360

• Prototype configuration design: See Through 360

• Capability to acquire completeness of multiple angle images including

horizontal, zenith and nadir.

Figure: “See Through 360” prototype configuration customization, program funded by Reina Imaging (2014).

(a) (b) (c)


Prototype (program name): See Through 360

• Prototype configuration design: See Through 360

• Capability to acquire completeness of multiple angle images including

horizontal, zenith and nadir.

Figure: “See Through 360” prototype configuration customization, program funded by Reina Imaging (2014).

(a) (b) (c)

Leveling

rotation nadir


Thank you

Research Discussion

Research Track

www.zisiangsee.com/research

http://www.zisiang.com/research


Recommendations and Suggestions

for Future Research

• Augmented Reality Process using Using Spherical Panorama

facilitated with High Dynamic Range Imaging (HDRI)

• Cultural Heritage Reproduction Using Spherical Panorama

facilitated with High Dynamic Range Imaging (HDRI)

• Macro Virtual Reality 360 Using Spherical Panorama facilitated with


• Interactive Advertising Visual Using Spherical Panorama facilitated

with High Dynamic Range Imaging (HDRI)