2
Focus Measurement on Programmable Graphics Hardware for All in-Focus Rendering from Light Fields Kaoru Sugita, Keita Takahashi, Takeshi Naemura and Hiroshi Harashima School of Information Science and Technology, The University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan {sugita, keita, naemura, hiro}@hc.t.u-tokyo.ac.jp Abstract This paper deals with a method for interactive rendering of photorealistic images, which is a fundamental technology in the field of virtual reality. Since the latest graphics pro- cessing units (GPUs) are programmable, they are expected to be useful for various applications including numerical computation and image processing. This paper proposes a method for focus measurement on light field rendering using a GPU as a fast processing unit for image processing and image-based rendering. It is confirmed that the proposed method enables interactive all in-focus rendering from light fields. This is because the latest DirectX 9 generation GPUs are much faster than CPUs in solving optimization prob- lems, and a GPU implementation can eliminate the latency for data transmission between video memory and system memory. Experimental results show that the GPU imple- mentation outperforms its CPU implementation. 1 Introduction The interactivity and the photorealism are essential for most of the virtual reality applications. The aim of this pa- per is to add the interactivity to a photo-realistic view syn- thesis system [4], which is based on the light field rendering method. Light field rendering (LFR) can generate free-viewpoint images from a dense set of reference images with no / little geometries. But it suffers from a focus like effect caused by the undersampling of the reference images [2]. Defocused regions in synthetic images are damaged with blurring and ghosting artifacts. Takahashi et al.[4] proposed a method for solving this problem. Their approach is to generate an all in focus image (free from blurring and ghosting) from a se- ries of differently-focused images which are synthesized by LFR from an undersampled light field. The optimal focused depth is selected for each pixel by the focus measurement (depth-search), but this process is much time-consuming. This paper explains novel techniques to implement their method [4] on the RADEON 9800 Pro GPU. By implement- ing all the processing on a GPU, we can derive the bene- fit of parallelism of the processing, as well as elimination of the latency for data transmission between video memory and system memory. Experimental results give quantitative comparisons of processing time between GPU and CPU im- plementations. 2 DirectX 9 generation GPUs Recently, graphics processing units (GPUs) as ATI RADEON series and NVIDIA GeForce FX series which support DirectX 9 are distributed as commodity peripheral equipments of desk-top PCs. The features of these GPUs are programmable rendering pipeline and floating point pre- cision computation as well as faster polygon filling oper- ation. These features allow GPUs to extend the range of applications [1]. For example, Yang and Pollefeys [5] pro- posed an algorithm for multi-resolution depth from stereo using GPUs. 3 Optimal focal depth search on GPU Figure 1 shows the procedure for generating all in-focus images from the series of multi focus images. Takahashi et al. [4] use a CPU for performing depth-search. In this case, multi-focus images must be transferred from video mem- ory to system memory. Our proposal is to perform entire processing on a GPU. In the original algorithm [4], block filters are used for smoothing. For the purpose of fast processing on a GPU, we can use “mip-map function” based multi-resolution op- eration [5]. Moreover, optimization can be performed by “depth replace function” [3]. Therefore we adopt multi-resolution optimization method for depth-search. It 255 IEEE Virtual Reality 2004 March 27-31, Chicago, IL USA 0-7803-8415-6/04/$20.00©2004 IEEE. Proceedings of the 2004 Virtual Reality (VR’04) 1087-8270/04 $ 20.00 IEEE

[IEEE IEEE Virtual Reality 2004 - Chicago, IL, USA (27-31 March 2004)] IEEE Virtual Reality 2004 - Focus measurement on programmable graphics hardware for all in-focus rendering from

  • Upload
    h

  • View
    212

  • Download
    0

Embed Size (px)

Citation preview

Page 1: [IEEE IEEE Virtual Reality 2004 - Chicago, IL, USA (27-31 March 2004)] IEEE Virtual Reality 2004 - Focus measurement on programmable graphics hardware for all in-focus rendering from

Focus Measurement on Programmable Graphics Hardwarefor All in-Focus Rendering from Light Fields

Kaoru Sugita, Keita Takahashi, Takeshi Naemura and Hiroshi HarashimaSchool of Information Science and Technology, The University of Tokyo

7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan{sugita, keita, naemura, hiro}@hc.t.u-tokyo.ac.jp

Abstract

This paper deals with a method for interactive renderingof photorealistic images, which is a fundamental technologyin the field of virtual reality. Since the latest graphics pro-cessing units (GPUs) are programmable, they are expectedto be useful for various applications including numericalcomputation and image processing. This paper proposes amethod for focus measurement on light field rendering usinga GPU as a fast processing unit for image processing andimage-based rendering. It is confirmed that the proposedmethod enables interactive all in-focus rendering from lightfields. This is because the latest DirectX 9 generation GPUsare much faster than CPUs in solving optimization prob-lems, and a GPU implementation can eliminate the latencyfor data transmission between video memory and systemmemory. Experimental results show that the GPU imple-mentation outperforms its CPU implementation.

1 Introduction

The interactivity and the photorealism are essential formost of the virtual reality applications. The aim of this pa-per is to add the interactivity to a photo-realistic view syn-thesis system [4], which is based on the light field renderingmethod.

Light field rendering (LFR) can generate free-viewpointimages from a dense set of reference images with no / littlegeometries. But it suffers from a focus like effect caused bythe undersampling of the reference images [2]. Defocusedregions in synthetic images are damaged with blurring andghosting artifacts. Takahashi et al.[4] proposed a method forsolving this problem. Their approach is to generate an allin focus image (free from blurring and ghosting) from a se-ries of differently-focused images which are synthesized byLFR from an undersampled light field. The optimal focuseddepth is selected for each pixel by the focus measurement

(depth-search), but this process is much time-consuming.This paper explains novel techniques to implement their

method [4] on the RADEON 9800 Pro GPU. By implement-ing all the processing on a GPU, we can derive the bene-fit of parallelism of the processing, as well as eliminationof the latency for data transmission between video memoryand system memory. Experimental results give quantitativecomparisons of processing time between GPU and CPU im-plementations.

2 DirectX 9 generation GPUs

Recently, graphics processing units (GPUs) as ATIRADEON series and NVIDIA GeForce FX series whichsupport DirectX 9 are distributed as commodity peripheralequipments of desk-top PCs. The features of these GPUsare programmable rendering pipeline and floating point pre-cision computation as well as faster polygon filling oper-ation. These features allow GPUs to extend the range ofapplications [1]. For example, Yang and Pollefeys [5] pro-posed an algorithm for multi-resolution depth from stereousing GPUs.

3 Optimal focal depth search on GPU

Figure 1 shows the procedure for generating all in-focusimages from the series of multi focus images. Takahashi etal. [4] use a CPU for performing depth-search. In this case,multi-focus images must be transferred from video mem-ory to system memory. Our proposal is to perform entireprocessing on a GPU.

In the original algorithm [4], block filters are used forsmoothing. For the purpose of fast processing on a GPU,we can use “mip-map function” based multi-resolution op-eration [5]. Moreover, optimization can be performedby “depth replace function” [3]. Therefore we adoptmulti-resolution optimization method for depth-search. It

255

IEEE Virtual Reality 2004 March 27-31, Chicago, IL USA 0-7803-8415-6/04/$20.00©2004 IEEE.

Proceedings of the 2004 Virtual Reality (VR’04) 1087-8270/04 $ 20.00 IEEE

Page 2: [IEEE IEEE Virtual Reality 2004 - Chicago, IL, USA (27-31 March 2004)] IEEE Virtual Reality 2004 - Focus measurement on programmable graphics hardware for all in-focus rendering from

Multi-view

point image

Multi-focus images reconstructed

with three different filters

Evaluation image

(difference)

All in-focus image

Wide

Normal

SkipSearch for

optimal focal distance

for each pixel

Multi.Depth-search

Diff. SearchDisplay

Read

Figure 1. The data flow of the method [4].

Table 1. Total processing time (ms).Total Multi. Depth-search Display

CPU 3170 355 2800 11.5GPU 412 53.8 3.17

Table 2. Processing time for depth-search(ms).

Total Read Diff. SearchCPU 2800 1300 91.4 1410GPU 53.8 — 16.1 37.7

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10111213141516

The number of focal planes N

Pro

cessing

tim

e rati

os

Multi.

Depth-search (CPU)

Depth-search (GPU)

Display (CPU)

Display (GPU)

Figure 2. The processing time ratio of eachitem in Table 1.

searches the optimal value, raising resolution by ”coarse-to-fine” basis. The effect of smoothing is acquired by usingthe solution at the lower resolution as the restriction condi-tion for the higher resolution operation.

4 Experiments

Our implementation environment is a PC with an IntelPentium 4 3.06GHz CPU and an ATI RADEON 9800 ProGPU (driver version is Catalyst 3.6). All C++ programs arecompiled with Intel C++ compiler version 7.1. Referenceimage set we use is the multiview image database courtesyof University of Tsukuba, Japan. Input images are arrangedin the resolution 256× 192 pixels of 9× 9 viewpoints. Theresolution of output image is 512 × 512 pixels.

Table 1 shows measured total processing time. “GPU” isthe proposed method. “CPU” is the same algorithm imple-

mented using a CPU. This shows that the GPU implementa-tion is about 7.7 times faster than the CPU implementation.And in “CPU” case, the processing time of depth-search oc-cupy the great portion of processing time.

Table 2 shows measured processing time for depth-search. In “CPU” case, data transfer from video memoryto system memory occupies 41.0 % of the total processingtime. This operation is not required in “GPU” case. In ad-dition, in “CPU” case, the processing time of searching is37.4 times longer than “GPU” case. These lead to increasethe processing time of “CPU” case compared with “GPU”case.

Figure 2 shows variation of the processing time. Eachvalue is expressed in the ratios. And the ratio is set to be onewhen the number of focal planes N = 1. This graph showsthat the great portion of the total processing time (Multi.and depth-search on a GPU) are mostly proportional to N .Therefore the reduction of N leads to improvement of in-teractivity.

5 Discussion

In this experiment, constants such as N etc. are deter-mined heuristically. Our future work will be on techniquesto determine suitable parameters for dynamic scenes, suchas video sequences acquired by multi-camera systems.

References

[1] M. J. Harris. GPGPU General Purpose Computation UsingGraphics Hardware. http://www.gpgpu.org/

[2] J. Stewart et al. A New Reconstruction Filter for Undersam-pled Light Fields. EGSR 2003, pages 150–156, Sep. 2003.

[3] K. Sugita et al. Performance Evaluation of ProgrammableGraphics Hardware for Image Filtering and Stereo Matching.ACM VRST 2003, pages 176–183, Oct. 2003.

[4] K. Takahashi et al. All in-Focus View Synthesis from Under-Sampled Light Fields. ICAT 2003, pages 249–256, Dec. 2003.

[5] R. Yang and M. Pollefeys. Multi-Resolution Real-TimeStereo on Commodity Graphics Hardware. IEEE CVPR 2003,pages 211–217, June 2003.

256Proceedings of the 2004 Virtual Reality (VR’04) 1087-8270/04 $ 20.00 IEEE