Upload
h
View
212
Download
0
Embed Size (px)
Citation preview
Focus Measurement on Programmable Graphics Hardwarefor All in-Focus Rendering from Light Fields
Kaoru Sugita, Keita Takahashi, Takeshi Naemura and Hiroshi HarashimaSchool of Information Science and Technology, The University of Tokyo
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan{sugita, keita, naemura, hiro}@hc.t.u-tokyo.ac.jp
Abstract
This paper deals with a method for interactive renderingof photorealistic images, which is a fundamental technologyin the field of virtual reality. Since the latest graphics pro-cessing units (GPUs) are programmable, they are expectedto be useful for various applications including numericalcomputation and image processing. This paper proposes amethod for focus measurement on light field rendering usinga GPU as a fast processing unit for image processing andimage-based rendering. It is confirmed that the proposedmethod enables interactive all in-focus rendering from lightfields. This is because the latest DirectX 9 generation GPUsare much faster than CPUs in solving optimization prob-lems, and a GPU implementation can eliminate the latencyfor data transmission between video memory and systemmemory. Experimental results show that the GPU imple-mentation outperforms its CPU implementation.
1 Introduction
The interactivity and the photorealism are essential formost of the virtual reality applications. The aim of this pa-per is to add the interactivity to a photo-realistic view syn-thesis system [4], which is based on the light field renderingmethod.
Light field rendering (LFR) can generate free-viewpointimages from a dense set of reference images with no / littlegeometries. But it suffers from a focus like effect caused bythe undersampling of the reference images [2]. Defocusedregions in synthetic images are damaged with blurring andghosting artifacts. Takahashi et al.[4] proposed a method forsolving this problem. Their approach is to generate an allin focus image (free from blurring and ghosting) from a se-ries of differently-focused images which are synthesized byLFR from an undersampled light field. The optimal focuseddepth is selected for each pixel by the focus measurement
(depth-search), but this process is much time-consuming.This paper explains novel techniques to implement their
method [4] on the RADEON 9800 Pro GPU. By implement-ing all the processing on a GPU, we can derive the bene-fit of parallelism of the processing, as well as eliminationof the latency for data transmission between video memoryand system memory. Experimental results give quantitativecomparisons of processing time between GPU and CPU im-plementations.
2 DirectX 9 generation GPUs
Recently, graphics processing units (GPUs) as ATIRADEON series and NVIDIA GeForce FX series whichsupport DirectX 9 are distributed as commodity peripheralequipments of desk-top PCs. The features of these GPUsare programmable rendering pipeline and floating point pre-cision computation as well as faster polygon filling oper-ation. These features allow GPUs to extend the range ofapplications [1]. For example, Yang and Pollefeys [5] pro-posed an algorithm for multi-resolution depth from stereousing GPUs.
3 Optimal focal depth search on GPU
Figure 1 shows the procedure for generating all in-focusimages from the series of multi focus images. Takahashi etal. [4] use a CPU for performing depth-search. In this case,multi-focus images must be transferred from video mem-ory to system memory. Our proposal is to perform entireprocessing on a GPU.
In the original algorithm [4], block filters are used forsmoothing. For the purpose of fast processing on a GPU,we can use “mip-map function” based multi-resolution op-eration [5]. Moreover, optimization can be performedby “depth replace function” [3]. Therefore we adoptmulti-resolution optimization method for depth-search. It
255
IEEE Virtual Reality 2004 March 27-31, Chicago, IL USA 0-7803-8415-6/04/$20.00©2004 IEEE.
Proceedings of the 2004 Virtual Reality (VR’04) 1087-8270/04 $ 20.00 IEEE
Multi-view
point image
Multi-focus images reconstructed
with three different filters
Evaluation image
(difference)
All in-focus image
Wide
Normal
SkipSearch for
optimal focal distance
for each pixel
Multi.Depth-search
Diff. SearchDisplay
Read
Figure 1. The data flow of the method [4].
Table 1. Total processing time (ms).Total Multi. Depth-search Display
CPU 3170 355 2800 11.5GPU 412 53.8 3.17
Table 2. Processing time for depth-search(ms).
Total Read Diff. SearchCPU 2800 1300 91.4 1410GPU 53.8 — 16.1 37.7
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10111213141516
The number of focal planes N
Pro
cessing
tim
e rati
os
Multi.
Depth-search (CPU)
Depth-search (GPU)
Display (CPU)
Display (GPU)
Figure 2. The processing time ratio of eachitem in Table 1.
searches the optimal value, raising resolution by ”coarse-to-fine” basis. The effect of smoothing is acquired by usingthe solution at the lower resolution as the restriction condi-tion for the higher resolution operation.
4 Experiments
Our implementation environment is a PC with an IntelPentium 4 3.06GHz CPU and an ATI RADEON 9800 ProGPU (driver version is Catalyst 3.6). All C++ programs arecompiled with Intel C++ compiler version 7.1. Referenceimage set we use is the multiview image database courtesyof University of Tsukuba, Japan. Input images are arrangedin the resolution 256× 192 pixels of 9× 9 viewpoints. Theresolution of output image is 512 × 512 pixels.
Table 1 shows measured total processing time. “GPU” isthe proposed method. “CPU” is the same algorithm imple-
mented using a CPU. This shows that the GPU implementa-tion is about 7.7 times faster than the CPU implementation.And in “CPU” case, the processing time of depth-search oc-cupy the great portion of processing time.
Table 2 shows measured processing time for depth-search. In “CPU” case, data transfer from video memoryto system memory occupies 41.0 % of the total processingtime. This operation is not required in “GPU” case. In ad-dition, in “CPU” case, the processing time of searching is37.4 times longer than “GPU” case. These lead to increasethe processing time of “CPU” case compared with “GPU”case.
Figure 2 shows variation of the processing time. Eachvalue is expressed in the ratios. And the ratio is set to be onewhen the number of focal planes N = 1. This graph showsthat the great portion of the total processing time (Multi.and depth-search on a GPU) are mostly proportional to N .Therefore the reduction of N leads to improvement of in-teractivity.
5 Discussion
In this experiment, constants such as N etc. are deter-mined heuristically. Our future work will be on techniquesto determine suitable parameters for dynamic scenes, suchas video sequences acquired by multi-camera systems.
References
[1] M. J. Harris. GPGPU General Purpose Computation UsingGraphics Hardware. http://www.gpgpu.org/
[2] J. Stewart et al. A New Reconstruction Filter for Undersam-pled Light Fields. EGSR 2003, pages 150–156, Sep. 2003.
[3] K. Sugita et al. Performance Evaluation of ProgrammableGraphics Hardware for Image Filtering and Stereo Matching.ACM VRST 2003, pages 176–183, Oct. 2003.
[4] K. Takahashi et al. All in-Focus View Synthesis from Under-Sampled Light Fields. ICAT 2003, pages 249–256, Dec. 2003.
[5] R. Yang and M. Pollefeys. Multi-Resolution Real-TimeStereo on Commodity Graphics Hardware. IEEE CVPR 2003,pages 211–217, June 2003.
256Proceedings of the 2004 Virtual Reality (VR’04) 1087-8270/04 $ 20.00 IEEE