Depth Map Super-Resolution by Deep Multi-Scale Guidance
Depth boundaries often lose sharpness when upsampling from low-resolution (LR) depth maps especially at large upscaling factors. We present a new method to address the problem of depth map super resolution in which a high-resolution (HR) depth map is infe
- PDF / 3,909,443 Bytes
- 17 Pages / 439.37 x 666.142 pts Page_size
- 13 Downloads / 243 Views
Department of Information Engineering, The Chinese University of Hong Kong, Sha Tin, Hong Kong {twhui,ccloy,xtang}@ie.cuhk.edu.hk 2 Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
Abstract. Depth boundaries often lose sharpness when upsampling from low-resolution (LR) depth maps especially at large upscaling factors. We present a new method to address the problem of depth map super resolution in which a high-resolution (HR) depth map is inferred from a LR depth map and an additional HR intensity image of the same scene. We propose a Multi-Scale Guided convolutional network (MSG-Net) for depth map super resolution. MSG-Net complements LR depth features with HR intensity features using a multi-scale fusion strategy. Such a multi-scale guidance allows the network to better adapt for upsampling of both fine- and large-scale structures. Specifically, the rich hierarchical HR intensity features at different levels progressively resolve ambiguity in depth map upsampling. Moreover, we employ a highfrequency domain training method to not only reduce training time but also facilitate the fusion of depth and intensity features. With the multiscale guidance, MSG-Net achieves state-of-art performance for depth map upsampling.
1
Introduction
The use of depth information of a scene is essential in many applications such as autonomous navigation, 3D reconstruction, human-computer interaction and virtual reality. The introduction of low-cost depth camera facilitates the use of depth information in our daily life. However, the resolution of depth maps which is provided in a low-cost depth camera is generally very limited. To facilitate the use of depth data, we often need to address an upsampling problem in which the corresponding high-resolution (HR) depth map is recovered from a given low-resolution (LR) depth map. Depth map super-resolution is a non-trivial task. Specifically, fine structures in HR image are either lost or severely distorted (depending on the scale factor used) in LR image because they cannot be fully represented by the limited spatial resolution. A brute-force upsampling of LR image simply causes those structures which are supposed to have sharp boundaries become blurred in the upsampled image. Ambiguity in super-resolving the severely distorted fine structures often c Springer International Publishing AG 2016 B. Leibe et al. (Eds.): ECCV 2016, Part III, LNCS 9907, pp. 353–369, 2016. DOI: 10.1007/978-3-319-46487-9 22
354
T.-W. Hui et al.
(a)
(b)
(c)
(d)
(e)
Fig. 1. Ambiguity in upsampling depth map. (a) Color image. (b) Ground truth. (c) (Enlarged) LR depth map downsampled by a factor of 8. Results for upsampling: (d) SRCNN [11], (e) Our solution without ambiguity problem.
(a)
(b)
(c)
(d)
(e)
Fig. 2. Over-texture transfer in depth map refinement and upsampling using intensity guidance. (a) Color image. (b) Ground truth. (c) Refinement of (b) using (a) by Guided Filtering [8] (r = 4, = 0.012 ). Results of using (a) to guide the 2× upsampling of (b): (d) Fers
Data Loading...
