Deep Learning Based Fence Segmentation and Removal from an Image Using a Video Sequence
Conventional approaches to image de-fencing use multiple adjacent frames for segmentation of fences in the reference image and are limited to restoring images of static scenes only. In this paper, we propose a de-fencing algorithm for images of dynamic sc
- PDF / 7,695,795 Bytes
- 16 Pages / 439.37 x 666.142 pts Page_size
- 16 Downloads / 219 Views
2
Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, Kharagpur, India [email protected] Department of Electrical Engineering, Indian Institute of Technology Kharagpur, Kharagpur, India [email protected], [email protected]
Abstract. Conventional approaches to image de-fencing use multiple adjacent frames for segmentation of fences in the reference image and are limited to restoring images of static scenes only. In this paper, we propose a de-fencing algorithm for images of dynamic scenes using an occlusion-aware optical flow method. We divide the problem of image de-fencing into the tasks of automated fence segmentation from a single image, motion estimation under known occlusions and fusion of data from multiple frames of a captured video of the scene. Specifically, we use a pre-trained convolutional neural network to segment fence pixels from a single image. The knowledge of spatial locations of fences is used to subsequently estimate optical flow in the occluded frames of the video for the final data fusion step. We cast the fence removal problem in an optimization framework by modeling the formation of the degraded observations. The inverse problem is solved using fast iterative shrinkage thresholding algorithm (FISTA). Experimental results show the effectiveness of proposed algorithm. Keywords: Image inpainting · De-fencing tional neural networks · Optical flow
1
·
Deep learning
·
Convolu-
Introduction
Images containing fences/occlusions occur in several situations such as photographing statues in museums, animals in a zoo etc. Image de-fencing involves the removal of fences or occlusions in images. De-fencing a single photo is strictly an image inpainting problem which uses data in the regions neighbouring fence pixels in the frame for filling-in occlusions. The works of [1–4] addressed the image inpainting problem wherein a portion of the image which is to be inpainted is specified by a mask manually. As shown in Fig. 1(a), in the image de-fencing problem it is difficult to manually mark all fence pixels since they are numerous c Springer International Publishing Switzerland 2016 G. Hua and H. J´ egou (Eds.): ECCV 2016 Workshops, Part III, LNCS 9915, pp. 836–851, 2016. DOI: 10.1007/978-3-319-49409-8 68
Deep Learning Based Fence Segmentation and Removal from an Image
(a)
(b)
(c)
837
(d)
Fig. 1. (a) A frame taken from a video. (b) Segmented binary fence mask obtained using proposed CNN-SVM algorithm. (c) Inpainted image corresponding to (a) using the method of [2]. (d) De-fenced image corresponding to (a) using the proposed algorithm.
and spread over the entire image. The segmented binary fence mask obtained using the proposed algorithm is shown in Fig. 1(b). These masks are used in our work to aid in occlusion-aware optical flow computation and background image reconstruction. In Fig. 1(c), we show the inpainted image corresponding to Fig. 1(a) obtained using the method of [2]. The de-fenced image obtained using the proposed algorithm is shown in F
Data Loading...
