Streaming Video Segmentation via Short-Term Hierarchical Segmentation and Frame-by-Frame Markov Random Field Optimizatio
An online video segmentation algorithm, based on short-term hierarchical segmentation (STHS) and frame-by-frame Markov random field (MRF) optimization, is proposed in this work. We develop the STHS technique, which generates initial segments by sliding a
- PDF / 2,300,561 Bytes
- 17 Pages / 439.37 x 666.142 pts Page_size
- 77 Downloads / 250 Views
Abstract. An online video segmentation algorithm, based on shortterm hierarchical segmentation (STHS) and frame-by-frame Markov random field (MRF) optimization, is proposed in this work. We develop the STHS technique, which generates initial segments by sliding a short window of frames. In STHS, we apply spatial agglomerative clustering to each frame, and then adopt inter-frame bipartite graph matching to construct initial segments. Then, we partition each frame into final segments, by minimizing an MRF energy function composed of unary and pairwise costs. We compute the unary cost using the STHS initial segments and the segmentation result at the previous frame. We set the pairwise cost to encourage similar nodes to have the same segment label. Experimental results on a video segmentation benchmark dataset, VSB100, demonstrate that the proposed algorithm outperforms state-of-the-art online video segmentation techniques significantly. Keywords: Video segmentation · Online segmentation · Streaming segmentation · Agglomerative clustering · Graph matching
1
Introduction
Segmentation, the task of partitioning data into disjoint subsets based on the underlying data structure, is one of the most fundamental problems in computer vision. For image segmentation, contour-based algorithms [1,2] have achieved great success recently. As the state-of-the-art contour detector [3] presents comparable performance to the human visual system, the contour-based image segmentation can provide more promising performance. On the other hand, video segmentation is the process to divide a video into volumetric segments. It is applicable to a wide variety of vision applications, such as action recognition, scene classification, video summarization, content-based video retrieval, and 3D reconstruction. However, video segmentation still remains a challenging problem due to object and camera motion, occlusion, and contour ambiguities. To overcome these issues, many attempts have been made. Video segmentation algorithms can be categorized into offline or online ones. Offline algorithms [4–10] divide a video into segments by processing all frames c Springer International Publishing AG 2016 B. Leibe et al. (Eds.): ECCV 2016, Part VI, LNCS 9910, pp. 599–615, 2016. DOI: 10.1007/978-3-319-46466-4 36
600
W.-D. Jang and C.-S. Kim
at once. On the other hand, online (or streaming) algorithms [11–13] extract segments sequentially from the first to the last frames. Note that the offline algorithms can achieve more accurate segmentation by exploiting the entire information in a video jointly, but they require huge memory space for a long video. Thus, the online algorithms, which use regular memory space regardless of the duration of a video, can be used more versatilely in practical applications. We propose a novel online video segmentation algorithm. The proposed algorithm consists of two steps: short-term hierarchical segmentation (STHS) and Markov random field (MRF) optimization. In the first pass, STHS generates initial segments sequentially, by sliding a s
Data Loading...
