We propose a Polarization Video Glass Segmentation network (PGVS-Net), which can exploit historical multi-view spectral
polarization cues to segment glass areas. The video results are shown below.
Abstract
In this paper, we present the first polarization-guided video glass segmentation propagation solution (PGVS-Net) that can robustly and coherently propagate glass segmentation in RGB-P video sequences. By leveraging spatio-temporal polarization and color information, our method effectively combines multi-view polarization cues and thus can alleviate the view dependence of single-input intensity variations on glass objects. We demonstrate that our model can outperform glass segmentation on RGB-only video sequences as well as produce more robust segmentation than per-frame RGB-P single-image segmentation methods. To train and validate PGVS-Net, we introduce the first RGB-P Glass Video dataset, termed PGV-117, of 117 video sequences, which will be publicly released upon publication.
Method
The RGB-P Memory and PGI fused keys are designed to store long-range multi-view polarization cues. The short-range spectral propagation are implemented by the CMTA and PTF modules.
Our Dataset
Summary of camera motions (left) and the PGV-117 dataset characteristics (right). To avoid biasing PGV-117 into certain types of camera motions, we implement different camera motion strategies. The blue areas represent the glass target. The 117 video sequences are captured with different types of camera paths, lighting conditions, dynamics, and materials (dark indicates ground glass).
Video Results
BibTex
coming soon