Abstract
Open-vocabulary querying in 3D space is crucial for enabling more intelligent perception in applications such as robotics, autonomous systems, and augmented reality. However, most existing methods rely on 2D pixel-level parsing, leading to multi-view inconsistencies and poor 3D object retrieval. Moreover, they are limited to static scenes and struggle with dynamic scenes due to the complexities of motion modeling. In this paper, we propose Segment-then-Splat, a 3D-aware open vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. Segment-then-Splat reverses the long established approach of segmentation after reconstruction by dividing Gaussians into distinct object sets before reconstruction. Once the reconstruction is complete, the scene is naturally segmented into individual objects, achieving true 3D segmentation. This approach not only eliminates Gaussian-object misalignment issues in dynamic scenes but also accelerates the optimization process, as it eliminates the need for learning a separate language field. After optimization, a CLIP embedding is assigned to each object to enable open-vocabulary querying. Extensive experiments on various datasets demonstrate the effectiveness of our proposed method in both static and dynamic scenarios.
Segment-then-Splat V.S. Splat-then-Segment
Comparison of our Segment-then-Splat pipeline and the traditional Splat-then-Segment approach. (a) The traditional Splat-then-Segment pipeline learns a language field alongside the reconstruction of the scene. During object queries, it renders Gaussian language embeddings into 2D feature maps to identify relevant pixels based on the input text embedding. (b) In contrast, our Segment-then-Splat pipeline first initialize Gaussians into object-specific sets before reconstruction, ensuring a precise object-Gaussian correspondence and better segmentation accuracy.
Pipeline Overview
A detailed demonstration of our proposed Segment then Splat pipeline. Our approach first extracts multi-view masks for each object through a robust tracking module, then object IDs are assigned to each initial Gaussian based on these masks, forming distinct object-specific sets. During optimization, object specific loss \(\mathcal{L}_{\text{obj}}\) is used to enforce Gaussian-object correspondence and thus resulting in more accurate object geometries. Finally, we associate each Gaussian group with CLIP embeddings, enabling open-vocabulary queries.
3D Segmentation V.S. Pixel-based Segmentation
Comparison of 3D Segmentation on Static Scene
Comparison of 3D Segmentation on Dynamic Scene
Citation
If you find our work helpful, please consider cite us:
@inproceedings{lusegment,
title={Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting},
author={Lu, Yiren and Zhou, Yunlai and Qiao, Yiran and Song, Chaoda and Liang, Tuo and Ma, Jing and Wang, Huan and Yin, Yu},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems}
}