Segment-then-Splat

Open-vocabulary querying in 3D space is crucial for enabling more intelligent perception in applications such as robotics, autonomous systems, and augmented reality. However, most existing methods rely on 2D pixel-level parsing, leading to multi-view inconsistencies and poor 3D object retrieval. Moreover, they are limited to static scenes and struggle with dynamic scenes due to the complexities of motion modeling. In this paper, we propose Segment-then-Splat, a 3D-aware open vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. Segment-then-Splat reverses the long established approach of segmentation after reconstruction by dividing Gaussians into distinct object sets before reconstruction. Once the reconstruction is complete, the scene is naturally segmented into individual objects, achieving true 3D segmentation. This approach not only eliminates Gaussian-object misalignment issues in dynamic scenes but also accelerates the optimization process, as it eliminates the need for learning a separate language field. After optimization, a CLIP embedding is assigned to each object to enable open-vocabulary querying. Extensive experiments on various datasets demonstrate the effectiveness of our proposed method in both static and dynamic scenarios.

Comparison of our Segment-then-Splat pipeline and the traditional Splat-then-Segment approach. (a) The traditional Splat-then-Segment pipeline learns a language field alongside the reconstruction of the scene. During object queries, it renders Gaussian language embeddings into 2D feature maps to identify relevant pixels based on the input text embedding. (b) In contrast, our Segment-then-Splat pipeline first initialize Gaussians into object-specific sets before reconstruction, ensuring a precise object-Gaussian correspondence and better segmentation accuracy.

A detailed demonstration of our proposed Segment then Splat pipeline. Our approach first extracts multi-view masks for each object through a robust tracking module, then object IDs are assigned to each initial Gaussian based on these masks, forming distinct object-specific sets. During optimization, object specific loss \(\mathcal{L}_{\text{obj}}\) is used to enforce Gaussian-object correspondence and thus resulting in more accurate object geometries. Finally, we associate each Gaussian group with CLIP embeddings, enabling open-vocabulary queries.

Citation

If you find our work helpful, please consider cite us:

@misc{lu2025segmentsplatunifiedapproach,
      title={Segment then Splat: A Unified Approach for 3D Open-Vocabulary Segmentation based on Gaussian Splatting}, 
      author={Yiren Lu and Yunlai Zhou and Yiran Qiao and Chaoda Song and Tuo Liang and Jing Ma and Yu Yin},
      year={2025},
      eprint={2503.22204},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2503.22204}, 
}

Segment-then-Splat: A Unified Approach for 3D Open-Vocabulary Segmentation based on Gaussian Splatting

Abstract

Segment-then-Splat V.S. Splat-then-Segment

Pipeline Overview

3D Segmentation V.S. Pixel-based Segmentation

Comparison of 3D Segmentation on Static Scene

Comparison of 3D Segmentation on Dynamic Scene

Citation