ACM MULTIMEDIA 2026 / RIO DE JANEIRO

Filling the Unseen

Scene Extrapolation via 3D Gaussian Splatting

Yunlai Zhou Yiren Lu Tuo Liang Disheng Liu Vipin Chaudhary Yu Yin†

Case Western Reserve University

† Corresponding author

Paper Code Video Explore results

Beyond the observed scene.

Drag to reveal
Our completed Bonsai scene, with the missing upper wall reconstructed and the tripod refined. Original 3DGS rendering of Bonsai, with missing geometry above the training views. Original 3DGSFilling the Unseen

Expand into unobserved regions.
Preserve the details already there.

Actual renderings from the paper. Black regions indicate missing content in the original reconstruction.

THE IDEA

A more complete world,
from an incomplete capture.

Novel views reveal what reconstruction leaves behind. We expand a 3D scene beyond its observed boundaries, while improving the quality of what is already visible.

01

Expand in parallel

Detect independent camera views in 3D, then generate and reconstruct non-overlapping regions in a single pass.

02

Complete the gaps

Fill the remaining regions between independent views with a second stage of residual completion.

03

Refine selectively

Use Quality-Aware Masks to guide scene updates toward under-reconstructed areas and protect existing detail.

Abstract

3D Gaussian Splatting achieves photorealistic reconstruction within the training view distribution, yet degrades on out-of-distribution novel views, exhibiting holes in unobserved regions and artifacts in observable areas. Recent works formulate this task as extrapolation and interpolation and try to address it with generative models, but remain limited in extrapolation scale and quality. They repeat a generate–reconstruct–shift cycle to progressively build a scene, which introduces accumulated errors with every step conditioning on previous outcomes.

In this work, we propose a holistic framework for extrapolation and interpolation. We devise an independent camera view detection mechanism to enable parallel, conflict-free extrapolation, circumventing reliance on the aforementioned error-prone cycle. Building upon this, we design a hierarchical pipeline that extrapolates independent and dependent camera views separately. Additionally, previous methods overlook inconsistency between generated and original images, compromising well-reconstructed areas. We propose a plug-and-play Quality-Aware Mask (QA-Mask) module, enabling selective utilization of generated data. By calibrating learning weights with pixel-wise rendering quality, it prevents generation-induced degradation in well-reconstructed areas. Extensive experiments demonstrate the superior performance of our framework, with QA-Mask generalizing across multiple generative reconstruction models.

SEE IT IN MOTION

Scene extrapolation results

SUPPLEMENTARY VIDEO · 1080P

Continuous camera trajectories on Mip-NeRF 360 and ScanNet++, with side-by-side baseline comparisons. The original supplementary video is silent; method names and additional render-time processing are labeled in the video.

Download video

QUALITATIVE COMPARISONS

A closer look at the unseen

Choose a scene, a view, and a method.
Drag the divider to compare.

View
Bonsai, view 1: Filling the Unseen. Bonsai, view 1: GenFusion. GenFusionOurs

Bonsai · Mip-NeRF 360 · View 1

Full comparison figure

Images are extracted directly from the manuscript figures at their original resolution. Mip-NeRF 360 has no ground-truth images for these extrapolation views; ScanNet++ uses a controlled partial-scene reconstruction protocol.

HOW IT WORKS

Two stages. One coherent scene.

A hierarchy of independent and dependent camera views reduces repeated generation cycles and the errors they accumulate.

Pipeline: independently inpaint camera views and generate videos in stage one, propagate results with depth, then complete dependent views in stage two. Video depth alignment provides depth priors.
Independent views support parallel generation and reconstruction. Dependent views complete the remaining gaps. DIBR-based video depth alignment supplies geometry priors throughout the process.

STAGE 01

Parallel extrapolation

Mesh-based frustum collision detection identifies views whose inpainted regions do not overlap. Each view is inpainted independently, then connected to original training views with generated videos.

STAGE 02

Residual completion

Depth-based propagation carries the generated content to intermediate views. A second pass fills the remaining gaps and connects neighboring views along a closed trajectory.

FINAL REFINEMENT

Scene interpolation

Reuse the videos generated during extrapolation and incorporate artifact-removal outputs to improve rendering within the original scene. QA-Mask guides where updates should happen.

How are independent camera views detected?
Mesh-based view frustums enclosing inpainted geometry. Non-colliding frustums identify independent views; colliding frustums identify dependent views.

For each candidate view, inpainting and depth estimation define a probable depth range for the missing region. A mesh frustum encloses that region in 3D. Non-colliding frustums indicate independent camera views, enabling parallel generation without overlapping generated geometry.

QUALITY-AWARE MASK

Know where to generate.
And where to preserve.

Generated images can improve missing regions while unintentionally changing well-reconstructed ones. QA-Mask makes scene updates selective.

QA-Mask initializes Gaussian base colors to red, trains only view-dependent spherical harmonic coefficients against black training images, and renders a mask that distinguishes supervised from under-supervised directions.
Spherical harmonics model which viewing directions have been supervised. The rendered mask spatially reweights the reconstruction loss so that generated data contributes where it is needed.
PLUG-AND-PLAY MODULE

Better preservation.
Across different models.

QA-Mask improves the quality of existing regions when integrated into both GenFusion and DiFix3D+, while retaining comparable extrapolation performance.

GenFusion existing-region PSNR26.49 28.33 dB
DiFix3D+ existing-region PSNR28.11 28.58 dB

ScanNet++ results from the paper. Arrows show baseline → baseline with QA-Mask.

GenFusion and DiFix3D+ results without QA-Mask on the left and with QA-Mask on the right. The mask preserves desks, cabinets, edges, and small details.

QUANTITATIVE EVALUATION

Measured across two benchmarks

Perceptual quality, consistency,
and reconstruction fidelity.

Reference metrics averaged across 10 ScanNet++ scenes. Best values are highlighted.
MethodPSNR ↑SSIM ↑LPIPS ↓
Few-Shot GS18.920.800.3438
DiFix3D+20.930.840.2554
GenFusion21.930.850.2502
Guidedvd-3dgs22.100.840.2662
Filling the Unseen Ours23.790.880.2300

Controlled evaluation: reconstruct half of each scene and extrapolate into the held-out half. Original images provide ground truth; all methods use ground-truth images as inpainting targets in this setting.

↑ Higher is better. ↓ Lower is better. Runtime for our method includes both extrapolation stages, interpolation, and QA-Mask training.

BIBTEX

Citation

If you find this work useful for your research, please consider citing:

@inproceedings{zhou2026filling,
  title     = {Filling the Unseen: Scene Extrapolation via
               3D Gaussian Splatting},
  author    = {Zhou, Yunlai and Lu, Yiren and Liang, Tuo and
               Liu, Disheng and Chaudhary, Vipin and Yin, Yu},
  booktitle = {Proceedings of the 34th ACM International
               Conference on Multimedia},
  series    = {MM '26},
  year      = {2026},
  doi       = {10.1145/3767308.3835425}
}

Open original PDF ↗