In this paper, we introduce Splatt3R, a pose-free, feed-forward method for in-the-wild 3D reconstruction and novel view synthesis from stereo pairs. Given uncalibrated natural images, Splatt3R can predict 3D Gaussian Splats without requiring any camera parameters or depth information. For generalizability, we build Splatt3R upon a foundation 3D geometry reconstruction method, MASt3R, by extending it to deal with both 3D structure and appearance. Specifically, unlike the original MASt3R which reconstructs only 3D point clouds, we predict the additional Gaussian attributes required to construct a Gaussian primitive for each point. Hence, unlike other novel view synthesis methods, Splatt3R is first trained by optimizing the 3D point cloud's geometry loss, and then a novel view synthesis objective. By doing this, we avoid the local minima present in training 3D Gaussian Splats from stereo views. We also propose a novel loss masking strategy that we empirically find is critical for strong performance on extrapolated viewpoints. We train Splatt3R on the ScanNet++ dataset and demonstrate excellent generalisation to uncalibrated, in-the-wild images. Splatt3R can reconstruct scenes at 4FPS at 512 x 512 resolution, and the resultant splats can be rendered in real-time.
在本文中，我们介绍了Splatt3R，这是一种无姿态、前馈方法，用于从立体图像对中进行野外场景的3D重建和新视角合成。在给定未校准的自然图像的情况下，Splatt3R可以预测3D高斯斑点，而不需要任何相机参数或深度信息。为了提高泛化能力，我们在一个“基础”3D几何重建方法MASt3R的基础上构建了Splatt3R，通过扩展它来处理3D结构和外观。具体来说，与原始MASt3R仅重建3D点云不同，我们预测了构造每个点所需的额外高斯属性。因此，与其他新视角合成方法不同，Splatt3R首先通过优化3D点云的几何损失进行训练，然后再进行新视角合成目标的训练。通过这种方式，我们避免了训练3D高斯斑点时存在的局部最小值问题。我们还提出了一种新颖的损失掩蔽策略，我们通过实验证明，这对在外推视点上获得强性能至关重要。我们在ScanNet++数据集上训练了Splatt3R，并展示了它在未校准的野外图像中的优异泛化能力。Splatt3R可以以512 x 512分辨率下每秒4帧的速度重建场景，并且生成的斑点可以实时渲染。