We present an efficient neural 3D scene representation for novel-view synthesis (NVS) in large-scale, dynamic urban areas. Existing works are not well suited for applications like mixed-reality or closed-loop simulation due to their limited visual quality and non-interactive rendering speeds. Recently, rasterization-based approaches have achieved high-quality NVS at impressive speeds. However, these methods are limited to small-scale, homogeneous data, i.e. they cannot handle severe appearance and geometry variations due to weather, season, and lighting and do not scale to larger, dynamic areas with thousands of images. We propose 4DGF, a neural scene representation that scales to large-scale dynamic urban areas, handles heterogeneous input data, and substantially improves rendering speeds. We use 3D Gaussians as an efficient geometry scaffold while relying on neural fields as a compact and flexible appearance model. We integrate scene dynamics via a scene graph at global scale while modeling articulated motions on a local level via deformations. This decomposed approach enables flexible scene composition suitable for real-world applications. In experiments, we surpass the state-of-the-art by over 3 dB in PSNR and more than 200 times in rendering speed.
我们提出了一种高效的神经3D场景表示方法，用于大规模动态城市区域的新视角合成（NVS）。现有工作由于视觉质量有限和渲染速度不具交互性，不适合混合现实或闭环模拟等应用。最近，基于光栅化的方法在保证高质量NVS的同时实现了令人印象深刻的速度。然而，这些方法仅限于小规模、同质化数据，即它们不能处理由于天气、季节和照明引起的严重外观和几何变化，也不适用于包含数千张图片的更大、动态的区域。我们提出了4DGF，这是一种神经场景表示，可以扩展到大规模动态城市区域，处理异质输入数据，并显著提高渲染速度。我们使用3D高斯作为高效的几何支架，同时依赖神经场作为一种紧凑且灵活的外观模型。我们通过全局尺度的场景图集成场景动态，同时通过变形在局部层面模拟关节运动。这种分解方法使场景组合灵活，适合实际应用。在实验中，我们的方法在峰值信噪比（PSNR）上超过了现有技术3dB以上，并在渲染速度上提高了200倍以上。