3DGStream: On-the-fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos
Constructing photo-realistic Free-Viewpoint Videos (FVVs) of dynamic scenes from multi-view videos remains a challenging endeavor. Despite the remarkable advancements achieved by current neural rendering techniques, these methods generally require complete video sequences for offline training and are not capable of real-time rendering. To address these constraints, we introduce 3DGStream, a method designed for efficient FVV streaming of real-world dynamic scenes. Our method achieves fast on-the-fly per-frame reconstruction within 12 seconds and real-time rendering at 200 FPS. Specifically, we utilize 3D Gaussians (3DGs) to represent the scene. Instead of the naïve approach of directly optimizing 3DGs per-frame, we employ a compact Neural Transformation Cache (NTC) to model the translations and rotations of 3DGs, markedly reducing the training time and storage required for each FVV frame. Furthermore, we propose an adaptive 3DG addition strategy to handle emerging objects in dynamic scenes. Experiments demonstrate that 3DGStream achieves competitive performance in terms of rendering speed, image quality, training time, and model storage when compared with state-of-the-art methods.
构建动态场景的逼真自由视点视频（FVVs）仍然是一项挑战性的任务。尽管当前神经渲染技术取得了显著进步，但这些方法通常需要完整的视频序列进行离线训练，并且无法实现实时渲染。为了解决这些限制，我们引入了3DGStream，一种为真实世界动态场景高效FVV流媒体设计的方法。我们的方法实现了快速的即时逐帧重建，在12秒内完成，并能以200 FPS的速度实时渲染。具体来说，我们使用3D高斯（3DGs）来表示场景。我们没有采用直接优化每帧3DGs的简单方法，而是采用了一个紧凑的神经转换缓存（NTC）来模拟3DGs的平移和旋转，显著减少了每个FVV帧所需的训练时间和存储空间。此外，我们提出了一种自适应3DG添加策略来处理动态场景中出现的新对象。实验表明，与最先进的方法相比，3DGStream在渲染速度、图像质量、训练时间和模型存储方面都达到了有竞争力的性能。