Free-DyGS: Camera-Pose-Free Scene Reconstruction based on Gaussian Splatting for Dynamic Surgical Videos
Reconstructing endoscopic videos is crucial for high-fidelity visualization and the efficiency of surgical operations. Despite the importance, existing 3D reconstruction methods encounter several challenges, including stringent demands for accuracy, imprecise camera positioning, intricate dynamic scenes, and the necessity for rapid reconstruction. Addressing these issues, this paper presents the first camera-pose-free scene reconstruction framework, Free-DyGS, tailored for dynamic surgical videos, leveraging 3D Gaussian splatting technology. Our approach employs a frame-by-frame reconstruction strategy and is delineated into four distinct phases: Scene Initialization, Joint Learning, Scene Expansion, and Retrospective Learning. We introduce a Generalizable Gaussians Parameterization module within the Scene Initialization and Expansion phases to proficiently generate Gaussian attributes for each pixel from the RGBD frames. The Joint Learning phase is crafted to concurrently deduce scene deformation and camera pose, facilitated by an innovative flexible deformation module. In the scene expansion stage, the Gaussian points gradually grow as the camera moves. The Retrospective Learning phase is dedicated to enhancing the precision of scene deformation through the reassessment of prior frames. The efficacy of the proposed Free-DyGS is substantiated through experiments on two datasets: the StereoMIS and Hamlyn datasets. The experimental outcomes underscore that Free-DyGS surpasses conventional baseline models in both rendering fidelity and computational efficiency.
重建内窥镜视频对于高保真可视化和手术操作的效率至关重要。尽管如此，现有的 3D 重建方法面临诸多挑战，包括对准确性的严格要求、相机定位不精确、复杂的动态场景以及快速重建的必要性。针对这些问题，本文提出了首个无相机姿态的场景重建框架 Free-DyGS，专为动态手术视频设计，利用 3D 高斯点喷射技术。我们的方法采用逐帧重建策略，并分为四个不同的阶段：场景初始化、联合学习、场景扩展和回顾学习。在场景初始化和扩展阶段，我们引入了通用高斯参数化模块，以高效地从 RGBD 帧生成每个像素的高斯属性。联合学习阶段旨在通过创新的灵活变形模块同时推导场景变形和相机姿态。在场景扩展阶段，随着相机移动，高斯点逐渐增长。回顾学习阶段则专注于通过重新评估先前的帧来提高场景变形的精度。通过对 StereoMIS 和 Hamlyn 数据集的实验验证，证明了 Free-DyGS 在渲染保真度和计算效率上超越了传统基线模型。