GaussianPU: A Hybrid 2D-3D Upsampling Framework for Enhancing Color Point Clouds via 3D Gaussian Splatting
Dense colored point clouds enhance visual perception and are of significant value in various robotic applications. However, existing learning-based point cloud upsampling methods are constrained by computational resources and batch processing strategies, which often require subdividing point clouds into smaller patches, leading to distortions that degrade perceptual quality. To address this challenge, we propose a novel 2D-3D hybrid colored point cloud upsampling framework (GaussianPU) based on 3D Gaussian Splatting (3DGS) for robotic perception. This approach leverages 3DGS to bridge 3D point clouds with their 2D rendered images in robot vision systems. A dual scale rendered image restoration network transforms sparse point cloud renderings into dense representations, which are then input into 3DGS along with precise robot camera poses and interpolated sparse point clouds to reconstruct dense 3D point clouds. We have made a series of enhancements to the vanilla 3DGS, enabling precise control over the number of points and significantly boosting the quality of the upsampled point cloud for robotic scene understanding. Our framework supports processing entire point clouds on a single consumer-grade GPU, such as the NVIDIA GeForce RTX 3090, eliminating the need for segmentation and thus producing high-quality, dense colored point clouds with millions of points for robot navigation and manipulation tasks. Extensive experimental results on generating million-level point cloud data validate the effectiveness of our method, substantially improving the quality of colored point clouds and demonstrating significant potential for applications involving large-scale point clouds in autonomous robotics and human-robot interaction scenarios.
密集的彩色点云能够提升视觉感知，并在各种机器人应用中具有重要价值。然而，现有的基于学习的点云上采样方法受到计算资源和批处理策略的限制，这些方法通常需要将点云划分为较小的块，导致失真，从而降低感知质量。为了解决这个问题，我们提出了一种新颖的基于 3D 高斯点喷射（3DGS）的 2D-3D 混合彩色点云上采样框架（GaussianPU），用于机器人感知。该方法利用 3DGS 将 3D 点云与机器人视觉系统中的 2D 渲染图像进行桥接。双尺度渲染图像恢复网络将稀疏点云渲染转换为密集表示，这些表示与精确的机器人相机姿态和插值的稀疏点云一起输入到 3DGS 中，以重建密集的 3D 点云。我们对原始 3DGS 进行了系列增强，实现了对点数的精确控制，并显著提升了点云上采样的质量，以便更好地理解机器人场景。我们的框架支持在单个消费级 GPU（如 NVIDIA GeForce RTX 3090）上处理整个点云，消除了对分割的需求，从而为机器人导航和操作任务生成高质量、密集的彩色点云，点数达到数百万级。大量生成百万级点云数据的实验结果验证了我们方法的有效性，显著提高了彩色点云的质量，并展示了在自主机器人和人机交互场景中应用大规模点云的巨大潜力。