We have recently seen tremendous progress in photo-real human modeling and rendering. Yet, efficiently rendering realistic human performance and integrating it into the rasterization pipeline remains challenging. In this paper, we present HiFi4G, an explicit and compact Gaussian-based approach for high-fidelity human performance rendering from dense footage. Our core intuition is to marry the 3D Gaussian representation with non-rigid tracking, achieving a compact and compression-friendly representation. We first propose a dual-graph mechanism to obtain motion priors, with a coarse deformation graph for effective initialization and a fine-grained Gaussian graph to enforce subsequent constraints. Then, we utilize a 4D Gaussian optimization scheme with adaptive spatial-temporal regularizers to effectively balance the non-rigid prior and Gaussian updating. We also present a companion compression scheme with residual compensation for immersive experiences on various platforms. It achieves a substantial compression rate of approximately 25 times, with less than 2MB of storage per frame. Extensive experiments demonstrate the effectiveness of our approach, which significantly outperforms existing approaches in terms of optimization speed, rendering quality, and storage overhead.
近期，在逼真人类建模和渲染方面取得了巨大进展。然而，高效地渲染逼真的人类表现并将其集成到光栅化流程中仍然具有挑战性。在这篇论文中，我们介绍了HiFi4G，这是一种基于高斯的明确且紧凑的方法，用于从密集的影像资料中渲染高保真度的人类表现。我们的核心直觉是将三维高斯表征与非刚性跟踪结合起来，实现一个紧凑且便于压缩的表征。首先，我们提出了一种双图机制来获取运动先验，包括一个粗略变形图用于有效初始化，以及一个细粒度高斯图用于实施后续约束。接着，我们利用一种具有自适应时空正则化器的4D高斯优化方案，有效平衡非刚性先验和高斯更新。我们还提出了一种伴随的压缩方案，带有残差补偿，以在各种平台上实现沉浸式体验。该方案实现了约25倍的显著压缩率，每帧存储量不到2MB。广泛的实验表明，我们的方法在优化速度、渲染质量和存储开销方面显著优于现有方法。