Recent text-guided generation of individual 3D object has achieved great success using diffusion priors. However, these methods are not suitable for object insertion and replacement tasks as they do not consider the background, leading to illumination mismatches within the environment. To bridge the gap, we introduce an illumination-aware 3D scene editing pipeline for 3D Gaussian Splatting (3DGS) representation. Our key observation is that inpainting by the state-of-the-art conditional 2D diffusion model is consistent with background in lighting. To leverage the prior knowledge from the well-trained diffusion models for 3D object generation, our approach employs a coarse-to-fine objection optimization pipeline with inpainted views. In the first coarse step, we achieve image-to-3D lifting given an ideal inpainted view. The process employs 3D-aware diffusion prior from a view-conditioned diffusion model, which preserves illumination present in the conditioning image. To acquire an ideal inpainted image, we introduce an Anchor View Proposal (AVP) algorithm to find a single view that best represents the scene illumination in target region. In the second Texture Enhancement step, we introduce a novel Depth-guided Inpainting Score Distillation Sampling (DI-SDS), which enhances geometry and texture details with the inpainting diffusion prior, beyond the scope of the 3D-aware diffusion prior knowledge in the first coarse step. DI-SDS not only provides fine-grained texture enhancement, but also urges optimization to respect scene lighting. Our approach efficiently achieves local editing with global illumination consistency without explicitly modeling light transport. We demonstrate robustness of our method by evaluating editing in real scenes containing explicit highlight and shadows, and compare against the state-of-the-art text-to-3D editing methods.
最近，文本引导的个体3D对象生成在扩散先验的帮助下取得了巨大成功。然而，这些方法不适用于对象插入和替换任务，因为它们没有考虑背景，导致环境中的光照不匹配。为此，我们提出了一种基于光照感知的3D场景编辑流程，适用于3D高斯点云（3DGS）表示。我们的关键观察是，最先进的条件2D扩散模型的图像修复与背景光照一致。为了利用训练良好的扩散模型在3D对象生成中的先验知识，我们的方法采用了一个从粗到细的目标优化流程，结合了修复视图。在第一个粗略步骤中，我们根据理想的修复视图实现图像到3D的提升。该过程使用来自视图条件扩散模型的3D感知扩散先验，保留了条件图像中的光照。为了获得理想的修复图像，我们引入了一种锚视图提议（AVP）算法，用于找到一个最佳视图，以代表目标区域的场景光照。在第二个纹理增强步骤中，我们引入了一种新型的深度引导修复评分蒸馏采样（DI-SDS），它通过修复扩散先验增强几何和纹理细节，超出了第一粗略步骤中3D感知扩散先验的范围。DI-SDS不仅提供了细粒度的纹理增强，还促进了优化以尊重场景光照。我们的方法高效地实现了局部编辑与全球光照一致性，无需明确建模光传输。通过在包含显著高光和阴影的真实场景中评估编辑效果，并与最先进的文本到3D编辑方法进行比较，我们展示了我们方法的鲁棒性。