SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction

Zechuan Zhang  Zongxin Yang✉  Yi Yang 

ReLER, CCAI, Zhejiang University

^✉Corresponding Author

CVPR 2024 Highlight

Figure 1. With just a single image, SIFU is capable of reconstructing a high-quality 3D clothed human model, making it well-suited for practical applications such as 3D printing and scene creation. At the heart of SIFU is a novel Side-view Conditioned Implicit Function, which is key to enhancing feature extraction and geometric precision. Furthermore, SIFU introduces a 3D Consistent Texture Refinement process, greatly improving texture quality and facilitating texture editing with the help of text-to-image diffusion models. Notably proficient in dealing with complex poses and loose clothing, SIFU stands out as an ideal solution for real-world applications.

📖 For more visual results, go checkout our project page

This repository will contain the official implementation of SIFU.

News

• [2024/6/18] Due to visa check problem, the author can not come to the conference center in person. We are sorry about this [sad][cry].
• [2024/4/5] Our paper has been accepted as Highlight (Top 11.9% of accepted papers)!
• [2024/2/28] We release the code of geometry reconstruction, including test and inference.
• [2024/2/27] SIFU has been accepted by CVPR 2024! See you in Seattle!
• [2023/12/13] We release the paper on arXiv.
• [2023/12/10] We build the Project Page.

Installation

• Ubuntu 20 / 18
• CUDA=11.6 or 11.7 or 11.8, GPU Memory > 16GB
• Python = 3.8
• PyTorch = 1.13.0 (official Get Started)

We thank @levnikolaevich and @GuangtaoLyu for provide valuable advice on the installation steps.

If you don't have conda or miniconda, please install that first:

sudo apt-get update && \
sudo apt-get upgrade -y && \
sudo apt-get install unzip libeigen3-dev ffmpeg build-essential nvidia-cuda-toolkit

mkdir -p ~/miniconda3 && \
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh && \
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 && \
rm -rf ~/miniconda3/miniconda.sh && \
~/miniconda3/bin/conda init bash && \
~/miniconda3/bin/conda init zsh

# close and reopen the shell
git clone https://github.com/River-Zhang/SIFU.git
sudo apt-get install libeigen3-dev ffmpeg
cd SIFU
conda env create -f environment.yaml
conda activate sifu
pip install -r requirements.txt

Please download the checkpoint (google drive) and place them in ./data/ckpt

Please follow ICON to download the extra data, such as HPS and SMPL (using fetch_hps.sh and fetch_data.sh). There may be missing files about SMPL, and you can download from here and put them in /data/smpl_related/smpl_data/.

Inference

python -m apps.infer -cfg ./configs/sifu.yaml -gpu 0 -in_dir ./examples -out_dir ./results -loop_smpl 100 -loop_cloth 200 -hps_type pixie

Testing

# 1. Register at http://icon.is.tue.mpg.de/ or https://cape.is.tue.mpg.de/
# 2. Download CAPE testset
bash fetch_cape.sh 

# evaluation
python -m apps.train -cfg ./configs/train/sifu.yaml -test

# TIP: the default "mcube_res" is 256 in apps/train.

Texture Refinement Module

The code is available for download on google drive. Please note that the current code structure may not be well-organized and may require some time to set up the environment. The author plans to reorganize it at their earliest convenience.

Applications of SIFU

Scene Building

3D Printing

Texture Editing

Animation

In-the-wild Reconstruction

Bibtex

If this work is helpful for your research, please consider citing the following BibTeX entry.

@InProceedings{Zhang_2024_CVPR,
    author    = {Zhang, Zechuan and Yang, Zongxin and Yang, Yi},
    title     = {SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {9936-9947}
}