Lizhen Wang, Zhiyuan Chen, Tao Yu, Chenguang Ma, Liang Li, Yebin Liu CVPR 2022
Tsinghua University & Ant Group
We present FaceVerse, a fine-grained 3D Neural Face Model, which is built from hybrid East Asian face datasets containing 60K fused RGB-D images and 2K high-fidelity 3D head scan models. A novel coarse-to-fine structure is proposed to take better advantage of our hybrid dataset. In the coarse module, we generate a base parametric model from large-scale RGB-D images, which is able to predict accurate rough 3D face models in different genders, ages, etc. Then in the fine module, a conditional StyleGAN architecture trained with high-fidelity scan models is introduced to enrich elaborate facial geometric and texture details. Note that different from previous methods, our base and detailed modules are both changeable, which enables an innovative application of adjusting both the basic attributes and the facial details of 3D face models. Furthermore, we propose a single-image fitting framework based on differentiable rendering. Rich experiments show that our method outperforms the state-of-the-art methods.
Fig.1 Single-image fitting results using FaceVerse model.
Please download the zip file of version 0 or version 1 (recommended) and unzip it in the ./data folder.
FaceVerse version 0 [download]: paper version.
Fig.2 Single-image reconstruction results of version 0 (base model, detail model and expression refined final model).
FaceVerse version 1 [download]:
-
Refine the shape of the base PCA model: symmetrical and more detailed.
-
Remove the points inside the mouth.
-
Refine the expression PCA components.
Fig.3 Single-image reconstruction results of version 1 (base model, detail model and expression refined final model).
- Python 3.9
- PyTorch 1.11.0
- torchvision 0.11.1
- PyTorch3D 0.6.0
- Cuda 11.3
- ONNX Runtime
- OpenCV
- Numpy
- tqdm
- ninja
You need to compile the ops provided by stylegan2-pytorch using ninja:
cd third_libs/stylegan_ops
python3 setup.py install
Reconstructing a 3D face from a single image. There are three processes: (a) reconstructed by PCA model; (b) refined by the detailed generator; (c) refined by the expression generator.
An example input with a image folder (sampled from the FFHQ dataset):
python3 fit_images.py --version 1 --input example/images --res_folder example/image_results --save_ply
Note: the detailed refinement is based on differentiable rendering, which is quite time-consuming (over 10 minutes).
Offline tracking input with a video (our code will crop the face region using the first frame):
python tracking_offline.py --input example/videos/test.mp4 --res_folder example/video_results
Online tracking using your PC camera (our code will crop the face region using the first frame):
python tracking_online.py
Note: the tracking is based on differentiable rendering and only has 2 fps.
If you use this dataset for your research, please consider citing:
@InProceedings{wang2022faceverse,
title={FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset},
author={Wang, Lizhen and Chen, Zhiyua and Yu, Tao and Ma, Chenguang and Li, Liang and Liu, Yebin},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR2022)},
month={June},
year={2022},
}
- Lizhen Wang (wlz18@mails.tsinghua.edu.cn)
- Zhiyuan Chen (juzhen.czy@antfin.com)
- Yebin Liu (liuyebin@mail.tsinghua.edu.cn)
The code is partially borrowed from 3DMM-Fitting-Pytorch, stylegan2-pytorch and OpenSeeFace. And many thanks to the volunteers participated in data collection. Our License