This work presents a novel autoregressive model, PointGrow, which generates realistic point cloud samples from scratch or conditioned on given semantic contexts. This model operates recurrently, with each point sampled according to a conditional distribution given its previously-generated points. It is further augmented with dedicated self-attention modules to capture long-range interpoint dependencies during the generation process.
We provided processed point clouds from 7 categories in ShapeNet, including airplane, car, table, chair, bench, cabinet and lamp. The coordinates of those point clouds, arranged as (z, y, x), range from 0 to 1. They are sorted in the order of z, y and x, and can be downloaded from here.
The model is trained per category, change the ShapeNet category id when working on different categories.
| Category | Id |
|---|---|
| Airplane | 02691156 |
| Car | 02958343 |
| Table | 04379243 |
| Chair | 03001627 |
| Bench | 02828884 |
| Cabinet | 02933112 |
| Lamp | 03636649 |
- Run unconditional PointGrow training script for airplane category with SACA-A module:
python train_unconditional.py --cat 02691156 --model unconditional_model_saca_aModel parameters will be stored under "log/unconditional_model_saca_a/02691156".
- For example, to generate 300 point clouds for airplane category using the pre-trained model:
python generate_unconditional.py --cat 02691156 --model unconditional_model_saca_a --tot_pc 300The generated point clouds will be stored in the format of numpy array under "res/unconditional_model_saca_a/res_02691156.npy".
Generate point clouds conditioned on additional one-hot vectors, with their non-empty elements indicating ShapeNet categories. For example, following the order in the above table, the one-hot vector for airplane can be expressed as [1, 0, 0, 0, 0, 0, 0].
- Train conditional PointGrow with one-hot vectors:
python train_conditional_one_hot.pyModel parameters will be saved under "log/conditional_model_one_hot".
- For example, to generate 50 point clouds for airplane (cat_idx = 0) using the pre-trained model:
python generate_conditional_one_hot.py --cat_idx 0 --tot_pc 50The generated point clouds will be stored in the format of numpy array under "res/conditional_model_one_hot/res_02691156.npy".
Generate point clouds conditioned on the embedding vectors of given 2D images. We still use ShapeNet point clouds, and obtain their 2D renderings from 3D-R2N2. A collection of 2D images of airplane and car categories and their shape ids matching the point clouds provided in this project can be found here.
- Train conditional PointGrow with 2D image embeddings for airplane category:
python train_conditional_im.py --cat 02691156Model parameters will be saved under "log/conditional_model_im/02691156".
- For example, to generate 50 point clouds for ShapeNet airplane testing images using the pre-trained model:
python generate_conditional_im.py --tot_pc 50 --batch_size 25 --cat 02691156The batch_size variable is recommened to be set less than 25 to fit GPU memory. The tot_pc variable will be truncated to a multiple of batch_size if tot_pc is larger than batch_size. The generated point clouds will be stored in the format of numpy array under "res/conditional_model_im/res_02691156.npy".
Please cite this paper if you want to use it in your work,
@inproceedings{sun2020pointgrow,
title={Pointgrow: Autoregressively learned point cloud generation with self-attention},
author={Sun, Yongbin and Wang, Yue and Liu, Ziwei and Siegel, Joshua and Sarma, Sanjay},
booktitle={The IEEE Winter Conference on Applications of Computer Vision},
pages={61--70},
year={2020}
}
MIT License