Skip to content
/ CISS Public

Unsupervised condition-level adaptation for semantic segmentation

License

Notifications You must be signed in to change notification settings

SysCV/CISS

Repository files navigation

Condition-Invariant Semantic Segmentation

A novel unsupervised domain adaptation method for semantic segmentation models, tailored for condition-level adaptation

by Christos Sakaridis, David Bruggemann, Fisher Yu and Luc Van Gool

CISS: Paper | arXiv

Overview

Adaptation of semantic segmentation networks to different visual conditions from those for which ground-truth annotations are available at training is vital for robust perception in autonomous cars and robots. However, previous work has shown that most feature-level adaptation methods, which employ adversarial training and are validated on synthetic-to-real adaptation, provide marginal gains in normal-to-adverse condition-level adaptation, being outperformed by simple pixel-level adaptation via stylization. Motivated by these findings, we propose to leverage stylization in performing feature-level adaptation by aligning the deep features extracted by the encoder of the network from the original and the stylized view of each input image with a novel feature invariance loss. In this way, we encourage the encoder to extract features that are invariant to the style of the input, allowing the decoder to focus on parsing these features and not on further abstracting from the specific style of the input.

We implement our method, named Condition-Invariant Semantic Segmentation (CISS), on the top-performing domain adaptation architecture and achieve outstanding results on condition-level adaptation. In particular, CISS sets the new state of the art in the popular daytime-to-nighttime Cityscapes→Dark Zurich benchmark. Furthermore, our method achieves the second-best performance on the normal-to-adverse Cityscapes→ACDC. benchmark. Our method is also shown to generalize well to domains unseen during training, outperforming competing state-of-the-art domain adaptation approaches on zero-shot nighttime generalization on BDD100K-night and ACDC-night.

This repository includes the source code for CISS.

For more information on CISS, please check our paper.

License

This software is made available for non-commercial use under a creative commons license. You can find a summary of the license here.

Contents

  1. Requirements
  2. Datasets
  3. Testing
  4. Training
  5. Checkpoints
  6. Framework Structure
  7. Acknowledgments
  8. Citation

Requirements

For implementing CISS, we use two virtual Python environments, one for training and the other for testing. The former involves Pytorch 1.7.1 and Python 3.8.5, while the latter involves Pytorch 1.11.1 and Python 3.10.4. These virtual environments should be set up based on the provided requirements.txt and requirements_test.txt files as follows:

  • Virtual environment for training:
python -m venv ~/venv/CISS
source ~/venv/CISS/bin/activate
pip install -r requirements_train.txt -f https://download.pytorch.org/whl/torch_stable.html
pip install mmcv-full==1.3.7  # requires the other packages to be installed first
  • Virtual environment for testing:
python -m venv ~/venv/CISS_test
source ~/venv/CISS_test/bin/activate
pip install -r requirements_test.txt -f https://download.pytorch.org/whl/torch_stable.html
pip install mmcv-full==1.3.7  # requires the other packages to be installed first

We have run training of CISS successfully on NVIDIA Tesla V100-SXM2-32GB, NVIDIA A100 40GB PCIe, and NVIDIA A100 80GB PCIe.

Testing is less memory-intensive and we have run it successfully on the above GPUs and on NVIDIA GeForce RTX 3090 and NVIDIA TITAN RTX.

Datasets and Initial Weights

Cityscapes: Download leftImg8bit_trainvaltest.zip and gt_trainvaltest.zip from here and extract them to data/cityscapes.

ACDC: Download rgb_anon_trainvaltest.zip and gt_trainval.zip from here and extract them to data/acdc. Further, please restructure the folders from condition/split/sequence/ to split/ using the following commands:

rsync -a data/acdc/rgb_anon/*/train/*/* data/acdc/rgb_anon/train/
rsync -a data/acdc/rgb_anon/*/val/*/* data/acdc/rgb_anon/val/
rsync -a data/acdc/gt/*/train/*/*_labelTrainIds.png data/acdc/gt/train/
rsync -a data/acdc/gt/*/val/*/*_labelTrainIds.png data/acdc/gt/val/

Dark Zurich: Download the Dark_Zurich_train_anon.zip and Dark_Zurich_val_anon.zip from here and extract it to data/dark_zurich.

The final folder structure should look like this:

CISS
├── ...
├── data
│   ├── acdc
│   │   ├── gt
│   │   │   ├── train
│   │   │   ├── val
│   │   ├── rgb_anon
│   │   │   ├── train
│   │   │   ├── val
│   ├── cityscapes
│   │   ├── leftImg8bit
│   │   │   ├── train
│   │   │   ├── val
│   │   ├── gtFine
│   │   │   ├── train
│   │   │   ├── val
│   ├── dark_zurich
│   │   ├── gt
│   │   │   ├── val
│   │   ├── rgb_anon
│   │   │   ├── train
│   │   │   ├── val
├── ...

For generalization evaluation, you need to also download BDD100K-night and Nighttime Driving which are used as test sets in that scenario.

Data Preprocessing: Finally, run the following scripts to convert the label IDs to the train IDs and to generate the class index for RCS:

python tools/convert_datasets/cityscapes.py data/cityscapes --nproc 8

Further, please download the MiT weights from SegFormer using the following script. If problems occur with the automatic download, please follow the instructions for a manual download within the script.

sh tools/download_checkpoints.sh

Testing

The provided CISS checkpoint trained on Cityscapes→ACDC (already downloaded by tools/download_checkpoints.sh) can be tested on the ACDC validation set using:

sh test.sh work_dirs/csHR2acdcHR_ciss_9fcab

The predictions are saved for inspection to work_dirs/csHR2acdcHR_ciss_9fcab/preds and the mIoU of the model is printed to the console. The provided checkpoint should achieve 68.67% mIoU on the validation set of ACDC. Refer to the end of work_dirs/csHR2acdcHR_ciss_9fcab/20230303_190813.log for more information, such as the IoU scores for individual classes.

The main results for Cityscapes→ACDC and Cityscapes→Dark Zurich in the paper are reported on the test split of the respective target dataset. To generate the predictions for the test set, please run:

source ~/venv/CISS_test/bin/activate
PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256
python -m tools.test path/to/config_file path/to/checkpoint_file --test-set --format-only --eval-options imgfile_prefix=path/to/trainids_dir to_label_id=False --show-dir path/to/preds_dir --opacity 1

The predictions can be submitted to the public evaluation server of the respective dataset to obtain the test score.

Training

For convenience, we provide an annotated config file of the final CISS instantiation for the main, Cityscapes→ACDC experiment. A training job can be launched using:

source ~/venv/CISS/bin/activate
python run_experiments.py --config configs/ciss/csHR2acdcHR_ciss.py

The logs and checkpoints are stored under work_dirs/.

For the other experiments in our paper, we use a script to automatically generate the configs and train with them:

python run_experiments.py --exp <ID>

More information about the available experiments and their assigned IDs can be found in experiments.py. The generated configs will be stored in configs/generated/.

Moreover, we provide bash scripts which prepare execution in compute clusters with SLURM and LSF architecture before calling the main training script run_experiments.py. Users can adapt these scripts according to their own configuration of computing infrastructure.

Checkpoints

Below, we provide checkpoints of CISS for different benchmarks.

The checkpoints come with the training logs. Please note that:

  • The logs provide the mIoU on validation sets. For Cityscapes→ACDC and Cityscapes→Dark Zurich the main results reported in the paper (besides those belonging to ablations studies) are calculated on the respective test sets. For Dark Zurich, the performance significantly differs between validation and test set. Please read the section above on how to obtain the test mIoU.

Framework Structure

This repository is based on mmsegmentation version 0.16.0. For more information about the framework structure and the config system, please refer to the mmsegmentation documentation and the mmcv documentation.

The most relevant files for CISS are:

Other relevant files are:

Acknowledgments

This work is funded by Toyota Motor Europe via the research project TRACE-Zürich.

The source code for CISS is built on top of the following open-source projects. We thank their authors for making the respective source code repositories publicly available.

Citation

If you use the source code of CISS in your work or find CISS useful in your research in general, please cite our publication as

@article{sakaridis2023ciss,
  title={Condition-Invariant Semantic Segmentation},
  author={Sakaridis, Christos and Bruggemann, David and Yu, Fisher and Van Gool, Luc},
  journal = {ArXiv e-prints},
  archivePrefix = "arXiv",
  eprint = {2305.17349},
  primaryClass = "cs.CV",
  year=2023
}