Skip to content

Latest commit

 

History

History
94 lines (61 loc) · 3.82 KB

README.md

File metadata and controls

94 lines (61 loc) · 3.82 KB
Raw

TASFormer: Task-aware Image Segmentation Transformer

Framework: PyTorch Lightning License

[Springer] [pdf] [BibTeX]

This repo contains the code for our paper TASFormer: Task-aware Image Segmentation Transformer.

Contents

  1. Notes
  2. Installation Instructions
  3. Dataset Preparation
  4. Execution Instructions
  5. Results

Notes

ADE20K_multitask_segmentation/ contains codes and configs for ADE20K dataset.

multitask_segmentation/ contains additional codes and configs for GDD, Sun, SceneNet and Trans10k datasets.

transformers_update_for_adapters/ contains files required for TASFormer (HF adapter) and TASFormer (HF adapter++).

You can find $bIoU$ metric implementation inside validation_step() in pl_module.py. Keep in mind, our TASFormer model works with binary masks, and averaging is performed over all masks, regardless of their class. More details about $bIoU$ metric can be found in our Paper.

Installation Instructions

  • We use Python 3.8, PyTorch 1.8.1 (CUDA 10.1 build).
  • We use PyTorch Lightning 1.5.0.
  • For complete installation instructions, please see Installation.

Dataset Preparation

  • We experiment on ADE20K benchmark and other datasets.
  • Please see Preparing Datasets for complete instructions for preparing the datasets.

Execution Instructions

Training

Evaluation

Results

You can find our pretrained models in Getting Started.

Results

ADE20K

Segmentation results on different Num Classes from ADE20K dataset, $bIoU$, %.

Method Params Crop Size 2 12 150
SegFormer (B0) 3.8M 320×320 63.2 52.4 37.9
TASFormer (emb) 4--14M 320×320 60.8 6.1 14.1
TASFormer (VSA emb) 4.1M 320×320 48.6 0.1 0.1
TASFormer (HF adapter) 7.3M 320×320 67.9 59.4 48.3
TASFormer (HF adapter++) 5.7M 320×320 67.9 58.4 47.6



Segmentation results of TASFormer (HF adapter) with different Crop Size on ADE20K dataset, $bIoU$, %.

Method Params Crop Size $mIoU$, % $bIoU$, %
TASFormer (HF adapter) 7.3M 320×320 14.6 48.3
TASFormer (HF adapter) 7.3M 640×640 14.7 51.1
TASFormer (HF adapter) 7.3M 896×896 14.4 52.0

Citation

If you found DAPS3D useful in your research, please consider starring ⭐ us on GitHub and citing 📚 us in your research!

@InProceedings{yudin2024tasformer,
title={TASFormer: Task-Aware Image Segmentation Transformer},
author={Yudin, Dmitry and Khorin, Aleksandr and Zemskova, Tatiana and Ovchinnikova, Darya},
booktitle={Neural Information Processing},
year={2024},
publisher={Springer Nature Singapore},
pages={305--317},
doi={10.1007/978-981-99-8073-4_24}
}