Awesome Vision Transformer Collection

Variants of Vision Transformer and Vision Transformer for Downstream Tasks

author: Runwei Guan

affiliation: University of Liverpool / JITRI-Institute of Deep Perception Technology

email: thinkerai@foxmail.com / Runwei.Guan@liverpool.ac.uk / guanrunwei@idpt.org

Image Backbone

• Vision Transformer paper code
• Swin Transformer paper code
• Swin Transformer V2: Scaling Up Capacity and Resolution paper code
• DVT paper code
• PVT paper code
• Lite Vision Transformer: LVT paper
• PiT paper code
• Twins paper code
• TNT paper code
• Mobile-ViT paper code
• Cross-ViT paper code
• LeViT paper code
• ViT-Lite paper
• Refiner paper code
• DeepViT paper code
• CaiT paper code
• LV-ViT paper code
• DeiT paper code
• CeiT paper code
• BoTNet paper
• ViTAE paper
• Visformer: The Vision-Friendly Transformer paper code
• Bootstrapping ViTs: Towards Liberating Vision Transformers from Pre-training paper
• AdaViT: Adaptive Tokens for Efficient Vision Transformer paper
• Improved Multiscale Vision Transformers for Classification and Detection paper
• Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding paper
• Towards End-to-End Image Compression and Analysis with Transformers paper
• MPViT: Multi-Path Vision Transformer for Dense Prediction paper
• Lite Vision Transformer with Enhanced Self-Attention paper
• PolyViT: Co-training Vision Transformers on Images, Videos and Audio paper
• MIA-Former: Efficient and Robust Vision Transformers via Multi-grained Input-Adaptation paper
• ELSA: Enhanced Local Self-Attention for Vision Transformer paper
• Vision Transformer for Small-Size Datasets paper
• SimViT: Exploring a Simple Vision Transformer with sliding windows paper
• SPViT: Enabling Faster Vision Transformers via Soft Token Pruning paper
• Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space paper
• Vision Transformer with Deformable Attention paper code
• PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture paper
• QuadTree Attention for Vision Transformers paper code
• TerViT: An Efficient Ternary Vision Transformer paper
• BViT: Broad Attention based Vision Transformer paper
• CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction paper
• EdgeFormer: Improving Light-weight ConvNets by Learning from Vision Transformers paper
• Dynamic Group Transformer: A General Vision Transformer Backbone with Dynamic Group Attention paper
• Coarse-to-Fine Vision Transformer paper
• ViT-P: Rethinking Data-efficient Vision Transformers from Locality paper
• MPViT: Multi-Path Vision Transformer for Dense Prediction paper
• Event Transformer paper
• DaViT: Dual Attention Vision Transformers paper
• LightViT: Towards Light-Weight Convolution-Free Vision Transformers paper
• UniNet: Unified Architecture Search with Convolution, Transformer, and MLP paper
• Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning paper
• EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications paper

Multi-label Classification

• Graph Attention Transformer Network for Multi-Label Image Classification paper

Point Cloud Processing

• Point Cloud Transformer paper
• Point Transformer paper
• Fast Point Transformer paper
• Adaptive Channel Encoding Transformer for Point Cloud Analysis paper
• PTTR: Relational 3D Point Cloud Object Tracking with Transformer paper
• Full Transformer Framework for Robust Point Cloud Registration with Deep Information Interaction paper
• LighTN: Light-weight Transformer Network for Performance-overhead Tradeoff in Point Cloud Downsampling paper
• Geometric Transformer for Fast and Robust Point Cloud Registration paper
• HiTPR: Hierarchical Transformer for Place Recognition in Point Cloud paper

Video Processing

• Video Transformers: A Survey paper
• ViViT: A Video Vision Transformer paper
• Vision Transformer Based Video Hashing Retrieval for Tracing the Source of Fake Videos paper
• LocFormer: Enabling Transformers to Perform Temporal Moment Localization on Long Untrimmed Videos With a Feature Sampling Approach paper
• Video Joint Modelling Based on Hierarchical Transformer for Co-summarization paper
• InverseMV: Composing Piano Scores with a Convolutional Video-Music Transformer paper
• TransVOD: End-to-end Video Object Detection with Spatial-Temporal Transformers paper
• Uniformer: Unified Transformer for Efficient Spatiotemporal Representation Learning paper
• Multiview Transformers for Video Recognition paper
• MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition paper
• Multi-direction and Multi-scale Pyramid in Transformer for Video-based Pedestrian Retrieval paper
• A Unified Transformer Framework for Group-based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection paper
• Learning Trajectory-Aware Transformer for Video Super-Resolution paper
• Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer paper

Model Compression

• A Unified Pruning Framework for Vision Transformers paper
• Multi-Dimensional Model Compression of Vision Transformer paper
• Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression paper

Transfer Learning & Pretraining

• Pre-Trained Image Processing Transformer paper code
• UP-DETR: Unsupervised Pre-training for Object Detection with Transformers paper code
• BEVT: BERT Pretraining of Video Transformers paper
• Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text paper
• On Efficient Transformer and Image Pre-training for Low-level Vision paper
• Pre-Training Transformers for Domain Adaptation paper
• RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-training paper
• Multiscale Convolutional Transformer with Center Mask Pretraining for Hyperspectral Image Classificationtion paper
• DiT: Self-supervised Pre-training for Document Image Transformer paper
• Underwater Image Enhancement Using Pre-trained Transformer paper

Multi-Modal

• Multi-Modal Fusion Transformer for End-to-End Autonomous Driving paper
• Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval paper
• LAVT: Language-Aware Vision Transformer for Referring Image Segmentation paper
• MTFNet: Mutual-Transformer Fusion Network for RGB-D Salient Object Detection paper
• Visual-Semantic Transformer for Scene Text Recognition paper
• Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text paper
• LaTr: Layout-Aware Transformer for Scene-Text VQA paper
• Multimodal Personality Recognition using Cross-Attention Transformer and Behaviour Encoding paper
• Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation paper
• Extended Self-Critical Pipeline for Transforming Videos to Text (TRECVID-VTT Task 2021) -- Team: MMCUniAugsburg paper
• On the Efficacy of Co-Attention Transformer Layers in Visual Question Answering paper
• DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers paper
• CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers paper
• VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer paper
• Knowledge Amalgamation for Object Detection with Transformers paper
• Are Multimodal Transformers Robust to Missing Modality? paper
• Self-supervised Vision Transformers for Joint SAR-optical Representation Learning paper
• Video Graph Transformer for Video Question Answering paper

Detection

• YOLOS: You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection paper code
• WB-DETR: Transformer-Based Detector without Backbone paper
• UP-DETR: Unsupervised Pre-training for Object Detection with Transformers paper
• TSP: Rethinking Transformer-based Set Prediction for Object Detection paper
• DETR paper code
• Deformable DETR paper code
• DN-DETR: Accelerate DETR Training by Introducing Query DeNoising paper code
• Rethinking Transformer-Based Set Prediction for Object Detection paper
• End-to-End Object Detection with Adaptive Clustering Transformer paper
• An End-to-End Transformer Model for 3D Object Detection paper
• End-to-End Human Object Interaction Detection with HOI Transformer paper code
• Adaptive Image Transformer for One-Shot Object Detection paper
• Improving 3D Object Detection With Channel-Wise Transformer paper
• TransPose: Keypoint Localization via Transformer paper
• Voxel Transformer for 3D Object Detection paper
• Embracing Single Stride 3D Object Detector with Sparse Transformer paper
• OW-DETR: Open-world Detection Transformer paper
• A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation paper
• Miti-DETR: Object Detection based on Transformers with Mitigatory Self-Attention Convergence paper
• Voxel Transformer for 3D Object Detection paper
• Short Range Correlation Transformer for Occluded Person Re-Identification paper
• TransVPR: Transformer-based place recognition with multi-level attention aggregation paper
• Pedestrian Detection: Domain Generalization, CNNs, Transformers and Beyond paper
• Arbitrary Shape Text Detection using Transformers paper
• A high-precision underwater object detection based on joint self-supervised deblurring and improved spatial transformer network paper
• A Unified Transformer Framework for Group-based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection paper
• Knowledge Amalgamation for Object Detection with Transformers paper
• SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection paper
• POSTER: A Pyramid Cross-Fusion Transformer Network for Facial Expression Recognition paper
• PSTR: End-to-End One-Step Person Search With Transformers paper
• Scaling Novel Object Detection with Weakly Supervised Detection Transformers paper
• OSFormer: One-Stage Camouflaged Instance Segmentation with Transformers paper
• Exploring Plain Vision Transformer Backbones for Object Detection paper

Segmentation

• Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation paper code
• Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention paper code
• MaX-DeepLab: End-to-End Panoptic Segmentation With Mask Transformers paper code
• Line Segment Detection Using Transformers without Edges paper
• VisTR: End-to-End Video Instance Segmentation with Transformers paper code
• SETR: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers paper code
• Segmenter: Transformer for Semantic Segmentation paper
• Fully Transformer Networks for Semantic Image Segmentation paper
• SOTR: Segmenting Objects with Transformers paper code
• GETAM: Gradient-weighted Element-wise Transformer Attention Map for Weakly-supervised Semantic segmentation paper
• Masked-attention Mask Transformer for Universal Image Segmentation paper
• A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation paper
• iSegFormer: Interactive Image Segmentation with Transformers paper
• SOIT: Segmenting Objects with Instance-Aware Transformers paper
• SeMask: Semantically Masked Transformers for Semantic Segmentation paper
• Siamese Network with Interactive Transformer for Video Object Segmentation paper
• Pyramid Fusion Transformer for Semantic Segmentation paper
• Swin transformers make strong contextual encoders for VHR image road extraction paper
• Transformers in Action:Weakly Supervised Action Segmentation paper
• Task-Adaptive Feature Transformer with Semantic Enrichment for Few-Shot Segmentation paper
• Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers paper
• Contextual Attention Network: Transformer Meets U-Net paper
• TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation paper

Pose Estimation

• Hand-Transformer: Non-Autoregressive Structured Modeling for 3D Hand Pose Estimation paper
• HOT-Net: Non-Autoregressive Transformer for 3D Hand-Object Pose Estimation paper
• End-to-End Human Pose and Mesh Reconstruction with Transformers paper code
• PE-former: Pose Estimation Transformer paper
• Pose Recognition with Cascade Transformers paper code
• Pose-guided Feature Disentangling for Occluded Person Re-identification Based on Transformer code
• Geometry-Contrastive Transformer for Generalized 3D Pose Transfer paper
• Temporal Transformer Networks with Self-Supervision for Action Recognition paper
• Co-training Transformer with Videos and Images Improves Action Recognition paper
• DProST: 6-DoF Object Pose Estimation Using Space Carving and Dynamic Projective Spatial Transformer paper
• Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition paper
• Motion-Aware Transformer For Occluded Person Re-identification paper
• HeadPosr: End-to-end Trainable Head Pose Estimation using Transformer Encoders paper
• ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers paper
• Zero-Shot Action Recognition with Transformer-based Video Semantic Embedding paper
• Spatial Transformer Network on Skeleton-based Gait Recognition paper

Tracking and Trajectory Prediction

• Transformer Tracking paper code
• Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking paper code
• MOTR: End-to-End Multiple-Object Tracking with TRansformer paper code
• SwinTrack: A Simple and Strong Baseline for Transformer Tracking paper
• Pedestrian Trajectory Prediction via Spatial Interaction Transformer Network paper
• PTTR: Relational 3D Point Cloud Object Tracking with Transformer paper
• Efficient Visual Tracking with Exemplar Transformers paper
• TransFollower: Long-Sequence Car-Following Trajectory Prediction through Transformer paper

Generative Model and Denoising

• 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds paper
• Spatial-Temporal Transformer for Dynamic Scene Graph Generation paper
• THUNDR: Transformer-Based 3D Human Reconstruction With Markers paper
• DoodleFormer: Creative Sketch Drawing with Transformers paper
• Image Transformer paper
• Taming Transformers for High-Resolution Image Synthesis paper code
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up code
• U2-Former: A Nested U-shaped Transformer for Image Restoration paper
• Neuromorphic Camera Denoising using Graph Neural Network-driven Transformers paper
• SiamTrans: Zero-Shot Multi-Frame Image Restoration with Pre-Trained Siamese Transformers paper
• StyleSwin: Transformer-based GAN for High-resolution Image Generation paper
• Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction paper
• SGTR: End-to-end Scene Graph Generation with Transformer paper
• Flow-Guided Sparse Transformer for Video Deblurring paper
• Spherical Transformer paper
• MaskGIT: Masked Generative Image Transformer paper
• Entroformer: A Transformer-based Entropy Model for Learned Image Compression paper
• UVCGAN: UNet Vision Transformer cycle-consistent GAN for unpaired image-to-image translation paper
• Stripformer: Strip Transformer for Fast Image Deblurring paper
• Vision Transformers for Single Image Dehazing paper
• Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer paper

Self-Supervised Learning

• Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning paper code
• iGPT paper code
• An Empirical Study of Training Self-Supervised Vision Transformers paper code
• Self-supervised Video Transformer paper
• TransMEF: A Transformer-Based Multi-Exposure Image Fusion Framework using Self-Supervised Multi-Task Learning paper
• TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning paper
• Transformers in Action:Weakly Supervised Action Segmentation paper
• Motion-Aware Transformer For Occluded Person Re-identification paper
• Transformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsics paper
• Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut paper
• Monocular Robot Navigation with Self-Supervised Pretrained Vision Transformers paper
• Multi-class Token Transformer for Weakly Supervised Semantic Segmentation paper
• Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers paper
• DiT: Self-supervised Pre-training for Document Image Transformer paper
• Self-supervised Vision Transformers for Joint SAR-optical Representation Learning paper
• DILEMMA: Self-Supervised Shape and Texture Learning with Transformers paper

Depth and Height Estimation

• Disentangled Latent Transformer for Interpretable Monocular Height Estimation paper
• Transformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsics paper
• SiaTrans: Siamese Transformer Network for RGB-D Salient Object Detection with Depth Image Classification paper

Explainable

• Development and testing of an image transformer for explainable autonomous driving systems paper
• Transformer Interpretability Beyond Attention Visualization paper code
• How Do Vision Transformers Work? paper
• eX-ViT: A Novel eXplainable Vision Transformer for Weakly Supervised Semantic Segmentation paper

Robustness

• Improved Robustness of Vision Transformer via PreLayerNorm in Patch Embedding paper

Deep Reinforcement Learning

• Evaluating Vision Transformer Methods for Deep Reinforcement Learning from Pixels paper

Calibration

• CTRL-C: Camera Calibration TRansformer With Line-Classification paper code

Radar

• Learning class prototypes from Synthetic InSAR with Vision Transformers paper
• Radar Transformer paper

Traffic

• SwinUNet3D -- A Hierarchical Architecture for Deep Traffic Prediction using Shifted Window Transformers paper

AI Medicine

• Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer paper
• 3D Medical Point Transformer: Introducing Convolution to Attention Networks for Medical Point Cloud Analysis paper
• Hformer: Pre-training and Fine-tuning Transformers for fMRI Prediction Tasks paper
• MT-TransUNet: Mediating Multi-Task Tokens in Transformers for Skin Lesion Segmentation and Classification paper
• MSHT: Multi-stage Hybrid Transformer for the ROSE Image Analysis of Pancreatic Cancer paper
• Generalized Wasserstein Dice Loss, Test-time Augmentation, and Transformers for the BraTS 2021 challenge paper
• D-Former: A U-shaped Dilated Transformer for 3D Medical Image Segmentation paper
• RFormer: Transformer-based Generative Adversarial Network for Real Fundus Image Restoration on A New Clinical Benchmark paper
• Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images paper
• Swin Transformer for Fast MRI paper code
• Automatic Segmentation of Head and Neck Tumor: How Powerful Transformers Are? paper
• ViTBIS: Vision Transformer for Biomedical Image Segmentation paper
• SegTransVAE: Hybrid CNN -- Transformer with Regularization for medical image segmentation paper
• Improving Across-Dataset Brain Tissue Segmentation Using Transformer paper
• Brain Cancer Survival Prediction on Treatment-naive MRI using Deep Anchor Attention Learning with Vision Transformer paper
• Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers paper
• AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation paper
• Uni4Eye: Unified 2D and 3D Self-supervised Pre-training via Masked Image Modeling Transformer for Ophthalmic Image Classification paper
• Characterizing Renal Structures with 3D Block Aggregate Transformers paper
• Multimodal Transformer for Nursing Activity Recognition paper
• RTN: Reinforced Transformer Network for Coronary CT Angiography Vessel-level Image Quality Assessment paper
• Radiomics-Guided Global-Local Transformer for Weakly Supervised Pathology Localization in Chest X-Rays paper

Hardware

• VAQF: Fully Automatic Software-hardware Co-design Framework for Low-bit Vision Transformer paper