SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery

⭐ This code has been completely released ⭐

⭐ our article ⭐

⭐ We also finish the work about the quantization based on SuperYOLO： Guided Hybrid Quantization for Object Detection in Multimodal Remote Sensing Imagery via One-to-one Self-teaching!!!⭐

Requirements

pip install -r requirements.txt

Train

1. Prepare training data

• 1.1 In order to realize the SR assisted branch, the input images of the network are downsampled from 1024 x 1024 size to 512 x 512 during the training process. In the test process, the image size is 512 x 512, which is consistent with the input of other algorithms compared.

• 1.2 Download VEDAI data for our experiment from baiduyun (code: hvi4) or google drive. And the path of dataset is like that

SuperYOLO
├── dataset
│   ├── VEDAI
│   │   ├── images
│   │   ├── labels
│   │   ├── fold01.txt
│   │   ├── fold01test.txt
│   │   ├── fold02.txt
│   │   ├── .....
│   ├── VEDAI_1024
│   │   ├── images
│   │   ├── labels

• 1.3 Note that we transform the labels of the dataset to be horizontal boxes by transform code. You shoud run transform.py before training the model. Change the PATH = './dataset/' and then run the code.

2. Begin to train multi images

python train.py --cfg models/SRyolo_MF.yaml --super --train_img_size 1024 --hr_input --data data/SRvedai.yaml --ch 64 --input_mode RGB+IR+MF

3. Begin to train RGB or IR images

python train.py --cfg models/SRyolo_noFocus_small.yaml --super --train_img_size 1024 --hr_input --data data/SRvedai.yaml --ch 3 --input_mode RGB

python train.py --cfg models/SRyolo_noFocus_small.yaml --super --train_img_size 1024 --hr_input --data data/SRvedai.yaml --ch 3 --input_mode IR

4. Begin to train multi images without SR branch

python train.py --cfg models/SRyolo_MF.yaml --train_img_size 512 --data data/SRvedai.yaml --ch 64 --input_mode RGB+IR+MF

5. Begin to train RGB or IR images without SR branch

python train.py --cfg models/SRyolo_noFocus_small.yaml --train_img_size 512 --data data/SRvedai.yaml --ch 3 --input_mode RGB

python train.py --cfg models/SRyolo_noFocus_small.yaml --train_img_size 512 --data data/SRvedai.yaml --ch 3 --input_mode IR

Test

1. Pretrained Checkpoints

You can use our pretrained checkpoints for test process. Download pre-trained model and put it in here.

2. Begin to test

python test.py --weights runs/train/exp/best.pt --input_mode RGB+IR+MF

Results

Method	Modality	Car	Pickup	Camping	Truck	Other	Tractor	Boat	Van	mAP50	Params. $\downarrow$	GFLOPs $\downarrow$
YOLOv3	IR	80.21	67.03	65.55	47.78	25.86	40.11	32.67	53.33	51.54	61.5351M	49.55
YOLOv3	RGB	83.06	71.54	69.14	59.30	48.93	67.34	33.48	55.67	61.06	61.5351M	49.55
YOLOv3	Multi	84.57	72.68	67.13	61.96	43.04	65.24	37.10	58.29	61.26	61.5354M	49.68
YOLOv4	IR	80.45	67.88	68.84	53.66	30.02	44.23	25.40	51.41	52.75	52.5082M	38.16
YOLOv4	RGB	83.73	73.43	71.17	59.09	51.66	65.86	34.28	60.32	62.43	52.5082M	38.16
YOLOv4	Multi	85.46	72.84	72.38	62.82	48.94	68.99	34.28	54.66	62.55	52.5085M	38.23
YOLOv5s	IR	77.31	65.27	66.47	51.56	25.87	42.36	21.88	48.88	49.94	7.0728M	5.24
YOLOv5s	RGB	80.07	68.01	66.12	51.52	45.76	64.38	21.62	40.93	54.82	7.0728M	5.24
YOLOv5s	Multi	80.81	68.48	69.06	54.71	46.76	64.29	24.25	45.96	56.79	7.0739M	5.32
YOLOv5m	IR	79.23	67.32	65.43	51.75	26.66	44.28	26.64	56.14	52.19	21.0659M	16.13
YOLOv5m	RGB	81.14	70.26	65.53	53.98	46.78	66.69	36.24	49.87	58.80	21.0659M	16.13
YOLOv5m	Multi	82.53	72.32	68.41	59.25	46.20	66.23	33.51	57.11	60.69	21.0677M	16.24
YOLOv5l	IR	80.14	68.57	65.37	53.45	30.33	45.59	27.24	61.87	54.06	46.6383M	36.55
YOLOv5l	RGB	81.36	71.70	68.25	57.45	45.77	70.68	35.89	55.42	60.81	46.6383M	36.55
YOLOv5l	Multi	82.83	72.32	69.92	63.94	48.48	63.07	40.12	56.46	62.16	46.6046M	36.70
YOLOv5x	IR	79.01	66.72	65.93	58.49	31.39	41.38	31.58	58.98	54.18	87.2458M	69.52
YOLOv5x	RGB	81.66	72.23	68.29	59.07	48.47	66.01	39.15	61.85	62.09	87.2458M	69.52
YOLOv5x	Multi	84.33	72.95	70.09	61.15	49.94	67.35	38.71	56.65	62.65	87.2487M	69.71
SuperYOLO	IR	87.90	81.39	76.90	61.56	39.39	60.56	46.08	71.00	65.60	4.8256M	16.61
SuperYOLO	RGB	90.30	82.66	76.69	68.55	53.86	79.48	58.08	70.30	72.49	4.8256M	16.61
SuperYOLO	Multi	91.13	85.66	79.30	70.18	57.33	80.41	60.24	76.50	75.09	4.8451M	17.98

Time

2024.4 SuperYOLO won the Highly Cited Paper and Hot paper ！！！！！

2023.2.14 open the train.py

2023.2.14 update the new fusion method (MF)

2023.2.16 update the test.py for visualization of detection results

Visualization of results

Acknowledgements

This code is built on YOLOv5 (PyTorch). We thank the authors for sharing the codes.

Licencing

Copyright (C) 2020 Jiaqing Zhang

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3 of the License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program.

Contact

If you have any questions, please contact me by email (jqzhang_2@stu.xidian.edu.cn).

Citation

If our code is helpful to you, please cite:

@ARTICLE{10075555,
  author={Zhang, Jiaqing and Lei, Jie and Xie, Weiying and Fang, Zhenman and Li, Yunsong and Du, Qian},
  journal={IEEE Transactions on Geoscience and Remote Sensing}, 
  title={SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery}, 
  year={2023},
  volume={61},
  number={},
  pages={1-15},
  doi={10.1109/TGRS.2023.3258666}}

@article{zhang2023guided,
  title={Guided Hybrid Quantization for Object Detection in Remote Sensing Imagery via One-to-one Self-teaching},
  author={Zhang, Jiaqing and Lei, Jie and Xie, Weiying and Li, Yunsong and Yang, Geng and Jia, Xiuping},
  journal={IEEE Transactions on Geoscience and Remote Sensing},
  year={2023},
  publisher={IEEE}
}