We introduce a novel approach to fine-grained cross-view geo-localization. Our method aligns a warped ground image with a corresponding GPS-tagged satellite image covering the same area using homography estimation. We first employ a differentiable spherical transform, adhering to geometric principles, to accurately align the perspective of the ground image with the satellite map. To address challenges such as occlusion, small overlapping range, and seasonal variations, we propose a robust correlation-aware homography estimator to align similar parts of the transformed ground image with the satellite image. Our method achieves sub-pixel resolution and meter-level GPS accuracy by mapping the center point of the transformed ground image to the satellite image using a homography matrix and determining the orientation of the ground camera using a point above the central axis. Operating at a speed of 30 FPS, our method outperforms state-of-the-art techniques, reducing the mean metric localization error by 21.3% and 32.4% in same-area and cross-area generalization tasks on the VIGOR benchmark, respectively, and by 34.4% on the KITTI benchmark in same-area evaluation.
- [2024-04-27] We release the training codes for the VIGOR dataset.
- [2023-10-27] We release the inferencing codes with checkpoints as well as the demo script. You can test HC-Net with your own machines.
- [2023-10-01] We release the code for implementing the spherical transform. For usage instructions, please refer to Spherical_transform.ipynb.
- [2023-09-21] HC-Net has been accepted by NeurIPS 2023! 🔥🔥🔥
- [2023-08-30] We release the paper of HC-Net and an online gradio demo.
HC-Net is online! Try it at this url.
Please use the demo script to deploy the demo locally for testing.
You can test our model using the data from the 'same_area_balanced_test.txt' split of the VIGOR dataset, or by providing your own Panorama image along with its corresponding Satellite image.
We train and test our codes under the following environment:
- Ubuntu 18.04
- CUDA 12.0
- Python 3.8.16
- PyTorch 1.13.0
To get started, follow these steps:
- Clone this repository.
git clone https://github.com/xlwangDev/HC-Net.git
cd HC-Net- Install the required packages.
conda create -n hcnet python=3.8 -y
conda activate hcnet
pip install -r requirements.txtsh train.shTo evaluate the HC-Net model, follow these steps:
- Download the VIGOR dataset and set its path to '/home/< usr >/Data/VIGOR'.
- Download the pretrained models and place them in the './checkpoints/VIGOR '.
- Run the following command:
chmod +x val.sh
# Usage: val.sh [same|cross]
# For same-area in VIGOR
./val.sh same 0
# For cross-area in VIGOR
./val.sh cross 0- You can also observe the visualization results of the model through a demo based on gradio. Use the following command to start the demo, and open the local URL: http://0.0.0.0:7860.
python demo_gradio.py🏷️ Label Correction for VIGOR Dataset
We propose the use of Mercator projection to directly compute the pixel coordinates of ground images on specified satellite images using the GPS information provided in the dataset. You can find the specific code at Mercator.py.
To use our corrected label, you can add the following content to the __getitem__ method of the VIGORDataset class in datasets.py file in the CCVPE project:
from Mercator import *
pano_gps = np.array(self.grd_list[idx][:-5].split(',')[-2:]).astype(float)
pano_gps = torch.from_numpy(pano_gps).unsqueeze(0)
sat_gps = np.array(self.sat_list[self.label[idx][pos_index]][:-4].split('_')[-2:]).astype(float)
sat_gps = torch.from_numpy(sat_gps).unsqueeze(0)
zoom = 20
y = get_pixel_tensor(sat_gps[:,0], sat_gps[:,1], pano_gps[:,0],pano_gps[:,1], zoom)
col_offset_, row_offset_ = y[0], y[1]
width_raw, height_raw = sat.size
col_offset, row_offset = width_raw/2 -col_offset_.item(), row_offset_.item() - height_raw/2We have released the code corresponding to section A.2 in the paper's Supplementary, along with an online testing platform
Compared to traditional Inverse Perspective Mapping (IPM), our approach does not require calibration of camera parameters. Instead, it allows for manual tuning to achieve an acceptable BEV projection result.
You can use Hugging Face Spaces for online testing, or run our code locally. Online testing utilizes CPU for computation, which is slower. If you run it locally with a GPU, the projection process takes less than 10ms.
python demo_gradio_kitti.pyOur projection process is implemented entirely in PyTorch, which means our projection method is differentiable and can be directly deployed in any network for gradient propagation.
- Add data preparation codes.
- Add inferencing and serving codes with checkpoints.
- Add evaluation codes.
- Add training codes.
If you find our work helpful, please cite:
@article{wang2024fine,
title={Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimator},
author={Wang, Xiaolong and Xu, Runsen and Cui, Zhuofan and Wan, Zeyu and Zhang, Yu},
journal={Advances in Neural Information Processing Systems},
volume={36},
year={2024}
}