RuntimeError: /io/build/temp.linux-x86_64-cpython-38/spconv/build/src/cumm/gemm/main/GemmMainUnitTest/GemmMainUnitTest_matmul2.cu 4489

[wenxi2]

[WARNING]your gpu arch (8, 6) isn't compiled in prebuilt, may cause invalid device function. available: {(6, 1), (3, 7), (7, 0), (5, 0), (6, 0), (7, 5), (5, 2)}
[Exception|indice_conv|subm]feat=torch.Size([100432, 256]),w=torch.Size([3, 3, 256, 256]),pair=torch.Size([2, 9, 100432]),pairnum=tensor([95659, 98114, 95665, 97724, 0, 0, 0, 0, 0],
dtype=torch.int32),act=100432,algo=ConvAlgo.Native
SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue.
x <spconv.pytorch.core.SparseConvTensor object at 0x7f730398bb20>
[WARNING]your gpu arch (8, 6) isn't compiled in prebuilt, may cause invalid device function. available: {(6, 1), (3, 7), (7, 0), (5, 0), (6, 0), (7, 5), (5, 2)}
[Exception|indice_conv|subm]feat=torch.Size([109840, 256]),w=torch.Size([3, 3, 256, 256]),pair=torch.Size([2, 9, 109840]),pairnum=tensor([105367, 107530, 105370, 107432, 0, 0, 0, 0, 0],
dtype=torch.int32),act=109840,algo=ConvAlgo.Native
SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue.
Traceback (most recent call last):
File "infer_visdrone.py", line 16, in
launch(
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/detectron2/engine/launch.py", line 67, in launch
mp.spawn(
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
while not context.join():
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/detectron2/engine/launch.py", line 126, in _distributed_worker
main_func(*args)
File "/home/abab/QueryDet-PyTorch-main/train_tools/visdrone_infer.py", line 170, in start_train
res = Trainer.test(cfg, model)
File "/home/abab/QueryDet-PyTorch-main/train_tools/visdrone_infer.py", line 77, in test
result = inference_on_dataset(model, data_loader, evaluator)
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/detectron2/evaluation/evaluator.py", line 158, in inference_on_dataset
outputs = model(inputs)
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/abab/QueryDet-PyTorch-main/models/querydet/detector.py", line 172, in forward
return self.test(batched_inputs)
File "/home/abab/QueryDet-PyTorch-main/models/querydet/detector.py", line 289, in test
results, total_time = self.test_forward(images) # normal test
File "/home/abab/QueryDet-PyTorch-main/models/querydet/detector.py", line 357, in test_forward
det_cls_query, det_bbox_query, query_anchors = self.qInfer.run_qinfer(params, CAS_feature, features_value,
File "/home/abab/QueryDet-PyTorch-main/models/querydet/qinfer.py", line 226, in run_qinfer
cls_result_i = self._run_spconvs(x, self.cls_spconv).view(-1, self.anchor_num * self.num_classes)[inds]
File "/home/abab/QueryDet-PyTorch-main/models/querydet/qinfer.py", line 201, in _run_spconvs
y = filters(x)
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/pytorch/modules.py", line 137, in forward
input = module(input)
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/pytorch/conv.py", line 330, in forward
out_features = Fsp.indice_subm_conv(
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 216, in decorate_fwd
return fwd(*args, **kwargs)
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/pytorch/functional.py", line 294, in forward
raise e
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/pytorch/functional.py", line 279, in forward
return ops.indice_conv(features,
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/pytorch/ops.py", line 695, in indice_conv
tuned_res, min_time = GEMM.tune_and_cache(
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/algo.py", line 358, in tune_and_cache
GemmMainUnitTest.matmul2(params)
RuntimeError: /io/build/temp.linux-x86_64-cpython-38/spconv/build/src/cumm/gemm/main/GemmMainUnitTest/GemmMainUnitTest_matmul2.cu 4489
cuda execution failed with error 209 no kernel image is available for execution on the device
Simt_f32f32f32f32f32tnt_m64n256k8m32n64k8_2_SAC00 error with params [100432, 256] [256, 256] [100432, 256]

Could you please help me with this problem? Thank you.

[wenxi2]

sys.platform linux
Python 3.8.17 (default, Jul 5 2023, 21:04:15) [GCC 11.2.0]
numpy 1.24.3
detectron2 0.6 @/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/detectron2
Compiler GCC 7.3
CUDA compiler CUDA 11.1
detectron2 arch flags 3.7, 5.0, 5.2, 6.0, 6.1, 7.0, 7.5, 8.0, 8.6
DETECTRON2_ENV_MODULE
PyTorch 1.8.0+cu111 @/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch
PyTorch debug build False
GPU available Yes
GPU 0,1 NVIDIA GeForce RTX 3080 (arch=8.6)
Driver version 535.183.01
CUDA_HOME /usr/local/cuda-11.7
Pillow 6.2.1
torchvision 0.9.0+cu111 @/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torchvision
torchvision arch flags 3.5, 5.0, 6.0, 7.0, 7.5, 8.0, 8.6
fvcore 0.1.5.post20221221
iopath 0.1.9
cv2 4.2.0

---

PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
• CuDNN 8.0.5
• Magma 2.5.2