Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add a check on grid before launching cuda kernels (pytorch#2639)
Summary: Pull Request resolved: pytorch#2639 We had errors like P1382114867 ``` c10::Error: CUDA error: invalid configuration argument CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Device-side assertion tracking was not enabled by user. Exception raised from c10_cuda_check_implementation at fbcode/caffe2/c10/cuda/CUDAException.cpp:43 (most recent call first): # 7 c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) # 8 c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) # 9 fbgemm_gpu::reorder_batched_ad_lengths_gpu(at::Tensor const&, at::Tensor const&, long, bool, long) ``` The `CUDA error: invalid configuration argument` usually means that the configuration we used to launch the cuda kernel is invalid https://fburl.com/wiki/cssgu0vs But the error message is not very helpful for debugging, this diff adds a check on grid size before launching the kernel, and logs the variables calculating the grid size to help quickly identify the errors and causes. Reviewed By: sryap, gnahzg Differential Revision: D57932413 fbshipit-source-id: d558a3eb2f0d7404ff5cda2eb9a80f8ffd17471d
- Loading branch information