Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error when run tools/train.py #6

Open
IAS-01 opened this issue Nov 23, 2021 · 21 comments
Open

error when run tools/train.py #6

IAS-01 opened this issue Nov 23, 2021 · 21 comments
Labels
good first issue Good for newcomers

Comments

@IAS-01
Copy link

IAS-01 commented Nov 23, 2021

[Warning] Your mxnet does not support ProposalTarget [Warning] Your mxnet does not support DecodeBBox [Warning] Your mxnet does not support BBoxNorm [Warning] Your mxnet does not support FocalLoss Traceback (most recent call last): File "tools/train.py", line 412, in <module> train_net(parse_args()) File "tools/train.py", line 407, in parse_args config = importlib.import_module(args.config.replace('.py', '').replace('/', '.')) File "/mmdetection3d/RangeDet/config/rangedet/rangedet_wo_aug_4_18e.py", line 22, in <module> from rangedet.symbol.head.builder import RangeRCNN as Detector File "/mmdetection3d/RangeDet/rangedet/symbol/head/builder.py", line 4, in <module> from operator_py import \ File "/mmdetection3d/RangeDet/operator_py/batch_rotated_iou.py", line 3, in <module> from mxnet.ndarray.contrib import RotatedIOU ImportError: cannot import name 'RotatedIOU' from 'mxnet.ndarray.contrib' (/usr/local/lib/python3.7/dist-packages/mxnet/ndarray/contrib.py)

the link to download MxNet is invalid, can you provide a new link?

this is provided by the SimpleDet, but it is invalid
https://1dv.aflat.top/mxnet_cu101-1.6.0b20191214-py2.py3-none-manylinux1_x86_64.whl

@Abyssaledge
Copy link
Collaborator

Sorry for the inconvenience. This error is raised because our customized MXNet operators in operator_cxx/contrib is not built.
First you need to clone MXNet from its repo and build it from the source. This is because the release version of MXNet is 1.8.0, which does not support numpy-style API. If you are not familiar with the whole installation process, please be patient and wait for a couple of days. We will provide detailed instructions after finishing this period of CVPR. A little bit busy now.

@IAS-01
Copy link
Author

IAS-01 commented Nov 23, 2021

Sorry for the inconvenience. This error is raised because our customized MXNet operators in operator_cxx/contrib is not built. First you need to clone MXNet from its repo and build it from the source. This is because the release version of MXNet is 1.8.0, which does not support numpy-style API. If you are not familiar with the whole installation process, please be patient and wait for a couple of days. We will provide detailed instructions after finishing this period of CVPR. A little bit busy now.

Thank you very much, can you provide a compiled Mxnet whl package, if it is convenient

@Abyssaledge
Copy link
Collaborator

We have not tried to make the wheel file before. This might be a solution, but we need time to check. I suggest you follow our upcoming instructions if you are still stuck at this moment and not in a hurry.

@Treemann
Copy link

Hi @Abyssaledge ,
I prepare all the dependencies and try with mxnet-2.0 and error occurs.
from mxnet import metric ImportError: cannot import name 'metric'
Because in mxnet-2.0 metrics is under mxnet.gloun, I believe your code is developed under version<2.
Then I install mxnet-1.9.x and install rangedet by "pip install -v -e ." and there is no error in the log, but I notice that your self-defined operators seems not be compiled and installed. Is there anything else to do for the installation

[Warning] Your mxnet does not support ProposalTarget
[Warning] Your mxnet does not support DecodeBBox
[Warning] Your mxnet does not support BBoxNorm
[Warning] Your mxnet does not support FocalLoss
.......
    from mxnet.ndarray.contrib import RotatedIOU
ImportError: cannot import name 'RotatedIOU'

@IAS-01 Have you solve the problem ?

@Abyssaledge
Copy link
Collaborator

@Treemann Could you please provide the detailed information how you install MXNet and customized operators?

@Treemann
Copy link

@Abyssaledge , I follow the instruction of mxnet

# https://mxnet.apache.org/versions/1.8.0/get_started/build_from_source.html
git clone -b v1.9.x https://github.com/apache/incubator-mxnet.git --recursive
cd incubator-mxnet
cp config/linux_gpu.cmake config.cmake
mkdir build; cd build
cmake ..
cmake --build .
cd ..
python3 -m pip install -e ./python

Then following your instruction:
pip install -v -e .

@Abyssaledge
Copy link
Collaborator

@Treemann
Thanks for information and sorry for this inconvenience. The error about metric import is weird, because we develope RangeDet with MXNet 2.0 and we never meet this error. We will check this in couple of days and answer you as soon as we figure it out.

@Treemann
Copy link

@Abyssaledge FYI
For trying mxnet-2.0, I used mxnet-2.0.0a0 with pip installation since there is a released pip version, I didn't build mxnet-2.0 from source code.

@Abyssaledge
Copy link
Collaborator

Abyssaledge commented Dec 26, 2021

@Treemann Hi, I found that the API of mxnet2.0 (in the master branch) has changed. You can pull the newest MXNet from the master branch and checkout to 1496c91871b9d81d6a18785bdc8a1c3450bedbca commit and build it from source, where we conduct our experiments and from mxnet import metric works fine.
As for the customized operators, you should modify the mx_home to your MXNet directory and simply run make in operator_cxx/contrib.
BTW, we have updated the train.py and test.py to import the customized operator.

@Treemann
Copy link

Thanks for your reply.
I'll try following your instruction and give some feedback.

@Treemann
Copy link

Hi @Abyssaledge, I followed your instruction to install all the dependencies (mxnet, openmpi, etc.), the aforementioned error disappeared, but some other error occurred.
Waiting for your update about the docs on environment setup.

[Warning] Your mxnet does not support ProposalTarget
[Warning] Your mxnet does not support DecodeBBox
[Warning] Your mxnet does not support BBoxNorm
[Warning] Your mxnet does not support FocalLoss
Please using new data mean and var
-------------------- meta_baseline_bias --------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
......
  ompi_mpi_init: ompi_rte_init failed
  --> Returned "(null)" (-43) instead of "Success" (0)

@Abyssaledge
Copy link
Collaborator

Abyssaledge commented Dec 30, 2021

Can you successfully run the code with single GPU training? @Treemann

@Treemann
Copy link

@Abyssaledge , I find the problem is related to horovod.
After I change the cuda and nccl version, horovod should be recompiled & reinstalled.
Now the training goes well. Thanks!

@ruanhao5
Copy link

Hi, @Abyssaledge, I follow the instruction of mxnet:

git clone --recursive https://github.com/apache/incubator-mxnet mxnet
cd mxnet
git checkout 1496c91871b9d81d6a18785bdc8a1c3450bedbca
sudo apt-get update
sudo apt-get install -y build-essential git ninja-build ccache libopenblas-dev libopencv-dev cmake
cp config/linux_gpu.cmake config.cmake
mkdir build; cd build
cmake ..

After running cmake .., it reported an error as follow:

-- The C compiler identification is GNU 7.4.0                         
-- The CXX compiler identification is GNU 7.4.0                       
-- Detecting C compiler ABI info                                      
-- Detecting C compiler ABI info - done                               
-- Check for working C compiler: /usr/bin/cc - skipped                
-- Detecting C compile features                                       
-- Detecting C compile features - done                                
-- Detecting CXX compiler ABI info                                    
-- Detecting CXX compiler ABI info - done                             
-- Check for working CXX compiler: /usr/bin/c++ - skipped             
-- Detecting CXX compile features                                     
-- Detecting CXX compile features - done                              
-- CMAKE_CROSSCOMPILING FALSE                                         
-- CMAKE_HOST_SYSTEM_PROCESSOR x86_64                                 
-- CMAKE_SYSTEM_PROCESSOR x86_64                                      
-- CMAKE_SYSTEM_NAME Linux                                            
-- CMake version '3.22.3' using generator 'Unix Makefiles'            
-- Looking for a CUDA compiler                                        
-- Looking for a CUDA compiler - /data/cuda/cuda-11.2/cuda/bin/nvcc   
-- The CUDA compiler identification is NVIDIA 11.2.142                
-- Detecting CUDA compiler ABI info                                   
-- Detecting CUDA compiler ABI info - done                            
-- Check for working CUDA compiler: /data/cuda/cuda-11.2/cuda/bin/nvcc
 - skipped                                                            
-- Detecting CUDA compile features                                    
-- Detecting CUDA compile features - done                             
CMake Error at CMakeLists.txt:255 (add_subdirectory):                 
  The source directory                                                
                                                                      
    /home/ruanhao/mxnet/3rdparty/mkldnn                               
                                                                      
  does not contain a CMakeLists.txt file.  

I find that mkldnn is a empty folder and its path is mxnet/3rdparty.
When branch is master, mxnet/3rdparty doesn't have 'mkldnn' folder. When checout 1496c91871b9d81d6a18785bdc8a1c3450bedbca, mxnet/3rdparty will create empty 'mkldnn' folder.

Could you please provide detailed installation instructions for mxnet? Thanks.

@Abyssaledge
Copy link
Collaborator

I did not occur this error if I recall correctly. I will try to reproduce this error in my free time. But I cannot guarantee the solution, it's indeed tricky to install MXNet in different environments . @ruanhao5

@ruanhao5
Copy link

@Abyssaledge Thanks for your reply.
I had solved that error.
I think mkldnn is a submodule in mxnet. So I run the command:

git submodule sync
git submodule update --init --recursive

After that, there is no error.

@ruanhao5
Copy link

@Abyssaledge Hi, I had installed all the dependencies (mxnet, openmpi, etc.).
As for the customized operators, I modified the mx_home to my MXNet directory and simply run make in operator_cxx/contrib.
Some other error occurred:

g++ -std=c++11 -shared -o contrib_cxx.so nms_3d.cc decode_3d_bbox.cc rotated_iou.cc nms_3d.o rotated_iou.o decode_3d_bbox.o -O2 -msse2 -msse3 -fPIC -I/home/ruanhao/range_view/mxnet/3rdparty/dmlc-core/include -I/home/ruanhao/range_view/mxnet/3rdparty/nnvm/include -I/home/ruanhao/range_view/mxnet/include -I/home/ruanhao/range_view/mxnet/3rdparty/mshadow -I/home/ruanhao/range_view/mxnet/3rdparty/dlpack/include -I/data/cuda/cuda-11.2/cuda/include -I/home/ruanhao/range_view/mxnet/src/operator -I/home/ruanhao/range_view/mxnet/src/common -I/data/cuda/cuda-11.2/cudnn/v8.1.0/include -D MSHADOW_USE_CBLAS -D MSHADOW_USE_CUDA -D MSHADOW_USE_CUDNN=1 -D MSHADOW_USE_CUSOLVER=1 -DMSHADOW_USE_F16C=0 -L/home/ruanhao/range_view/mxnet/lib -lmxnet
/usr/bin/ld: cannot find -lmxnet
collect2: error: ld returned 1 exit status
Makefile:30: recipe for target 'contrib_cxx.so' failed
make: *** [contrib_cxx.so] Error 1

I opened path /home/ruanhao/range_view/RangeDet/operator_cxx/contrib/ and checked Makefile file:

mx_home=/home/ruanhao/range_view/mxnet
dmlc_inc=${mx_home}/3rdparty/dmlc-core/include
nnvm_inc=${mx_home}/3rdparty/nnvm/include
mshadow_inc=${mx_home}/3rdparty/mshadow
dlpack_inc=${mx_home}/3rdparty/dlpack/include
mxnet_src_inc=${mx_home}/src/operator
mxnet_src_com=${mx_home}/src/common
mxnet_inc=${mx_home}/include
cuda_inc=/data/cuda/cuda-11.2/cuda/include
cudnn_inc=/data/cuda/cuda-11.2/cudnn/v8.1.0/include

mxnet_lib=${mx_home}

INC = -I${dmlc_inc} -I${nnvm_inc} -I${mxnet_inc} -I${mshadow_inc} -I${dlpack_inc} -I${cuda_inc} -I${mxnet_src_inc} -I${mxnet_src_com} -I${cudnn_inc}
DEFINE = -D MSHADOW_USE_CBLAS -D MSHADOW_USE_CUDA -D MSHADOW_USE_CUDNN=1 -D MSHADOW_USE_CUSOLVER=1 -DMSHADOW_USE_F16C=0
LIB = -L${mxnet_lib}/lib -lmxnet
# LIB = -L/home/ruanhao/range_view/mxnet/build -lmxnet

all: clean build cleancache

build: contrib_cxx.so

%.o: %.cu
	nvcc -std=c++11 -c -o $@ $? -O2 -x cu -Xcompiler -fPIC ${INC} ${DEFINE} 
contrib_cxx.so: $(wildcard *.cc) $(subst .cu,.o, $(wildcard *.cu))
	g++ -std=c++11 -shared -o $@ $? -O2 -msse2 -msse3 -fPIC ${INC} ${DEFINE} ${LIB}

clean:
	rm -f *.o *.so *.pyc

cleancache:
	rm -f *.o

I think the problem is the LIB path and then I opened the path ${mxnet_lib}. There was no lib folder.
But, path /home/ruanhao/range_view/mxnet/build had libmxnet.so and libmxnet.a files.
I guess LIB should be LIB = -L/home/ruanhao/range_view/mxnet/build -lmxnet. It didn't occur errors after running make in operator_cxx/contrib.

I run python tools/train.py --config "config/rangedet/rangedet_veh_wo_aug_all_36e.py" and error occurred:

Traceback (most recent call last):
  File "tools/train.py", line 2, in <module>
    _=ctypes.CDLL('./operator_cxx/contrib/contrib_cxx.so')
  File "/home/ruanhao/miniconda3/envs/rangedet/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libmxnet.so: cannot open shared object file: No such file or directory

Hence, customized operators did not install successfully.
Could you please provide some advice?

@happynear
Copy link
Collaborator

Now It seems that you already get './operator_cxx/contrib/contrib_cxx.so'. The problem is the program cannot find libmxnet.so. Maybe you can try to add the folder of libmxnet.so into PATH?

@ruanhao5
Copy link

@happynear Hi, I added the folder of libmxnet.so into LD_LIBRARY_PATH:

export LD_LIBRARY_PATH="/home/ruanhao/range_view/mxnet/build:$LD_LIBRARY_PATH"

The error disappeared. Thanks for your reply.

@ruanhao5
Copy link

ruanhao5 commented Apr 20, 2022

@happynear @Abyssaledge Hi, I run python tools/train.py --config "config/rangedet/rangedet_veh_wo_aug_all_36e.py" and error occurred:

Click to expand error!
[Warning] Your mxnet does not support ProposalTarget
[Warning] Your mxnet does not support DecodeBBox
[Warning] Your mxnet does not support BBoxNorm
[Warning] Your mxnet does not support FocalLoss
Please using new data mean and var
-------------------- meta_baseline_bias --------------------
04-19 11:01:54 rank:0/1 gpu list:[0, 1, 2, 3, 4, 5, 6, 7]
cpu allowed list len:8, [36, 37, 38, 39, 40, 41, 42, 43]
04-19 11:01:54 rank: 0, pid:237, affinity to cpu [36, 37, 38, 39, 40, 41, 42, 43]
04-19 11:01:54 [0/1] local sync flag:True
04-19 11:01:54 input batch size:16
load all training data with 39447 records
loader rank:0, partition count:39447
{'input_data': (16, 8, 64, 2656), 'rpn_reg_target_s1': (16, 8, 64, 2656), 'rpn_reg_target_s2': (16, 8, 64, 1328), 'rpn_reg_target_s4': (16, 8, 64, 664), 'rpn_reg_weight_s1': (16, 8, 64, 2656), 'rpn_reg_weight_s2': (16, 8, 64, 1328), 'rpn_reg_weight_s4': (16, 8, 64, 664), 'range_image_mask_s1': (16, 1, 64, 2656), 'range_image_mask_s2': (16, 1, 64, 1328), 'range_image_mask_s4': (16, 1, 64, 664), 'reg_normalize_weight_s1': (16, 8, 64, 2656), 'reg_normalize_weight_s2': (16, 8, 64, 1328), 'reg_normalize_weight_s4': (16, 8, 64, 664), 'pc_vehicle_frame_s1': (16, 169984, 3), 'pc_vehicle_frame_s2': (16, 84992, 3), 'pc_vehicle_frame_s4': (16, 42496, 3), 'gt_bbox_veh_for_iou_pred': (16, 200, 8), 'coord_s1': (16, 3, 64, 2656)}
04-19 11:01:59 parameter shape
04-19 11:01:59 [('input_data', (2, 8, 64, 2656)),
 ('res1_unit1_conv1_weight', (64, 8, 3, 3)),
 ('res1_unit1_bn1_gamma', (64,)),
 ('res1_unit1_bn1_beta', (64,)),
 ('res1_unit1_bn1_moving_mean', (64,)),
 ('res1_unit1_bn1_moving_var', (64,)),
 ('res1_unit1_conv2_weight', (64, 64, 3, 3)),
 ('res1_unit1_bn2_gamma', (64,)),
 ('res1_unit1_bn2_beta', (64,)),
 ('res1_unit1_bn2_moving_mean', (64,)),
 ('res1_unit1_bn2_moving_var', (64,)),
 ('res1_unit1_sc_weight', (64, 8, 1, 1)),
 ('res1_unit1_sc_bn_gamma', (64,)),
 ('res1_unit1_sc_bn_beta', (64,)),
 ('res1_unit1_sc_bn_moving_mean', (64,)),
 ('res1_unit1_sc_bn_moving_var', (64,)),
 ('coord_s1', (2, 3, 64, 2656)),
 ('res1_unit2_2656_mlp0_weight', (32, 3, 1, 1)),
 ('res1_unit2_2656_mlp0_bias', (32,)),
 ('res1_unit2_2656_mlp1_weight', (64, 32, 1, 1)),
 ('res1_unit2_2656_mlp1_bias', (64,)),
 ('res1_unit2point_wise_mlp_bn1_gamma', (576,)),
 ('res1_unit2point_wise_mlp_bn1_beta', (576,)),
 ('res1_unit2point_wise_mlp_bn1_moving_mean', (576,)),
 ('res1_unit2point_wise_mlp_bn1_moving_var', (576,)),
 ('res1_unit2aggregation_conv1_weight', (64, 576, 1, 1)),
 ('res1_unit2aggregation_bn1_gamma', (64,)),
 ('res1_unit2aggregation_bn1_beta', (64,)),
 ('res1_unit2aggregation_bn1_moving_mean', (64,)),
 ('res1_unit2aggregation_bn1_moving_var', (64,)),
 ('res1_unit2_conv2_weight', (64, 64, 3, 3)),
 ('res1_unit2_bn2_gamma', (64,)),
 ('res1_unit2_bn2_beta', (64,)),
 ('res1_unit2_bn2_moving_mean', (64,)),
 ('res1_unit2_bn2_moving_var', (64,)),
 ('res2a_unit1_conv1_weight', (64, 64, 3, 3)),
 ('res2a_unit1_bn1_gamma', (64,)),
 ('res2a_unit1_bn1_beta', (64,)),
 ('res2a_unit1_bn1_moving_mean', (64,)),
 ('res2a_unit1_bn1_moving_var', (64,)),
 ('res2a_unit1_conv2_weight', (64, 64, 3, 3)),
 ('res2a_unit1_bn2_gamma', (64,)),
 ('res2a_unit1_bn2_beta', (64,)),
 ('res2a_unit1_bn2_moving_mean', (64,)),
 ('res2a_unit1_bn2_moving_var', (64,)),
 ('res2a_unit1_sc_weight', (64, 64, 1, 1)),
 ('res2a_unit1_sc_bn_gamma', (64,)),
 ('res2a_unit1_sc_bn_beta', (64,)),
 ('res2a_unit1_sc_bn_moving_mean', (64,)),
 ('res2a_unit1_sc_bn_moving_var', (64,)),
 ('res2a_unit2_conv1_weight', (64, 64, 3, 3)),
 ('res2a_unit2_bn1_gamma', (64,)),
 ('res2a_unit2_bn1_beta', (64,)),
 ('res2a_unit2_bn1_moving_mean', (64,)),
 ('res2a_unit2_bn1_moving_var', (64,)),
 ('res2a_unit2_conv2_weight', (64, 64, 3, 3)),
 ('res2a_unit2_bn2_gamma', (64,)),
 ('res2a_unit2_bn2_beta', (64,)),
 ('res2a_unit2_bn2_moving_mean', (64,)),
 ('res2a_unit2_bn2_moving_var', (64,)),
 ('res2a_unit3_conv1_weight', (64, 64, 3, 3)),
 ('res2a_unit3_bn1_gamma', (64,)),
 ('res2a_unit3_bn1_beta', (64,)),
 ('res2a_unit3_bn1_moving_mean', (64,)),
 ('res2a_unit3_bn1_moving_var', (64,)),
 ('res2a_unit3_conv2_weight', (64, 64, 3, 3)),
 ('res2a_unit3_bn2_gamma', (64,)),
 ('res2a_unit3_bn2_beta', (64,)),
 ('res2a_unit3_bn2_moving_mean', (64,)),
 ('res2a_unit3_bn2_moving_var', (64,)),
 ('res2_unit1_conv1_weight', (128, 64, 3, 3)),
 ('res2_unit1_bn1_gamma', (128,)),
 ('res2_unit1_bn1_beta', (128,)),
 ('res2_unit1_bn1_moving_mean', (128,)),
 ('res2_unit1_bn1_moving_var', (128,)),
 ('res2_unit1_conv2_weight', (128, 128, 3, 3)),
 ('res2_unit1_bn2_gamma', (128,)),
 ('res2_unit1_bn2_beta', (128,)),
 ('res2_unit1_bn2_moving_mean', (128,)),
 ('res2_unit1_bn2_moving_var', (128,)),
 ('res2_unit1_sc_weight', (128, 64, 1, 1)),
 ('res2_unit1_sc_bn_gamma', (128,)),
 ('res2_unit1_sc_bn_beta', (128,)),
 ('res2_unit1_sc_bn_moving_mean', (128,)),
 ('res2_unit1_sc_bn_moving_var', (128,)),
 ('res2_unit2_conv1_weight', (128, 128, 3, 3)),
 ('res2_unit2_bn1_gamma', (128,)),
 ('res2_unit2_bn1_beta', (128,)),
 ('res2_unit2_bn1_moving_mean', (128,)),
 ('res2_unit2_bn1_moving_var', (128,)),
 ('res2_unit2_conv2_weight', (128, 128, 3, 3)),
 ('res2_unit2_bn2_gamma', (128,)),
 ('res2_unit2_bn2_beta', (128,)),
 ('res2_unit2_bn2_moving_mean', (128,)),
 ('res2_unit2_bn2_moving_var', (128,)),
 ('res2_unit3_conv1_weight', (128, 128, 3, 3)),
 ('res2_unit3_bn1_gamma', (128,)),
 ('res2_unit3_bn1_beta', (128,)),
 ('res2_unit3_bn1_moving_mean', (128,)),
 ('res2_unit3_bn1_moving_var', (128,)),
 ('res2_unit3_conv2_weight', (128, 128, 3, 3)),
 ('res2_unit3_bn2_gamma', (128,)),
 ('res2_unit3_bn2_beta', (128,)),
 ('res2_unit3_bn2_moving_mean', (128,)),
 ('res2_unit3_bn2_moving_var', (128,)),
 ('agg1_deconv_weight', (128, 64, 3, 8)),
 ('agg1_deconv_bn_gamma', (64,)),
 ('agg1_deconv_bn_beta', (64,)),
 ('agg1_deconv_bn_moving_mean', (64,)),
 ('agg1_deconv_bn_moving_var', (64,)),
 ('agg1_res_unit1_conv1_weight', (64, 64, 3, 3)),
 ('agg1_res_unit1_bn1_gamma', (64,)),
 ('agg1_res_unit1_bn1_beta', (64,)),
 ('agg1_res_unit1_bn1_moving_mean', (64,)),
 ('agg1_res_unit1_bn1_moving_var', (64,)),
 ('agg1_res_unit1_conv2_weight', (64, 64, 3, 3)),
 ('agg1_res_unit1_bn2_gamma', (64,)),
 ('agg1_res_unit1_bn2_beta', (64,)),
 ('agg1_res_unit1_bn2_moving_mean', (64,)),
 ('agg1_res_unit1_bn2_moving_var', (64,)),
 ('agg1_res_unit1_sc_weight', (64, 64, 1, 1)),
 ('agg1_res_unit1_sc_bn_gamma', (64,)),
 ('agg1_res_unit1_sc_bn_beta', (64,)),
 ('agg1_res_unit1_sc_bn_moving_mean', (64,)),
 ('agg1_res_unit1_sc_bn_moving_var', (64,)),
 ('agg1_res_unit2_conv1_weight', (64, 64, 3, 3)),
 ('agg1_res_unit2_bn1_gamma', (64,)),
 ('agg1_res_unit2_bn1_beta', (64,)),
 ('agg1_res_unit2_bn1_moving_mean', (64,)),
 ('agg1_res_unit2_bn1_moving_var', (64,)),
 ('agg1_res_unit2_conv2_weight', (64, 64, 3, 3)),
 ('agg1_res_unit2_bn2_gamma', (64,)),
 ('agg1_res_unit2_bn2_beta', (64,)),
 ('agg1_res_unit2_bn2_moving_mean', (64,)),
 ('agg1_res_unit2_bn2_moving_var', (64,)),
 ('res3a_unit1_conv1_weight', (128, 128, 3, 3)),
 ('res3a_unit1_bn1_gamma', (128,)),
 ('res3a_unit1_bn1_beta', (128,)),
 ('res3a_unit1_bn1_moving_mean', (128,)),
 ('res3a_unit1_bn1_moving_var', (128,)),
 ('res3a_unit1_conv2_weight', (128, 128, 3, 3)),
 ('res3a_unit1_bn2_gamma', (128,)),
 ('res3a_unit1_bn2_beta', (128,)),
 ('res3a_unit1_bn2_moving_mean', (128,)),
 ('res3a_unit1_bn2_moving_var', (128,)),
 ('res3a_unit1_sc_weight', (128, 128, 1, 1)),
 ('res3a_unit1_sc_bn_gamma', (128,)),
 ('res3a_unit1_sc_bn_beta', (128,)),
 ('res3a_unit1_sc_bn_moving_mean', (128,)),
 ('res3a_unit1_sc_bn_moving_var', (128,)),
 ('res3a_unit2_conv1_weight', (128, 128, 3, 3)),
 ('res3a_unit2_bn1_gamma', (128,)),
 ('res3a_unit2_bn1_beta', (128,)),
 ('res3a_unit2_bn1_moving_mean', (128,)),
 ('res3a_unit2_bn1_moving_var', (128,)),
 ('res3a_unit2_conv2_weight', (128, 128, 3, 3)),
 ('res3a_unit2_bn2_gamma', (128,)),
 ('res3a_unit2_bn2_beta', (128,)),
 ('res3a_unit2_bn2_moving_mean', (128,)),
 ('res3a_unit2_bn2_moving_var', (128,)),
 ('res3a_unit3_conv1_weight', (128, 128, 3, 3)),
 ('res3a_unit3_bn1_gamma', (128,)),
 ('res3a_unit3_bn1_beta', (128,)),
 ('res3a_unit3_bn1_moving_mean', (128,)),
 ('res3a_unit3_bn1_moving_var', (128,)),
 ('res3a_unit3_conv2_weight', (128, 128, 3, 3)),
 ('res3a_unit3_bn2_gamma', (128,)),
 ('res3a_unit3_bn2_beta', (128,)),
 ('res3a_unit3_bn2_moving_mean', (128,)),
 ('res3a_unit3_bn2_moving_var', (128,)),
 ('res3a_unit4_conv1_weight', (128, 128, 3, 3)),
 ('res3a_unit4_bn1_gamma', (128,)),
 ('res3a_unit4_bn1_beta', (128,)),
 ('res3a_unit4_bn1_moving_mean', (128,)),
 ('res3a_unit4_bn1_moving_var', (128,)),
 ('res3a_unit4_conv2_weight', (128, 128, 3, 3)),
 ('res3a_unit4_bn2_gamma', (128,)),
 ('res3a_unit4_bn2_beta', (128,)),
 ('res3a_unit4_bn2_moving_mean', (128,)),
 ('res3a_unit4_bn2_moving_var', (128,)),
 ('res3a_unit5_conv1_weight', (128, 128, 3, 3)),
 ('res3a_unit5_bn1_gamma', (128,)),
 ('res3a_unit5_bn1_beta', (128,)),
 ('res3a_unit5_bn1_moving_mean', (128,)),
 ('res3a_unit5_bn1_moving_var', (128,)),
 ('res3a_unit5_conv2_weight', (128, 128, 3, 3)),
 ('res3a_unit5_bn2_gamma', (128,)),
 ('res3a_unit5_bn2_beta', (128,)),
 ('res3a_unit5_bn2_moving_mean', (128,)),
 ('res3a_unit5_bn2_moving_var', (128,)),
 ('res3_unit1_conv1_weight', (128, 128, 3, 3)),
 ('res3_unit1_bn1_gamma', (128,)),
 ('res3_unit1_bn1_beta', (128,)),
 ('res3_unit1_bn1_moving_mean', (128,)),
 ('res3_unit1_bn1_moving_var', (128,)),
 ('res3_unit1_conv2_weight', (128, 128, 3, 3)),
 ('res3_unit1_bn2_gamma', (128,)),
 ('res3_unit1_bn2_beta', (128,)),
 ('res3_unit1_bn2_moving_mean', (128,)),
 ('res3_unit1_bn2_moving_var', (128,)),
 ('res3_unit1_sc_weight', (128, 128, 1, 1)),
 ('res3_unit1_sc_bn_gamma', (128,)),
 ('res3_unit1_sc_bn_beta', (128,)),
 ('res3_unit1_sc_bn_moving_mean', (128,)),
 ('res3_unit1_sc_bn_moving_var', (128,)),
 ('res3_unit2_conv1_weight', (128, 128, 3, 3)),
 ('res3_unit2_bn1_gamma', (128,)),
 ('res3_unit2_bn1_beta', (128,)),
 ('res3_unit2_bn1_moving_mean', (128,)),
 ('res3_unit2_bn1_moving_var', (128,)),
 ('res3_unit2_conv2_weight', (128, 128, 3, 3)),
 ('res3_unit2_bn2_gamma', (128,)),
 ('res3_unit2_bn2_beta', (128,)),
 ('res3_unit2_bn2_moving_mean', (128,)),
 ('res3_unit2_bn2_moving_var', (128,)),
 ('res3_unit3_conv1_weight', (128, 128, 3, 3)),
 ('res3_unit3_bn1_gamma', (128,)),
 ('res3_unit3_bn1_beta', (128,)),
 ('res3_unit3_bn1_moving_mean', (128,)),
 ('res3_unit3_bn1_moving_var', (128,)),
 ('res3_unit3_conv2_weight', (128, 128, 3, 3)),
 ('res3_unit3_bn2_gamma', (128,)),
 ('res3_unit3_bn2_beta', (128,)),
 ('res3_unit3_bn2_moving_mean', (128,)),
 ('res3_unit3_bn2_moving_var', (128,)),
 ('res3_unit4_conv1_weight', (128, 128, 3, 3)),
 ('res3_unit4_bn1_gamma', (128,)),
 ('res3_unit4_bn1_beta', (128,)),
 ('res3_unit4_bn1_moving_mean', (128,)),
 ('res3_unit4_bn1_moving_var', (128,)),
 ('res3_unit4_conv2_weight', (128, 128, 3, 3)),
 ('res3_unit4_bn2_gamma', (128,)),
 ('res3_unit4_bn2_beta', (128,)),
 ('res3_unit4_bn2_moving_mean', (128,)),
 ('res3_unit4_bn2_moving_var', (128,)),
 ('res3_unit5_conv1_weight', (128, 128, 3, 3)),
 ('res3_unit5_bn1_gamma', (128,)),
 ('res3_unit5_bn1_beta', (128,)),
 ('res3_unit5_bn1_moving_mean', (128,)),
 ('res3_unit5_bn1_moving_var', (128,)),
 ('res3_unit5_conv2_weight', (128, 128, 3, 3)),
 ('res3_unit5_bn2_gamma', (128,)),
 ('res3_unit5_bn2_beta', (128,)),
 ('res3_unit5_bn2_moving_mean', (128,)),
 ('res3_unit5_bn2_moving_var', (128,)),
 ('agg2_deconv_weight', (128, 128, 3, 8)),
 ('agg2_deconv_bn_gamma', (128,)),
 ('agg2_deconv_bn_beta', (128,)),
 ('agg2_deconv_bn_moving_mean', (128,)),
 ('agg2_deconv_bn_moving_var', (128,)),
 ('agg2_res_unit1_conv1_weight', (128, 128, 3, 3)),
 ('agg2_res_unit1_bn1_gamma', (128,)),
 ('agg2_res_unit1_bn1_beta', (128,)),
 ('agg2_res_unit1_bn1_moving_mean', (128,)),
 ('agg2_res_unit1_bn1_moving_var', (128,)),
 ('agg2_res_unit1_conv2_weight', (128, 128, 3, 3)),
 ('agg2_res_unit1_bn2_gamma', (128,)),
 ('agg2_res_unit1_bn2_beta', (128,)),
 ('agg2_res_unit1_bn2_moving_mean', (128,)),
 ('agg2_res_unit1_bn2_moving_var', (128,)),
 ('agg2_res_unit1_sc_weight', (128, 128, 1, 1)),
 ('agg2_res_unit1_sc_bn_gamma', (128,)),
 ('agg2_res_unit1_sc_bn_beta', (128,)),
 ('agg2_res_unit1_sc_bn_moving_mean', (128,)),
 ('agg2_res_unit1_sc_bn_moving_var', (128,)),
 ('agg2_res_unit2_conv1_weight', (128, 128, 3, 3)),
 ('agg2_res_unit2_bn1_gamma', (128,)),
 ('agg2_res_unit2_bn1_beta', (128,)),
 ('agg2_res_unit2_bn1_moving_mean', (128,)),
 ('agg2_res_unit2_bn1_moving_var', (128,)),
 ('agg2_res_unit2_conv2_weight', (128, 128, 3, 3)),
 ('agg2_res_unit2_bn2_gamma', (128,)),
 ('agg2_res_unit2_bn2_beta', (128,)),
 ('agg2_res_unit2_bn2_moving_mean', (128,)),
 ('agg2_res_unit2_bn2_moving_var', (128,)),
 ('agg2a_deconv_weight', (128, 64, 3, 4)),
 ('agg2a_deconv_bn_gamma', (64,)),
 ('agg2a_deconv_bn_beta', (64,)),
 ('agg2a_deconv_bn_moving_mean', (64,)),
 ('agg2a_deconv_bn_moving_var', (64,)),
 ('agg2a_res_unit1_conv1_weight', (64, 64, 3, 3)),
 ('agg2a_res_unit1_bn1_gamma', (64,)),
 ('agg2a_res_unit1_bn1_beta', (64,)),
 ('agg2a_res_unit1_bn1_moving_mean', (64,)),
 ('agg2a_res_unit1_bn1_moving_var', (64,)),
 ('agg2a_res_unit1_conv2_weight', (64, 64, 3, 3)),
 ('agg2a_res_unit1_bn2_gamma', (64,)),
 ('agg2a_res_unit1_bn2_beta', (64,)),
 ('agg2a_res_unit1_bn2_moving_mean', (64,)),
 ('agg2a_res_unit1_bn2_moving_var', (64,)),
 ('agg2a_res_unit1_sc_weight', (64, 64, 1, 1)),
 ('agg2a_res_unit1_sc_bn_gamma', (64,)),
 ('agg2a_res_unit1_sc_bn_beta', (64,)),
 ('agg2a_res_unit1_sc_bn_moving_mean', (64,)),
 ('agg2a_res_unit1_sc_bn_moving_var', (64,)),
 ('agg3_deconv_weight', (64, 64, 3, 4)),
 ('agg3_deconv_bn_gamma', (64,)),
 ('agg3_deconv_bn_beta', (64,)),
 ('agg3_deconv_bn_moving_mean', (64,)),
 ('agg3_deconv_bn_moving_var', (64,)),
 ('agg3_res_unit1_conv1_weight', (64, 64, 3, 3)),
 ('agg3_res_unit1_bn1_gamma', (64,)),
 ('agg3_res_unit1_bn1_beta', (64,)),
 ('agg3_res_unit1_bn1_moving_mean', (64,)),
 ('agg3_res_unit1_bn1_moving_var', (64,)),
 ('agg3_res_unit1_conv2_weight', (64, 64, 3, 3)),
 ('agg3_res_unit1_bn2_gamma', (64,)),
 ('agg3_res_unit1_bn2_beta', (64,)),
 ('agg3_res_unit1_bn2_moving_mean', (64,)),
 ('agg3_res_unit1_bn2_moving_var', (64,)),
 ('agg3_res_unit1_sc_weight', (64, 64, 1, 1)),
 ('agg3_res_unit1_sc_bn_gamma', (64,)),
 ('agg3_res_unit1_sc_bn_beta', (64,)),
 ('agg3_res_unit1_sc_bn_moving_mean', (64,)),
 ('agg3_res_unit1_sc_bn_moving_var', (64,)),
 ('agg3_res_unit2_conv1_weight', (64, 64, 3, 3)),
 ('agg3_res_unit2_bn1_gamma', (64,)),
 ('agg3_res_unit2_bn1_beta', (64,)),
 ('agg3_res_unit2_bn1_moving_mean', (64,)),
 ('agg3_res_unit2_bn1_moving_var', (64,)),
 ('agg3_res_unit2_conv2_weight', (64, 64, 3, 3)),
 ('agg3_res_unit2_bn2_gamma', (64,)),
 ('agg3_res_unit2_bn2_beta', (64,)),
 ('agg3_res_unit2_bn2_moving_mean', (64,)),
 ('agg3_res_unit2_bn2_moving_var', (64,)),
 ('rpn_reg_conv_0_lvl_0_weight', (128, 72, 3, 3)),
 ('rpn_reg_conv_0_lvl_0_bn_gamma', (128,)),
 ('rpn_reg_conv_0_lvl_0_bn_beta', (128,)),
 ('rpn_reg_conv_0_lvl_0_bn_moving_mean', (128,)),
 ('rpn_reg_conv_0_lvl_0_bn_moving_var', (128,)),
 ('rpn_reg_conv_1_lvl_0_weight', (128, 128, 3, 3)),
 ('rpn_reg_conv_1_lvl_0_bn_gamma', (128,)),
 ('rpn_reg_conv_1_lvl_0_bn_beta', (128,)),
 ('rpn_reg_conv_1_lvl_0_bn_moving_mean', (128,)),
 ('rpn_reg_conv_1_lvl_0_bn_moving_var', (128,)),
 ('rpn_reg_conv_2_lvl_0_weight', (128, 128, 3, 3)),
 ('rpn_reg_conv_2_lvl_0_bn_gamma', (128,)),
 ('rpn_reg_conv_2_lvl_0_bn_beta', (128,)),
 ('rpn_reg_conv_2_lvl_0_bn_moving_mean', (128,)),
 ('rpn_reg_conv_2_lvl_0_bn_moving_var', (128,)),
 ('rpn_reg_conv_3_lvl_0_weight', (128, 128, 3, 3)),
 ('rpn_reg_conv_3_lvl_0_bn_gamma', (128,)),
 ('rpn_reg_conv_3_lvl_0_bn_beta', (128,)),
 ('rpn_reg_conv_3_lvl_0_bn_moving_mean', (128,)),
 ('rpn_reg_conv_3_lvl_0_bn_moving_var', (128,)),
 ('rpn_reg_delta_lvl_0_weight', (8, 128, 1, 1)),
 ('rpn_reg_delta_lvl_0_bias', (8,)),
 ('pc_vehicle_frame_s1', (2, 169984, 3)),
 ('decode3dbbox0_decoded_bbox', (2, 169984, 10)),
 ('gt_bbox_veh_for_iou_pred', (2, 200, 8)),
 ('batch_rotated_iou_veh_lvl_0_iou_map', (2, 169984)),
 ('rpn_cls_conv_0_lvl_0_weight', (128, 72, 3, 3)),
 ('rpn_cls_conv_0_lvl_0_bn_gamma', (128,)),
 ('rpn_cls_conv_0_lvl_0_bn_beta', (128,)),
 ('rpn_cls_conv_0_lvl_0_bn_moving_mean', (128,)),
 ('rpn_cls_conv_0_lvl_0_bn_moving_var', (128,)),
 ('rpn_cls_conv_1_lvl_0_weight', (128, 128, 3, 3)),
 ('rpn_cls_conv_1_lvl_0_bn_gamma', (128,)),
 ('rpn_cls_conv_1_lvl_0_bn_beta', (128,)),
 ('rpn_cls_conv_1_lvl_0_bn_moving_mean', (128,)),
 ('rpn_cls_conv_1_lvl_0_bn_moving_var', (128,)),
 ('rpn_cls_conv_2_lvl_0_weight', (128, 128, 3, 3)),
 ('rpn_cls_conv_2_lvl_0_bn_gamma', (128,)),
 ('rpn_cls_conv_2_lvl_0_bn_beta', (128,)),
 ('rpn_cls_conv_2_lvl_0_bn_moving_mean', (128,)),
 ('rpn_cls_conv_2_lvl_0_bn_moving_var', (128,)),
 ('rpn_cls_conv_3_lvl_0_weight', (128, 128, 3, 3)),
 ('rpn_cls_conv_3_lvl_0_bn_gamma', (128,)),
 ('rpn_cls_conv_3_lvl_0_bn_beta', (128,)),
 ('rpn_cls_conv_3_lvl_0_bn_moving_mean', (128,)),
 ('rpn_cls_conv_3_lvl_0_bn_moving_var', (128,)),
 ('rpn_cls_logit_lvl_0_weight', (1, 128, 1, 1)),
 ('rpn_cls_logit_lvl_0_bias', (1,)),
 ('range_image_mask_s1', (2, 1, 64, 2656)),
 ('rpn_reg_conv_0_lvl_1_weight', (128, 64, 3, 3)),
 ('rpn_reg_conv_0_lvl_1_bn_gamma', (128,)),
 ('rpn_reg_conv_0_lvl_1_bn_beta', (128,)),
 ('rpn_reg_conv_0_lvl_1_bn_moving_mean', (128,)),
 ('rpn_reg_conv_0_lvl_1_bn_moving_var', (128,)),
 ('rpn_reg_conv_1_lvl_1_weight', (128, 128, 3, 3)),
 ('rpn_reg_conv_1_lvl_1_bn_gamma', (128,)),
 ('rpn_reg_conv_1_lvl_1_bn_beta', (128,)),
 ('rpn_reg_conv_1_lvl_1_bn_moving_mean', (128,)),
 ('rpn_reg_conv_1_lvl_1_bn_moving_var', (128,)),
 ('rpn_reg_conv_2_lvl_1_weight', (128, 128, 3, 3)),
 ('rpn_reg_conv_2_lvl_1_bn_gamma', (128,)),
 ('rpn_reg_conv_2_lvl_1_bn_beta', (128,)),
 ('rpn_reg_conv_2_lvl_1_bn_moving_mean', (128,)),
 ('rpn_reg_conv_2_lvl_1_bn_moving_var', (128,)),
 ('rpn_reg_conv_3_lvl_1_weight', (128, 128, 3, 3)),
 ('rpn_reg_conv_3_lvl_1_bn_gamma', (128,)),
 ('rpn_reg_conv_3_lvl_1_bn_beta', (128,)),
 ('rpn_reg_conv_3_lvl_1_bn_moving_mean', (128,)),
 ('rpn_reg_conv_3_lvl_1_bn_moving_var', (128,)),
 ('rpn_reg_delta_lvl_1_weight', (8, 128, 1, 1)),
 ('rpn_reg_delta_lvl_1_bias', (8,)),
 ('pc_vehicle_frame_s2', (2, 84992, 3)),
 ('decode3dbbox1_decoded_bbox', (2, 84992, 10)),
 ('batch_rotated_iou_veh_lvl_1_iou_map', (2, 84992)),
 ('rpn_cls_conv_0_lvl_1_weight', (128, 64, 3, 3)),
 ('rpn_cls_conv_0_lvl_1_bn_gamma', (128,)),
 ('rpn_cls_conv_0_lvl_1_bn_beta', (128,)),
 ('rpn_cls_conv_0_lvl_1_bn_moving_mean', (128,)),
 ('rpn_cls_conv_0_lvl_1_bn_moving_var', (128,)),
 ('rpn_cls_conv_1_lvl_1_weight', (128, 128, 3, 3)),
 ('rpn_cls_conv_1_lvl_1_bn_gamma', (128,)),
 ('rpn_cls_conv_1_lvl_1_bn_beta', (128,)),
 ('rpn_cls_conv_1_lvl_1_bn_moving_mean', (128,)),
 ('rpn_cls_conv_1_lvl_1_bn_moving_var', (128,)),
 ('rpn_cls_conv_2_lvl_1_weight', (128, 128, 3, 3)),
 ('rpn_cls_conv_2_lvl_1_bn_gamma', (128,)),
 ('rpn_cls_conv_2_lvl_1_bn_beta', (128,)),
 ('rpn_cls_conv_2_lvl_1_bn_moving_mean', (128,)),
 ('rpn_cls_conv_2_lvl_1_bn_moving_var', (128,)),
 ('rpn_cls_conv_3_lvl_1_weight', (128, 128, 3, 3)),
 ('rpn_cls_conv_3_lvl_1_bn_gamma', (128,)),
 ('rpn_cls_conv_3_lvl_1_bn_beta', (128,)),
 ('rpn_cls_conv_3_lvl_1_bn_moving_mean', (128,)),
 ('rpn_cls_conv_3_lvl_1_bn_moving_var', (128,)),
 ('rpn_cls_logit_lvl_1_weight', (1, 128, 1, 1)),
 ('rpn_cls_logit_lvl_1_bias', (1,)),
 ('range_image_mask_s2', (2, 1, 64, 1328)),
 ('rpn_reg_conv_0_lvl_2_weight', (128, 128, 3, 3)),
 ('rpn_reg_conv_0_lvl_2_bn_gamma', (128,)),
 ('rpn_reg_conv_0_lvl_2_bn_beta', (128,)),
 ('rpn_reg_conv_0_lvl_2_bn_moving_mean', (128,)),
 ('rpn_reg_conv_0_lvl_2_bn_moving_var', (128,)),
 ('rpn_reg_conv_1_lvl_2_weight', (128, 128, 3, 3)),
 ('rpn_reg_conv_1_lvl_2_bn_gamma', (128,)),
 ('rpn_reg_conv_1_lvl_2_bn_beta', (128,)),
 ('rpn_reg_conv_1_lvl_2_bn_moving_mean', (128,)),
 ('rpn_reg_conv_1_lvl_2_bn_moving_var', (128,)),
 ('rpn_reg_conv_2_lvl_2_weight', (128, 128, 3, 3)),
 ('rpn_reg_conv_2_lvl_2_bn_gamma', (128,)),
 ('rpn_reg_conv_2_lvl_2_bn_beta', (128,)),
 ('rpn_reg_conv_2_lvl_2_bn_moving_mean', (128,)),
 ('rpn_reg_conv_2_lvl_2_bn_moving_var', (128,)),
 ('rpn_reg_conv_3_lvl_2_weight', (128, 128, 3, 3)),
 ('rpn_reg_conv_3_lvl_2_bn_gamma', (128,)),
 ('rpn_reg_conv_3_lvl_2_bn_beta', (128,)),
 ('rpn_reg_conv_3_lvl_2_bn_moving_mean', (128,)),
 ('rpn_reg_conv_3_lvl_2_bn_moving_var', (128,)),
 ('rpn_reg_delta_lvl_2_weight', (8, 128, 1, 1)),
 ('rpn_reg_delta_lvl_2_bias', (8,)),
 ('pc_vehicle_frame_s4', (2, 42496, 3)),
 ('decode3dbbox2_decoded_bbox', (2, 42496, 10)),
 ('batch_rotated_iou_veh_lvl_2_iou_map', (2, 42496)),
 ('rpn_cls_conv_0_lvl_2_weight', (128, 128, 3, 3)),
 ('rpn_cls_conv_0_lvl_2_bn_gamma', (128,)),
 ('rpn_cls_conv_0_lvl_2_bn_beta', (128,)),
 ('rpn_cls_conv_0_lvl_2_bn_moving_mean', (128,)),
 ('rpn_cls_conv_0_lvl_2_bn_moving_var', (128,)),
 ('rpn_cls_conv_1_lvl_2_weight', (128, 128, 3, 3)),
 ('rpn_cls_conv_1_lvl_2_bn_gamma', (128,)),
 ('rpn_cls_conv_1_lvl_2_bn_beta', (128,)),
 ('rpn_cls_conv_1_lvl_2_bn_moving_mean', (128,)),
 ('rpn_cls_conv_1_lvl_2_bn_moving_var', (128,)),
 ('rpn_cls_conv_2_lvl_2_weight', (128, 128, 3, 3)),
 ('rpn_cls_conv_2_lvl_2_bn_gamma', (128,)),
 ('rpn_cls_conv_2_lvl_2_bn_beta', (128,)),
 ('rpn_cls_conv_2_lvl_2_bn_moving_mean', (128,)),
 ('rpn_cls_conv_2_lvl_2_bn_moving_var', (128,)),
 ('rpn_cls_conv_3_lvl_2_weight', (128, 128, 3, 3)),
 ('rpn_cls_conv_3_lvl_2_bn_gamma', (128,)),
 ('rpn_cls_conv_3_lvl_2_bn_beta', (128,)),
 ('rpn_cls_conv_3_lvl_2_bn_moving_mean', (128,)),
 ('rpn_cls_conv_3_lvl_2_bn_moving_var', (128,)),
 ('rpn_cls_logit_lvl_2_weight', (1, 128, 1, 1)),
 ('rpn_cls_logit_lvl_2_bias', (1,)),
 ('range_image_mask_s4', (2, 1, 64, 664)),
 ('rpn_reg_target_s1', (2, 8, 64, 2656)),
 ('rpn_reg_weight_s1', (2, 8, 64, 2656)),
 ('reg_normalize_weight_s1', (2, 8, 64, 2656)),
 ('rpn_reg_target_s2', (2, 8, 64, 1328)),
 ('rpn_reg_weight_s2', (2, 8, 64, 1328)),
 ('reg_normalize_weight_s2', (2, 8, 64, 1328)),
 ('rpn_reg_target_s4', (2, 8, 64, 664)),
 ('rpn_reg_weight_s4', (2, 8, 64, 664)),
 ('reg_normalize_weight_s4', (2, 8, 64, 664))]
04-19 11:01:59 intermediate output shape
04-19 11:01:59 [('data_fp16_output', (2, 8, 64, 2656)),
 ('res1_unit1_conv1_output', (2, 64, 64, 2656)),
 ('res1_unit1_bn1_output', (2, 64, 64, 2656)),
 ('res1_unit1_relu1_output', (2, 64, 64, 2656)),
 ('res1_unit1_conv2_output', (2, 64, 64, 2656)),
 ('res1_unit1_bn2_output', (2, 64, 64, 2656)),
 ('res1_unit1_sc_output', (2, 64, 64, 2656)),
 ('res1_unit1_sc_bn_output', (2, 64, 64, 2656)),
 ('res1_unit1_plus_output', (2, 64, 64, 2656)),
 ('res1_unit1_relu_output', (2, 64, 64, 2656)),
 ('res1_unit2_data_sampler_output', (2, 576, 169984)),
 ('reshape3_output', (2, 64, 9, 64, 2656)),
 ('res1_unit2coord_data_fp16_output', (2, 3, 64, 2656)),
 ('res1_unit2_coord_sampler_output', (2, 27, 169984)),
 ('reshape0_output', (2, 3, 9, 64, 2656)),
 ('expand_dims0_output', (2, 3, 1, 64, 2656)),
 ('relative_dis_output', (2, 3, 9, 64, 2656)),
 ('reshape1_output', (2, 3, 576, 2656)),
 ('res1_unit2_2656_mlp0_output', (2, 32, 576, 2656)),
 ('res1_unit2_2656_mlp_relu0_output', (2, 32, 576, 2656)),
 ('res1_unit2_2656_mlp1_output', (2, 64, 576, 2656)),
 ('reshape2_output', (2, 64, 9, 64, 2656)),
 ('_mul0_output', (2, 64, 9, 64, 2656)),
 ('reshape4_output', (2, 576, 64, 2656)),
 ('res1_unit2point_wise_mlp_bn1_output', (2, 576, 64, 2656)),
 ('res1_unit2point_wise_mlp_relu1_output', (2, 576, 64, 2656)),
 ('res1_unit2aggregation_conv1_output', (2, 64, 64, 2656)),
 ('res1_unit2aggregation_bn1_output', (2, 64, 64, 2656)),
 ('res1_unit2aggregation_relu1_output', (2, 64, 64, 2656)),
 ('res1_unit2_conv2_output', (2, 64, 64, 2656)),
 ('res1_unit2_bn2_output', (2, 64, 64, 2656)),
 ('res1_unit2_plus_output', (2, 64, 64, 2656)),
 ('res1_unit2_relu_output', (2, 64, 64, 2656)),
 ('res2a_unit1_conv1_output', (2, 64, 64, 2656)),
 ('res2a_unit1_bn1_output', (2, 64, 64, 2656)),
 ('res2a_unit1_relu1_output', (2, 64, 64, 2656)),
 ('res2a_unit1_conv2_output', (2, 64, 64, 1328)),
 ('res2a_unit1_bn2_output', (2, 64, 64, 1328)),
 ('res2a_unit1_sc_output', (2, 64, 64, 1328)),
 ('res2a_unit1_sc_bn_output', (2, 64, 64, 1328)),
 ('res2a_unit1_plus_output', (2, 64, 64, 1328)),
 ('res2a_unit1_relu_output', (2, 64, 64, 1328)),
 ('res2a_unit2_conv1_output', (2, 64, 64, 1328)),
 ('res2a_unit2_bn1_output', (2, 64, 64, 1328)),
 ('res2a_unit2_relu1_output', (2, 64, 64, 1328)),
 ('res2a_unit2_conv2_output', (2, 64, 64, 1328)),
 ('res2a_unit2_bn2_output', (2, 64, 64, 1328)),
 ('res2a_unit2_plus_output', (2, 64, 64, 1328)),
 ('res2a_unit2_relu_output', (2, 64, 64, 1328)),
 ('res2a_unit3_conv1_output', (2, 64, 64, 1328)),
 ('res2a_unit3_bn1_output', (2, 64, 64, 1328)),
 ('res2a_unit3_relu1_output', (2, 64, 64, 1328)),
 ('res2a_unit3_conv2_output', (2, 64, 64, 1328)),
 ('res2a_unit3_bn2_output', (2, 64, 64, 1328)),
 ('res2a_unit3_plus_output', (2, 64, 64, 1328)),
 ('res2a_unit3_relu_output', (2, 64, 64, 1328)),
 ('res2_unit1_conv1_output', (2, 128, 64, 1328)),
 ('res2_unit1_bn1_output', (2, 128, 64, 1328)),
 ('res2_unit1_relu1_output', (2, 128, 64, 1328)),
 ('res2_unit1_conv2_output', (2, 128, 64, 664)),
 ('res2_unit1_bn2_output', (2, 128, 64, 664)),
 ('res2_unit1_sc_output', (2, 128, 64, 664)),
 ('res2_unit1_sc_bn_output', (2, 128, 64, 664)),
 ('res2_unit1_plus_output', (2, 128, 64, 664)),
 ('res2_unit1_relu_output', (2, 128, 64, 664)),
 ('res2_unit2_conv1_output', (2, 128, 64, 664)),
 ('res2_unit2_bn1_output', (2, 128, 64, 664)),
 ('res2_unit2_relu1_output', (2, 128, 64, 664)),
 ('res2_unit2_conv2_output', (2, 128, 64, 664)),
 ('res2_unit2_bn2_output', (2, 128, 64, 664)),
 ('res2_unit2_plus_output', (2, 128, 64, 664)),
 ('res2_unit2_relu_output', (2, 128, 64, 664)),
 ('res2_unit3_conv1_output', (2, 128, 64, 664)),
 ('res2_unit3_bn1_output', (2, 128, 64, 664)),
 ('res2_unit3_relu1_output', (2, 128, 64, 664)),
 ('res2_unit3_conv2_output', (2, 128, 64, 664)),
 ('res2_unit3_bn2_output', (2, 128, 64, 664)),
 ('res2_unit3_plus_output', (2, 128, 64, 664)),
 ('res2_unit3_relu_output', (2, 128, 64, 664)),
 ('agg1_deconv_output', (2, 64, 64, 2656)),
 ('agg1_deconv_bn_output', (2, 64, 64, 2656)),
 ('agg1_relu_output', (2, 64, 64, 2656)),
 ('agg1_plus_output', (2, 64, 64, 2656)),
 ('agg1_res_unit1_conv1_output', (2, 64, 64, 2656)),
 ('agg1_res_unit1_bn1_output', (2, 64, 64, 2656)),
 ('agg1_res_unit1_relu1_output', (2, 64, 64, 2656)),
 ('agg1_res_unit1_conv2_output', (2, 64, 64, 2656)),
 ('agg1_res_unit1_bn2_output', (2, 64, 64, 2656)),
 ('agg1_res_unit1_sc_output', (2, 64, 64, 2656)),
 ('agg1_res_unit1_sc_bn_output', (2, 64, 64, 2656)),
 ('agg1_res_unit1_plus_output', (2, 64, 64, 2656)),
 ('agg1_res_unit1_relu_output', (2, 64, 64, 2656)),
 ('agg1_res_unit2_conv1_output', (2, 64, 64, 2656)),
 ('agg1_res_unit2_bn1_output', (2, 64, 64, 2656)),
 ('agg1_res_unit2_relu1_output', (2, 64, 64, 2656)),
 ('agg1_res_unit2_conv2_output', (2, 64, 64, 2656)),
 ('agg1_res_unit2_bn2_output', (2, 64, 64, 2656)),
 ('agg1_res_unit2_plus_output', (2, 64, 64, 2656)),
 ('agg1_res_unit2_relu_output', (2, 64, 64, 2656)),
 ('res3a_unit1_conv1_output', (2, 128, 64, 664)),
 ('res3a_unit1_bn1_output', (2, 128, 64, 664)),
 ('res3a_unit1_relu1_output', (2, 128, 64, 664)),
 ('res3a_unit1_conv2_output', (2, 128, 64, 332)),
 ('res3a_unit1_bn2_output', (2, 128, 64, 332)),
 ('res3a_unit1_sc_output', (2, 128, 64, 332)),
 ('res3a_unit1_sc_bn_output', (2, 128, 64, 332)),
 ('res3a_unit1_plus_output', (2, 128, 64, 332)),
 ('res3a_unit1_relu_output', (2, 128, 64, 332)),
 ('res3a_unit2_conv1_output', (2, 128, 64, 332)),
 ('res3a_unit2_bn1_output', (2, 128, 64, 332)),
 ('res3a_unit2_relu1_output', (2, 128, 64, 332)),
 ('res3a_unit2_conv2_output', (2, 128, 64, 332)),
 ('res3a_unit2_bn2_output', (2, 128, 64, 332)),
 ('res3a_unit2_plus_output', (2, 128, 64, 332)),
 ('res3a_unit2_relu_output', (2, 128, 64, 332)),
 ('res3a_unit3_conv1_output', (2, 128, 64, 332)),
 ('res3a_unit3_bn1_output', (2, 128, 64, 332)),
 ('res3a_unit3_relu1_output', (2, 128, 64, 332)),
 ('res3a_unit3_conv2_output', (2, 128, 64, 332)),
 ('res3a_unit3_bn2_output', (2, 128, 64, 332)),
 ('res3a_unit3_plus_output', (2, 128, 64, 332)),
 ('res3a_unit3_relu_output', (2, 128, 64, 332)),
 ('res3a_unit4_conv1_output', (2, 128, 64, 332)),
 ('res3a_unit4_bn1_output', (2, 128, 64, 332)),
 ('res3a_unit4_relu1_output', (2, 128, 64, 332)),
 ('res3a_unit4_conv2_output', (2, 128, 64, 332)),
 ('res3a_unit4_bn2_output', (2, 128, 64, 332)),
 ('res3a_unit4_plus_output', (2, 128, 64, 332)),
 ('res3a_unit4_relu_output', (2, 128, 64, 332)),
 ('res3a_unit5_conv1_output', (2, 128, 64, 332)),
 ('res3a_unit5_bn1_output', (2, 128, 64, 332)),
 ('res3a_unit5_relu1_output', (2, 128, 64, 332)),
 ('res3a_unit5_conv2_output', (2, 128, 64, 332)),
 ('res3a_unit5_bn2_output', (2, 128, 64, 332)),
 ('res3a_unit5_plus_output', (2, 128, 64, 332)),
 ('res3a_unit5_relu_output', (2, 128, 64, 332)),
 ('res3_unit1_conv1_output', (2, 128, 64, 332)),
 ('res3_unit1_bn1_output', (2, 128, 64, 332)),
 ('res3_unit1_relu1_output', (2, 128, 64, 332)),
 ('res3_unit1_conv2_output', (2, 128, 64, 166)),
 ('res3_unit1_bn2_output', (2, 128, 64, 166)),
 ('res3_unit1_sc_output', (2, 128, 64, 166)),
 ('res3_unit1_sc_bn_output', (2, 128, 64, 166)),
 ('res3_unit1_plus_output', (2, 128, 64, 166)),
 ('res3_unit1_relu_output', (2, 128, 64, 166)),
 ('res3_unit2_conv1_output', (2, 128, 64, 166)),
 ('res3_unit2_bn1_output', (2, 128, 64, 166)),
 ('res3_unit2_relu1_output', (2, 128, 64, 166)),
 ('res3_unit2_conv2_output', (2, 128, 64, 166)),
 ('res3_unit2_bn2_output', (2, 128, 64, 166)),
 ('res3_unit2_plus_output', (2, 128, 64, 166)),
 ('res3_unit2_relu_output', (2, 128, 64, 166)),
 ('res3_unit3_conv1_output', (2, 128, 64, 166)),
 ('res3_unit3_bn1_output', (2, 128, 64, 166)),
 ('res3_unit3_relu1_output', (2, 128, 64, 166)),
 ('res3_unit3_conv2_output', (2, 128, 64, 166)),
 ('res3_unit3_bn2_output', (2, 128, 64, 166)),
 ('res3_unit3_plus_output', (2, 128, 64, 166)),
 ('res3_unit3_relu_output', (2, 128, 64, 166)),
 ('res3_unit4_conv1_output', (2, 128, 64, 166)),
 ('res3_unit4_bn1_output', (2, 128, 64, 166)),
 ('res3_unit4_relu1_output', (2, 128, 64, 166)),
 ('res3_unit4_conv2_output', (2, 128, 64, 166)),
 ('res3_unit4_bn2_output', (2, 128, 64, 166)),
 ('res3_unit4_plus_output', (2, 128, 64, 166)),
 ('res3_unit4_relu_output', (2, 128, 64, 166)),
 ('res3_unit5_conv1_output', (2, 128, 64, 166)),
 ('res3_unit5_bn1_output', (2, 128, 64, 166)),
 ('res3_unit5_relu1_output', (2, 128, 64, 166)),
 ('res3_unit5_conv2_output', (2, 128, 64, 166)),
 ('res3_unit5_bn2_output', (2, 128, 64, 166)),
 ('res3_unit5_plus_output', (2, 128, 64, 166)),
 ('res3_unit5_relu_output', (2, 128, 64, 166)),
 ('agg2_deconv_output', (2, 128, 64, 664)),
 ('agg2_deconv_bn_output', (2, 128, 64, 664)),
 ('agg2_relu_output', (2, 128, 64, 664)),
 ('agg2_plus_output', (2, 128, 64, 664)),
 ('agg2_res_unit1_conv1_output', (2, 128, 64, 664)),
 ('agg2_res_unit1_bn1_output', (2, 128, 64, 664)),
 ('agg2_res_unit1_relu1_output', (2, 128, 64, 664)),
 ('agg2_res_unit1_conv2_output', (2, 128, 64, 664)),
 ('agg2_res_unit1_bn2_output', (2, 128, 64, 664)),
 ('agg2_res_unit1_sc_output', (2, 128, 64, 664)),
 ('agg2_res_unit1_sc_bn_output', (2, 128, 64, 664)),
 ('agg2_res_unit1_plus_output', (2, 128, 64, 664)),
 ('agg2_res_unit1_relu_output', (2, 128, 64, 664)),
 ('agg2_res_unit2_conv1_output', (2, 128, 64, 664)),
 ('agg2_res_unit2_bn1_output', (2, 128, 64, 664)),
 ('agg2_res_unit2_relu1_output', (2, 128, 64, 664)),
 ('agg2_res_unit2_conv2_output', (2, 128, 64, 664)),
 ('agg2_res_unit2_bn2_output', (2, 128, 64, 664)),
 ('agg2_res_unit2_plus_output', (2, 128, 64, 664)),
 ('agg2_res_unit2_relu_output', (2, 128, 64, 664)),
 ('agg2a_deconv_output', (2, 64, 64, 1328)),
 ('agg2a_deconv_bn_output', (2, 64, 64, 1328)),
 ('agg2a_relu_output', (2, 64, 64, 1328)),
 ('agg2a_plus_output', (2, 64, 64, 1328)),
 ('agg2a_res_unit1_conv1_output', (2, 64, 64, 1328)),
 ('agg2a_res_unit1_bn1_output', (2, 64, 64, 1328)),
 ('agg2a_res_unit1_relu1_output', (2, 64, 64, 1328)),
 ('agg2a_res_unit1_conv2_output', (2, 64, 64, 1328)),
 ('agg2a_res_unit1_bn2_output', (2, 64, 64, 1328)),
 ('agg2a_res_unit1_sc_output', (2, 64, 64, 1328)),
 ('agg2a_res_unit1_sc_bn_output', (2, 64, 64, 1328)),
 ('agg2a_res_unit1_plus_output', (2, 64, 64, 1328)),
 ('agg2a_res_unit1_relu_output', (2, 64, 64, 1328)),
 ('agg3_deconv_output', (2, 64, 64, 2656)),
 ('agg3_deconv_bn_output', (2, 64, 64, 2656)),
 ('agg3_relu_output', (2, 64, 64, 2656)),
 ('agg3_plus_output', (2, 64, 64, 2656)),
 ('agg3_res_unit1_conv1_output', (2, 64, 64, 2656)),
 ('agg3_res_unit1_bn1_output', (2, 64, 64, 2656)),
 ('agg3_res_unit1_relu1_output', (2, 64, 64, 2656)),
 ('agg3_res_unit1_conv2_output', (2, 64, 64, 2656)),
 ('agg3_res_unit1_bn2_output', (2, 64, 64, 2656)),
 ('agg3_res_unit1_sc_output', (2, 64, 64, 2656)),
 ('agg3_res_unit1_sc_bn_output', (2, 64, 64, 2656)),
 ('agg3_res_unit1_plus_output', (2, 64, 64, 2656)),
 ('agg3_res_unit1_relu_output', (2, 64, 64, 2656)),
 ('agg3_res_unit2_conv1_output', (2, 64, 64, 2656)),
 ('agg3_res_unit2_bn1_output', (2, 64, 64, 2656)),
 ('agg3_res_unit2_relu1_output', (2, 64, 64, 2656)),
 ('agg3_res_unit2_conv2_output', (2, 64, 64, 2656)),
 ('agg3_res_unit2_bn2_output', (2, 64, 64, 2656)),
 ('agg3_res_unit2_plus_output', (2, 64, 64, 2656)),
 ('agg3_res_unit2_relu_output', (2, 64, 64, 2656)),
 ('data_concat_output', (2, 72, 64, 2656)),
 ('rpn_reg_conv_0_lvl_0_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_0_lvl_0_bn_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_0_lvl_0_relu_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_1_lvl_0_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_1_lvl_0_bn_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_1_lvl_0_relu_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_2_lvl_0_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_2_lvl_0_bn_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_2_lvl_0_relu_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_3_lvl_0_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_3_lvl_0_bn_output', (2, 128, 64, 2656)),
 ('rpn_reg_conv_3_lvl_0_relu_output', (2, 128, 64, 2656)),
 ('rpn_reg_delta_lvl_0_output', (2, 8, 64, 2656)),
 ('rpn_reg_delta_lvl_0_fp32_output', (2, 8, 64, 2656)),
 ('bbox_delta_reshape_lvl_0_output', (2, 1, 8, 169984)),
 ('slice_axis1_output', (2, 1, 8, 169984)),
 ('squeeze1_output', (2, 8, 169984)),
 ('transpose0_output', (2, 169984, 8)),
 ('reshape5_output', (2, 1, 64, 2656)),
 ('stop_gradient0_output', (2, 1, 64, 2656)),
 ('concat0_output', (2, 1, 64, 2656)),
 ('_mulscalar2_output', (2, 1, 64, 2656)),
 ('rpn_cls_conv_0_lvl_0_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_0_lvl_0_bn_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_0_lvl_0_relu_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_1_lvl_0_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_1_lvl_0_bn_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_1_lvl_0_relu_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_2_lvl_0_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_2_lvl_0_bn_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_2_lvl_0_relu_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_3_lvl_0_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_3_lvl_0_bn_output', (2, 128, 64, 2656)),
 ('rpn_cls_conv_3_lvl_0_relu_output', (2, 128, 64, 2656)),
 ('rpn_cls_logit_lvl_0_output', (2, 1, 64, 2656)),
 ('rpn_cls_logit_lvl_0_fp32_output', (2, 1, 64, 2656)),
 ('sigmoid1_output', (2, 1, 64, 2656)),
 ('clip0_output', (2, 1, 64, 2656)),
 ('log0_output', (2, 1, 64, 2656)),
 ('_mul3_output', (2, 1, 64, 2656)),
 ('_rminusscalar0_output', (2, 1, 64, 2656)),
 ('_mulscalar3_output', (2, 1, 64, 2656)),
 ('_mulscalar0_output', (2, 1, 64, 2656)),
 ('_zeros0_output', (1,)),
 ('broadcast_greater_equal0_output', (2, 1, 64, 2656)),
 ('_mul1_output', (2, 1, 64, 2656)),
 ('_mulscalar1_output', (2, 1, 64, 2656)),
 ('_mul2_output', (2, 1, 64, 2656)),
 ('_minus0_output', (2, 1, 64, 2656)),
 ('activation0_output', (2, 1, 64, 2656)),
 ('_minus1_output', (2, 1, 64, 2656)),
 ('_mul4_output', (2, 1, 64, 2656)),
 ('_plus0_output', (2, 1, 64, 2656)),
 ('_mulscalar4_output', (2, 1, 64, 2656)),
 ('_mulscalar5_output', (2, 1, 64, 2656)),
 ('_mulscalar6_output', (2, 1, 64, 2656)),
 ('sigmoid0_output', (2, 1, 64, 2656)),
 ('_minus2_output', (2, 1, 64, 2656)),
 ('abs0_output', (2, 1, 64, 2656)),
 ('_powerscalar0_output', (2, 1, 64, 2656)),
 ('_mul7_output', (2, 1, 64, 2656)),
 ('_zeros2_output', (1, 1)),
 ('broadcast_equal0_output', (2, 1, 64, 2656)),
 ('_mul8_output', (2, 1, 64, 2656)),
 ('_mul5_output', (2, 1, 64, 2656)),
 ('_zeros1_output', (1, 1)),
 ('broadcast_greater0_output', (2, 1, 64, 2656)),
 ('_mul6_output', (2, 1, 64, 2656)),
 ('_plus1_output', (2, 1, 64, 2656)),
 ('stop_gradient3_output', (2, 1, 64, 2656)),
 ('reshape8_output', (2, 1, 64, 2656)),
 ('broadcast_mul0_output', (2, 1, 64, 2656)),
 ('sum0_output', (1,)),
 ('_plusscalar0_output', (1,)),
 ('broadcast_div0_output', (2, 1, 64, 2656)),
 ('rpn_cls_loss_s1_output', (2, 1, 64, 2656)),
 ('rpn_reg_conv_0_lvl_1_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_0_lvl_1_bn_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_0_lvl_1_relu_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_1_lvl_1_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_1_lvl_1_bn_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_1_lvl_1_relu_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_2_lvl_1_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_2_lvl_1_bn_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_2_lvl_1_relu_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_3_lvl_1_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_3_lvl_1_bn_output', (2, 128, 64, 1328)),
 ('rpn_reg_conv_3_lvl_1_relu_output', (2, 128, 64, 1328)),
 ('rpn_reg_delta_lvl_1_output', (2, 8, 64, 1328)),
 ('rpn_reg_delta_lvl_1_fp32_output', (2, 8, 64, 1328)),
 ('bbox_delta_reshape_lvl_1_output', (2, 1, 8, 84992)),
 ('slice_axis3_output', (2, 1, 8, 84992)),
 ('squeeze3_output', (2, 8, 84992)),
 ('transpose1_output', (2, 84992, 8)),
 ('reshape6_output', (2, 1, 64, 1328)),
 ('stop_gradient1_output', (2, 1, 64, 1328)),
 ('concat1_output', (2, 1, 64, 1328)),
 ('_mulscalar10_output', (2, 1, 64, 1328)),
 ('rpn_cls_conv_0_lvl_1_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_0_lvl_1_bn_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_0_lvl_1_relu_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_1_lvl_1_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_1_lvl_1_bn_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_1_lvl_1_relu_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_2_lvl_1_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_2_lvl_1_bn_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_2_lvl_1_relu_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_3_lvl_1_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_3_lvl_1_bn_output', (2, 128, 64, 1328)),
 ('rpn_cls_conv_3_lvl_1_relu_output', (2, 128, 64, 1328)),
 ('rpn_cls_logit_lvl_1_output', (2, 1, 64, 1328)),
 ('rpn_cls_logit_lvl_1_fp32_output', (2, 1, 64, 1328)),
 ('sigmoid3_output', (2, 1, 64, 1328)),
 ('clip1_output', (2, 1, 64, 1328)),
 ('log1_output', (2, 1, 64, 1328)),
 ('_mul13_output', (2, 1, 64, 1328)),
 ('_rminusscalar1_output', (2, 1, 64, 1328)),
 ('_mulscalar11_output', (2, 1, 64, 1328)),
 ('_mulscalar8_output', (2, 1, 64, 1328)),
 ('_zeros3_output', (1,)),
 ('broadcast_greater_equal1_output', (2, 1, 64, 1328)),
 ('_mul11_output', (2, 1, 64, 1328)),
 ('_mulscalar9_output', (2, 1, 64, 1328)),
 ('_mul12_output', (2, 1, 64, 1328)),
 ('_minus4_output', (2, 1, 64, 1328)),
 ('activation1_output', (2, 1, 64, 1328)),
 ('_minus5_output', (2, 1, 64, 1328)),
 ('_mul14_output', (2, 1, 64, 1328)),
 ('_plus2_output', (2, 1, 64, 1328)),
 ('_mulscalar12_output', (2, 1, 64, 1328)),
 ('_mulscalar13_output', (2, 1, 64, 1328)),
 ('_mulscalar14_output', (2, 1, 64, 1328)),
 ('sigmoid2_output', (2, 1, 64, 1328)),
 ('_minus6_output', (2, 1, 64, 1328)),
 ('abs1_output', (2, 1, 64, 1328)),
 ('_powerscalar1_output', (2, 1, 64, 1328)),
 ('_mul17_output', (2, 1, 64, 1328)),
 ('_zeros5_output', (1, 1)),
 ('broadcast_equal1_output', (2, 1, 64, 1328)),
 ('_mul18_output', (2, 1, 64, 1328)),
 ('_mul15_output', (2, 1, 64, 1328)),
 ('_zeros4_output', (1, 1)),
 ('broadcast_greater1_output', (2, 1, 64, 1328)),
 ('_mul16_output', (2, 1, 64, 1328)),
 ('_plus3_output', (2, 1, 64, 1328)),
 ('stop_gradient6_output', (2, 1, 64, 1328)),
 ('reshape9_output', (2, 1, 64, 1328)),
 ('broadcast_mul1_output', (2, 1, 64, 1328)),
 ('sum2_output', (1,)),
 ('_plusscalar2_output', (1,)),
 ('broadcast_div2_output', (2, 1, 64, 1328)),
 ('rpn_cls_loss_s2_output', (2, 1, 64, 1328)),
 ('rpn_reg_conv_0_lvl_2_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_0_lvl_2_bn_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_0_lvl_2_relu_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_1_lvl_2_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_1_lvl_2_bn_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_1_lvl_2_relu_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_2_lvl_2_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_2_lvl_2_bn_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_2_lvl_2_relu_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_3_lvl_2_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_3_lvl_2_bn_output', (2, 128, 64, 664)),
 ('rpn_reg_conv_3_lvl_2_relu_output', (2, 128, 64, 664)),
 ('rpn_reg_delta_lvl_2_output', (2, 8, 64, 664)),
 ('rpn_reg_delta_lvl_2_fp32_output', (2, 8, 64, 664)),
 ('bbox_delta_reshape_lvl_2_output', (2, 1, 8, 42496)),
 ('slice_axis5_output', (2, 1, 8, 42496)),
 ('squeeze5_output', (2, 8, 42496)),
 ('transpose2_output', (2, 42496, 8)),
 ('reshape7_output', (2, 1, 64, 664)),
 ('stop_gradient2_output', (2, 1, 64, 664)),
 ('concat2_output', (2, 1, 64, 664)),
 ('_mulscalar18_output', (2, 1, 64, 664)),
 ('rpn_cls_conv_0_lvl_2_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_0_lvl_2_bn_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_0_lvl_2_relu_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_1_lvl_2_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_1_lvl_2_bn_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_1_lvl_2_relu_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_2_lvl_2_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_2_lvl_2_bn_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_2_lvl_2_relu_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_3_lvl_2_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_3_lvl_2_bn_output', (2, 128, 64, 664)),
 ('rpn_cls_conv_3_lvl_2_relu_output', (2, 128, 64, 664)),
 ('rpn_cls_logit_lvl_2_output', (2, 1, 64, 664)),
 ('rpn_cls_logit_lvl_2_fp32_output', (2, 1, 64, 664)),
 ('sigmoid5_output', (2, 1, 64, 664)),
 ('clip2_output', (2, 1, 64, 664)),
 ('log2_output', (2, 1, 64, 664)),
 ('_mul23_output', (2, 1, 64, 664)),
 ('_rminusscalar2_output', (2, 1, 64, 664)),
 ('_mulscalar19_output', (2, 1, 64, 664)),
 ('_mulscalar16_output', (2, 1, 64, 664)),
 ('_zeros6_output', (1,)),
 ('broadcast_greater_equal2_output', (2, 1, 64, 664)),
 ('_mul21_output', (2, 1, 64, 664)),
 ('_mulscalar17_output', (2, 1, 64, 664)),
 ('_mul22_output', (2, 1, 64, 664)),
 ('_minus8_output', (2, 1, 64, 664)),
 ('activation2_output', (2, 1, 64, 664)),
 ('_minus9_output', (2, 1, 64, 664)),
 ('_mul24_output', (2, 1, 64, 664)),
 ('_plus4_output', (2, 1, 64, 664)),
 ('_mulscalar20_output', (2, 1, 64, 664)),
 ('_mulscalar21_output', (2, 1, 64, 664)),
 ('_mulscalar22_output', (2, 1, 64, 664)),
 ('sigmoid4_output', (2, 1, 64, 664)),
 ('_minus10_output', (2, 1, 64, 664)),
 ('abs2_output', (2, 1, 64, 664)),
 ('_powerscalar2_output', (2, 1, 64, 664)),
 ('_mul27_output', (2, 1, 64, 664)),
 ('_zeros8_output', (1, 1)),
 ('broadcast_equal2_output', (2, 1, 64, 664)),
 ('_mul28_output', (2, 1, 64, 664)),
 ('_mul25_output', (2, 1, 64, 664)),
 ('_zeros7_output', (1, 1)),
 ('broadcast_greater2_output', (2, 1, 64, 664)),
 ('_mul26_output', (2, 1, 64, 664)),
 ('_plus5_output', (2, 1, 64, 664)),
 ('stop_gradient9_output', (2, 1, 64, 664)),
 ('reshape10_output', (2, 1, 64, 664)),
 ('broadcast_mul2_output', (2, 1, 64, 664)),
 ('sum4_output', (1,)),
 ('_plusscalar4_output', (1,)),
 ('broadcast_div4_output', (2, 1, 64, 664)),
 ('rpn_cls_loss_s4_output', (2, 1, 64, 664)),
 ('_minus3_output', (2, 8, 64, 2656)),
 ('rpn_reg_smoothl1_output', (2, 8, 64, 2656)),
 ('stop_gradient4_output', (2, 8, 64, 2656)),
 ('_mul9_output', (2, 8, 64, 2656)),
 ('stop_gradient5_output', (2, 8, 64, 2656)),
 ('_mul10_output', (2, 8, 64, 2656)),
 ('sum1_output', (1,)),
 ('_plusscalar1_output', (1,)),
 ('broadcast_div1_output', (2, 8, 64, 2656)),
 ('_mulscalar7_output', (2, 8, 64, 2656)),
 ('rpn_reg_loss_s1_output', (2, 8, 64, 2656)),
 ('_minus7_output', (2, 8, 64, 1328)),
 ('rpn_reg_smoothl1_output', (2, 8, 64, 1328)),
 ('stop_gradient7_output', (2, 8, 64, 1328)),
 ('_mul19_output', (2, 8, 64, 1328)),
 ('stop_gradient8_output', (2, 8, 64, 1328)),
 ('_mul20_output', (2, 8, 64, 1328)),
 ('sum3_output', (1,)),
 ('_plusscalar3_output', (1,)),
 ('broadcast_div3_output', (2, 8, 64, 1328)),
 ('_mulscalar15_output', (2, 8, 64, 1328)),
 ('rpn_reg_loss_s2_output', (2, 8, 64, 1328)),
 ('_minus11_output', (2, 8, 64, 664)),
 ('rpn_reg_smoothl1_output', (2, 8, 64, 664)),
 ('stop_gradient10_output', (2, 8, 64, 664)),
 ('_mul29_output', (2, 8, 64, 664)),
 ('stop_gradient11_output', (2, 8, 64, 664)),
 ('_mul30_output', (2, 8, 64, 664)),
 ('sum5_output', (1,)),
 ('_plusscalar5_output', (1,)),
 ('broadcast_div5_output', (2, 8, 64, 664)),
 ('_mulscalar23_output', (2, 8, 64, 664)),
 ('rpn_reg_loss_s4_output', (2, 8, 64, 664))]
04-19 11:01:59 terminal output shape
04-19 11:01:59 [('rpn_cls_loss_s1_output', (2, 1, 64, 2656)),
 ('rpn_cls_loss_s2_output', (2, 1, 64, 1328)),
 ('rpn_cls_loss_s4_output', (2, 1, 64, 664)),
 ('rpn_reg_loss_s1_output', (2, 8, 64, 2656)),
 ('rpn_reg_loss_s2_output', (2, 8, 64, 1328)),
 ('rpn_reg_loss_s4_output', (2, 8, 64, 664))]
[11:01:59] /home/ruanhao/range_view/mxnet/src/executor/graph_executor.cc:1036: Disabling fusion due to altered topological order of inputs.
[1]    237 killed     python tools/train.py --config 

As mentioned above, it has two major errors:

  • [Warning] Your mxnet does not support ProposalTarget
    [Warning] Your mxnet does not support DecodeBBox
    [Warning] Your mxnet does not support BBoxNorm
    [Warning] Your mxnet does not support FocalLoss
    Please using new data mean and var
  • [11:01:59] /home/ruanhao/range_view/mxnet/src/executor/graph_executor.cc:1036: Disabling fusion due to altered topological order of inputs.
    [1] 237 killed python tools/train.py --config

Here are some of the results of my pip list:

rangedet                    1.0.0     /home/ruanhao/range_view/RangeDet
mxnet                       2.0.0     /home/ruanhao/range_view/mxnet/python
horovod                     0.19.0

cuda and nccl version are:

cuda-11.2
cudnn - 8.1.0
nccl - 2.8.4
openmpi - 4.0.7

Could you please provide some advice?

@BAI-Yeqi
Copy link

##About the Reproducibility

@happynear @ruanhao5 @Treemann @IAS-01

Thank you for the great discussion!

May I check with you if you are able to reproduce the reported results on Waymo or Kitti Dataset?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

6 participants