val_nusc_tracking.py AssertionError

[yancie-yjr]

I used the default model to eval, but triger a warming and a AssertionError, sincerely look forward to your answer. Also, when will you release the full version of the code?

python ./tools/val_nusc_tracking.py examples/point_pillars/configs/nusc_all_pp_centernet_tracking.py --checkpoint model_zoo/simtrack_pillar.pth --work_dir /data/simtrack_output/

Use HM Bias: -2.19
====== Loading NuScenes tables for version v1.0-trainval...
23 category,
8 attribute, 4 visibility, 64386 instance,
12 sensor,
10200 calibrated_sensor,
2631083 ego_pose,
68 log,
850 scene, 34149 sample,
2631083 sample_data,
1166187 sample_annotation,
4 map,
Done loading in 135.2 seconds.

Reverse indexing ...
Done reverse indexing in 10.4 seconds.

/data/simtrack/det3d/core/bbox/geometry.py:160: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function() found for signature:

getitem(array(float64, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))

There are 22 candidate implementations:

• Of which 20 did not match due to:
  Overload of function 'getitem': File: : Line N/A.
  With argument(s): '(array(float64, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))':
  No match.
• Of which 2 did not match due to:
  Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 166.
  With argument(s): '(array(float64, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))':
  Rejected as the implementation raised a specific error:
  NumbaTypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>)
  raised from /home/yangjinrong/miniconda3/envs/det3d/lib/python3.7/site-packages/numba/core/typing/arraydecl.py:73

During: typing of intrinsic-call at /data/simtrack/det3d/core/bbox/geometry.py (179)

File "det3d/core/bbox/geometry.py", line 179:
def points_in_convex_polygon_jit(points, polygon, clockwise=True):

:,
[num_points_of_polygon - 1] + list(range(num_points_of_polygon - 1)),
^

@numba.jit
/data/simtrack/det3d/core/bbox/geometry.py:160: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: Cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>
File "det3d/core/bbox/geometry.py", line 196: def points_in_convex_polygon_jit(points, polygon, clockwise=True):

cross = 0.0 for i in range(num_points):
^
@numba.jit
/home/yangjinrong/miniconda3/envs/det3d/lib/python3.7/site-packages/numba/core/object_mode_passes.py:152: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops. File "det3d/core/bbox/geometry.py", line 171:
def points_in_convex_polygon_jit(points, polygon, clockwise=True):

# first convert polygon to directed lines
num_points_of_polygon = polygon.shape[1]
^
state.func_ir.loc))
/home/yangjinrong/miniconda3/envs/det3d/lib/python3.7/site-packages/numba/core/object_mode_passes.py:162: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "det3d/core/bbox/geometry.py", line 171:
def points_in_convex_polygon_jit(points, polygon, clockwise=True):

# first convert polygon to directed lines
num_points_of_polygon = polygon.shape[1]
^

state.func_ir.loc))

Loading NuScenes tables for version v1.0-trainval...
23 category,
8 attribute,
4 visibility,
64386 instance,
12 sensor,
10200 calibrated_sensor,
2631083 ego_pose,
68 log,
850 scene,
34149 sample,
2631083 sample_data,
1166187 sample_annotation,
4 map,
Done loading in 35.9 seconds.

Reverse indexing ...
Done reverse indexing in 9.5 seconds.

Finish generate predictions for testset, save to /data/simtrack_output/tracking_results.json

Loading NuScenes tables for version v1.0-trainval...
23 category, 8 attribute, 4 visibility,
64386 instance,
12 sensor, 10200 calibrated_sensor,
2631083 ego_pose,68 log,
850 scene,
34149 sample, 2631083 sample_data, 1166187 sample_annotation,
4 map,
Done loading in 36.0 seconds.

Reverse indexing ...
Done reverse indexing in 8.5 seconds. ======
Initializing nuScenes tracking evaluation
Loaded results from /data/simtrack_output/tracking_results.json. Found detections for 6019 samples.
Loading annotations for val split from nuScenes version: v1.0-trainval
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6019/6019 [00:06<00:00, 871.16it/s]
Loaded ground truth annotations for 6019 samples.
Filtering tracks
=> Original number of boxes: 227984
=> After distance based filtering: 190099
=> After LIDAR points based filtering: 190099
=> After bike rack filtering: 189972
Filtering ground truth tracks
=> Original number of boxes: 142261
=> After distance based filtering: 103564
=> After LIDAR points based filtering: 93885
=> After bike rack filtering: 93875
Accumulating metric data...
Computing metrics for class bicycle...

Computed thresholds
[15/167]
MOTAR MOTP Recall Frames GT GT-Mtch GT-Miss GT-IDS Pred Pred-TP Pred-FP Pred-IDS
thr_0.1681 0.000 0.278 0.507 1923 1993 971 982 40 2431 971 1420 40

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS   

thr_0.1975 0.000 0.271 0.488 1769 1993 939 1021 33 1999 939 1027 33

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS                  

thr_0.2212 0.127 0.266 0.462 1686 1993 893 1073 27 1700 893 780 27

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.2547 0.450 0.262 0.441 1546 1993 857 1114 22 1350 857 471 22

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS                    

thr_0.2824 0.535 0.262 0.414 1514 1993 804 1167 22 1200 804 374 22

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.2922 0.551 0.260 0.395 1501 1993 766 1205 22 1132 766 344 22

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.3012 0.538 0.277 0.368 1490 1993 712 1260 21 1062 712 329 21

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.3335 0.603 0.266 0.346 1470 1993 673 1303 17 957 673 267 17

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.3816 0.741 0.267 0.316 1422 1993 617 1364 12 789 617 160 12

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.4070 0.769 0.248 0.293 1413 1993 575 1410 8 716 575 133 8

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.4231 0.781 0.241 0.276 1407 1993 544 1443 6 669 544 119 6

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.4741 0.841 0.231 0.243 1385 1993 479 1508 6 561 479 76 6

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.4873 0.837 0.223 0.221 1385 1993 435 1553 5 511 435 71 5

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.5002 0.860 0.206 0.199 1378 1993 394 1596 3 452 394 55 3

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.5331 0.926 0.202 0.183 1351 1993 363 1628 2 392 363 27 2

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.5464 0.944 0.199 0.153 1347 1993 303 1688 2 322 303 17 2

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.5668 0.940 0.206 0.134 1347 1993 266 1726 1 283 266 16 1

            MOTAR   MOTP    Recall  Frames  GT      GT-Mtch GT-Miss GT-IDS  Pred    Pred-TP Pred-FP Pred-IDS

thr_0.5879 0.956 0.207 0.104 1343 1993 206 1786 1 216 206 9 1

Traceback (most recent call last):
File "./tools/val_nusc_tracking.py", line 202, in
tracking()
File "./tools/val_nusc_tracking.py", line 148, in tracking
dataset.evaluation_tracking(copy.deepcopy(predictions), output_dir=args.work_dir, testset=False)
File "/data/simtrack/det3d/datasets/nuscenes/nuscenes.py", line 382, in evaluation_tracking
metrics_summary = nusc_eval.main()
File "/home/yangjinrong/miniconda3/envs/det3d/lib/python3.7/site-packages/nuscenes/eval/tracking/evaluate.py", line 205, in main
metrics, metric_data_list = self.evaluate()
File "/home/yangjinrong/miniconda3/envs/det3d/lib/python3.7/site-packages/nuscenes/eval/tracking/evaluate.py", line 135, in evaluate
accumulate_class(class_name)
File "/home/yangjinrong/miniconda3/envs/det3d/lib/python3.7/site-packages/nuscenes/eval/tracking/evaluate.py", line 131, in accumulate_class
curr_md = curr_ev.accumulate()
File "/home/yangjinrong/miniconda3/envs/det3d/lib/python3.7/site-packages/nuscenes/eval/tracking/algo.py", line 156, in accumulate
assert unachieved_thresholds + duplicate_thresholds + len(thresh_metrics) == self.num_thresholds
AssertionError

[YumengXiu]

I met the same problem, how did you solve it?

[12w2]

I met the same problem, how did you solve it?

upgrade the nuscenes-devkit
pip install --upgrade nuscenes-devkit

btw,are you trying to revive the job? If so, can we communicate with each other?

[YumengXiu]

I met the same problem, how did you solve it?

upgrade the nuscenes-devkit pip install --upgrade nuscenes-devkit

btw,are you trying to revive the job? If so, can we communicate with each other?

Yes, I'm trying to involve it in ROS simulation, I'm happy for any discussion

[12w2]

I met the same problem, how did you solve it?

upgrade the nuscenes-devkit pip install --upgrade nuscenes-devkit
btw,are you trying to revive the job? If so, can we communicate with each other?

Yes, I'm trying to involve it in ROS simulation, I'm happy for any discussion

I am now trying to reproduce this work. I am curious about the motion update branch, especially how to implement update if multiple frames are lost. The update here seems to only predict the detection target in the previous frame, but How to predict when multiple frames are lost. The results of the paper show the prediction results when multiple frames of the vehicle are lost. Do you understand how this operates or what part of the code is there?