Simulator requirements

[WJaworskiRobotec]

Discussion about requirements for the Simulator to enable testing of the Autoware components in Bus ODD.
Agreed requirements will be compiled into the document in the Autoware wiki and any Simulator provider can integrate with Autoware following the requirements document.

Groups of requirements to specify (If anyone has other ideas, feel free to add):

• Simulation environment
• Vehicle model
• Sensors specifications
• Integration interface with AWF/Universe (topics and messages description)
• Supported scenarios / scenario formats
• Which components to be tested in ss2 and in E2E Simulator
• Metrics to calculate (To be defined in the future with E2E and metric calculation nodes)

[brkay54]

About Integration interface with AWF/Universe, I don't know we can access them or not in current design but I think we should be able to access the vehicles state (position, orientation, steering tire angle etc.) without any noise to evaluate the modules.

[WJaworskiRobotec]

Integration interface description prepared by TierIV (still under development, table will be updated according to any changes). CC: @brkay54

publish / subscribe	category	topic	msg	frame_id	hz
publish	clock	/clock	rosgraph_msgs/Clock		100~1000
subscribe	control	/control/command/turn_indicators_cmd	autoware_auto_vehicle_msgs/TurnIndicatorsCommand		10
subscribe	control	/control/command/hazard_lights_cmd	autoware_auto_vehicle_msgs/HazardLightsCommand		10
subscribe	control	/control/command/control_cmd	autoware_auto_control_msgs/AckermannControlCommand		60
subscribe	control	/control/command/gear_cmd	autoware_auto_vehicle_msgs/GearCommand		10
subscribe	control	/control/command/emergency_cmd	tier4_vehicle_msgs/msg/VehicleEmergencyStamped		60
publish	camera	/sensing/camera/traffic_light/camera_info	sensor_msgs/CameraInfo	traffic_light_left_camera/camera_link	10
publish	camera	/sensing/camera/traffic_light/image_raw	sensor_msgs/Image	traffic_light_left_camera/camera_link	10
publish	gps	/sensing/gnss/pose	geometry_msgs/PoseStamped	gnss_link	1
publish	gps	/sensing/gnss/pose_with_covariance	geometry_msgs/PoseWithCovarianceStamped	gnss_link	1
publish	imu	/sensing/imu/tamagawa/imu_raw	sensor_msgs/Imu	tamagawa/imu_link	30
publish	lidar	/sensing/lidar/left/pointcloud_raw	sensor_msgs/PointCloud2	velodyne_left_base_link	10
publish	lidar	/sensing/lidar/left/pointcloud_raw_ex	sensor_msgs/PointCloud2	velodyne_left_base_link	10
publish	lidar	/sensing/lidar/right/pointcloud_raw	sensor_msgs/PointCloud2	velodyne_right_base_link	10
publish	lidar	/sensing/lidar/right/pointcloud_raw_ex	sensor_msgs/PointCloud2	velodyne_right_base_link	10
publish	lidar	/sensing/lidar/top/pointcloud_raw	sensor_msgs/PointCloud2	sensor_kit_base_link	10
publish	lidar	/sensing/lidar/top/pointcloud_raw_ex	sensor_msgs/PointCloud2	sensor_kit_base_link	10
publish	vehicle status	/vehicle/status/control_mode	autoware_auto_vehicle_msgs/ControlModeReport		30
publish	vehicle status	/vehicle/status/gear_status	autoware_auto_vehicle_msgs/GearReport		30
publish	vehicle status	/vehicle/status/steering_status	autoware_auto_vehicle_msgs/SteeringReport		30
publish	vehicle status	/vehicle/status/turn_indicators_status	autoware_auto_vehicle_msgs/TurnIndicatorsReport		30
publish	vehicle status	/vehicle/status/hazard_lights_status	autoware_auto_vehicle_msgs/HazardLightsReport		30
publish	vehicle status	/vehicle/status/velocity_status	autoware_auto_vehicle_msgs/VehicleReport	base_link	30

[brkay54]

@WJaworskiRobotec I guess that these outputs will be noisy in autoware right? Actually I wanted to ask that there will be any gate to access these outputs of simulation without any noise? Or will we able to determine the noises of these outputs? Because in LGSVL we couldn't access the raw outputs of simulation (I guess).

[WJaworskiRobotec]

@brkay54 Ok I see your point. You mean to add to the simulation publication of Ground Truth information about position/orientation/steering angle etc. to be able to evaluate e.g. Localization algorithms accuracy ? I fully agree that such topics should be added to the simulation

[sglee-morai]

1. About Simulation Environment

1.1 The simulator shall support 3D environment that can match lanelet2 data which are used for SUT (Autoware-based autonomous driving software)

For the bus-odd, the following maps in lanelet2 are available. Which map will be used for validation isn't decided yet.

• Hsinchu-city
• ITRI Campus

1.2 The simulator shall support traffic lights

(TBD) Which types of traffic lights should be supported by the simulator

Related Use-cases for this (bus odd document)

• UC-PB-001-0009
• UC-PB-008-0001 (not 100% sure)
• UC-PB-008-0007 (not 100% sure)
• UC-PB-009-0001 (not 100% sure)

1.3 The simulator shall support traffic signs (TBD)

Need a discussion. The bus-odd document does not specify requirements where the bus shall do a certain action when detecting a certain traffic sign

[sglee-morai]

2. About Vehicle Model

2.1 The simulator shall support a vehicle model that can meet performance requirements specified by the bus odd document

• Acceleration: 1.75 m/s^2
• Jerk: 2.5 m/s^3
• Stopping distance from 60km/h: 78m (approx.)
• Stopping distance from 50km/h: 54m (approx.)
• Stopping distance from 40km/h: 34m (approx.)
• Stopping distance from 30km/h: 19m (approx.)

2.2 The vehicle model in the simulator shall have state variables that are published to SUT as vehicle status

A short list for vehicle status ([Warning] The list must be updated when the integration interface section is updated)

• /vehicle/status/control_mode (autoware_auto_vehicle_msgs/ControlModeReport)
• /vehicle/status/gear_status (autoware_auto_vehicle_msgs/GearReport)
• /vehicle/status/steering_status (autoware_auto_vehicle_msgs/SteeringReport)
• /vehicle/status/turn_indicators_status (autoware_auto_vehicle_msgs/TurnIndicatorsReport )
• /vehicle/status/hazard_lights_status (autoware_auto_vehicle_msgs/HazardLightsReport)
• /vehicle/status/velocity_status(autoware_auto_vehicle_msgs/VehicleReport)

For more detail, refer to the integration interface section

2.3 The vehicle model in the simulator shall propagate its internal dynamics using control commands from SUT as an input

A short list for vehicle commands ([Warning] The list above must be updated when the integration interface section is updated)

• /control/command/turn_indicators_cmd | autoware_auto_vehicle_msgs/TurnIndicatorsCommand
• /control/command/hazard_lights_cmd | autoware_auto_vehicle_msgs/HazardLightsCommand
• /control/command/control_cmd | autoware_auto_control_msgs/AckermannControlCommand
• /control/command/gear_cmd | autoware_auto_vehicle_msgs/GearCommand
• /control/command/emergency_cmd | tier4_vehicle_msgs/msg/VehicleEmergencyStamped

For more detail, refer to the integration interface section

2.4 The vehicle model in the simulator shall propagate its internal dynamics using collision with the road surface

2.5 The vehicle model shall have a similar physical look and dimensions as real-world buses to be used for the bus odd project

The following models can be considered

• Bus from ITRI
• ISUZU Bus model (for the sister project) (can be considered later)
• Other commercial bus models (can be considered later)

[Igata-ctrl]

I heard from ITRI that their AD bus is based on the following bus model.
https://www.racev.com/product/k5
But, last week I received an update from ITRI and the followings are the most accurate vehicle specs.
wheel_radius: 0.302875
wheel_width: 0.215
wheel_base: 3.81 # between front wheel center and rear wheel center
wheel_tread: 1.71 # between left wheel center and right wheel center
front_overhang: 1.16 # between front wheel center and vehicle front
rear_overhang: 2.01 # between rear wheel center and vehicle rear
left_overhang: 0.12 # between left wheel center and vehicle left
right_overhang: 0.12 # between right wheel center and vehicle right
vehicle_height: 2.8
As for the exterior image, please refer to p1 and p5 of the following file.
https://drive.google.com/drive/folders/1NEN3tgExwGJGTsDwVH4q37mFRoIAByDQ
Thanks!

[WJaworskiRobotec]

@Igata-ctrl I cannot access the link. Could you use AWF google drive : https://drive.google.com/drive/folders/1YynTJKBC4ULsbvfbOqV7ipRKgMStAYI_ ?

[Igata-ctrl]

Sorry for the trouble, but actually,
https://drive.google.com/drive/folders/1NEN3tgExwGJGTsDwVH4q37mFRoIAByDQ
is an AFW Google Drive and it works for me. I granted you access to the entire ODD WG folder.
Please refer to the file "ITRI BUS ODD Proposal_20210831.pdf" there.

I tried to upload the file to:
https://drive.google.com/drive/folders/1YynTJKBC4ULsbvfbOqV7ipRKgMStAYI_
but it seems that I don't have upload permission to this folder.

[Igata-ctrl]

The file "ITRI_ADV.docx" in
https://drive.google.com/drive/folders/1NEN3tgExwGJGTsDwVH4q37mFRoIAByDQ
has more exterior information of the ITRI bus including the back-shot.

[doganulus]

Not specific to the Bus ODD but I think of the following discussion items (+ those are listed above) for simulator requirements:

• Simulation determinism
• Simulation time model (discrete?, fixed/variable step?)
• Clock synchronization between the simulator and Autoware (if any)
• NPC control in simulation (fully reactive?, scenario-only?, trajectory-only?)
• Supported log formats other than rosbags (if any)

Any requirements on these items? Or non-requirements?

[ralwing]

I think we should also mention the format of scenarios. Currently the OpenScenario 1.x is the most common format for simulator and editors.
Speaking of the editor; we should also consider its availability and licensing.
Another requirement IMHO is the availability of sensor models. The Lidars(solid-state, rotating etc), Radars, IMU...

[WJaworskiRobotec]

@ralwing regarding sensors, there is/will be a list of sensors that needs to be simulated for CargoODD.
As far as i know TierIV has some editor for creating scenarios, let's discuss the details of availability on the Simualtion WG meeting tomorrow

[WJaworskiRobotec]

I've created the wiki page where we will put the final requirements decided in this discussion : link

One more topic that should be written there, is general description of what and how are we going to test using the simulator. Initial high level description of testing procedures :

Planning & Control
Planning module is tested using scenario_simulator_v2 based on scenarios created by ODD Working Group
SS2 provides two runners:
scenario_test_runner – OpenScenario format based scenario runner
random_test_runner – Runner for performing configurable tests with random traffic

Localization
Vision/LIDAR based localization algorithms needs to be tested using the high fidelity simulator with satisfying quality of sensors simulation. Simulator should provide Ground Truth information about current location of the vehicle under test, as well as input data for localization algorithms. Accuracy of the localization algorithms can be tested using localization_evaluation_node (Still in development, link to be shared once it’s merged into Autoware Universe repository)

Perception
For all perception algorithms it should be possible to perform tests in the simulator. To simplify the process, several Ground Truth sensors should be implemented in the simulator:
2D/3D Bounding Box with labels for object detection/classification algorithms validation
Instance Segmentation for segmentation algorithms validation
Then Ground Truth can be compared with the output of perception algorithms of Autoware using perception_evaluation_node(s) (Still in development, link to be shared once it’s merged into Autoware Universe repository)

End-to-end testing
Simulator should be used for End-to-end testing of the entire Autoware stack. For this purpose, the simulator should support easy and flexible way of configuration of traffic in the test scenario (both scenario based and random).

Please let me know your opinion, or anything that we can add/specify here.

[sglee-morai]

Localization + Planning / Control Testing

Localization Module Testing

Need to discuss: While the localization node is being tested, should we keep the ego-vehicle moving? If that's the case, the localization node (or the ego-vehicle) should be moving along a predefined path.

MORAI SIM: Drive has a feature that allows an ego-vehicle to follow a certain path using ground truth data. Is that gonna be enough for aking ego's path?

[sglee-morai]

Test Scenario for E2E Testing

1) Random Traffic (TierIV SIM, MORAI SIM)

• The ego-vehicle would follow a certain path
• Random NPC vehicles are moving around the ego-vehicle
• Density of the random traffic can be adjusted (At least, 3 levels of traffic density should be available: low, mid, and high)
• Traffic Behavior could be randomly chosen (speed of the surrounding vehicles: min, max value)

2) Unit-level Mission (MORAI SIM, TierIV(potential))

• Re-usage of scenarios defined by ODD WG
• A pedestrian coming out of the road suddenly behind the parked car

[sglee-morai]

Perception

Closed-loop Testing (TierIV SIM, MORAI SIM)

Openloop Testing (MORAI SIM)

• Creating synthetic dataset -> rosbag file -> playback with perception node running -> see the result
  - Creating a launcher so that only the perception part is running
• For bus ODD, MORAI SIM: Drive will support that
• Benefit: it doesn't require any real-time performance.

Environment / Scene

• Create a special scene?
  • For Closed-loop testing: Preferred one - use the same scenario as defined above

[WJaworskiRobotec]

Let's discuss the message types here :

• Ground Truth Localization - /tf message
• Ground Truth Instance Segmentation - Pointcloud with additional field "label"
• Ground Truth 2D/3D Bounding Boxes - Autoware messages for Detected Objects

[sglee-morai]

DetectedObjects

PredictedObjects