This project was created to solve the need for a MY99 BMW 540i (E39) to drive autonomously.
OPENPILOT is an open source driver assistance system that offers Automated Lane Centering and Adaptive Cruise Control for over 200 supported car makes and models. To function properly, OPENPILOT needs to be able to control the longitudinal (gas and brake) and lateral (steering) movements of the car using the CAN bus. For more information, see https://comma.ai/ and https://github.com/commaai/openpilot.
Video introducing OPENPILOT.
OPENPILOT is a complex robotics platform that can be difficult to explain or quickly learn about. For those interested in taking a deeper look at how OPENPILOT works, the following resources may be helpful: https://blog.comma.ai/openpilot-in-2021/ and https://github.com/commaai/openpilot/wiki/Introduction-to-openpilot.
In short, OPENPILOT is a LEVEL 2 ADAS (Advanced Driver Assistance System) software that runs on a device with two cameras: one for recording the view of the road and another for driver monitoring. The device must also be connected to or have an integrated device that can communicate with the CAN bus and has high-precision GPS (e.g. Panda). OPENPILOT takes in the camera feeds, runs them through an ML model, evaluates the car's state using sensor and CAN bus data, and outputs longitudinal and lateral control commands based on that information. This allows the system to make intelligent, real-time decisions about how to control the car.
To run OPENPILOT, you need hardware that is compatible with it. COMMA AI produces their own device, the COMMA THREE (https://github.com/commaai/openpilot/wiki/comma-three), but older hardware has also been developed by COMMA, such as the COMMA TWO (https://github.com/commaai/openpilot/wiki/comma-two) and EON. Additionally, it is possible to run OPENPILOT on a Linux PC.
Video of COMMA TWO:
FrEON is the name of a phone that is capable of running OPENPILOT, and is encased in a 3D-printed case with some type of DIY cooling system. Only two phone models can currently run the older version of the OPENPILOT software: the OnePlus 3T and the LeEco Le Pro 3 (what a great name don't you think).
Panda is a crucial part of the OPENPILOT ecosystem, along with the COMMA/FrEON device. It is a CAN interfacing device that has high-precision GPS and is used to power the COMMA/FrEON (https://github.com/commaai/panda). There have been four different types of Pandas, including White, Grey, Black, and the newest version, Red. The following is a hardware guide for the White Panda (an older version): https://github.com/commaai/panda/blob/master/docs/guide.pdf.
Video showing Grey and Black Panda:
On cars that have ADAS, the working principle of Pandas is slightly different, as they need to intercept CAN bus messages. In my simpler use case, Panda simply communicates with a single CAN bus and requires connections to 12V, GND, and IGN lines to function properly. I won't go into more detail about the differences in the working principle of Pandas in this context.
For OPENPILOT to work correctly, it needs to have both lateral (steering) and longitudinal (gas and brake) control of the car. In newer cars with ADAS capabilities, this is achieved by intercepting the CAN bus messages, but in older cars, additional hardware is usually required to add this control. In my case, this is done by intercepting the gas pedal sensor signals with a COMMA PEDAL type of hardware for the gas (https://github.com/commaai/openpilot/wiki/comma-pedal), and using my own designed BrakeModule for the brake (https://github.com/killinen/BrakeModule). I currently do not have lateral control.
The following video provides a good overview of the requirements for making OPENPILOT work on an older car. The gas solution shown in the video is for mechanical throttle bodies, interceptors can be used with electronic throttle bodies, like in my case. Additionally, the video does not show the implementation of braking capabilities, but it still provides a good illustration of the basic idea behind the system:
So devices that are interfacing with my car are:
- Panda (Grey)
- Gas interceptor
- BrakeModule
These all talk to each other via BMW's original CAN bus:
On the same CAN bus are as shown in the picture are:
- Panda (OP)
- BrakeModule (BM)
- Steering Angle Sensor (LEW)
- BOSCH ABS module (DSCIII)
- Motor contor unit (DME)
Communication between Panda and FrEON takes place via USB.
OPENPILOT listen following states from CAN bus:
- Wheel speeds (car speed)
- Gas pedal state
- Brake pedal state (disengage OP)
- Blinkers state (lane change detection)
- Door states (warning)
- Seatbelt states (warning)
- DSC state (disengage OP if not on)
- Steering Wheel Angle sensor (needed for determine car state)
- Gear selector state (OP won't engage on P)
- Cruise state (engage/disengage OP)
OPENPILOT sends to CAN bus:
- Gas pedal request message (to gas interceptor)
- Brake demand message (to BrakeModule)
- Cruise control cancel request message
- Lead car info message (debug)
- TOYOTA instrument cluster ui message (legacy, not really needed)
Few mods have been done to this fork order to work with my car and my likenings. On high level:
- Add E39 CAN msgs configuration to opendbc
- Detect BMW as OLD_CAR which is detected as TOYOTA COROLLA (fingerprinting)
- Panda SW mods to accept BMW CAN msg's
- Disable steering requests
- Enable vision only longitudinal control and improve it (ACC wo radar)
- Made stop and go work good enough
- Made smoothing of gas output to lessen jerk when accelerating
- Add OP3T support
Default branch is based on openpilot version 0.8.2. In December of 2022 there is already openpilot version 0.9, which is much more capable than this older one. However, my hardware is no longer supported by the newer version of the software.
To be clear my car does not currently have steering capabilities. However, the system I am using offers vision-based ACC with stop and go functionality, which greatly improves my driving experience. In fact, if for some reason OPENPILOT is not working, it bums me.
Below is notes from Shane Smiskol whos Stock Additions fork is integrated in my fork, which has great additions to stock OPENPILOT
Stock Additions Update 3 (0.8.2)
Stock Additions is a fork of openpilot designed to be minimal in design while boasting various feature additions and behavior improvements over stock. I have a 2017 Toyota Corolla with comma pedal, so most of my changes are designed to improve the longitudinal performance.
Want to request a feature or create a bug report? Open an issue here! Want to reach me to debug an issue or have a question? Shane#6175 on Discord!
View Stock Additions Changelog
- Dynamic follow (now with profiles!) - 3 + auto profiles to control distance
- Lane Speed ❗ALL LANE SPEED FEATURES REMOVED TEMPORARILY❗
- Lane Speed Alerts - alerts for when an adjacent lane is faster
- Dynamic camera offsetting - moves you over if adjacent lane has oncoming traffic
- Dynamic gas - smoother gas control
- Adding derivative to PI for better control - lat: smoother control in turns; long: fix for comma pedal overshoot
- NEW❗ Smoother long control using delay - using an accel delay, just like for lateral
- Customize this fork - easily edit fork parameters with support for live tuning
- Automatic updates
- ZSS Support - takes advantage of your high-precision Zorrobyte Steering Sensor
- Offline crash logging - check out
/data/community/crashes
- Colored the lane lines - based on distance from car
- Colored model path - based on curvature
Dynamic follow aims to provide the stock (Toyota) experience of having three different distance settings. Dynamic follow works by dynamically changing the distance in seconds which is sent to the long MPC to predict a speed to travel at. Basically, if the lead is decelerating or might soon, increase distance to prepare. And if the lead is accelerating, reduce distance to get up to speed quicker.
Dynamic follow works if openpilot can control your vehicle's gas and brakes (longitudinal). Check if openpilot can control your vehicle's longitudinal from this list.
Just use the button on the button right of the screen while driving to change between these profiles:
traffic- Meant to keep you a bit closer in traffic, hopefully reducing cut-ins. Always be alert, as you are with any driving assistance software.relaxed- This is the default dynamic follow profile for casual driving.stock- This is the stock 1.8 second profile default in stock openpilot, with no dynamic follow mods. The previous roadtrip profile was closer than a true road trip profile, this is more in line with that intention.auto- The auto dynamic follow model was trained on about an hour of me manually cycling through the different profiles based on driving conditions, this profile tries to replicate those decisions entirely on its own.
I've trained a custom model with Keras that takes in the past 35 seconds of your speed, the lead's speed and the lead's distance. With these inputs, it tries to correctly predict which profile is the best for your current situation.
It's only been trained on about an hour of data, so it's not perfect yet, but it's great for users who just want to set it and forget it. To enable the auto profile, simply tap the profile changing button for dynamic follow until it reaches the auto profile!
If you're annoyed by the silent alerts that show when the model has changed the profile automatically, just use opEdit and set hide_auto_df_alerts to True. Auto profile and model will remain functional but will not show alerts.
Resources:
- The auto-df repo.
- The model file.
- I converted the Keras model to be able to run with pure NumPy using Konverter.
This feature alerts you of faster-travelling adjacent lanes and can be configured using the on-screen LS button on the bottom right to either be disabled, audible, or silent.
The idea behind this feature is since we often become very relaxed behind the wheel when being driven by openpilot, we don't always notice when we've become stuck behind a slower-moving vehicle. When either the left or right adjacent lane is moving faster than your current lane, LaneSpeed alerts the user that a faster lane is available so that they can make a lane change, overtaking the slower current lane. Thus saving time in the long run on long road trips or in general highway driving!
The original idea is thanks to Greengree#5537 on Discord. This feature is available at 35 mph and up.
This feature automatically adjusts your position in the lane if an adjacent lane has oncoming traffic. For example, if you're on a two-lane highway and the left adjacent lane has oncoming cars, LaneSpeed recognizes those cars and applies an offset to your CAMERA_OFFSET to move you over in the lane, keeping you farther from oncoming cars.
This feature is available from 35 to ~60 mph due to a limitation with the Toyota radar. It may not recognize oncoming traffic above 60 mph or so. To enable or disable this feature, use opEdit and change this parameter: dynamic_camera_offset.
Dynamic gas aims to provide a smoother driving experience in stop and go traffic (under 20 mph) by reducing the maximum gas that can be applied based on your current velocity, the relative velocity of the lead, the acceleration of the lead, and the distance of the lead. This usually results in quicker and smoother acceleration from a standstill without the jerking you get in stock openpilot with comma pedal (ex. taking off from a traffic light). It tries to coast if the lead is just inching up, it doesn’t use maximum gas as soon as the lead inches forward. When you are above 20 mph, relative velocity and the current following distance in seconds is taken into consideration.
All cars that have a comma pedal are supported! However to get the smoothest acceleration, I've custom tuned gas curve profiles for the following cars:
pedal cars:
- 2017 Toyota Corolla (non-TSS2)
- Toyota RAV4 (non-TSS2)
- 2017 Honda Civic
- 2019 Honda Pilot
non-pedal cars:
- None yet
If you have a car without a pedal, or you do have one but I haven't created a profile for you yet, please let me know and we can develop one for your car to test.
(long: longitudinal, speed control. lat: latitudinal, steering control)
Changes for lat control: (NEW❗)
-
Adding the derivative componenet to lat control greatly improves the turning performance of openpilot, I've found it loses control much less frequently in both slight and sharp curves and smooths out steering in all situations. Basically it ramps down torque as your wheel approaches the desired angle, and ramps up torque quicky when your wheel is moving away from desired.
Currently Supported Cars: (when param
use_lqris False)- 2017 Toyota Corolla
- TSS2 Toyota Corolla (when param
corollaTSS2_use_indiis False) - tune from birdman! - All Prius years (when param
prius_use_pidis True) - tune from Trae!
Changes for long control:
-
I've added a custom implementation of derivative to the PI loop controlling the gas and brake output sent to your car. Derivative (change in error) is calculated based on the current and last error and added to the class's integral variable. It's essentially winding down integral according to derivative. It helps fix overshoot on some cars with the comma pedal and increases responsiveness (like when going up and down hills) on all other cars! Still need to figure out the tuning, right now it's using the same derivative gain for all cars. Test it out and let me know what you think!
Long derivative is disabled by default due to only one tune for all cars, but can be enabled by using opEdit and setting the
enable_long_derivativeparameter toTrue. It works well on my '17 Corolla with pedal.
This just simply uses desired future acceleration for feedforward rather than current desired acceleration. openpilot already compensates for steering delay, but not longitudinal. This adds that, replacing the previous experimental feature called eager accel which tried to fix the same issues; jerky and late braking. Now we more correctly compensate for delay.
By default, we assume a 0.4 second delay from sending acceleration to seeing it realized, which is tunable with the opEdit param long_accel_delay. Raise if braking too late, lower if braking too early. Stock openpilot is 0.0 (no delay).
This is a handy tool to change your opParams parameters without diving into any json files or code. You can specify parameters to be used in any fork's operation that supports opParams. First, ssh in to your EON and make sure you're in /data/openpilot, then start opEdit:
cd /data/openpilot
python op_edit.py # or ./op_edit.pyTo see what features opEdit has, click me!
🆕 All params now update live while driving, and params that are marked with static need a reboot of the device, or ignition.
Here are the main parameters you can change with this fork:
- Tuning params:
camera_offset(live!): Your camera offset to use in lane_planner.py. Helps fix lane huggingsteer_ratio(live!): The steering ratio you want to use with openpilot. If you enter None, it will use the learned steer ratio from openpilot insteadenable_long_derivative: This enables derivative-based integral wind-down to help overshooting within the PID loop. Useful for Toyotas with pedals or cars with bad long tuninguse_lqr: Enable this to use LQR for lateral control with any car. It uses the RAV4 tuning, but has proven to work well for many cars
- General fork params:
alca_no_nudge_speed: Above this speed (mph), lane changes initiate IMMEDIATELY after turning on the blinker. Behavior is stock under this speed (waits for torque)upload_on_hotspot: Controls whether your EON will upload driving data on your phone's hotspotupdate_behavior:offwill never update,alertshows an alert on-screen.autowill reboot the device when an update is seendisengage_on_gas: Whether you want openpilot to disengage on gas input or not
- Dynamic params:
dynamic_gas: Whether to use dynamic gas if your car is supportedglobal_df_mod(live!): The multiplier for the current distance used by dynamic follow. The range is limited from 0.85 to 2.5. Smaller values will get you closer, larger will get you farther. This is applied to ALL profiles!min_TR(live!): The minimum allowed following distance in seconds. Default is 0.9 seconds, the range of this mod is limited from 0.85 to 1.3 seconds. This is applied to ALL profiles!hide_auto_df_alerts: Hides the alert that shows what profile the model has chosendynamic_camera_offset: Whether to automatically keep away from oncoming traffic. Works from 35 to ~60 mphdynamic_camera_offset_time: How long to keep the offset after losing the oncoming lane/radar track in seconds
dynamic_follow: Deprecated, use the on-screen button to change profiles
- Experimental params:
support_white_panda: This allows users with the original white panda to use openpilot above 0.7.7. The high precision localizer's performance may be reduced due to a lack of GPSprius_use_pid: This enables the PID lateral controller with new a experimental derivative tunecorollaTSS2_use_indi: Enable this to use INDI for lat with your TSS2 Corolla (can be enabled for all years by request)standstill_hack: Some cars support stop and go, you just need to enable this
A full list of parameters that you can modify are located here.
An archive of opParams lives here.
Parameters are stored at /data/op_params.json
When a new update is available on GitHub for Stock Additions, your EON/C2 will pull and reset your local branch to the remote. It then queues a reboot to occur when the following is true:
- your EON has been inactive or offroad for more than 5 minutes
update_behaviorparam is set toauto
Therefore, if your device sees an update while you're driving it will reboot approximately 5 to 10 minutes after you finish your drive, it resets the timer if you start driving again before the time is up.
If you have a Prius with a ZSS (Zorrobyte Steer Sensor), you can use this fork to take full advantage of your high-precision angle sensor! Added support for ZSS with PR #198, there's nothing you need to do. Special thanks to Trae for helping testing the addition!
If you have a ZSS but not a Prius, let me know and I can add support for your car.
If you experience a crash or exception while driving with this fork, and you're not on internet for the error to be uploaded to Sentry, you should be able to check out the directory /data/community/crashes to see any and all logs of exceptions caught in openpilot. Simply view the logs with ls -lah and then cat the file you wish to view by date. This does not catch all errors, for example scons compilation errors or some Python syntax errors will not be caught, tmux a is usually best to view these (if openpilot didn't start).
❗Quickly view the latest crash: cat /data/community/crashes/latest.log
Feel free to reach out to me on Discord if you're having any issues with the fork!
To install Stock Additions, just run the following on your EON/C2 (make sure to press enter after each line):
cd /data/
mv openpilot openpilot.old # or equivalent
git clone -b stock_additions --single-branch https://github.com/shanesmiskol/openpilot --depth 1
reboot
The --depth 1 flag shallow clones the fork, it ends up being about 90 Mb so you can get the fork up and running quickly. Once you install Stock Additions, automatic updating should always keep openpilot up to date with the latest from my fork!
Or use the emu CLI to easily switch to this fork's default branch: emu fork switch ShaneSmiskol. The initial setup may take longer than the above command, but you gain the ability to switch to any fork you want.
Or (last or, I promise!) you can use my handy fork installation link during NEOS setup after a factory reset: https://smiskol.com/fork/shane
Most of the branches on this fork are development branches I use as various openpilot tests. The few that more permanent are the following:
stock_additions: This is similar to stock openpilot's release branch. Will receive occasional and tested updates to Stock Additions.stock_additions-develorSA-staging: My development branch of Stock Additions I use to test new features or changes; similar to the master branch. Not recommendeded as a daily driver.
- Stock Additions 0.7
- Stock Additions 0.7.1
- Stock Additions 0.7.4
- Stock Additions 0.7.5
- Stock Additions 0.7.7
- Stock Additions 0.7.10
- Stock Additions 0.8
Here's a short video showing how close the traffic profile was in 0.7.4. In 0.7.5, the traffic profile is an average of 7.371 feet closer from 18 mph to 90 mph. Video thanks to @rolo01!
If you'd like to support my development of Stock Additions with a dollar for a RaceTrac ICEE (my PayPal link). Thanks! 🥰