Version 1.2: New Improvements (#17)

* Added engine RPM logging
* Added gear logging
* Added fuel calcuation
* Added boost logging
* Added yaw rate logging
* Added helper scripts (`run.ps1`, `run.bat`, `run.command`, `run.sh`) for Windows, Mac and Linux
* Added lap saving as JSON
* Add missing documentation
* Upgrade to bokeh 3.2.0
* Upgrade scipy 1.10.0
* Add doc about speed peaks and valleys
* Fixed background logging. Gt7dashboard will now log laps in background even if the browser window is not active
* Major refactoring of the main method

.gitignore

@@ -135,3 +135,4 @@ session.csv
 
 test_out
 db/cars.csv
+gt7dashboard/test/data

Makefile

@@ -25,8 +25,8 @@ car_lists:
 	python3 helper/download_cars_csv.py
 
 serve:
-	GT7_PLAYSTATION_IP=ps5wifi bokeh serve .
+	bokeh serve .
 
 deploy:
 	git push
-	ssh ${MK_SERVER_USER}@${MK_SERVER_HOST} "cd work/gt7telemetry; git pull; cd ~/git/conf/docker; sudo -S CONTAINER_NAME=gt7telemetry make build"
+	ssh ${MK_SERVER_USER}@${MK_SERVER_HOST} "cd work/gt7dashboard && git pull && git switch '$(shell git rev-parse --abbrev-ref HEAD)' && cd ~/git/conf/docker && sudo -S CONTAINER_NAME=gt7dashboard make build"

README.md

@@ -33,25 +33,31 @@ See the [Manual](#manual) for detailed instructions.
 
 ### Get Telemetry of a Demonstration lap or Replay
 
-Enable the "Always Record" checkbox to always record replays. Otherwise will only the laps you are actually driving are recorded.
+Enable the "Always Record" checkbox to always record replays. Otherwise, will only the laps you are actually driving are recorded.
 
 ## How to run
 
 You will have to have a running Python installation. Look [here](https://wiki.python.org/moin/BeginnersGuide/Download) for instructions.
 
-* If you are on Windows edit the file `run.bat` and replace `<...>` with your IP address. Run the file with a double click afterwards.
+* If you are on Windows 
+  * Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
 
-* If you are on MacOS edit the file `run.command` and replace `<...>` with your IP address. Run the file with a double click afterwards.
-
-* If you are on Linux edit the file `run.sh` and replace `<...>` with your IP address. Run the file with a double click afterwards.
+  * Run the file `run.bat` with a double click
+* If you are on MacOS run the file `run.command`
+* If you are on Linux run the file `run.sh`
 
 The commands `pip3` or `python3` may be different on your OS. Try `pip` or `python` instead.
 
 ## How to run for experienced users
 
 1. (Once) `pip3 install -r requirements.txt` to install Python dependencies
-2. (Optional) Download the list of car names with `python3 helper/download_cars_csv.py`. Without this file, car names will only show as `CAR-ID-123`.
+   1. On Windows: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
+
+2. (Optional, Once) Download the list of car names with `python3 helper/download_cars_csv.py`. Without this file, car names will only show as `CAR-ID-123`.
 3. Running the Dashboard
+    - (Mac/Linux) `bokeh serve .` (when inside the  `gt7dashboard` folder)
+    - (Windows) `python -m bokeh serve .`  (when inside the `gt7dashboard` folder)
+4. (Optional) Running the Dashboard with a custom IP
    - (Mac/Linux) `GT7_PLAYSTATION_IP=<CONSOLE IP ADDRESS> bokeh serve .` (when inside the  `gt7dashboard` folder)
    - (Windows) `set GT7_PLAYSTATION_IP=<CONSOLE IP ADDRESS>` and `python -m bokeh serve .`  (when inside the `gt7dashboard` folder)
 
@@ -97,6 +103,13 @@ This is a sample `docker-compose` configuration:
             - GT7_PLAYSTATION_IP=<playstation ip>
             - TZ=Europe/Berlin
 ```
+
+Hint: You should set the `GT7_PLAYSTATION_IP` env var since Docker containers are not allowed to send UDP broadcasts by default. This is the default behaviour when no IP is set.
+
+## Lap Files
+
+If you want to edit your lap files, use a JSON editor. For example ` cat ... | jq -c '.[0:4]' > ...` will shorten the laps to the first 4 laps in the save file.
+
 ## Contributing
 
 Please add unit tests for all new features, calculations etc.
@@ -159,6 +172,12 @@ This map is helpful if you are using the index number of a graph to quickly dete
 
 See the tab 'Race Line' for a more detailed race line.
 
+#### Peaks and Valleys
+
+![screenshot_header](README.assets/screenshot_peaks_and_valleys.png)
+
+A list of speed peaks and valleys for the selected laps. We assume peaks are straights (s) and valleys are turns (T). Use this to compare the difference in speed between the last lap and the reference lap on given positions of the race track.
+
 #### Speed Deviation (Spd. Dev.)
 
 ![screenshot_header](README.assets/screenshot_speeddeviation.png)
@@ -184,6 +203,14 @@ If they had one graph it would be the deviation in the (best) laps of the same d
 
 This is the amount of throttle pressure from 0% to 100% of the laps selected.
 
+#### Yaw Rate / Second
+
+![screenshot_header](README.assets/screenshot_yaw.png)
+
+This is the yaw rate per second of your car. Use this to determine the Maximum Rotation Point (MRP). At this point you should normally accelerate.
+
+[Suellio Almeida](https://suellioalmeida.ca) introduced this concept to me. See [here](https://www.youtube.com/watch?v=B92vFKKjyB0) for more information.
+
 #### Braking
 
 ![screenshot_header](README.assets/screenshot_braking.png)
@@ -196,6 +223,24 @@ This is the amount of braking pressure from 0% to 100% of the laps selected.
 
 This is the amount of coasting from 0% to 100% of the laps selected. Coasting is when neither throttle nor brake are engaged.
 
+#### Gear
+
+![screenshot_header](README.assets/screenshot_gear.png)
+
+This is the current gear of the laps selected.
+
+#### RPM
+
+![screenshot_header](README.assets/screenshot_rpm.png)
+
+This is the current RPM of the laps selected.
+
+#### Boost
+
+![screenshot_header](README.assets/screenshot_boost.png)
+
+This is the current Boost in x100 kPa of the laps selected.
+
 #### Tire Speed / Car Speed
 
 ![screenshot_header](README.assets/screenshot_tirespeed.png)
@@ -238,5 +283,3 @@ Here is some useful information you may use for tuning. Such as Max Speed and mi
 This is a race line map with the last lap (blue) and the reference lap (magenta). This diagram does also feature spead peaks (▴) and valleys (▾) as well as throttle, brake and coasting zones.
 
 The thinner line of the two is your last lap. The reference line is the thicker translucent line. If you want to make out differences in the race line have a look at the middle of the reference lap line and your line. You may zoom in to spot the differences and read the values on peaks and valleys.
-
-

generate_doc.py

@@ -41,6 +41,10 @@ def add_screenshot(filename):
     out_markdown += add_screenshot("screenshot_raceline.png") + "\n\n"
     out_markdown += gt7help.RACE_LINE_MINI + "\n\n"
 
+    out_markdown += "#### Peaks and Valleys\n\n"
+    out_markdown += add_screenshot("screenshot_peaks_and_valleys.png") + "\n\n"
+    out_markdown += gt7help.SPEED_PEAKS_AND_VALLEYS + "\n\n"
+
     out_markdown += "#### Speed Deviation (Spd. Dev.)\n\n"
     out_markdown += add_screenshot("screenshot_speeddeviation.png") + "\n\n"
     out_markdown += gt7help.SPEED_VARIANCE + "\n\n"
@@ -53,6 +57,11 @@ def add_screenshot(filename):
     out_markdown += add_screenshot("screenshot_throttle.png") + "\n\n"
     out_markdown += gt7help.THROTTLE_DIAGRAM + "\n\n"
 
+    out_markdown += "#### Yaw Rate / Second\n\n"
+    out_markdown += add_screenshot("screenshot_yaw.png") + "\n\n"
+    out_markdown += gt7help.YAW_RATE_DIAGRAM + "\n\n"
+    out_markdown += "[Suellio Almeida](https://suellioalmeida.ca) introduced this concept to me. See [here](https://www.youtube.com/watch?v=B92vFKKjyB0) for more information.\n\n"
+
     out_markdown += "#### Braking\n\n"
     out_markdown += add_screenshot("screenshot_braking.png") + "\n\n"
     out_markdown += gt7help.BRAKING_DIAGRAM + "\n\n"
@@ -61,6 +70,18 @@ def add_screenshot(filename):
     out_markdown += add_screenshot("screenshot_coasting.png") + "\n\n"
     out_markdown += gt7help.COASTING_DIAGRAM + "\n\n"
 
+    out_markdown += "#### Gear\n\n"
+    out_markdown += add_screenshot("screenshot_gear.png") + "\n\n"
+    out_markdown += gt7help.GEAR_DIAGRAM + "\n\n"
+
+    out_markdown += "#### RPM\n\n"
+    out_markdown += add_screenshot("screenshot_rpm.png") + "\n\n"
+    out_markdown += gt7help.RPM_DIAGRAM + "\n\n"
+
+    out_markdown += "#### Boost\n\n"
+    out_markdown += add_screenshot("screenshot_boost.png") + "\n\n"
+    out_markdown += gt7help.BOOST_DIAGRAM + "\n\n"
+
     out_markdown += "#### Tire Speed / Car Speed\n\n"
     out_markdown += add_screenshot("screenshot_tirespeed.png") + "\n\n"
     out_markdown += gt7help.TIRE_DIAGRAM + "\n\n"

gt7dashboard/gt7communication.py

@@ -1,9 +1,11 @@
 import datetime
 import json
 import logging
+import math
 import socket
 import struct
 import time
+import copy
 import traceback
 from datetime import timedelta
 from threading import Thread
@@ -166,6 +168,9 @@ def __init__(self, playstation_ip):
         # True will always quit with the main process
         self.daemon = True
 
+        # Set lap callback function as none
+        self.lap_callback_function = None
+
         self.playstation_ip = playstation_ip
         self.send_port = 33739
         self.receive_port = 33740
@@ -189,6 +194,10 @@ def run(self):
             try:
                 self._shall_restart = False
                 s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+
+                if self.playstation_ip == "255.255.255.255":
+                    s.setsockopt (socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
+
                 s.bind(('0.0.0.0', self.receive_port))
                 self._send_hb(s)
                 s.settimeout(10)
@@ -239,10 +248,13 @@ def run(self):
                         # Handler for package exceptions
                         self._send_hb(s)
                         package_nr = 0
+                        # Reset package id for new connections
+                        package_id = 0
 
             except Exception as e:
                 # Handler for general socket exceptions
-                logging.info("No connection to %s:%d: %s" % (self.playstation_ip, self.send_port, e))
+                # TODO logging not working
+                print("Error while connecting to %s:%d: %s" % (self.playstation_ip, self.send_port, e))
                 s.close()
                 # Wait before reconnect
                 time.sleep(5)
@@ -315,7 +327,6 @@ def _log_data(self, data):
         self.current_lap.data_braking.append(data.brake)
         self.current_lap.data_throttle.append(data.throttle)
         self.current_lap.data_speed.append(data.car_speed)
-        self.current_lap.data_rpm.append(data.rpm)
 
         delta_divisor = data.car_speed
         if data.car_speed == 0:
@@ -331,12 +342,35 @@ def _log_data(self, data):
 
         self.current_lap.data_tires.append(delta_fl + delta_fr + delta_rl + delta_rr)
 
+        ## RPM and shifting
+
+        self.current_lap.data_rpm.append(data.rpm)
+        self.current_lap.data_gear.append(data.current_gear)
+
         ## Log Position
 
         self.current_lap.data_position_x.append(data.position_x)
         self.current_lap.data_position_y.append(data.position_y)
         self.current_lap.data_position_z.append(data.position_z)
 
+        ## Log Boost
+
+        self.current_lap.data_boost.append(data.boost)
+
+        ## Log Yaw Rate
+
+        # This is the interval to collection yaw rate
+        interval = 1 * 60 # 1 second has 60 fps and 60 data ticks
+        self.current_lap.data_rotation_yaw.append(data.rotation_yaw)
+
+        # Collect yaw rate, skip first interval with all zeroes
+        if len(self.current_lap.data_rotation_yaw) > interval:
+            yaw_rate_per_second = data.rotation_yaw - self.current_lap.data_rotation_yaw[-interval]
+        else:
+            yaw_rate_per_second = 0
+
+        self.current_lap.data_absolute_yaw_rate_per_second.append(abs(yaw_rate_per_second))
+
         # Adapted from https://www.gtplanet.net/forum/threads/gt7-is-compatible-with-motion-rig.410728/post-13810797
         self.current_lap.lap_live_time = (self.current_lap.lap_ticks * 1. / 60.) - (self.session.special_packet_time / 1000.)
 
@@ -365,21 +399,31 @@ def finish_lap(self, manual=False):
         self.current_lap.fuel_at_end = self.last_data.current_fuel
         self.current_lap.fuel_consumed = self.current_lap.fuel_at_start - self.current_lap.fuel_at_end
         self.current_lap.lap_finish_time = self.current_lap.lap_finish_time
+        self.current_lap.total_laps = self.last_data.total_laps
         self.current_lap.title = seconds_to_lap_time(self.current_lap.lap_finish_time / 1000)
         self.current_lap.car_id = self.last_data.car_id
         self.current_lap.number = self.last_data.current_lap - 1  # Is not counting the same way as the in-game timetable
+        # TODO Proper pythonic name
         self.current_lap.EstimatedTopSpeed = self.last_data.estimated_top_speed
 
+        self.current_lap.lap_end_timestamp = datetime.datetime.now()
+
         # Race is not in 0th lap, which is before starting the race.
         # We will only persist those laps that have crossed the starting line at least once
+        # And those laps which have data for speed logged. This will prevent empty laps.
         # TODO Correct this comment, this is about Laptime not lap numbers
-        if self.current_lap.lap_finish_time > 0:
+        if self.current_lap.lap_finish_time > 0 and len(self.current_lap.data_speed) > 0:
             self.laps.insert(0, self.current_lap)
 
+            # Make a copy of this lap and call the callback function if set
+            if self.lap_callback_function:
+                self.lap_callback_function(copy.deepcopy(self.current_lap))
+
         # Reset current lap with an empty one
         self.current_lap = Lap()
         self.current_lap.fuel_at_start = self.last_data.current_fuel
 
+
     def reset(self):
         """
         Resets the current lap, all stored laps and the current session.
@@ -389,6 +433,9 @@ def reset(self):
         self.last_data = GTData(None)
         self.laps = []
 
+    def set_lap_callback(self, new_lap_callback):
+        self.lap_callback_function = new_lap_callback
+
 
 # data stream decoding
 def salsa20_dec(dat):