Merge Master branch changes to develop (#159)

* initial queue implementation cc bridge

* add some comments

* fix unsub logic (#126)

* init

* cc bridge simplify logic

* Cloud bridge investigation (#131)

* initial queue implementation cc bridge

* add some comments

* cc bridge simplify logic

* back to queue, add in file listener

* Fix unit status and UI notification and event live status (#129)

* init

* add comments

* fix warning

* update text

* Remove credentials (#134)

* updating log levels, docker compose log size parameter

* revert local changes

* init

* remove hardcoded values

* add analysis scripts

* messaging server log size parameter

* init

* add nan

* Fix json key value convertor logic (#146)

fix comparison

* Fix/drop nan value fields (#148)

drop fields with NaN values

* clean up readme.md

---------

Co-authored-by: Abey <abey.yoseph@leidos.com>
Co-authored-by: Anish_deva <51463994+adev4a@users.noreply.github.com>
Co-authored-by: dan-du-car <dan.du@leidos.com>
Co-authored-by: abey-yoseph <59704440+abey-yoseph@users.noreply.github.com>
Co-authored-by: dan-du-car <62157949+dan-du-car@users.noreply.github.com>
Co-authored-by: Anish <Anishdeva17@gmail.com>
Co-authored-by: Cody Garver <codygarver@users.noreply.github.com>

README.md

@@ -1,29 +1,18 @@
-| Sonar Code Quality |
-|--------------------|
-[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=usdot-fhwa-stol_cda-telematics&metric=alert_status)](https://sonarcloud.io/dashboard?id=usdot-fhwa-stol_cda-telematics) |
-
-# GitHub Actions Build status
+## GitHub Actions Build status
 | Carma-streets-bridge | Carma-vehicle-bridge | telematics-cloud-messaging |
 |-----|-----|-----|
 [![carma-streets-bridge](https://github.com/usdot-fhwa-stol/cda-telematics/actions/workflows/carma-streets-bridge.yml/badge.svg?branch=feature_gha)](https://github.com/usdot-fhwa-stol/cda-telematics/actions/workflows/carma-streets-bridge.yml) | [![carma-vehicle-bridge](https://github.com/usdot-fhwa-stol/cda-telematics/actions/workflows/carma-vehicle-bridge.yml/badge.svg)](https://github.com/usdot-fhwa-stol/cda-telematics/actions/workflows/carma-vehicle-bridge.yml) | [![telematic-cloud-messaging](https://github.com/usdot-fhwa-stol/cda-telematics/actions/workflows/telematic-cloud-messaging.yml/badge.svg)](https://github.com/usdot-fhwa-stol/cda-telematics/actions/workflows/telematic-cloud-messaging.yml)
 
-# DockerHub Release Builds
-| Carma streets nats bridge | Carma streets vehicle nats bridge | telematic cloud messaging |
-|-----|-----|-----|
-[![Docker Cloud Build Status](https://img.shields.io/docker/cloud/build/usdotfhwastoldev/carma_street_nats_bridge?label=carma_street_nats_bridge)](https://hub.docker.com/repository/docker/usdotfhwastoldev/carma_street_nats_bridge) | [![Docker Cloud Build Status](https://img.shields.io/docker/cloud/build/usdotfhwastoldev/carma_vehicle_nats_bridge?label=carma_vehicle_nats_bridge)](https://hub.docker.com/repository/docker/usdotfhwastoldev/carma_vehicle_nats_bridge) | [![Docker Cloud Build Status](https://img.shields.io/docker/cloud/build/usdotfhwastoldev/telematic_cloud_messaging?label=telematic_cloud_messaging)](https://hub.docker.com/repository/docker/usdotfhwastoldev/telematic_cloud_messaging)
-# DockerHub Release Candidate Builds
-Need to add the build status checks for release candidate builds during first demo release.
-# DockerHub Develop Builds
-Need to add the build status checks for official release builds during first demo release.
-
-# cda-telematics
+# CDA-Telematics
 This project will create an open-source Module that can be installed on any vehicle (e.g. a CARMA Platform and/or Messenger vehicle, an L0 or L1 production vehicle, etc.) that will collect data about the vehicle and wirelessly send it out in real time for data analysis. The same Module, with any modifications, if necessary, will also be compatible with CARMA Streets and CARMA Cloud. On the receiving end of this data, a user will have a Data Processing & Visualization Tool available to visualize and/or plot the data that was sent using the Module(s). This Module can be thought of as a Fleet Management tool with extra capabilities to support CDA research and education.
 
 ## Architecture Diagram
 [Detailed Design](https://usdot-carma.atlassian.net/wiki/spaces/WFD2/pages/2230321179/Detailed+System+Design)
 
 ![architecture](https://user-images.githubusercontent.com/34483068/171265484-67177ebb-69f7-4286-9602-016043079958.png)
 
+## Release Notes
+The current version of CDA-Telematics tool and release history of the CARMA software platform: [CARMA Release Notes](<docs/Release_notes.md>)
 
 ## Documentation
 Documentation of the setup, operation, and design of the CDA Telematics can be found on the project [Confluence](https://usdot-carma.atlassian.net/wiki/spaces/WFD2/overview) pages. 

telematic_system/docker-compose.cloud.servers.yml

@@ -5,7 +5,7 @@ services:
     restart: always
     logging:
       options:
-        max-size: "10m"
+        max-size: "2g"
         max-file: "1"
     network_mode: host
 
@@ -17,8 +17,8 @@ services:
     restart: always
     logging:
       options:
-        max-size: "10m"
+        max-size: "2g"
         max-file: "1"
     network_mode: host
     environment:
-      - NATS_URI=nats://localhost:4222
+      - NATS_URI=nats://localhost:4222

telematic_system/docker-compose.units.yml

@@ -7,7 +7,7 @@ services:
     image: usdotfhwastoldev/carma_vehicle_nats_bridge:develop
     logging:
       options:
-        max-size: "10m"
+        max-size: "2g"
         max-file: "1"
     container_name: carma_vehicle_bridge
     network_mode: host
@@ -35,7 +35,7 @@ services:
     image: usdotfhwastoldev/carma_street_nats_bridge:develop
     logging:
       options:
-        max-size: "10m"
+        max-size: "2g"
         max-file: "1"
     container_name: carma_street_bridge
     network_mode: host
@@ -63,7 +63,7 @@ services:
     image: usdotfhwastoldev/carma_cloud_nats_bridge:develop
     logging:
       options:
-        max-size: "10m"
+        max-size: "2g"
         max-file: "1"
     container_name: carma_cloud_bridge
     network_mode: host

telematic_system/telematic.env

@@ -1,10 +1,10 @@
 INFLUXDB_DEV_TAG=latest #Required: The tag for influxDB image version. Current latest version is 2.4
-INFLUXDB_DEV_INIT_USERNAME=admin #Required: Create a credential username on container initial startup.
-INFLUXDB_DEV_INIT_PASSWORD=adminpwd #Required: Create a credential password on container initial startup.
+INFLUXDB_DEV_INIT_USERNAME=<USERNAME> #Required: Create a credential username on container initial startup.
+INFLUXDB_DEV_INIT_PASSWORD=<PASSWORD> #Required: Create a credential password on container initial startup.
 INFLUXDB_DEV_INIT_BUCKET=platform-dev #Required: Create an bucket on container initial startup. You can create more buckets inside the influxDB container.
 INFLUXDB_DEV_ORG=my-org #Required: Create an organization on container startup. You can create more orgs inside the same influxDB container.
 INFLUXDB_DEV_RETENTION=7d #Optional: If not set, default is 0s meaning the initial bucket will retain data forever.
-NATS_SERVER_IP_PORT=44.206.13.7:4222 #change to amazon ec2 ipv4 address
+NATS_SERVER_IP_PORT=<NATS-IP>:<PORT> #change to amazon ec2 ipv4 address:nats_port
 ##Cloud Bridge params
 CARMA_CLOUD_LOG=/opt/tomcat/logs/carmacloud.log #Required for carma cloud bridge, containing the directory where carma cloud log is located
 CARMA_CLOUD_BRIDGE_UNIT_ID=cloud_id

telematic_system/telematic_apps/web_app/client/src/api/api-topics.js

@@ -54,7 +54,7 @@ const requestSelectedLiveUnitsTopics = async (seletedUnitTopicListToConfirm) =>
         } catch (err) {
             sentStatus.push({
                 errCode: err.response !== undefined ? err.response.status : "404",
-                errMsg: err.response !== undefined ? unitStatus + err.response.statusText + err.response.data : unitStatus + "No reponse from server."
+                errMsg: err.response !== undefined ? unitStatus + err.response.statusText +". " + err.response.data : unitStatus + "No reponse from server."
             });
         }
         return sentStatus;

telematic_system/telematic_apps/web_app/client/src/api/user_topic_request.js

@@ -30,10 +30,11 @@ const upsertUserTopicRequestForEventUnits = async (seletedUnitsTopics, user_id)
 /**
  *@brief Load or find the user topic request for the given event and list of units
  * @Params event id used to uniquely identifer each event
+ * @Params current user id to exclude
  * @Params selectedUnitIdentifiers: A list of unit identifiers. Each unit identifier is a string and is used to uniquely identify each unit.
  * @Return Response status and load a bulk of topics for each event and list of units for the event
  */
-const findAllUserTopicRequestByEventUnits = async (event_id, selectedUnitIdentifiers) => {
+const findUsersTopicRequestByEventUnits = async (event_id, selectedUnitIdentifiers, exclude_user_id) => {
   const URL = `${process.env.REACT_APP_WEB_SERVER_URI}/api/user_topic_request/all`
   if (selectedUnitIdentifiers.length === 0 || event_id === 0 || event_id === undefined) {
     return { errCode: CanceledError.ERR_BAD_REQUEST, errMsg: "Event id or units cannot be empty" };
@@ -44,6 +45,7 @@ const findAllUserTopicRequestByEventUnits = async (event_id, selectedUnitIdentif
       withCredentials: true,
       params: {
         event_id: event_id,
+        exclude_user_id: exclude_user_id,
         unit_identifiers: selectedUnitIdentifiers,
       },
     },);
@@ -83,4 +85,4 @@ const findAllUserTopicRequestByEventUnits = async (event_id, selectedUnitIdentif
   }
 }
 
-export { upsertUserTopicRequestForEventUnits,findUserTopicRequestByUserEventUnits, findAllUserTopicRequestByEventUnits,  }
+export { upsertUserTopicRequestForEventUnits,findUserTopicRequestByUserEventUnits, findUsersTopicRequestByEventUnits,  }

telematic_system/telematic_apps/web_app/client/src/pages/TopicPage.js

@@ -20,7 +20,7 @@ import { findAllEvents } from '../api/api-events';
 import { findAllLocations } from '../api/api-locations';
 import { findAllTestingTypes } from '../api/api-testing-types';
 import { requestSelectedLiveUnitsTopics } from '../api/api-topics';
-import { findAllUserTopicRequestByEventUnits, upsertUserTopicRequestForEventUnits } from '../api/user_topic_request';
+import { findUsersTopicRequestByEventUnits, upsertUserTopicRequestForEventUnits } from '../api/user_topic_request';
 import { VALID_UNIT_TYPES } from '../components/events/EventMetadata';
 import InfrastructureTopicList from '../components/topics/InfrastructureTopicList';
 import { NOTIFICATION_STATUS } from '../components/topics/TopicMetadata';
@@ -117,12 +117,11 @@ const TopicPage = React.memo(() => {
       event_id = item.event_id;
     });
 
-    //Find all existing users's topic request before sending the current user's topic request
-    findAllUserTopicRequestByEventUnits(event_id, unit_identifiers).then(data => {
+    //Find all existing users's topic request except current user topic request before sending the current user's topic request
+    findUsersTopicRequestByEventUnits(event_id, unit_identifiers, authCtx.user_id).then(data => {
       let updatedSeletedUnitTopicListToConfirm = [];
       if (data !== undefined
         && Array.isArray(data)
-        && data.length > 0
         && seletedUnitTopicListToConfirm !== undefined
         && Array.isArray(seletedUnitTopicListToConfirm)) {
         seletedUnitTopicListToConfirm.forEach(item => {
@@ -145,7 +144,6 @@ const TopicPage = React.memo(() => {
               });
             });
           }
-
           //Loop through existing all users' topic request
           data.forEach(dataItem => {
             //Check the same unit for current user topics request and all users' topics request
@@ -187,9 +185,12 @@ const TopicPage = React.memo(() => {
             }
           });
         }
-
+        if(num_failed === 0 && num_success === 0)
+        {
+            messageList.push("Failed to send request. Please click the confirm selected topics button again.")
+        }
         //Notification
-        let severity = num_failed === 0 ? NOTIFICATION_STATUS.SUCCESS : (num_success === 0 ? NOTIFICATION_STATUS.ERROR : NOTIFICATION_STATUS.WARNING);
+        let severity = num_failed === 0 && num_success === 0 ? NOTIFICATION_STATUS.ERROR : (num_failed === 0 && num_success !== 0 ? NOTIFICATION_STATUS.SUCCESS: (num_failed !==0 && num_success===0? NOTIFICATION_STATUS.ERROR : NOTIFICATION_STATUS.WARNING));
         setAlertStatus({
           open: true,
           severity: severity,

telematic_system/telematic_apps/web_app/server/controllers/user_topic_request.controller.js

@@ -115,10 +115,12 @@ exports.findUserRequestByUserEventUnit = (req, res) => {
  exports.findAllUserRequestByEventUnit = (req, res) => {
     const event_id = req.query.event_id;
     const unit_identifiers = req.query.unit_identifiers;
-    var condition = [];
-    condition.push({ event_id: event_id });
-    condition.push({ unit_identifier: unit_identifiers });
-    user_topic_request.findAll({ where: condition })
+    const exclude_user_id = req.query.exclude_user_id;
+    user_topic_request.findAll({ where: {
+        event_id: event_id,
+        unit_identifier: unit_identifiers,
+        updated_by :{ [Op.ne]: exclude_user_id}
+    } })
         .then(data => {
             res.status(200).send(data);
         }).catch(err => {

telematic_system/telematic_cloud_messaging/src/main/java/com/telematic/telematic_cloud_messaging/message_converters/JSON2KeyValuePairsConverter.java

@@ -1,5 +1,6 @@
 package com.telematic.telematic_cloud_messaging.message_converters;
 
+import java.util.List;
 import org.json.simple.JSONObject;
 import org.json.simple.parser.JSONParser;
 import org.json.simple.parser.ParseException;
@@ -9,9 +10,10 @@
 public class JSON2KeyValuePairsConverter {
     /**
      * @param json_str String JSON format consumed from NATS subject
+     * @param to_str_values Array of String to match in the JSON and convert the match to String data type value
      * @return String of key value pairs separated by commas.
      */
-    public String convertJson2KeyValuePairs(String json_str) {
+    public String convertJson2KeyValuePairs(String json_str, List<String> to_str_values) {
         String pairs = "";
         JSONParser parser = new JSONParser();
         try {
@@ -26,22 +28,38 @@ public String convertJson2KeyValuePairs(String json_str) {
                     pairs += key + "=\"NA\"";
                 }
                 else {
-                    // Regex matching integers
-                    if (value.toString().matches("[-+]?\\d*")) {
-                        pairs += key + "=" + value;
-                    }
-                    // Regex matching decimals
-                    else if (value.toString().matches("[-+]?\\d*\\.?\\d+")) {
-                        pairs += key + "=" + value;
-                    }
-                    //Regex matching scientific notation. InfluxDB does not support scientific notation float syntax, temporarily set this kind of value = 0.0 
-                    else if (value.toString().matches("^[+-]?\\d+(?:\\.\\d*(?:[eE][+-]?\\d+)?)?$")) {
-                        pairs += key + "=" + 0.0;
-                    }
-                    // If none of the above Regex matches, considering it as string
-                    else {
-                        pairs += key + "=\"" + value.toString().replaceAll("\\s", "") + "\"";
+                    boolean is_skip = false;
+                    for (String value_item: to_str_values)
+                    {
+                        if (key.toString().strip().equals(value_item)){
+                            pairs += key + "=\"" + value.toString().replaceAll("\\s", "") + "\"";
+                            is_skip = true;
+                        }
                     }
+                    if(!is_skip)
+                    {
+                        // Regex matching integers
+                        if (value.toString().matches("[-+]?\\d*")) {
+                            pairs += key + "=" + value;
+                        }
+                        // Regex matching decimals
+                        else if (value.toString().matches("[-+]?\\d*\\.?\\d+")) {
+                            pairs += key + "=" + value;
+                        }
+                        //Regex matching scientific notation. InfluxDB does not support scientific notation float syntax, temporarily set this kind of value = 0.0 
+                        else if (value.toString().matches("^[+-]?\\d+(?:\\.\\d*(?:[eE][+-]?\\d+)?)?$")) {
+                            pairs += key + "=" + 0.0;
+                        }
+                        //Leave NaN value as float (Assuming influx treats NaN value as float type) and do not convert to string
+                        else if (value.toString().toLowerCase().strip().equals("nan")){
+                            // Drop fields with NaN values
+                            continue;
+                        }
+                        // If none of the above Regex matches, considering it as string
+                        else {
+                            pairs += key + "=\"" + value.toString().replaceAll("\\s", "") + "\"";
+                        }
+                    }                    
                 }              
 
                 if (json.keySet().size() != key_count) {

telematic_system/telematic_cloud_messaging/src/main/java/com/telematic/telematic_cloud_messaging/nats_influx_connection/Config.java

@@ -1,5 +1,7 @@
 package com.telematic.telematic_cloud_messaging.nats_influx_connection;
 
+import java.util.*;
+
 /**
  * The Config object instantiates a configuration object which stores information to create a connection to the telematic nats server 
  * and influxdb bucket.
@@ -61,6 +63,8 @@ public enum BucketType{
     String streets_unit_id_list;
     // List of cloud unit ids
     String cloud_unit_id_list;
+    //List of values in the stream that should only be set to string data type
+    List<String> to_str_values;
 
     public Config(){}
 
@@ -88,7 +92,8 @@ public String ToString(){
         "\nnats_topic_per_dispatcher: " + topics_per_dispatcher+
         "\nvehicle_unit_id_list: " + vehicle_unit_id_list +
         "\nstreets_unit_id_list: " + streets_unit_id_list +
-        "\ncloud_unit_id_list: " + cloud_unit_id_list);
+        "\ncloud_unit_id_list: " + cloud_unit_id_list + 
+        "\nto_str_values:" + to_str_values.toString());
 
         return config_str;
 

telematic_system/telematic_cloud_messaging/src/main/java/com/telematic/telematic_cloud_messaging/nats_influx_connection/InfluxDataWriter.java

@@ -1,27 +1,19 @@
 package com.telematic.telematic_cloud_messaging.nats_influx_connection;
 
-import java.io.*;
-import java.util.Properties;
+
 import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-import java.util.Arrays;
 import java.util.List;
 import java.util.ArrayList;
 
-import com.fasterxml.jackson.annotation.JsonValue;
-import com.google.gson.JsonObject;
 import com.influxdb.client.*;
 import com.influxdb.client.InfluxDBClientOptions;
-import com.influxdb.client.domain.Authorization;
 import com.influxdb.client.domain.WritePrecision;
 import com.telematic.telematic_cloud_messaging.message_converters.JSONFlattenerHelper;
 
 import okhttp3.OkHttpClient;
 
 import com.telematic.telematic_cloud_messaging.message_converters.JSON2KeyValuePairsConverter;
 
-import org.json.simple.JSONValue;
-import org.json.simple.parser.JSONParser;
 import org.json.*;  
 import org.slf4j.LoggerFactory;
 import org.slf4j.Logger;
@@ -202,7 +194,7 @@ public String influxStringConverter(String publishData) {
         JSONObject payloadJson = publishDataJson.getJSONObject("payload");
 
         String flattenedPayloadJson = jsonFlattener.flattenJsonStr(payloadJson.toString());
-        String keyValuePairs = keyValueConverter.convertJson2KeyValuePairs(flattenedPayloadJson);
+        String keyValuePairs = keyValueConverter.convertJson2KeyValuePairs(flattenedPayloadJson, config_.to_str_values);
 
         String unit_id = publishDataJson.getString("unit_id").replaceAll("\\s", "_");
         String unit_type = publishDataJson.getString("unit_type").replaceAll("\\s", "_");

telematic_system/telematic_cloud_messaging/src/main/java/com/telematic/telematic_cloud_messaging/nats_influx_connection/NatsConsumer.java

@@ -1,15 +1,9 @@
 package com.telematic.telematic_cloud_messaging.nats_influx_connection;
 
 import io.nats.client.*;
-import java.io.*;
-import java.util.Properties;
 import java.util.List;
 import java.util.ArrayList;
 import java.nio.charset.StandardCharsets;
-import org.springframework.boot.CommandLineRunner;
-import com.telematic.telematic_cloud_messaging.nats_influx_connection.InfluxDataWriter;
-import com.telematic.telematic_cloud_messaging.message_converters.JSONFlattenerHelper;
-import com.telematic.telematic_cloud_messaging.message_converters.JSON2KeyValuePairsConverter;
 import org.slf4j.LoggerFactory;
 import org.slf4j.Logger;
 import org.apache.commons.lang3.exception.ExceptionUtils;
@@ -102,7 +96,6 @@ public void nats_connect() {
      * the topic list variable
      */
     public void updateAvailableTopicList() {
-        String error_msg = "";
 
         for (String unit_id: unit_id_list) {
             try 

telematic_system/telematic_cloud_messaging/src/main/java/com/telematic/telematic_cloud_messaging/nats_influx_connection/NatsInfluxPush.java

@@ -5,6 +5,8 @@
 import org.springframework.context.annotation.Profile;
 
 import java.io.*;
+import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Properties;
 import com.telematic.telematic_cloud_messaging.nats_influx_connection.Config;
 import com.telematic.telematic_cloud_messaging.nats_influx_connection.InfluxDataWriter;
@@ -71,6 +73,7 @@ static Config getConfigValues() {
             config.vehicle_unit_id_list = prop.getProperty("VEHICLE_UNIT_ID_LIST");
             config.streets_unit_id_list = prop.getProperty("STREETS_UNIT_ID_LIST");
             config.cloud_unit_id_list = prop.getProperty("CLOUD_UNIT_ID_LIST");
+            config.to_str_values = Arrays.asList(prop.getProperty("TO_STR_VALUES").split(","));
 
             try{
                 config.influx_bucket_type = BucketType.valueOf(prop.getProperty("INFLUX_BUCKET_TYPE"));