Retain IEEE 1609.2 Security Headers

[drewjj]

PR Details

Description

Includes updates to both the UDP endpoints and the log ingest endpoint to ensure ASN.1 encoded messages are recorded in Kafka with signed IEEE 1609.2 security headers on them (when present).

The log ingestion no longer deposits J2735 message types directly to the topic.Asn1DecoderInput topic, instead routes the messages depending upon their message type to the corresponding topic.OdeRawEncodedXJson topic.

The UDP endpoints now build out proper OdeData objects before depositing ASN.1 messages to their corresponding topic.OdeRawEncodedXJson topic. This is to ensure all data within the RawEncoded topics conform to the same message structure.

Everything downstream from the RawEncoded topics remain unaffected by these changes. (topic.Asn1DecoderInput and beyond)

Related Issue

N/A

Motivation and Context

• Support misbehavior detection and allow reporting of the original messages and their security headers
• Allow for easier troubleshooting and message replay for all endpoints

How Has This Been Tested?

• Unit tests
• Local jpo-ode Dockerized environment
• CDOT GCP K8s cluster with Confluent Cloud

Types of changes

• Defect fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that cause existing functionality to change)

Checklist:

• I have added any new packages to the sonar-scanner.properties file
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
  ODE Contributing Guide
• I have added tests to cover my changes.
• All new and existing tests passed.

[dmccoystephenson]

These are some pretty exciting changes! I've left a few questions/comments.

Also, I ran the unit tests in the dev container and a few of the tests for the File Upload Controller failed:

Note: I've yet to run the solution and send data through with these changes.

[drewjj]

These are some pretty exciting changes! I've left a few questions/comments.

Also, I ran the unit tests in the dev container and a few of the tests for the File Upload Controller failed:

Note: I've yet to run the solution and send data through with these changes.

I fixed these test failures. I originally removed several threads that weren't clear to me on their purpose. I think they are actually for the port 8080 site's log upload so it is good I add these back.

All your comments should now be addressed as well.

[dmccoystephenson]

Looking good & the unit tests pass now, just a few comments:

1. The ODE Data Flow Overview diagram appears to have an inaccuracy.
2. The changes to the data flow still need to be reflected in the corresponding data flow explanations in the relevant README file.

Note: I've yet to run the solution and send data through with these changes.

[dmccoystephenson]

Looks good!

[Michael7371]

Looks good. Locally tested and works for all Commsignia log files.

[payneBrandon]

Looks good, just the one question!

[payneBrandon]

lgtm!