Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PSM Data Feed Support, DockerHub Docs, Commsignia Log Offloading Support #522

Merged
merged 87 commits into from
Feb 5, 2024
Merged
Show file tree
Hide file tree
Changes from 84 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
b5f8d21
started updates for PSM data feed
Michael7371 Feb 17, 2023
dcd785e
updated enums to uppercase
Michael7371 Feb 17, 2023
95d1782
updates to psmBuilder for required parameters
Michael7371 Feb 18, 2023
413934d
unit test progress
Michael7371 Feb 22, 2023
2bd88df
updates for psmbuilder and associated POJOs
Michael7371 Feb 23, 2023
91aeaa4
working psm ingest and processing
Michael7371 Mar 13, 2023
f479c21
adding topic creation for psm
Michael7371 Mar 13, 2023
3e186b2
updates
Michael7371 Jul 31, 2023
e316993
updated reference to submodules
Michael7371 Aug 17, 2023
f6df692
Merge remote-tracking branch 'upstream/develop' into psm-data-feed
Michael7371 Aug 23, 2023
a4f695a
remove unnecessary changes
Michael7371 Sep 21, 2023
ac69c72
Merge remote-tracking branch 'upstream/develop' into psm-data-feed
Michael7371 Sep 21, 2023
4ffc684
added unit tests and removed unnecessary changes.
Michael7371 Sep 21, 2023
71c66e9
un-commented services
Michael7371 Sep 21, 2023
e05ea5f
updated udp_ip to use env variable
Michael7371 Sep 21, 2023
56d7811
removed unnecessary comment
Michael7371 Sep 21, 2023
36de40d
initial commit with working ingest on bsm type messages
Michael7371 Sep 21, 2023
d1ad1e4
added schema link for psm to doc readme
Michael7371 Sep 21, 2023
acf570b
added generic receiver classes
Michael7371 Sep 25, 2023
92cd295
updates for rx tim message processing
Michael7371 Sep 26, 2023
aeca341
updates for more generic receiver.
Michael7371 Sep 27, 2023
fe8866e
updates to dataflow
Michael7371 Sep 27, 2023
a28c5ec
addressing PR comments.
Michael7371 Sep 27, 2023
5f9af98
updated TMC environment figure
Michael7371 Sep 27, 2023
7749c03
updates for driverAlert logs and refactoring new line handling
Michael7371 Sep 29, 2023
7181f9f
updates to allow processing of old logs (without new line characters)
Michael7371 Nov 2, 2023
dc765c2
updates to processing
Michael7371 Nov 3, 2023
638ab84
removing extra changes
Michael7371 Nov 3, 2023
6b5df85
updated readme to include additional feature.
Michael7371 Nov 3, 2023
599790c
fix bug in processing wydot logs
Michael7371 Nov 3, 2023
bcd4c3e
Merge branch 'develop' into log-offloading-updates
Michael7371 Nov 3, 2023
26eb9da
updates with new firmware log files
Michael7371 Nov 6, 2023
31c2da4
Added image documentation to `docs/dockerhub.md`
dmccoystephenson Nov 10, 2023
21a55bc
add protocols, update snmp4j version
mwodahl Nov 13, 2023
e1e6e75
update fourdot1 and snmp1218 pdu creation
mwodahl Nov 13, 2023
dd6eff8
add engineID discovery, snmpsession test update
mwodahl Nov 13, 2023
3f3eb65
remove unused imports
mwodahl Nov 13, 2023
a861912
Deprecated some test TIM messages that did not get deposited successf…
dmccoystephenson Nov 20, 2023
466ed21
Added logging variables for PPM/ACM to docker-compose files & sample.env
dmccoystephenson Nov 21, 2023
d6882e1
addressing drew's comments
Michael7371 Nov 21, 2023
d424a67
Merge pull request #43 from Michael7371/psm-data-feed
payneBrandon Nov 27, 2023
57da3fd
updates from brandon's comments
Michael7371 Nov 27, 2023
17ac4aa
Modified 'Direct Dependencies' section & added the 'Configuration' he…
dmccoystephenson Nov 27, 2023
03a9c91
Merge pull request #46 from CDOT-CV/docs/dockerhub-description
payneBrandon Nov 27, 2023
05889e7
Merge branch 'dev' into deprecating-test-messages
payneBrandon Nov 27, 2023
9a6a5b5
Updated OdeTravelerInformationMessage and TravelerDataFrame to confor…
dmccoystephenson Dec 22, 2023
6ba271f
Updated test data files (JSON/XML) to conform with J2735 2020
dmccoystephenson Dec 22, 2023
7c21c26
Updated TIM schema & associated test
dmccoystephenson Dec 22, 2023
402ae59
Updated test data in 'docs' directory
dmccoystephenson Dec 22, 2023
87ca65d
Merge pull request #47 from CDOT-CV/update/snmp-1218
payneBrandon Dec 26, 2023
aa6fa65
Merge pull request #49 from CDOT-CV/docker-compose/adding-new-logging…
payneBrandon Dec 26, 2023
65dcdb4
Merge pull request #48 from CDOT-CV/deprecating-test-messages
payneBrandon Dec 26, 2023
760c581
Merge pull request #45 from CDOT-CV/log-offloading-updates
payneBrandon Dec 26, 2023
2947c50
Merge remote-tracking branch 'origin/dev' into j2735/conform-to-2020-…
dmccoystephenson Dec 26, 2023
f7589e1
pom.xml update
mwodahl Jan 3, 2024
729489b
Dockerfile + devcontainer file updates
mwodahl Jan 3, 2024
96a8893
Class deprecated method updates
mwodahl Jan 3, 2024
8d94ea8
Testing Updates
mwodahl Jan 3, 2024
2d562c4
Merge branch 'dev' into update/java-lts-version
mwodahl Jan 3, 2024
cf41ef6
Updated commit reference for 'ode-output-validator-library' git submo…
dmccoystephenson Jan 3, 2024
9548148
Merge branch 'update/java-lts-version' into j2735/conform-to-2020-sta…
dmccoystephenson Jan 3, 2024
4d5d670
Merge branch 'usdot-jpo-ode:develop' into dev
drewjj Jan 4, 2024
4a0cbe3
Remove redundant test that no longer functions
drewjj Jan 4, 2024
3aef947
Updated data flow overview diagram to reflect 'offloaded files' input…
dmccoystephenson Jan 5, 2024
fe0ed8d
Update sonar plugin version
mwodahl Jan 10, 2024
3eecfba
sonar-maven-plugin version update
mwodahl Jan 10, 2024
4047c1b
Updated message-specific data flow diagrams to reflect log offloading…
dmccoystephenson Jan 10, 2024
6de4f3d
Update github workflow java version
mwodahl Jan 11, 2024
f14a82d
Merge pull request #50 from CDOT-CV/update/java-lts-version
drewjj Jan 15, 2024
ea6d468
Merge remote-tracking branch 'origin/dev' into j2735/conform-to-2020-…
drewjj Jan 15, 2024
851204d
Merge pull request #51 from CDOT-CV/j2735/conform-to-2020-standard
drewjj Jan 16, 2024
83f07a4
Merge pull request #52 from CDOT-CV/docs/data-flow-offloaded-files
payneBrandon Jan 17, 2024
b3ad7ef
Removed DOCKER_SHARED_VOLUME_WINDOWS env var
dmccoystephenson Jan 25, 2024
6a57ddb
Merge pull request #53 from CDOT-CV/env/remove-docker-shared-volume-w…
payneBrandon Jan 26, 2024
4157157
Added release notes for v1.6.0
dmccoystephenson Jan 30, 2024
5f5dd9d
Removed hashtags from v1.6.0 release notes
dmccoystephenson Jan 30, 2024
2726205
Merge branch 'usdot-jpo-ode:develop' into dev
dmccoystephenson Jan 30, 2024
829a956
Merge remote-tracking branch 'origin/dev' into docs/release-notes-for…
dmccoystephenson Jan 30, 2024
65fcfa6
Changed version to 1.6.0-SNAPSHOT
dmccoystephenson Jan 30, 2024
deee10a
Merge pull request #54 from CDOT-CV/docs/release-notes-for-1.6.0
drewjj Jan 31, 2024
7540b6c
Merge pull request #55 from CDOT-CV/version/change-version-to-1.6.0-S…
dmccoystephenson Jan 31, 2024
7d659ce
Merge remote-tracking branch 'usdot/develop' into resolve-merge-confl…
dmccoystephenson Jan 31, 2024
353ba10
Updated logback-core version to 1.4.14
dmccoystephenson Jan 31, 2024
e673011
Merge pull request #56 from CDOT-CV/resolve-merge-conflicts-1-31-2024
drewjj Jan 31, 2024
239fafd
initial fixes from pr comments
Michael7371 Feb 2, 2024
ae1cbe6
pulled wsmp header length into a constant
Michael7371 Feb 2, 2024
86f3d48
Merge pull request #57 from CDOT-CV/usdot-merge-fixes
dmccoystephenson Feb 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Licensed under the MIT License. See https://go.microsoft.com/fwlink/?linkid=2090316 for license information.
#-------------------------------------------------------------------------------------------------------------

FROM openjdk:11-jdk
FROM eclipse-temurin:21-jdk

# This Dockerfile adds a non-root user with sudo access. Use the "remoteUser"
# property in devcontainer.json to use it. On Linux, the container user's GID/UIDs
Expand Down Expand Up @@ -45,7 +45,7 @@ RUN apt-get -y install snmp

#-------------------Install Kafka----------------------------------
RUN mkdir ~/Downloads
RUN curl "https://archive.apache.org/dist/kafka/2.1.1/kafka_2.11-2.1.1.tgz" -o ~/Downloads/kafka.tgz
RUN curl "https://archive.apache.org/dist/kafka/3.6.0/kafka_2.12-3.6.0.tgz" -o ~/Downloads/kafka.tgz
RUN mkdir ~/kafka \
&& cd ~/kafka \
&& tar -xvzf ~/Downloads/kafka.tgz --strip 1
Expand Down
43 changes: 23 additions & 20 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,30 +1,33 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.154.0/containers/java-8
{
"name": "Java 11 and C++",
"name": "Java 21 and C++",
"dockerFile": "Dockerfile",
"overrideCommand": false,
"shutdownAction": "stopContainer",
"settings": {
"terminal.integrated.shell.linux": "/bin/bash"
"customizations": {
"vscode": {
"settings": {
"terminal.integrated.shell.linux": "/bin/bash"
},
"extensions": [
"vscjava.vscode-java-pack",
"vscjava.vscode-java-debug",
"vscjava.vscode-maven",
"vscjava.vscode-java-dependency",
"vscjava.vscode-java-test",
"hbenl.vscode-test-explorer",
"ms-vscode.test-adapter-converter",
"esbenp.prettier-vscode",
"mhutchie.git-graph",
"tabnine.tabnine-vscode",
"redhat.java",
"redhat.vscode-commons",
"ms-vscode.cpptools",
"ms-vscode.cmake-tools"
]
}
},
// Add the IDs of extensions you want installed when the container is created.
"extensions": [
"vscjava.vscode-java-pack",
"vscjava.vscode-java-debug",
"vscjava.vscode-maven",
"vscjava.vscode-java-dependency",
"vscjava.vscode-java-test",
"hbenl.vscode-test-explorer",
"ms-vscode.test-adapter-converter",
"esbenp.prettier-vscode",
"mhutchie.git-graph",
"tabnine.tabnine-vscode",
"redhat.java",
"redhat.vscode-commons",
"ms-vscode.cpptools",
"ms-vscode.cmake-tools"
],
// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [8080, 9090, 46753, 46800, 5555, 6666, 8090, 2181, 9092],
// Use 'postCreateCommand' to run commands after the container is created.
Expand Down
28 changes: 14 additions & 14 deletions .devcontainer/post-create.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,17 @@ bin/kafka-server-start.sh -daemon config/server.properties
# wait 2 seconds for the server to start and be able to add partitions
sleep 2s
# add topics
bin/kafka-topics.sh --create --topic "topic.OdeBsmPojo" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeBsmJson" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.FilteredOdeBsmJson" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeTimJson" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeTimBroadcastJson" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.J2735TimBroadcastJson" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeDriverAlertJson" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.Asn1DecoderInput" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.Asn1DecoderOutput" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.Asn1EncoderInput" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.Asn1EncoderOutput" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.SDWDepositorInput" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeTIMCertExpirationTimeJson" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeRawEncodedMessageJson" --zookeeper localhost:2181 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeBsmPojo" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeBsmJson" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.FilteredOdeBsmJson" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeTimJson" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeTimBroadcastJson" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.J2735TimBroadcastJson" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeDriverAlertJson" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.Asn1DecoderInput" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.Asn1DecoderOutput" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.Asn1EncoderInput" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.Asn1EncoderOutput" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.SDWDepositorInput" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
bin/kafka-topics.sh --create --topic "topic.OdeTIMCertExpirationTimeJson" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
dan-du-car marked this conversation as resolved.
Show resolved Hide resolved
bin/kafka-topics.sh --create --topic "topic.OdeRawEncodedMessageJson" --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
- name: Set up JDK
uses: actions/setup-java@v3
with:
java-version: "11"
java-version: "21"
distribution: "temurin"
- name: Run Sonar
env:
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM maven:3.8.1-openjdk-11 as builder
FROM maven:3.8-eclipse-temurin-21-alpine as builder
MAINTAINER 583114@bah.com

WORKDIR /home
Expand All @@ -16,7 +16,7 @@ COPY ./jpo-ode-svcs/src ./jpo-ode-svcs/src

RUN mvn clean package -DskipTests

FROM eclipse-temurin:11-jre-alpine
FROM eclipse-temurin:21-jre-alpine

WORKDIR /home

Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Once the ODE is deployed and running locally, you may access the ODE's demonstra
- [WYDOT Log Records](data/wydotLogRecords.h)
3. Press `Upload` button to upload the file to ODE.

Upload records within the files must be embedding BSM and/or TIM messages wrapped in J2735 MessageFrame and ASN.1 UPER encoded, wrapped in IEEE 1609.2 envelope and ASN.1 COER encoded binary format. Please review the files in the [data](data) folder for samples of each supported type. By uploading a valid data file, you will be able to observe the decoded messages contained within the file appear in the web UI page while connected to the WebSocket interface.
Upload records within the files can be embedding BSM, MAP and/or TIM messages wrapped in J2735 MessageFrame and ASN.1 UPER encoded, wrapped in IEEE 1609.2 envelope and ASN.1 COER encoded binary format. Log processing of files that contain messages with WSMP headers within the ASN.1 UPER encoded messages is supported but the header will be removed before processing. Please review the files in the [data](data) folder for samples of each supported type. By uploading a valid data file, you will be able to observe the decoded messages contained within the file appear in the web UI page while connected to the WebSocket interface.

Another way data can be uploaded to the ODE is through copying the file to the location specified by the `ode.uploadLocationRoot/ode.uploadLocationObuLog`property. If not specified, Default locations would be `uploads/bsmlog`sub-directory off of the location where ODE is launched.

Expand All @@ -90,6 +90,7 @@ Supported message types:
- SPaT
- SRM
- SSM
- PSM

1. Navigate to the [UDP sender Python scripts](<./scripts/tests/>) in the project.
2. Ensure the environment variable "DOCKER_HOST_IP" has been set in the shell that will be running the script. This must be set to the same IP that the ODE deployments are using.
Expand Down
Loading
Loading