feat: Loki Log Streaming #1385

amanji · 2024-10-09T15:10:46Z

No description provided.

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

loneil · 2024-10-10T23:59:29Z

Rebased and resolved package conflicts

loneil · 2024-10-15T17:59:40Z

@amanji there's a pull request here that refactors some startup to a manage script. Any thoughts on if we merge that if that would cause any issues (or should have change to include) the docker compose startup instructions in this PR?

https://github.com/bcgov/traction/pull/1388/files

amanji · 2024-10-15T18:22:06Z

Let's give it a try. I doubt there will be a huge impact

amanji · 2024-10-15T20:09:38Z

@loneil we'd need to add this to the manage script:

traction/scripts/README.md

Line 132 in 86fb430

docker compose -f docker-compose.logs.yml -f docker-compose.yml up

github-actions · 2024-10-16T14:40:09Z

Deployment	Resource	Location
Traction
	Tenant UI	https://pr-1385-traction-tenant-ui-dev.apps.silver.devops.gov.bc.ca
	Innkeeper UI	https://pr-1385-traction-tenant-ui-dev.apps.silver.devops.gov.bc.ca/innkeeper
Aca-Py
	Admin Swagger	https://pr-1385-traction-acapy-admin-dev.apps.silver.devops.gov.bc.ca/api/doc
	Tenant Proxy Swagger	https://pr-1385-traction-tenant-proxy-dev.apps.silver.devops.gov.bc.ca/api/doc

Deployment URLs ready for review.

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

…log-streaming

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

…log-streaming

esune · 2024-10-21T23:31:11Z

scripts/docker-compose.yml

+      - SERVER_LOKI_URL=${SERVER_LOKI_URL}
      - FRONTEND_TENANT_PROXY_URL=${FRONTEND_TENANT_PROXY_URL}
+      - FRONTEND_LOG_STREAM_URL=${FRONTEND_LOG_STREAM_URL}


These envvars need to be added to the chart as well

See chart if its correct.

@esune can you confirm if the included chart changes look correct for these variables?

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

amanji · 2024-10-22T14:26:49Z

FYI trying on the deployed PR and hitting start on the logs doesn't seem to show output or an error
See an aggregateError in the tenant UI backend logs:

We need to build in three more services for the logging. I'm assuming the deployed services dont account for those yet

@i5okie do you have any suggestiono n how to set things up for the supporting services? i.e.: we list them as pre-requisites or do we include (optional) depenedncies in the Helm chart?

In my opinion, I think Traction should work without throwing exceptions or failing when it runs without Loki backend being configured. As such, I think Loki should be an optional component setup separately by the user. I'd imagine, if loki endpoint is not provided/log-streaming is disabled, the feature should be turned off in someway. Or at least say logs unavailable in a friendly (not app crashing) manner on the page.

As for the promtail, it should be included as an optional sidecar. We could put together documentation, with example configs on how to set Traction up with Loki. I can help with the latter two.

I'll update the PR to hide the log feature when the FRONTEND_LOG_STREAM_URL is not set.

github-actions · 2024-12-07T02:08:26Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

esune · 2024-12-12T19:26:47Z

I'll update the PR to hide the log feature when the FRONTEND_LOG_STREAM_URL is not set.

@amanji it does not look like changes have been pushed since this - can you confirm? Would you be able to push the above tweak so we can move forward with reviewing/merging the pr?

Signed-off-by: Akiff Manji <amanji@petridish.dev>

amanji · 2025-01-10T16:11:02Z

@esune @i5okie @loneil Please take a look at this again. The frontend has been updated to filter out the Log component/routes from display if the logStreamUrl is not configured.

Signed-off-by: Akiff Manji <amanji@petridish.dev>

Signed-off-by: Lucas ONeil <lucasoneil@gmail.com>

loneil · 2025-01-10T18:40:30Z

Getting a fail on one of the loki fields during Helm deployment

Error: template: traction/templates/ui/deployment.yaml:9:24: executing "traction/templates/ui/deployment.yaml" at <include (print $.Template.BasePath "/ui/configmap.yaml") .>: error calling include: template: traction/templates/ui/configmap.yaml:33:29: executing "traction/templates/ui/configmap.yaml" at <include "loki.fullname" .>: error calling include: template: no template "loki.fullname" associated with template "gotpl"

esune

A couple minor tweaks, looks good otherwise.

esune · 2025-01-10T19:02:15Z

charts/traction/templates/ui/configmap.yaml

@@ -29,6 +30,7 @@ data:
  SERVER_SMTP_SECURE: {{ .Values.ui.smtp.secure | quote }}
  SERVER_SMTP_USER: {{ .Values.ui.smtp.user | quote }}
  SERVER_TRACTION_URL: http://{{ include "tenant_proxy.fullname" . }}:{{ .Values.tenant_proxy.service.port }}
+  SERVER_LOKI_URL: http://{{ include "loki.fullname" . }}:{{ .Values.loki.service.port }}


loki.fullname will not be available unless Loki is deployed at the same time as Traction.
I think it might be best to just use a full URL from the values file in case Loki is not being deployed with the solution. Otherwise you'll have to add Loki as a dependent chart to Traction and handle on/off scenarios.

esune · 2025-01-10T19:03:56Z

charts/traction/values.yaml

+## @section Loki configuration
+##
+loki:
+  ## Loki service configuration
+  ##
+  service:
+    ## @param loki.service.type Kubernetes Service type
+    ##
+    type: ClusterIP
+    ## @param loki.service.port Port to expose for http services
+    port: 3100
+


If we go with a "hard-coded" Loki url this should be able to go.

loneil · 2025-01-10T19:45:29Z

Getting a local docker error as well for me on fresh setuo
Error response from daemon: error looking up logging plugin loki: plugin "loki" not found

Built fresh (docker compose build) and then any of
docker compose -f docker-compose.logs.yml -f docker-compose.yml up
docker compose up
./manage start

If I install the loki plugin
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
it starts up

I guess we could add the plugin install to the manage script?
That makes starting up need the manage script, and installs a plugin on people's systems even if they're not using the logging feature.

Want to support people starting up without needing logging setup so I'm not sure if there's a way to make that plugin more optional?

Signed-off-by: Lucas ONeil <lucasoneil@gmail.com>

amanji requested review from loneil and esune October 9, 2024 15:21

amanji mentioned this pull request Oct 9, 2024

Log consumption for tenants #1043

Open

amanji linked an issue Oct 9, 2024 that may be closed by this pull request

Log consumption for tenants #1043

Open

amanji added 10 commits October 10, 2024 16:54

feat: docker compose config for promtail, grafana and loki services

59cb815

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

feat: websocket package

61ee803

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

feat: API websocket support

803955d

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

update: vscode settings

d43478b

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

feat: add jwonwebtoken package

bfddbb0

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

feat: API websocket token authentication

504056f

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

feat: log store and component

bbe6591

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

chore: update test snapshots

da4c622

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

chore: suppress unrelated ts build error

19b03fc

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

chore: parameterize URLs

592789b

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

loneil force-pushed the bcgov/feature/loki-log-streaming branch from 227c6a2 to 592789b Compare October 10, 2024 23:59

Merge branch 'main' into bcgov/feature/loki-log-streaming

86fb430

amanji marked this pull request as ready for review October 11, 2024 21:33

amanji temporarily deployed to development October 16, 2024 14:37 — with GitHub Actions Inactive

amanji force-pushed the bcgov/feature/loki-log-streaming branch from bea4259 to c8e0a77 Compare October 16, 2024 14:40

chore: suppress unrelated ts build error

48818a4

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

amanji force-pushed the bcgov/feature/loki-log-streaming branch from c8e0a77 to 48818a4 Compare October 16, 2024 14:47

amanji temporarily deployed to development October 16, 2024 14:50 — with GitHub Actions Inactive

amanji added 2 commits October 16, 2024 07:55

Merge remote-tracking branch 'upstream/main' into bcgov/feature/loki-…

8ba3e44

…log-streaming

chore: audit and update dependencies

2a94893

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

amanji temporarily deployed to development October 16, 2024 15:00 — with GitHub Actions Inactive

amanji added 2 commits October 21, 2024 13:38

chore: update env variables

f3729c7

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

Merge remote-tracking branch 'upstream/main' into bcgov/feature/loki-…

200cdd5

…log-streaming

amanji temporarily deployed to development October 21, 2024 20:46 — with GitHub Actions Inactive

esune reviewed Oct 21, 2024

View reviewed changes

update: chart values

57edbcf

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

amanji had a problem deploying to development October 22, 2024 14:12 — with GitHub Actions Failure

update: chart values

155a340

Signed-off-by: Akiff Manji <akiff.manji@quartech.com>

amanji requested a review from esune October 22, 2024 14:25

amanji had a problem deploying to development October 22, 2024 14:27 — with GitHub Actions Failure

github-actions bot added the Stale label Dec 7, 2024

esune removed the Stale label Dec 12, 2024

Merge branch 'bcgov/main' into bcgov/feature/loki-log-streaming

2b6b472

amanji had a problem deploying to development January 9, 2025 21:56 — with GitHub Actions Failure

fix: filter out log if logStreamUrl is not configured

2e3909f

Signed-off-by: Akiff Manji <amanji@petridish.dev>

amanji and others added 2 commits January 10, 2025 08:13

fix: test snapshots

96ab1cc

Signed-off-by: Akiff Manji <amanji@petridish.dev>

Fix TS no-unsafe-optional-chaining errors

ffbb082

Signed-off-by: Lucas ONeil <lucasoneil@gmail.com>

loneil had a problem deploying to development January 10, 2025 18:30 — with GitHub Actions Failure

esune requested changes Jan 10, 2025

View reviewed changes

esune assigned esune and loneil Jan 13, 2025

Optionalizing logging setup

4f31372

Signed-off-by: Lucas ONeil <lucasoneil@gmail.com>

loneil temporarily deployed to development January 16, 2025 22:57 — with GitHub Actions Inactive

blank default

aa263fd

Signed-off-by: Lucas ONeil <lucasoneil@gmail.com>

loneil temporarily deployed to development January 16, 2025 23:06 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Loki Log Streaming #1385

feat: Loki Log Streaming #1385

amanji commented Oct 9, 2024

loneil commented Oct 10, 2024

loneil commented Oct 15, 2024

amanji commented Oct 15, 2024

amanji commented Oct 15, 2024

github-actions bot commented Oct 16, 2024

esune Oct 21, 2024

amanji Oct 22, 2024 •

edited

Loading

amanji Jan 9, 2025

amanji commented Oct 22, 2024

github-actions bot commented Dec 7, 2024

esune commented Dec 12, 2024

amanji commented Jan 10, 2025

loneil commented Jan 10, 2025

esune left a comment

esune Jan 10, 2025

esune Jan 10, 2025

loneil commented Jan 10, 2025

feat: Loki Log Streaming #1385

Are you sure you want to change the base?

feat: Loki Log Streaming #1385

Conversation

amanji commented Oct 9, 2024

loneil commented Oct 10, 2024

loneil commented Oct 15, 2024

amanji commented Oct 15, 2024

amanji commented Oct 15, 2024

github-actions bot commented Oct 16, 2024

esune Oct 21, 2024

Choose a reason for hiding this comment

amanji Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

amanji Jan 9, 2025

Choose a reason for hiding this comment

amanji commented Oct 22, 2024

github-actions bot commented Dec 7, 2024

esune commented Dec 12, 2024

amanji commented Jan 10, 2025

loneil commented Jan 10, 2025

esune left a comment

Choose a reason for hiding this comment

esune Jan 10, 2025

Choose a reason for hiding this comment

esune Jan 10, 2025

Choose a reason for hiding this comment

loneil commented Jan 10, 2025

amanji Oct 22, 2024 •

edited

Loading