Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: standardise logs #371

Merged
merged 1 commit into from
Oct 2, 2023
Merged

feat: standardise logs #371

merged 1 commit into from
Oct 2, 2023

Conversation

fenos
Copy link
Contributor

@fenos fenos commented Sep 29, 2023

What kind of change does this PR introduce?

Improvement

What is the current behavior?

Currently, logs are a bit all over the place with different shapes.

What is the new behavior?

This PR tries to standardize logs across the service and include more useful information such as:

  • owner
  • reqId
  • headers:
    • 'content-type',
    • 'if-none-match',
    • 'if-modified-since',
    • 'upload-metadata',
    • 'upload-length',
    • 'tus-resumable',

Lifecycle events are now also logged and you can find the lifecycle of a specific asset by filtering with:

m.objectPath:"{ref}/{bucket}/{object}

additionally, you can filter by the following properties:

m.type:"event" m.event:~"ObjectCreated" m.project:""

All events will also include a reqId field which can be correlated to the request that was issued from.

m.type:"request" m.reqId:"reqId"

@fenos fenos force-pushed the feat/standard-logs branch 2 times, most recently from 4e98ef5 to 9d8de08 Compare September 29, 2023 12:12
@@ -20,5 +20,8 @@ export default function () {
return createLogFlareWriteStream({
apiKey: logflareApiKey,
sourceToken: logflareSourceToken,
onPreparePayload: (payload: any) => {
return payload
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ziinc the project key is already inferred from the log-middleware https://github.com/supabase/storage-api/pull/371/files#diff-7657d30e7d33d62772ae092233fbe7844f1601fbda2d5ebe6252b220988928abR7

do you think is still worth calling defaultPreparePayload(payload, meta)?

If yes, would this make sense:

onPreparePayload: (payload: any, meta: any) => {
      const item = defaultPreparePayload(payload, meta)
      item.project = payload.project
      return item
    },

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea it is needed as the key needs to be a top level field for enabling clustering

@fenos fenos force-pushed the feat/standard-logs branch 2 times, most recently from 06aa132 to f2b24e0 Compare September 29, 2023 12:55
@coveralls
Copy link

Pull Request Test Coverage Report for Build 6351879663

  • 110 of 176 (62.5%) changed or added relevant lines in 18 files are covered.
  • 5 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-0.4%) to 84.337%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/http/routes/tus/index.ts 3 4 75.0%
src/queue/events/multipart-upload-completed.ts 1 2 50.0%
src/queue/events/object-admin-delete.ts 12 13 92.31%
src/storage/uploader.ts 4 5 80.0%
src/storage/object.ts 16 18 88.89%
src/storage/errors.ts 6 9 66.67%
src/http/routes/tus/handlers.ts 0 4 0.0%
src/queue/events/webhook.ts 2 15 13.33%
src/http/plugins/db.ts 1 41 2.44%
Files with Coverage Reduction New Missed Lines %
src/storage/errors.ts 1 80.0%
src/http/plugins/db.ts 4 64.96%
Totals Coverage Status
Change from base Build 6337917527: -0.4%
Covered Lines: 7390
Relevant Lines: 8692

💛 - Coveralls

@@ -20,5 +22,10 @@ export default function () {
return createLogFlareWriteStream({
apiKey: logflareApiKey,
sourceToken: logflareSourceToken,
onPreparePayload: (payload: any, meta: any) => {
const item = defaultPreparePayload(payload, meta)
item.project = payload.project
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@supabase/logging I want to avoid project as the top level key. Can we cluster by tenant instead?

@fenos fenos merged commit 27c3b38 into master Oct 2, 2023
1 check passed
@fenos fenos deleted the feat/standard-logs branch October 2, 2023 09:21
@github-actions
Copy link

github-actions bot commented Oct 2, 2023

🎉 This PR is included in version 0.42.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants