-
Notifications
You must be signed in to change notification settings - Fork 90
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Demonstrate internal job row type using struct conversion
I wanted to throw this one up as a possible alternative to a public facing `rivertype` module in #36, in which we demonstrate an alternate strategy using struct conversions. The problems it aims to solve are the same: * A way to access core job types from within subpackages, allowing more modularity (e.g. `Rescuer` could live with other maintenance services). * Long term, a way of giving drivers access to job primitives without having to reference the top-level `river`. The advantage of this approach is there's fewer user-facing changes, and the user never has to interact with an alternate package. It works by doing a struct conversion as necessary when moving from an internal package to one that bubbles up to `river`. Go allows pure struct conversions as long as their fields are the same, but there are some limitations: * Substructs like `AttemptError` can't be converted, even if identical. * Custom types like `JobState` can't be converted, even if identical. We work around these issues by: * Keeping errors as a `[][]byte` instead of it being unmarshaled by pgx. A helper called `ErrorsUnmarshaled()` is added to the top-level `JobRow` to give easy access to unmarshaled errors. * `State` is changed to a basic string akin to functions in `http`. This is technically a little less type-safe, but practically speaking would make very little difference because all these string-based types can be converted to and from each other with no checks anyway. Also, users aren't expected to access job state all that often. When they do, they can still use `job.State == river.JobStateAvailable` because the constants have been made available as strings. I think this approach does also work reasonably well, although I worry a little bit that there may be some sharp edge in the future that ends up limiting us in some unexpected way. Also, having to use helpers like `ErrorsUnmarshaled()` is definitely a little bit worse. That said, ~no user-facing API changes, and has some marginal performance benefits: * Converting structs between each other should be ~free. * Errors probably aren't used that much, so by not unmarshaling them most of the time, we save on a lot of unnecessary JSON parsing.
- Loading branch information
Showing
17 changed files
with
200 additions
and
143 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| package dbsqlc | ||
|
|
||
| import ( | ||
| "github.com/riverqueue/river/internal/rivertype" | ||
| ) | ||
|
|
||
| func JobRowFromInternal(internal *RiverJob) *rivertype.JobRow { | ||
| tags := internal.Tags | ||
| if tags == nil { | ||
| tags = []string{} | ||
| } | ||
| return &rivertype.JobRow{ | ||
| ID: internal.ID, | ||
| Attempt: max(int(internal.Attempt), 0), | ||
| AttemptedAt: internal.AttemptedAt, | ||
| AttemptedBy: internal.AttemptedBy, | ||
| CreatedAt: internal.CreatedAt, | ||
| EncodedArgs: internal.Args, | ||
| Errors: internal.Errors, | ||
| FinalizedAt: internal.FinalizedAt, | ||
| Kind: internal.Kind, | ||
| MaxAttempts: max(int(internal.MaxAttempts), 0), | ||
| Priority: max(int(internal.Priority), 0), | ||
| Queue: internal.Queue, | ||
| ScheduledAt: internal.ScheduledAt.UTC(), // TODO(brandur): Very weird this is the only place a UTC conversion happens. | ||
| State: string(internal.State), | ||
| Tags: tags, | ||
|
|
||
| // metadata: internal.Metadata, | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| package rivertype | ||
|
|
||
| import "time" | ||
|
|
||
| // JobRow contains the properties of a job that are persisted to the database. | ||
| // Use of `Job[T]` will generally be preferred in user-facing code like worker | ||
| // interfaces. | ||
| type JobRow struct { | ||
| // ID of the job. Generated as part of a Postgres sequence and generally | ||
| // ascending in nature, but there may be gaps in it as transactions roll | ||
| // back. | ||
| ID int64 | ||
|
|
||
| // Attempt is the attempt number of the job. Jobs are inserted at 0, the | ||
| // number is incremented to 1 the first time work its worked, and may | ||
| // increment further if it's either snoozed or errors. | ||
| Attempt int | ||
|
|
||
| // AttemptedAt is the time that the job was last worked. Starts out as `nil` | ||
| // on a new insert. | ||
| AttemptedAt *time.Time | ||
|
|
||
| // AttemptedBy is the set of worker IDs that have worked this job. A worker | ||
| // ID differs between different programs, but is shared by all executors | ||
| // within any given one. (i.e. Different Go processes have different IDs, | ||
| // but IDs are shared within any given process.) A process generates a new | ||
| // ULID (an ordered UUID) worker ID when it starts up. | ||
| AttemptedBy []string | ||
|
|
||
| // CreatedAt is when the job record was created. | ||
| CreatedAt time.Time | ||
|
|
||
| // EncodedArgs is the job's JobArgs encoded as JSON. | ||
| EncodedArgs []byte | ||
|
|
||
| // Errors is a set of errors that occurred when the job was worked, one for | ||
| // each attempt. Ordered from earliest error to the latest error. | ||
| Errors [][]byte | ||
|
|
||
| // FinalizedAt is the time at which the job was "finalized", meaning it was | ||
| // either completed successfully or errored for the last time such that | ||
| // it'll no longer be retried. | ||
| FinalizedAt *time.Time | ||
|
|
||
| // Kind uniquely identifies the type of job and instructs which worker | ||
| // should work it. It is set at insertion time via `Kind()` on the | ||
| // `JobArgs`. | ||
| Kind string | ||
|
|
||
| // MaxAttempts is the maximum number of attempts that the job will be tried | ||
| // before it errors for the last time and will no longer be worked. | ||
| // | ||
| // Extracted (in order of precedence) from job-specific InsertOpts | ||
| // on Insert, from the worker level InsertOpts from JobArgsWithInsertOpts, | ||
| // or from a client's default value. | ||
| MaxAttempts int | ||
|
|
||
| // Priority is the priority of the job, with 1 being the highest priority and | ||
| // 4 being the lowest. When fetching available jobs to work, the highest | ||
| // priority jobs will always be fetched before any lower priority jobs are | ||
| // fetched. Note that if your workers are swamped with more high-priority jobs | ||
| // then they can handle, lower priority jobs may not be fetched. | ||
| Priority int | ||
|
|
||
| // Queue is the name of the queue where the job will be worked. Queues can | ||
| // be configured independently and be used to isolate jobs. | ||
| // | ||
| // Extracted from either specific InsertOpts on Insert, or InsertOpts from | ||
| // JobArgsWithInsertOpts, or a client's default value. | ||
| Queue string | ||
|
|
||
| // ScheduledAt is when the job is scheduled to become available to be | ||
| // worked. Jobs default to running immediately, but may be scheduled | ||
| // for the future when they're inserted. They may also be scheduled for | ||
| // later because they were snoozed or because they errored and have | ||
| // additional retry attempts remaining. | ||
| ScheduledAt time.Time | ||
|
|
||
| // State is the state of job like `available` or `completed`. Jobs are | ||
| // `available` when they're first inserted. | ||
| State string | ||
|
|
||
| // Tags are an arbitrary list of keywords to add to the job. They have no | ||
| // functional behavior and are meant entirely as a user-specified construct | ||
| // to help group and categorize jobs. | ||
| Tags []string | ||
|
|
||
| // metadata is a field that'll eventually be used to store arbitrary data on | ||
| // a job for flexible use and use with plugins. It's currently unexported | ||
| // until we get a chance to more fully flesh out this feature. | ||
| // metadata []byte | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.