-
Notifications
You must be signed in to change notification settings - Fork 93
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Demonstrate use of goreadme to generate README
Here, we demonstrate the use of goreadme [1] to generate a README for the project. This is not mergable, but it's good to see what it looks like. Top problems: * goreadme hasn't been updated to allow for internal Godoc links, so links like `[Client]` that Godoc will point to a local reference, don't get a link target in the README. * Any line that beings with `-` is interpreted as a diff block, which unfortunately includes our bullet point lists. I actually prefer the use of `*` for lists, but something in VSCode (gofmt I think?) doesn't let me use `*` and always converts to `-`, so those not being handle is a non-starter. Command used to generate this: goreadme -badge-godoc -credit=false -skip-examples -skip-sub-packages -import-path=github.com/riverqueue/river > README.md [1] https://github.com/posener/goreadme
- Loading branch information
Showing
3 changed files
with
200 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,28 +1,182 @@ | ||
# River | ||
# river | ||
|
||
River is an experimental Postgres queue for Go. | ||
[![GoDoc](https://img.shields.io/badge/pkg.go.dev-doc-blue)](http://pkg.go.dev/github.com/riverqueue/river) | ||
|
||
## Development | ||
Package river is a robust high-performance job processing system for Go. | ||
|
||
### Run tests | ||
Because it is built using Postgres, River enables you to use the same database | ||
for both your application data and your job queue. This simplifies operations, | ||
but perhaps more importantly it makes it possible to enqueue jobs | ||
transactionally with other database changes. This avoids a whole class of | ||
distributed systems issues like jobs that execute before the database | ||
transaction that enqueued them has even committed, or jobs that attempt to | ||
utilize database changes which were rolled back. It also makes it possible for | ||
your job to make database changes atomically with the job being marked as | ||
complete. | ||
|
||
Raise test databases: | ||
# Job args | ||
|
||
go run ./internal/cmd/testdbman create | ||
Jobs need to be able to serialize their state to JSON so that they can round | ||
tripped from the database and back. Each job has an args struct with JSON tags | ||
on its properties to allow for this: | ||
|
||
Run tests: | ||
```go | ||
// SortArgs are arguments for SortWorker. | ||
type SortArgs struct { | ||
// Strings is a slice of strings to sort. | ||
Strings []string `json:"strings"` | ||
} | ||
|
||
go test ./... | ||
func (SortArgs) Kind() string { return "sort_job" } | ||
``` | ||
|
||
### Run lint | ||
Args are created to enqueue a new job and are what a worker receives to work | ||
one. Each one implements [JobArgs].Kind, which returns a unique string that's | ||
used to recognize the job as it round trips from the database. | ||
|
||
Run the linter and try to autofix: | ||
# Job workers | ||
|
||
golangci-lint run --fix | ||
Each job kind also has a corresponding worker struct where its core work | ||
function is defined: | ||
|
||
### Generate sqlc | ||
```go | ||
// SortWorker is a job worker for sorting strings. | ||
type SortWorker struct { | ||
river.WorkerDefaults[SortArgs] | ||
} | ||
|
||
The project uses sqlc (`brew install sqlc`) to generate Go targets for Postgres | ||
queries. After changing an sqlc `.sql` file, generate Go with: | ||
func (w *SortWorker) Work(ctx context.Context, job *river.Job[SortArgs]) error { | ||
sort.Strings(job.Args.Strings) | ||
fmt.Printf("Sorted strings: %+v\n", job.Args.Strings) | ||
return nil | ||
} | ||
``` | ||
|
||
make generate | ||
A few details to notice: | ||
|
||
```go | ||
- Although not strictly necessary, workers embed [WorkerDefaults] with a | ||
reference to their args type. This allows them to inherit defaults for the | ||
[Worker] interface, and helps futureproof in case its ever expanded. | ||
``` | ||
|
||
- Each worker implements [Worker].Work, which is where the async heavy-lifting | ||
for a background job is done. Work implementations receive a generic like | ||
river.Job[SortArgs] for easy access to job arguments. | ||
|
||
# Registering workers | ||
|
||
As a program is initially starting up, worker structs are registered so that | ||
River can know how to work them: | ||
|
||
```go | ||
workers := river.NewWorkers() | ||
river.AddWorker(workers, &SortWorker{}) | ||
``` | ||
|
||
# River client | ||
|
||
The main River client takes a [pgx] connection pool wrapped with River's Pgx v5 | ||
driver using [riverpgxv5.New] and a set of registered workers (see above). Each | ||
queue can receive configuration like the maximum number of goroutines that'll be | ||
used to work it: | ||
|
||
```go | ||
dbConfig, err := pgxpool.ParseConfig("postgres://localhost/river") | ||
if err != nil { | ||
return err | ||
} | ||
|
||
dbPool, err := pgxpool.NewWithConfig(ctx, dbConfig) | ||
if err != nil { | ||
return err | ||
} | ||
defer dbPool.Close() | ||
|
||
riverClient, err := river.NewClient(&river.Config{ | ||
Driver: riverpgxv5.New(dbPool), | ||
Queues: map[string]river.QueueConfig{ | ||
river.DefaultQueue: {MaxWorkers: 100}, | ||
}, | ||
Workers: workers, | ||
}) | ||
|
||
if err := riverClient.Start(ctx); err != nil { | ||
... | ||
} | ||
|
||
... | ||
|
||
// Before program exit, try to shut down cleanly. | ||
if err := riverClient.Shutdown(ctx); err != nil { | ||
return err | ||
} | ||
``` | ||
|
||
For programs that'll be inserting jobs only, the Queues and Workers | ||
configuration keys can be omitted for brevity: | ||
|
||
```go | ||
riverClient, err := river.NewClient(&river.Config{ | ||
DBPool: dbPool, | ||
}) | ||
``` | ||
|
||
However, if Workers is specified, the client can validate that an inserted job | ||
has a worker that's registered with the workers bundle, so it's recommended that | ||
Workers is configured anyway if your project is set up to easily allow it. | ||
|
||
See [Config] for details on all configuration options. | ||
|
||
# Inserting jobs | ||
|
||
Insert jobs by opening a transaction and calling [Client.InsertTx] with a job | ||
args instance (a non-transactional [Client.Insert] is also available) and the | ||
transaction wrapped with [riverpgxv5Tx]: | ||
|
||
```go | ||
tx, err := dbPool.Begin(ctx) | ||
if err != nil { | ||
return err | ||
} | ||
defer tx.Rollback(ctx) | ||
|
||
_, err = riverClient.InsertTx(ctx, tx, SortArgs{ | ||
Strings: []string{ | ||
"whale", "tiger", "bear", | ||
}, | ||
}, nil) | ||
if err != nil { | ||
return err | ||
} | ||
|
||
if err := tx.Commit(ctx); err != nil { | ||
return err | ||
} | ||
``` | ||
|
||
Due to rules around transaction visibility, inserted jobs aren't visible to | ||
workers until the transaction that inserted them is committed. This prevents | ||
a whole host of problems like workers trying to work a job before its viable to | ||
do so because not all its requisite data has been persisted yet. | ||
|
||
See the InsertAndWork example for all this code in one place. | ||
|
||
# Other features | ||
|
||
```go | ||
- Periodic jobs that run on a predefined interval. See the PeriodicJob example | ||
below. | ||
``` | ||
|
||
# Verifying inserted jobs | ||
|
||
See the rivertest package for test helpers that can be used to easily verified | ||
inserted jobs in a test suite. For example: | ||
|
||
```go | ||
job := rivertest.RequireInserted(ctx, t, dbPool, &RequiredArgs{}, nil) | ||
fmt.Printf("Test passed with message: %s\n", job.Args.Message) | ||
``` | ||
|
||
[pgx]: [https://github.com/jackc/pgx](https://github.com/jackc/pgx) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
# River | ||
|
||
River is an experimental Postgres queue for Go. | ||
|
||
## Development | ||
|
||
### Run tests | ||
|
||
Raise test databases: | ||
|
||
go run ./internal/cmd/testdbman create | ||
|
||
Run tests: | ||
|
||
go test ./... | ||
|
||
### Run lint | ||
|
||
Run the linter and try to autofix: | ||
|
||
golangci-lint run --fix | ||
|
||
### Generate sqlc | ||
|
||
The project uses sqlc (`brew install sqlc`) to generate Go targets for Postgres | ||
queries. After changing an sqlc `.sql` file, generate Go with: | ||
|
||
make generate |