From e9613458c99147fff5613887a4fbe88e220d1ef0 Mon Sep 17 00:00:00 2001 From: Brandur Date: Sun, 12 Nov 2023 19:54:03 -0800 Subject: [PATCH] Demonstrate use of goreadme to generate README Here, we demonstrate the use of goreadme [1] to generate a README for the project. This is not mergable, but it's good to see what it looks like. Top problems: * goreadme hasn't been updated to allow for internal Godoc links, so links like `[Client]` that Godoc will point to a local reference, don't get a link target in the README. * Any line that beings with `-` is interpreted as a diff block, which unfortunately includes our bullet point lists. I actually prefer the use of `*` for lists, but something in VSCode (gofmt I think?) doesn't let me use `*` and always converts to `-`, so those not being handle is a non-starter. Command used to generate this: goreadme -badge-godoc -credit=false -skip-examples -skip-sub-packages -import-path=github.com/riverqueue/river > README.md [1] https://github.com/posener/goreadme --- README.md | 184 ++++++++++++++++++++++++++++++++++++++++---- doc.go | 6 +- docs/development.md | 28 +++++++ 3 files changed, 200 insertions(+), 18 deletions(-) create mode 100644 docs/development.md diff --git a/README.md b/README.md index 380d46a8..b89fb685 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,182 @@ -# River +# river -River is an experimental Postgres queue for Go. +[![GoDoc](https://img.shields.io/badge/pkg.go.dev-doc-blue)](http://pkg.go.dev/github.com/riverqueue/river) -## Development +Package river is a robust high-performance job processing system for Go. -### Run tests +Because it is built using Postgres, River enables you to use the same database +for both your application data and your job queue. This simplifies operations, +but perhaps more importantly it makes it possible to enqueue jobs +transactionally with other database changes. This avoids a whole class of +distributed systems issues like jobs that execute before the database +transaction that enqueued them has even committed, or jobs that attempt to +utilize database changes which were rolled back. It also makes it possible for +your job to make database changes atomically with the job being marked as +complete. -Raise test databases: +# Job args - go run ./internal/cmd/testdbman create +Jobs need to be able to serialize their state to JSON so that they can round +tripped from the database and back. Each job has an args struct with JSON tags +on its properties to allow for this: -Run tests: +```go +// SortArgs are arguments for SortWorker. +type SortArgs struct { + // Strings is a slice of strings to sort. + Strings []string `json:"strings"` +} - go test ./... +func (SortArgs) Kind() string { return "sort_job" } +``` -### Run lint +Args are created to enqueue a new job and are what a worker receives to work +one. Each one implements [JobArgs].Kind, which returns a unique string that's +used to recognize the job as it round trips from the database. -Run the linter and try to autofix: +# Job workers - golangci-lint run --fix +Each job kind also has a corresponding worker struct where its core work +function is defined: -### Generate sqlc +```go +// SortWorker is a job worker for sorting strings. +type SortWorker struct { + river.WorkerDefaults[SortArgs] +} -The project uses sqlc (`brew install sqlc`) to generate Go targets for Postgres -queries. After changing an sqlc `.sql` file, generate Go with: +func (w *SortWorker) Work(ctx context.Context, job *river.Job[SortArgs]) error { + sort.Strings(job.Args.Strings) + fmt.Printf("Sorted strings: %+v\n", job.Args.Strings) + return nil +} +``` - make generate \ No newline at end of file +A few details to notice: + +```go +- Although not strictly necessary, workers embed [WorkerDefaults] with a + reference to their args type. This allows them to inherit defaults for the + [Worker] interface, and helps futureproof in case its ever expanded. +``` + +- Each worker implements [Worker].Work, which is where the async heavy-lifting +for a background job is done. Work implementations receive a generic like +river.Job[SortArgs] for easy access to job arguments. + +# Registering workers + +As a program is initially starting up, worker structs are registered so that +River can know how to work them: + +```go +workers := river.NewWorkers() +river.AddWorker(workers, &SortWorker{}) +``` + +# River client + +The main River client takes a [pgx] connection pool wrapped with River's Pgx v5 +driver using [riverpgxv5.New] and a set of registered workers (see above). Each +queue can receive configuration like the maximum number of goroutines that'll be +used to work it: + +```go +dbConfig, err := pgxpool.ParseConfig("postgres://localhost/river") +if err != nil { + return err +} + +dbPool, err := pgxpool.NewWithConfig(ctx, dbConfig) +if err != nil { + return err +} +defer dbPool.Close() + +riverClient, err := river.NewClient(&river.Config{ + Driver: riverpgxv5.New(dbPool), + Queues: map[string]river.QueueConfig{ + river.DefaultQueue: {MaxWorkers: 100}, + }, + Workers: workers, +}) + +if err := riverClient.Start(ctx); err != nil { + ... +} + +... + +// Before program exit, try to shut down cleanly. +if err := riverClient.Shutdown(ctx); err != nil { + return err +} +``` + +For programs that'll be inserting jobs only, the Queues and Workers +configuration keys can be omitted for brevity: + +```go +riverClient, err := river.NewClient(&river.Config{ + DBPool: dbPool, +}) +``` + +However, if Workers is specified, the client can validate that an inserted job +has a worker that's registered with the workers bundle, so it's recommended that +Workers is configured anyway if your project is set up to easily allow it. + +See [Config] for details on all configuration options. + +# Inserting jobs + +Insert jobs by opening a transaction and calling [Client.InsertTx] with a job +args instance (a non-transactional [Client.Insert] is also available) and the +transaction wrapped with [riverpgxv5Tx]: + +```go +tx, err := dbPool.Begin(ctx) +if err != nil { + return err +} +defer tx.Rollback(ctx) + +_, err = riverClient.InsertTx(ctx, tx, SortArgs{ + Strings: []string{ + "whale", "tiger", "bear", + }, +}, nil) +if err != nil { + return err +} + +if err := tx.Commit(ctx); err != nil { + return err +} +``` + +Due to rules around transaction visibility, inserted jobs aren't visible to +workers until the transaction that inserted them is committed. This prevents +a whole host of problems like workers trying to work a job before its viable to +do so because not all its requisite data has been persisted yet. + +See the InsertAndWork example for all this code in one place. + +# Other features + +```go +- Periodic jobs that run on a predefined interval. See the PeriodicJob example + below. +``` + +# Verifying inserted jobs + +See the rivertest package for test helpers that can be used to easily verified +inserted jobs in a test suite. For example: + +```go +job := rivertest.RequireInserted(ctx, t, dbPool, &RequiredArgs{}, nil) +fmt.Printf("Test passed with message: %s\n", job.Args.Message) +``` + +[pgx]: [https://github.com/jackc/pgx](https://github.com/jackc/pgx) diff --git a/doc.go b/doc.go index dfcf6bc1..8bf893f7 100644 --- a/doc.go +++ b/doc.go @@ -51,9 +51,9 @@ A few details to notice: reference to their args type. This allows them to inherit defaults for the [Worker] interface, and helps futureproof in case its ever expanded. - - Each worker implements [Worker].Work, which is where the async heavy-lifting - for a background job is done. Work implementations receive a generic like - river.Job[SortArgs] for easy access to job arguments. +- Each worker implements [Worker].Work, which is where the async heavy-lifting +for a background job is done. Work implementations receive a generic like +river.Job[SortArgs] for easy access to job arguments. # Registering workers diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 00000000..380d46a8 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,28 @@ +# River + +River is an experimental Postgres queue for Go. + +## Development + +### Run tests + +Raise test databases: + + go run ./internal/cmd/testdbman create + +Run tests: + + go test ./... + +### Run lint + +Run the linter and try to autofix: + + golangci-lint run --fix + +### Generate sqlc + +The project uses sqlc (`brew install sqlc`) to generate Go targets for Postgres +queries. After changing an sqlc `.sql` file, generate Go with: + + make generate \ No newline at end of file