Skip to content

Commit

Permalink
Demonstrate use of goreadme to generate README
Browse files Browse the repository at this point in the history
Here, we demonstrate the use of goreadme [1] to generate a README for
the project. This is not mergable, but it's good to see what it looks
like.

Top problems:

* goreadme hasn't been updated to allow for internal Godoc links, so
  links like `[Client]` that Godoc will point to a local reference,
  don't get a link target in the README.

* Any line that beings with `-` is interpreted as a diff block, which
  unfortunately includes our bullet point lists. I actually prefer the
  use of `*` for lists, but something in VSCode (gofmt I think?) doesn't
  let me use `*` and always converts to `-`, so those not being handle
  is a non-starter.

Command used to generate this:

    goreadme -badge-godoc -credit=false -skip-examples -skip-sub-packages -import-path=github.com/riverqueue/river > README.md

[1] https://github.com/posener/goreadme
  • Loading branch information
brandur committed Nov 13, 2023
1 parent 285a510 commit e961345
Show file tree
Hide file tree
Showing 3 changed files with 200 additions and 18 deletions.
184 changes: 169 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,182 @@
# River
# river

River is an experimental Postgres queue for Go.
[![GoDoc](https://img.shields.io/badge/pkg.go.dev-doc-blue)](http://pkg.go.dev/github.com/riverqueue/river)

## Development
Package river is a robust high-performance job processing system for Go.

### Run tests
Because it is built using Postgres, River enables you to use the same database
for both your application data and your job queue. This simplifies operations,
but perhaps more importantly it makes it possible to enqueue jobs
transactionally with other database changes. This avoids a whole class of
distributed systems issues like jobs that execute before the database
transaction that enqueued them has even committed, or jobs that attempt to
utilize database changes which were rolled back. It also makes it possible for
your job to make database changes atomically with the job being marked as
complete.

Raise test databases:
# Job args

go run ./internal/cmd/testdbman create
Jobs need to be able to serialize their state to JSON so that they can round
tripped from the database and back. Each job has an args struct with JSON tags
on its properties to allow for this:

Run tests:
```go
// SortArgs are arguments for SortWorker.
type SortArgs struct {
// Strings is a slice of strings to sort.
Strings []string `json:"strings"`
}

go test ./...
func (SortArgs) Kind() string { return "sort_job" }
```

### Run lint
Args are created to enqueue a new job and are what a worker receives to work
one. Each one implements [JobArgs].Kind, which returns a unique string that's
used to recognize the job as it round trips from the database.

Run the linter and try to autofix:
# Job workers

golangci-lint run --fix
Each job kind also has a corresponding worker struct where its core work
function is defined:

### Generate sqlc
```go
// SortWorker is a job worker for sorting strings.
type SortWorker struct {
river.WorkerDefaults[SortArgs]
}

The project uses sqlc (`brew install sqlc`) to generate Go targets for Postgres
queries. After changing an sqlc `.sql` file, generate Go with:
func (w *SortWorker) Work(ctx context.Context, job *river.Job[SortArgs]) error {
sort.Strings(job.Args.Strings)
fmt.Printf("Sorted strings: %+v\n", job.Args.Strings)
return nil
}
```

make generate
A few details to notice:

```go
- Although not strictly necessary, workers embed [WorkerDefaults] with a
reference to their args type. This allows them to inherit defaults for the
[Worker] interface, and helps futureproof in case its ever expanded.
```

- Each worker implements [Worker].Work, which is where the async heavy-lifting
for a background job is done. Work implementations receive a generic like
river.Job[SortArgs] for easy access to job arguments.

# Registering workers

As a program is initially starting up, worker structs are registered so that
River can know how to work them:

```go
workers := river.NewWorkers()
river.AddWorker(workers, &SortWorker{})
```

# River client

The main River client takes a [pgx] connection pool wrapped with River's Pgx v5
driver using [riverpgxv5.New] and a set of registered workers (see above). Each
queue can receive configuration like the maximum number of goroutines that'll be
used to work it:

```go
dbConfig, err := pgxpool.ParseConfig("postgres://localhost/river")
if err != nil {
return err
}

dbPool, err := pgxpool.NewWithConfig(ctx, dbConfig)
if err != nil {
return err
}
defer dbPool.Close()

riverClient, err := river.NewClient(&river.Config{
Driver: riverpgxv5.New(dbPool),
Queues: map[string]river.QueueConfig{
river.DefaultQueue: {MaxWorkers: 100},
},
Workers: workers,
})

if err := riverClient.Start(ctx); err != nil {
...
}

...

// Before program exit, try to shut down cleanly.
if err := riverClient.Shutdown(ctx); err != nil {
return err
}
```

For programs that'll be inserting jobs only, the Queues and Workers
configuration keys can be omitted for brevity:

```go
riverClient, err := river.NewClient(&river.Config{
DBPool: dbPool,
})
```

However, if Workers is specified, the client can validate that an inserted job
has a worker that's registered with the workers bundle, so it's recommended that
Workers is configured anyway if your project is set up to easily allow it.

See [Config] for details on all configuration options.

# Inserting jobs

Insert jobs by opening a transaction and calling [Client.InsertTx] with a job
args instance (a non-transactional [Client.Insert] is also available) and the
transaction wrapped with [riverpgxv5Tx]:

```go
tx, err := dbPool.Begin(ctx)
if err != nil {
return err
}
defer tx.Rollback(ctx)

_, err = riverClient.InsertTx(ctx, tx, SortArgs{
Strings: []string{
"whale", "tiger", "bear",
},
}, nil)
if err != nil {
return err
}

if err := tx.Commit(ctx); err != nil {
return err
}
```

Due to rules around transaction visibility, inserted jobs aren't visible to
workers until the transaction that inserted them is committed. This prevents
a whole host of problems like workers trying to work a job before its viable to
do so because not all its requisite data has been persisted yet.

See the InsertAndWork example for all this code in one place.

# Other features

```go
- Periodic jobs that run on a predefined interval. See the PeriodicJob example
below.
```

# Verifying inserted jobs

See the rivertest package for test helpers that can be used to easily verified
inserted jobs in a test suite. For example:

```go
job := rivertest.RequireInserted(ctx, t, dbPool, &RequiredArgs{}, nil)
fmt.Printf("Test passed with message: %s\n", job.Args.Message)
```

[pgx]: [https://github.com/jackc/pgx](https://github.com/jackc/pgx)
6 changes: 3 additions & 3 deletions doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,9 @@ A few details to notice:
reference to their args type. This allows them to inherit defaults for the
[Worker] interface, and helps futureproof in case its ever expanded.
- Each worker implements [Worker].Work, which is where the async heavy-lifting
for a background job is done. Work implementations receive a generic like
river.Job[SortArgs] for easy access to job arguments.
- Each worker implements [Worker].Work, which is where the async heavy-lifting
for a background job is done. Work implementations receive a generic like
river.Job[SortArgs] for easy access to job arguments.
# Registering workers
Expand Down
28 changes: 28 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# River

River is an experimental Postgres queue for Go.

## Development

### Run tests

Raise test databases:

go run ./internal/cmd/testdbman create

Run tests:

go test ./...

### Run lint

Run the linter and try to autofix:

golangci-lint run --fix

### Generate sqlc

The project uses sqlc (`brew install sqlc`) to generate Go targets for Postgres
queries. After changing an sqlc `.sql` file, generate Go with:

make generate

0 comments on commit e961345

Please sign in to comment.