Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix producer sample so that it picks up usable defaults #38

Merged
merged 1 commit into from
Nov 18, 2023

Conversation

brandur
Copy link
Contributor

@brandur brandur commented Nov 18, 2023

The producer sample currently fails with this error:

time=2023-11-18T13:37:05.945-08:00 level=ERROR msg=failed error="error inserting jobs: error inserting many jobs: ERROR: new row for relation \"river_job\" violates check constraint \"max_attempts_is_positive\" (SQLSTATE 23514)"

Digging into it, it turns out that this is because (1) batch insert's
raw query uses COPY FROM so it's not allowed to have coalesce in it
like the single insert, (2) defaults are added for bulk inserts by
River's Go code, but at the client level and not the adapter level, and
(3) the producer sample manually creates an adapter to insert jobs with.

So it worked around the use of normal client inserts using an adapter
instead, wasn't adding its own MaxAttempts and no default was provided,
so it failed the on the positive check constraint.

I could fix this by putting a MaxAttempts into the insert parameters,
but I don't think this is the right way to go -- instead, the sample
should just bulk insert through the client. There might be some nominal
performance benefit to inserting through the adapter instead, but even
if there is, I don't think it'd be that honest to use it in a benchmark
because our users would be inserting through the client anyway. Using
the client also simplifies the code quite a bit.

The producer sample currently fails with this error:

> `time=2023-11-18T13:37:05.945-08:00 level=ERROR msg=failed error="error inserting jobs: error inserting many jobs: ERROR: new row for relation \"river_job\" violates check constraint \"max_attempts_is_positive\" (SQLSTATE 23514)"`

Digging into it, it turns out that this is because (1) batch insert's
raw query uses `COPY FROM` so it's not allowed to have `coalesce` in it
like the single insert, (2) defaults are added for bulk inserts by
River's Go code, but at the client level and not the adapter level, and
(3) the producer sample manually creates an adapter to insert jobs with.

So it worked around the use of normal client inserts using an adapter
instead, wasn't adding its own `MaxAttempts` and no default was provided,
so it failed the on the positive check constraint.

I could fix this by putting a `MaxAttempts` into the insert parameters,
but I don't think this is the right way to go -- instead, the sample
should just bulk insert through the client. There might be some nominal
performance benefit to inserting through the adapter instead, but even
if there is, I don't think it'd be that honest to use it in a benchmark
because our users would be inserting through the client anyway. Using
the client also simplifies the code quite a bit.
@brandur brandur requested a review from bgentry November 18, 2023 21:49
Copy link
Contributor

@bgentry bgentry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep this is fine, it's a vestige of earlier testing. We have a separate bench tool now specifically for working jobs which is the only place I could see us wanting to take a shortcut when inserting i.e. 1 million jobs at once.

@bgentry bgentry merged commit fd4b93e into master Nov 18, 2023
5 checks passed
@bgentry bgentry deleted the brandur-fix-producer-sample branch November 18, 2023 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants