Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A few fixes/changes for the graceful shutdown example #4

Merged
merged 1 commit into from
Nov 10, 2023

Conversation

brandur
Copy link
Contributor

@brandur brandur commented Nov 10, 2023

Follows up #2 with a few fixes/changes. As I was writing documentation
for graceful shutdown I realized there were a few things that weren't
quite ideal:

  • The job should return an error in the event of cancellation so that it
    can be persisted as errored and be worked again.

  • The example now respects either SIGINT or SIGTERM. SIGTERM is
    what's used on Heroku, but SIGINT is the standard signal from
    Ctrl+C in a terminal, so by respecting both we can have a program
    that works well in either development or a common hosted environment.

  • Add a third phase in which the program initiates an unclean stop by
    not waiting on stop any longer. This is probably something that most
    programs should have because it's going to be reasonably easy to write
    workers that accidentally don't respect context cancellation and get
    stuck.

  • Add a 10 second timeout to each phase. This is for Heroku's benefit.
    It'll send one SIGTERM and wait 30 seconds before issuing SIGKILL.
    So the program here waits 10 seconds for a soft stop, another 10
    seconds for a hard stop, and then exits uncleanly on its own volition
    before getting SIGKILLed.

@brandur brandur requested a review from bgentry November 10, 2023 02:07
@brandur brandur force-pushed the brandur-graceful-shutdown-fixes branch from 1112413 to b8acf94 Compare November 10, 2023 02:08
@brandur brandur force-pushed the brandur-graceful-shutdown-fixes branch from b8acf94 to 4c01984 Compare November 10, 2023 02:34
Follows up #2 with a few fixes/changes. As I was writing documentation
for graceful shutdown I realized there were a few things that weren't
quite ideal:

* The job should return an error in the event of cancellation so that it
  can be persisted as errored and be worked again.

* The example now respects either `SIGINT` or `SIGTERM`. `SIGTERM` is
  what's used on Heroku, but `SIGINT` is the standard signal from
  `Ctrl+C` in a terminal, so by respecting both we can have a program
  that works well in either development or a common hosted environment.

* Add a third phase in which the program initiates an unclean stop by
  not waiting on stop any longer. This is probably something that most
  programs should have because it's going to be reasonably easy to write
  workers that accidentally don't respect context cancellation and get
  stuck.

* Add a 10 second timeout to each phase. This is for Heroku's benefit.
  It'll send one `SIGTERM` and wait 30 seconds before issuing `SIGKILL`.
  So the program here waits 10 seconds for a soft stop, another 10
  seconds for a hard stop, and then exits uncleanly on its own volition
  before getting `SIGKILL`ed.
@brandur brandur force-pushed the brandur-graceful-shutdown-fixes branch from 4c01984 to 67495cb Compare November 10, 2023 02:35
@brandur
Copy link
Contributor Author

brandur commented Nov 10, 2023

Thanks! I also updated to use ctx.Err() as you pointed out in the docs review.

@brandur brandur merged commit 250ce47 into master Nov 10, 2023
5 checks passed
@brandur brandur deleted the brandur-graceful-shutdown-fixes branch November 10, 2023 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants