Granian/Django (WSGI) - Suggestions to implement gracefull shutdown after N request processed

[ducdetronquito]

Hi everyone,

First thanks for your work on granian !
I replaced uWSGI with it yersterday on a small web service at work and it currently seems to work quiet well !

One of the feature of uWSGI that saves our asses in production is the ability to restart a worker after it has processed N HTTP requests, which is a dirty but effective way to deal with memory leaks in our code.

As I understand from this issue (#34), it seems that this feature requires quiet a lot of work and is therefore not prioritized which its understandable.

The last comment of @cirospaciari made me realize that I should be able to implement this feature from within Django instead of realying on Granian to do it, but I am not sure what is the proper way to do it.

I thought about creating a dedicated HTTP middleware at the top of the HTTP middleware stack which counts requests as they are processed, and after N requests it triggers granian to gracefully shutdown its worker, with a SIGTERM or SIGINT signal sent to the granian process.

By starting at least 2 instances of our web service running with 1 granian worker per service I would expect it to behave nicely and not end up having a disruption of service, but I admit it's just a guess.

What do you think about it ? Does someone has implemented something like this before ?

Thanks in advance and have a nice day ! :)

[gi0baro]

So, theoretically speaking if your app emit a sigterm the worker might catch it and exit, and given that you use the --respawn-failed-workers, the main process should respawn that worker. There are a few caveats here though as:

• this is not a graceful shutdown
• chances might be low, but you might incur in a condition in which all the workers exit at the same time if they processed the same amount of requests

I think using some sort of timeout-based restart is more easy here, as you can have some cron or loop task which does something like this (given you have the pname extra installed):
kill -SIGHUP $(ps aux | grep 'granian' | grep -v 'worker' | grep -v 'multiprocessing' | grep -v 'grep' | awk '{print $2}')

That might look a bit messy, but it gets the PID of the Granian master process and issue a graceful respawn of all the workers. If you run that at a constant interval (eg: 60 minutes) you should have something quite close to your original aim.

In future releases Granian might add this feature, but as you said the priority is quite low at the moment.

[ducdetronquito]

Hi @gi0baro !

Thanks for your answer, I will give it a try in the upcoming days !

I keep this discussion open if you don't mind to give updates for others depending on the result.