fix(cove): Reduce workers to a more sensible level (can/should probab…

…ly be lower), to avoid OOM if e.g. a bot sends tons of 404s

threads=2
- Threads was set to 1 in cd16ce7, so that processes "can be killed without affecting other requests"
- It seems like common advice is to set threads to 2 (or to the number of CPU?), to avoid CPU affinity issues
  https://www.cloudbees.com/blog/getting-every-microsecond-out-of-uwsgi#multithreading
- Note that a cpu-affinity option is also available
  https://uwsgi-docs-additions.readthedocs.io/en/latest/Options.html#cpu-affinity

workers=16
- The original maximum workers of 100 is from the original server transfer cd16ce7
- Today, uWSGI processes use between 2.1% (84MB) and 0.9% (38MB) memory (presumably those that handled requests, and those that didn't).
- So, workers=16 with threads=2 is not expected to exhaust the 4GB memory, though the default reload-on-rss of 256MB means up to 8GB can be requested.

cheaper=4 and cheaper-initial=8
- These were increased to 50 in 88369e1, I think because low values led to poor performance during traffic spikes

cheaper-rss-limit-soft-ratio is set to 0.9 to match the Prometheus monitoring rules.

docs/develop/update/python.rst

@@ -147,15 +147,42 @@ max-requests
   Number of requests before a worker is reloaded. This can help address memory leaks. Default: ``1000``.
 max-worker-lifetime
   Number of seconds before a worker is reloaded. This can help address memory leaks. Default: ``3600`` (1 hour).
+reload-on-rss
+  Reload a worker (after processing a request) if using too much memory, in MB. Default: ``256``.
 worker-reload-mercy
   The maximum time for a worker to reload/shutdown. Default: ``60`` (1 minute).
 limit-as
   Limit uWSGI memory usage, in MB. Default: 3/4 of RAM. This assumes no other process uses significant memory.
-reload-on-rss
-  Reload a worker (after processing a request) if using too much memory, in MB. Default: ``256``.
+
+The optional values are:
+
+workers
+  Set it to the maximum number of workers that the server's CPU and RAM can support.
+cheaper
+  Set it to the minimum number of works to keep at all times.
+cheaper-initial
+  Set it to the number of workers to handle an average load.
+cheaper-rss-limit-soft-ratio
+  Set to the ratio of total RAM to use as the `cheaper-rss-limit-soft <https://uwsgi-docs.readthedocs.io/en/latest/Cheaper.html#setting-memory-limits>`__.
+cheaper-step
+  Not in use.
+cheaper-overload
+  Not in use.
+cheaper-busyness-multiplier
+  Not in use.
+threads
+  The number of threads per worker.
+smart-attach-daemon
+  Not in use.
+stats
+  Enables the `uWSGI Stats Server <https://uwsgi-docs.readthedocs.io/en/latest/StatsServer.html>`__. Set it to a Unix socket path, like ``/home/USER/stats.sock``. Monitor with `uswgitop <https://pypi.org/project/uwsgitop/>`__, like ``uwsgitop /home/USER/stats.sock``.
 
 Alternatively, you can write your own configuration file in ``salt/uwsgi/files``, and reference it from the ``configuration`` variable.
 
+.. seealso::
+
+   `Configuration Options <https://uwsgi-docs-additions.readthedocs.io/en/latest/Options.html>`__ from uWSGI Docs Additions
+
 Configure Apache
 ----------------
 

pillar/cove.sls

@@ -22,8 +22,9 @@ python_apps:
     uwsgi:
       configuration: django
       harakiri: 1800 # 30 min
-      cheaper: 50
-      cheaper-initial: 50
-      workers: 100
-      threads: 1
+      workers: 16
+      cheaper: 4
+      cheaper-initial: 8
+      cheaper-rss-limit-soft-ratio: 0.9
+      threads: 2
       stats: /home/cove/stats.sock

salt/uwsgi/files/django.ini

@@ -4,7 +4,7 @@
 ; https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html
 ; https://docs.djangoproject.com/en/3.2/howto/deployment/wsgi/uwsgi/
 
-{# Only use on Ubuntu 22.04 systems and newer. #}
+{# Only use on Ubuntu 22.04 systems and newer. -#}
 {%- if grains.osmajorrelease >= 22 %}
 plugin = python3
 strict = true
@@ -45,6 +45,9 @@ cheaper = {{ entry.uwsgi.cheaper }}
 ; Number of workers to start with.
 cheaper-initial = {{ entry.uwsgi['cheaper-initial'] }}
 {%- endif %}
+{%- if 'cheaper-rss-limit-soft-ratio' in entry.uwsgi %}
+cheaper-rss-limit-soft = {{ entry.uwsgi['cheaper-rss-limit-soft-ratio'] * grains.mem_total }}
+{%- endif %}
 {%- if 'cheaper-step' in entry.uwsgi %}
 ; Number of workers to spawn at a time. Default 1.
 cheaper-step = {{ entry.uwsgi['cheaper-step'] }}
@@ -67,12 +70,11 @@ smart-attach-daemon = /tmp/celery.pid {{ appdir }}/.ve/bin/{{ entry.uwsgi['smart
 ; App specific settings.
 uid = {{ entry.user }}
 gid = {{ entry.user }}
-;logto = /var/log/uwsgi/%n.log
 chdir = {{ appdir }}
 virtualenv = .ve
 module = {{ entry.django.app }}.wsgi:application
+; Default logto location: /var/log/uwsgi/app/
 ; Default socket location: /run/uwsgi/app/
-;socket = {{ appdir }}/{{ entry.user }}.sock
 chmod-socket = 666
 
 ; Environment settings. During deployment, uWSGI reloads rather than restarts. However, deleted environment variables
@@ -88,7 +90,7 @@ env = {{ key }}={{ value }}
 ; Otherwise, directories and files are written with permissions 777 and 666, respectively (e.g. to Django's media directory).
 umask = 022
 
-{% if 'stats' in entry.uwsgi %}
+{% if 'stats' in entry.uwsgi -%}
 ; Report running uWSGI stats for additional debugging.
 stats = {{ entry.uwsgi.stats }}
-{% endif %}
+{%- endif %}