You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Following #28, I was trying to figure out where to mount the volume for joblogs, and noticed that while the example kubernetes manifest mounts this at /data (though commented out), the files generated live in temporary file space
According to gettempdir(), this is configured in the environment.
But I think this needs a dedicated volume, because other temporary files might be generated too, and if these may be used elsewhere, that comes in the way of computing the required storage requirements.
So let's configure this in the config file for joblogs, with a fallback to the temporary directory.
Note that it is possible that the scrapyd-k8s pod is evicted and restarted on another node (e.g. when a node fails, or during upgrades), so to not lose logs, this shouldn't be a host-local path that disappears when switching hosts.
The text was updated successfully, but these errors were encountered:
On Kubernetes, with multiple nodes (that have no shared filesystem), one must use a persistent volume (PVC). For this, and perhaps also other scenario's, it is important that the user running scrapyd-k8s can write there for joblogs.
When using PVCs, Kubernetes uses the securityContextfsGroup to set PVC ownership (see here). This can be used to make sure joblogs can be written. Put it in the example k8s manifest (avoid having yet another place to comment out to enable joblogs - so it could be time for making a full helm chart where conditionals can be handled more cleanly; or have two manifests, one without and with joblogs (these can then also be tested in #33).
Following #28, I was trying to figure out where to mount the volume for joblogs, and noticed that while the example kubernetes manifest mounts this at
/data
(though commented out), the files generated live in temporary file spacescrapyd-k8s/scrapyd_k8s/joblogs/log_handler_k8s.py
Line 178 in 7975a70
According to
gettempdir()
, this is configured in the environment.But I think this needs a dedicated volume, because other temporary files might be generated too, and if these may be used elsewhere, that comes in the way of computing the required storage requirements.
So let's configure this in the config file for joblogs, with a fallback to the temporary directory.
Note that it is possible that the scrapyd-k8s pod is evicted and restarted on another node (e.g. when a node fails, or during upgrades), so to not lose logs, this shouldn't be a host-local path that disappears when switching hosts.
The text was updated successfully, but these errors were encountered: