Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure joblogs storage dir #34

Open
wvengen opened this issue Oct 31, 2024 · 1 comment
Open

Configure joblogs storage dir #34

wvengen opened this issue Oct 31, 2024 · 1 comment

Comments

@wvengen
Copy link
Member

wvengen commented Oct 31, 2024

Following #28, I was trying to figure out where to mount the volume for joblogs, and noticed that while the example kubernetes manifest mounts this at /data (though commented out), the files generated live in temporary file space

temp_dir = tempfile.gettempdir()

According to gettempdir(), this is configured in the environment.
But I think this needs a dedicated volume, because other temporary files might be generated too, and if these may be used elsewhere, that comes in the way of computing the required storage requirements.
So let's configure this in the config file for joblogs, with a fallback to the temporary directory.

Note that it is possible that the scrapyd-k8s pod is evicted and restarted on another node (e.g. when a node fails, or during upgrades), so to not lose logs, this shouldn't be a host-local path that disappears when switching hosts.

@wvengen wvengen mentioned this issue Dec 17, 2024
@wvengen
Copy link
Member Author

wvengen commented Dec 17, 2024

On Kubernetes, with multiple nodes (that have no shared filesystem), one must use a persistent volume (PVC). For this, and perhaps also other scenario's, it is important that the user running scrapyd-k8s can write there for joblogs.

When using PVCs, Kubernetes uses the securityContext fsGroup to set PVC ownership (see here). This can be used to make sure joblogs can be written. Put it in the example k8s manifest (avoid having yet another place to comment out to enable joblogs - so it could be time for making a full helm chart where conditionals can be handled more cleanly; or have two manifests, one without and with joblogs (these can then also be tested in #33).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant