Update sfs_load.overlay: Workarounds for /etc, /run, and /var

[rizalmart]

The problem with symlink method for loading sfs modules on the fly on overlay is that files under /etc, /run, and /var where got read-only. To fix this issue simply exclude those folder when creating symlinks. Then perform copy of files of /etc, /run, and /var from sfs module to rootfs

[peabee]

How and when does this (files under /etc, /run, and /var where got read-only) cause problems?

@dimkr review requested

[dimkr]

The symlinks point to read-only files, yes. That's a limitation of the symlinks method.

But why restrict this to these arbitrary directories and not copy the entire SFS? (By the way, I see no reason for a SFS to include files under /var and /run)

[rizalmart]

But why restrict this to these arbitrary directories and not copy the entire SFS?

Copy the entire SFS contents defeats the purpose of SFS on the fly loading and will eat a lot of disk space. Symlink method saves disk space because it was just a symlink. Also /etc was config files it requires write access. /var was self-explanatory on its name. The files inside of that folder always change.

(By the way, I see no reason for a SFS to include files under /var and /run)

Some linux apps have files on /var and /run.

[dimkr]

Copy the entire SFS contents defeats the purpose of SFS on the fly loading

True, but users who use sfs_load.overlay instead of loading the SFS at boot time probably know its limitations.

Some linux apps have files on /var and /run.

But /run is a tmpfs and /var is mostly for things like state or cache.

[rizalmart]

But /run is a tmpfs and /var is mostly for things like state or cache.
Not so fast. flatpak apps sometimes installed at /var folder, apache server stores hosted webpages on /var folder, mysql database and other rdms stored data on /var folder. If these apps were made into SFS modules, just copy contents of /var folder to make it writable makes sense

[dimkr]

I can see why you'd want directories in /var (for example, a package-specific subdirectory of /var/cache), but not files. If your SFS contains an application but also its cache or state, these files will always get copied up. The entire database will be copied to RAM on first change, if you have a snapshot of the database in /var of a SFS. Same with Flatpak - the package database is huge, changes often and gets copied to RAM if you actually use Flatpak.

(This is only my opinion, I don't have veto power but I'm not going to cherry-pick this PR into my fork when it's merged)

[peabee]

This proposed change needs to be exposed to wider scrutiny and fully tested. The best way to do this is probably to release a test build on the forum with requests to test apps which are likely to produce problems.

(If flatpak apps are converted to sfs then presumably they can be reconfigured during the conversion to conform to sfs_load.overlay constraints?)

[dimkr]

(If flatpak apps are converted to sfs then presumably they can be reconfigured during the conversion to conform to sfs_load.overlay constraints?)

You can put whatever you want in a SFS, but sometimes it doesn't make sense. For example, if you put /var/cache/apt in a SFS and apt update, everything is in RAM (because the files change), the files in the SFS are unused and you need to recreate the SFS if you want the updated files on next boot. In such cases, I wouldn't put these files in a SFS, because it's just a waste of disk space (second copy of the files) and RAM (if the SFS is copied to RAM, but the files are unused) without any benefit.

Same thing with Flatpak, its cache is huge and you'll need to rebuild the SFS with the current cache every time you install or update something, unless you want to download everything and waste RAM again.

[rizalmart]

(If flatpak apps are converted to sfs then presumably they can be reconfigured during the conversion to conform to sfs_load.overlay constraints?)

You can put whatever you want in a SFS, but sometimes it doesn't make sense. For example, if you put /var/cache/apt in a SFS and apt update, everything is in RAM (because the files change), the files in the SFS are unused and you need to recreate the SFS if you want the updated files on next boot. In such cases, I wouldn't put these files in a SFS, because it's just a waste of disk space (second copy of the files) and RAM (if the SFS is copied to RAM, but the files are unused) without any benefit.

Same thing with Flatpak, its cache is huge and you'll need to rebuild the SFS with the current cache every time you install or update something, unless you want to download everything and waste RAM again.

Including /var/cache on sfs module is a bad idea for dev's perspective. If someone converts installed flatpak apps to sfs then the repo objects in flatpak folder on /var must not be included in the first place

[dimkr]

Including /var/cache on sfs module is a bad idea for dev's perspective. If someone converts installed flatpak apps to sfs then the repo objects in flatpak folder on /var must not be included in the first place

I'd say the same logic applies to all files under /var. A SFS shouldn't contain state and cache (= files under /var) if it's supposed to ship an application in its "clean slate" state.