ghost of wato-drive2?

[mxpoch]

Hi all, I have some simulation results on wato-drive2 but it doesn't appear under the disks when I login.
Does anybody else have this issue?

[ben-z]

A gigantic out-of-memory issue brought the node hosting wato-drive2 down yesterday :(

The root cause appears to be an arbitrary number of kcopyd_job threads are allowed to spawn to write back data from the cache disk to the main disk. Combined with a cache writeback threshold (512GB) larger than the available RAM (64GB), kcopyd hogged up all available memory, and critical processes could not run.

Here's the full logs from that day for the record:
2023-02-07-oom.log

We got it back up 30 minutes ago by manually rebooting the machine. But it went down again when we tried to flush and disable our cache disk. We'll get the drive back this afternoon.

cc @WATonomous/compute-cluster-users

For internal reference, the steps that will be taken are:

• Hard reboot wato-derek2 (the machine that's hosting wato-drive2)
• Change the cache writeback job limit to a sensible number (e.g. 40GiB, or 10485760 pages): lvchange --cachesettings "writeback_jobs=10485760" wato-drive2-vg/wato-drive2-lv-main
• Verify the RAM usage is sane by filling up the cache disk to the writeback trigger (50% full, or 512GB)

[Jimmyj30]

Verify the RAM usage is sane by filling up the cache disk to the writeback trigger (50% full, or 512GB)

@ben-z Just wondering how we would go about this step—are there some commands to run or do we wait until the cache disk fills up (if that is even a thing?)

Thanks!

[ben-z]

We can manually fill it up using something like yes | pv > test.txt! And we can monitor the cache disk usage using dmsetup status and disk activity using dstat -tdD total,sda,nvme0n.

[ben-z]

wato-drive2 is up with reduced performance (the caching drive is disabled until we can test limiting kcopyd's resource usage). The reduction in performance will be the most prominent in random r/w workloads.

Internal reference