WATcloud Ubuntu 22.04 Upgrade and New Features

[ben-z]

Hi @WATonomous/watcloud-compute-cluster-users,

The WATcloud team is doing an upgrade of all general-use machines from Ubuntu 20.04 to Ubuntu 22.04 in the next week. To do this, we are scheduling a maintenance window for trpro-ubuntu1 on Sunday, Nov 26, from 11am-11pm Eastern Time. The other machines are less heavily-utilized, so we will perform ad-hoc rolling upgrades based on usage. Please let us know via Discord or email if you have any questions or concerns regarding this upgrade.

On another note, we've been pushing out a lot of new improvements in the past few months. We are working on polishing these features and a newsletter that will cover them in detail, but in the mean time, here's a sneak peak:

• New website! This website will serve as a source-of-truth for anything related to WATcloud, including machine inventory and onboarding guides. We've built an end-to-end pipeline to automatically extract machine specs so that the machine inventory and the SSH access instructions will never go out of date!
• New communication channels! We've opened up our Discord server for anyone to ask questions and chat about WATcloud. Here's an invite link. The main support channels are #tech-support for specific questions and #server-use for general chatting. These channels may be renamed, and we will clarify that in our next newsletter.
• New server access instructions! We've written up a guide and an SSH command generator to outline various ways to access machines in the compute cluster. It turns out, bastion is not the only entrypoint!
• User profile editor! We've built a pipeline to add editing functionality to the information that you filled out in the onboarding form. You can see an alpha version here. Click "Try it out" and enter your contact email address to receive an edit link.
• Ability to add additional SSH keys without going through the approval process again! We've revamped our SSH key management and added the ability for you to self-manage SSH keys. The WATcloud-managed SSH keys (the ones that you filled out on the onboarding form) are now stored in /etc/cluster/users/<uid>/.ssh/authorized_keys. You can put additional SSH keys in $HOME/.ssh/authorized_keys and it can be used for authentication. As always, anything that you put in your home directory will be synchronized to all of the general-use machines. There's one caveat: this feature is not ready for bastion yet. In the mean time, you may be able to work around this by using an alternative entrypoints shown in the SSH command generator.
• We are rolling out a "trusted affiliation" list. This is a way to allow the leads of different groups approve their own member access requests instead of having to go through a WATcloud contact. We are currently beta testing this with WATonomous subteams, and will have a general release really soon!

There are many other exciting upgrades that we will cover in a future newsletter. Stay tuned!

The WATcloud Team

[ben-z]

Please note that during this upgrade, the local storage (mounted at /) will be replaced, including the Docker storage for images and containers (/var/lib/docker). /home, /mnt/wato-drive, and /mnt/wato-drive2 will be persisted. Please move important files into these locations.

[ben-z]

We ran into some issues with disk provisioning and will need a bit more time. Will keep this thread updated.

[ben-z]

trpro-ubuntu1 has been replaced by trpro-ubuntu2. The VM has been upgraded to Ubuntu 22.04.

There was an unrelated disk failure on wato-thor, which caused thor-ubuntu1 to go down. During debugging, the bare-metal machine backing thor-ubuntu1, wato-thor, went down and as a result, /mnt/wato-drive became inaccessible. /mnt/wato-drive2 is still accessible. wato-thor currently doesn't have a remotely manageable power switch. We will go in person to boot it up in the morning.

The SSH access instructions will be updated once the cluster is healthy again (required for the provisioning pipeline to progress).

[ben-z]

We will also mark trpro-ubuntu2 as experimental until the cluster is healthy again and we can perform a full provision.

[ben-z]

The cluster is healthy again, with the exception of thor-ubuntu1 and wato-drive1, which will be brought up once our in-person team finishes their work in the server room. The machines page is updated with the latest available resources.

[ben-z]

/mnt/wato-drive is back online and thor-ubuntu1 has been upgraded to Ubuntu 22.04 and renamed to thor-ubuntu2. The machines page is updated with the latest available resources. The remaining machines will be upgraded in the coming days.

[ben-z]

derek3-ubuntu1 has been reprovisioned and became derek3-ubuntu2. You can find a list of the latest machines here: https://cloud.watonomous.ca/machines

[ben-z]

delta-ubuntu1 has been replaced with delta-ubuntu2. The OS was upgraded from Ubuntu 20.04 to Ubuntu 22.04. Here's the updated list of machines.

The filesystem quota implementation is taking a bit more effort than expected. In the mean time, please regularly clean up your docker setup (e.g. with docker system prune -a) so that our drives don't fill up as quickly.