wato-delta outage + unexpected cluster outage

[Jimmyj30]

Hi @WATonomous/watcloud-compute-cluster-users,

This evening (Feb 28, 2024) around 8-9 pm EST, there was an unplanned outage of wato-delta (and delta-ubuntu2). Our plan is to reboot the system and return to regular operation either tomorrow or the following day (Feb 29, 2024 or March 1, 2024).

Thanks for your understanding,
WATcloud

[Jimmyj30]

Hi @WATonomous/watcloud-compute-cluster-users,

We are currently working on restoring wato-delta but ran into a few issues, expect the machine to be down until further notice.

Thanks for your understanding,
WATcloud

[ben-z]

Let's pile on some more unfortunate news--30 minutes ago, there seemed to be a split-second power outage that brought down all machines except for trpro. We are working to restore the machines.

[ben-z]

A few machines in the cluster are back up (trpro-ubuntu2, thor-ubuntu2). The cluster is currently operating in a degraded state due to two simultaneous network failures. This requires physical access to the cluster to diagnose and we are working hard to restore the rest of the cluster functionality.

[ben-z]

trpro just went down by itself again. There's definitely something fishy in the server room. The cluster will be down until morning, when @hepromark checks what's going on in the server room.

[ben-z]

trpro-ubuntu2 is up again. However, we are still diagnosing networking issues on a few other computers in the cluster. The cluster is still in a degraded state.

[Jimmyj30]

Hi @WATonomous/watcloud-compute-cluster-users,

The networking issue was resolved yesterday around 8:30 pm EST and the dev VMs are up again!

The cluster status can be tracked at https://status.watonomous.ca/. Note that wato2 and tr are decommissioned currently (#38).

Thanks for your patience,
WATcloud