Add details about motherboard replacement

changes/2023-11-disk-failure-incident-response/index.md

@@ -4,7 +4,7 @@ description: |
   New storage was added and a new pool was configured in response to this incident.
 ---
 
-## Storage changes
+## Hardware changes
 
 The failed 1TB HDD was removed.
 

changes/2023-11-node-failure-incident-response/index.md

@@ -1,11 +1,34 @@
 ---
 title: "2023-11: Node failure incident response"
 description: |
-  TODO
+  Motherboard was replaced in response to this incident.
 ---
 
-# TODO
+## Hardware changes
 
-* Buy a rackmount UPS with spare capacity (the current UPS probably is undersized)
-* Buy a rack
-* Rack mount servers
+* Ordered a replacement `Supermicro X9SCL-F rev1.11a` from eBay (~$20 + shipping).
+* TODO: Install
+
+
+## Diagnosis
+
+Diagnosed a motherboard failure. Rationale:
+
+  * Symptoms began after a full shutdown.
+  * Symptoms:
+      * Fans spin for ~1 second on pushing power button, then stop. Repeats after ~1
+        second delay, indefinitely.
+      * No display
+      * No beep from built-in speaker
+      * No POST
+      * Power cycling stops after holding powr button for 5 seconds.
+  * Symptoms unchanged when removing all components (one by one): disks, RAM sticks,
+    CPU.
+  * Symptoms unchanged with multiple PSUs, PSU cables, and wall outlets.
+
+
+## TODO
+
+- [x] Diagnose failed part(s)
+- [x] Order replacement part(s)
+- [ ] Install replacement part(s)