Add documentation for migrating systems for the 2023-07 release (#148)

* Add documentation for migrating systems This release is coming with a breaking change. All the connected client systems will be disconnected after the upgrade because we've switched to the new salt-master implementation. We are not providing an automated migration process within the product, we are providing manual steps. Signed-off-by: Cristian Hotea <cristian.hotea@ni.com> * Implemented feedback Signed-off-by: Cristian Hotea <cristian.hotea@ni.com> * Update release-notes/2023-07/SystemsConnectionMigration.md Co-authored-by: Melissa Hilliard <63749096+MelissaHilliard@users.noreply.github.com> * Update release-notes/2023-07/SystemsConnectionMigration.md Co-authored-by: Melissa Hilliard <63749096+MelissaHilliard@users.noreply.github.com> * Update release-notes/2023-07/SystemsConnectionMigration.md Co-authored-by: Melissa Hilliard <63749096+MelissaHilliard@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Melissa Hilliard <63749096+MelissaHilliard@users.noreply.github.com> * Add clean up PVC step Signed-off-by: Cristian Hotea <cristian.hotea@ni.com> * Update SystemsConnectionMigration.md --------- Signed-off-by: Cristian Hotea <cristian.hotea@ni.com> Co-authored-by: Melissa Hilliard <63749096+MelissaHilliard@users.noreply.github.com> Co-authored-by: = <mark.black@ni.com>
ni · Jun 30, 2023 · bf61d80 · bf61d80
1 parent 59cb5d5
commit bf61d80
Show file tree

Hide file tree

Showing 3 changed files with 213 additions and 0 deletions.
diff --git a/getting-started/templates/systemlink-secrets.yaml b/getting-started/templates/systemlink-secrets.yaml
@@ -347,6 +347,30 @@ sysmgmtevent:
       ##
       replicaSetKey: ""  # <ATTENTION>
 
+## Secret configuration for salt master.
+##
+saltmaster:
+  secrets:
+    ## Credentials for the MongoDB cluster.
+    ##  
+    mongodb:
+      # Root user password for the database cluster.
+      rootPassword: ""
+      # Limited user password to allow the service to access the minions database. This password cannot contain commas or any character that must be escaped in a URL.
+      minionsPassword: ""
+      # Limited user password to allow the service to access the pillars database. This password cannot contain commas or any character that must be escaped in a URL.
+      pillarsPassword: ""
+      # Key used to authenticate pods in the database cluster.
+      # Refer to MongoDB documentation for key generation: https://www.mongodb.com/docs/manual/tutorial/enforce-keyfile-access-control-in-existing-replica-set/#create-a-keyfile
+      replicaSetKey: ""
+    ## RSA keys used by saltmaster in the communication with client systems.
+    ##      
+    saltmaster:
+      # RSA Private key using PKCS1 padding for salt-master.
+      privateKey:
+      # RSA Public key using PKCS1 padding for salt-master.
+      publicKey:
+
 ## Secret configuration for systems management.
 ##
 systems:

diff --git a/release-notes/2023-07/NewSaltMaster-SystemsMigration.ipynb b/release-notes/2023-07/NewSaltMaster-SystemsMigration.ipynb
@@ -0,0 +1,134 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "4daa69fa-5250-4d00-8456-99e464827e1e",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Getting the managed systems\n",
+      "Found 16 managed systems\n",
+      "\n",
+      "Getting the systems keys\n",
+      "Found 6 approved systems\n",
+      "Found 38 pending systems\n",
+      "Found 0 denied systems\n",
+      "Found 0 rejected systems\n",
+      "\n",
+      "Approving the pending systems that are currently managed\n",
+      "Found 2 managed systems that are in the pending state\n",
+      "\n",
+      "Approving 1 batches of 50 systems\n",
+      "Approving 2 minions, batch: 1\n",
+      "[{'action': 'ACCEPT',\n",
+      " 'id': 'BTS-LabPC-1',\n",
+      " 'key': None,\n",
+      " 'workspace': 'ccae109b-38dc-4ba5-894b-93d0625326e2'}, {'action': 'ACCEPT',\n",
+      " 'id': 'Precision_5570--SN-GPP8WT3--MAC-14-75-5B-DC-5F-AA',\n",
+      " 'key': None,\n",
+      " 'workspace': 'ccae109b-38dc-4ba5-894b-93d0625326e2'}]\n",
+      "Manage Systems Keys Result: {} {'error': None}\n"
+     ]
+    }
+   ],
+   "source": [
+    "import math\n",
+    "import os\n",
+    "import json\n",
+    "import requests\n",
+    "import time\n",
+    "\n",
+    "import systemlink.clients.nisysmgmt.api.systems_api as systems_api\n",
+    "from systemlink.clients.nisysmgmt.models import QuerySystemsRequest, ManageKeysRequest, KeyRequest, KeyAction\n",
+    "\n",
+    "systems_handle = systems_api.SystemsApi(systems_api.ApiClient())\n",
+    "\n",
+    "print(\"Getting the managed systems\")\n",
+    "projection = 'new(id, workspace)'\n",
+    "query_systems_request = QuerySystemsRequest(projection = projection)\n",
+    "managed_systems = (await systems_handle.get_systems_by_query(query_systems_request)).data\n",
+    "print(\"Found {0} managed systems\\n\".format(len(managed_systems)))\n",
+    "# print(managed_systems)\n",
+    "\n",
+    "print(\"Getting the systems keys\")\n",
+    "systems_keys = await systems_handle.get_systems_keys()\n",
+    "print(\"Found {0} approved systems\".format(len(systems_keys.systems_approved) if systems_keys.systems_approved else 0))\n",
+    "print(\"Found {0} pending systems\".format(len(systems_keys.systems_pending) if systems_keys.systems_pending else 0))\n",
+    "print(\"Found {0} denied systems\".format(len(systems_keys.systems_denied) if systems_keys.systems_denied else 0))\n",
+    "print(\"Found {0} rejected systems\\n\".format(len(systems_keys.systems_rejected) if systems_keys.systems_rejected else 0 ))\n",
+    "# print(systems_keys.systems_pending)\n",
+    "\n",
+    "print(\"Approving the pending systems that are currently managed\")\n",
+    "if systems_keys.systems_pending is None:\n",
+    "    systems_keys.systems_pending = []\n",
+    "managed_systems_to_approve = [system for system in managed_systems if system.get(\"id\") in systems_keys.systems_pending]\n",
+    "print(\"Found {} managed systems that are in the pending state\\n\".format(len(managed_systems_to_approve)))\n",
+    "# managed_systems_to_approve = [{\"id\":\"Precision_5570--SN-GPP8WT3--MAC-14-75-5B-DC-5F-AA\",\"workspace\":\"846e294a-a007-47ac-9fc2-fac07eab240e\"}]\n",
+    "\n",
+    "batch_size = 50\n",
+    "batches = math.ceil(len(managed_systems_to_approve) / batch_size)\n",
+    "print(\"Approving {} batches of {} systems\".format(batches, batch_size))\n",
+    "\n",
+    "for i in range(0, batches):\n",
+    "    minions = managed_systems_to_approve[i * batch_size : (i+1) * batch_size]\n",
+    "    print(\"Approving {} minions, batch: {}\".format(len(minions), i+1))\n",
+    "    keys_actions = [KeyRequest(minion.get(\"id\"), KeyAction.ACCEPT, None, minion.get(\"workspace\")) for minion in minions]\n",
+    "    print(keys_actions)\n",
+    "    manage_keys_request = ManageKeysRequest(is_async=True, key_actions = keys_actions)\n",
+    "    result = await systems_handle.manage_systems_keys(manage_keys_request)\n",
+    "    print(\"Manage Systems Keys Result: {}\", result)\n",
+    "    time.sleep(10)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ed471076-bc05-4b9b-b0a0-b2a41c5cb1fe",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9e88969c-35f6-40aa-a06e-0aabcc0ca8c4",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a7d1876b-ed90-4efb-99e4-a275911ce434",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/release-notes/2023-07/SystemsConnectionMigration.md b/release-notes/2023-07/SystemsConnectionMigration.md
@@ -0,0 +1,55 @@
+# Migrating Connected Systems to SLE 2023-07 or later
+
+The 2023-07 release introduces a breaking change in systems connectivity that affects prior releases. The saltmaster service which manages the connectivity with the client systems has been replaced with a new implementation that disconnects all connected clients upon upgrade. Complete the following steps to re-approve and re-connect all previously connected clients.
+
+## Prior to upgrading to SLE 2023-07
+
+The old saltmaster uses a pair of RSA keys to secure communication. The public key is already cached on all
+accepted clients. You must persist these keys between the upgrade of SLE, so the client systems do not reject the new saltmaster's public key.
+
+1. Open a terminal to the current `saltmaster-0` pod.
+2. Navigate to `/etc/salt/pki/master`
+3. Make a note of the `master.pem` (private key) and `master.pub` (public key) values.
+4. Set these values to the secrets definition object of the saltmaster chart in the top level secrets values yaml file. Make sure
+you use the '|' indicator because these are multiline strings. Refer to <https://yaml-multiline.info/> for details on this character's usage.
+
+The updated chart will create a Kubernetes secret using these two keys.
+
+## Post SLE Upgrade
+
+After the upgrade, all previously connected (green/orange systems) will be shown as **Disconnected**. Additionally, these clients will also be shown in the **Pending systems** list. You can use the [Jupyter Notebook](./NewSaltMaster-SystemsMigration.ipynb) provided to automate system approval. This notebook will add systems to the workspace they were previously a member of. The user who runs the notebook must have the **Add systems** privilege in all the necessary workspaces.  
+
+1. Ensure all systems are disconnected and appear in the pending systems list.
+1. Upload **NewSaltMaster-SystemsMigration.ipynb** to JupyterHub.
+1. Identify a single system to validate the notebook operation.
+1. Override `managed_systems_to_approve` with its id and workspace. The notebook already
+contains this line, we only need to uncomment it and add the id and workspace:
+
+    ```python
+    managed_systems_to_approve = [{"id":"Precision_5570--SN-GPP8WT3--MAC-15-75-5B-DC-5F-BB","workspace":"846e294a-a007-47ac-9fc2-fac07eab240a"}]
+    ```
+
+1. Run the notebook and check the output. It should say that one system was approved.
+
+    ```python
+    Approving 1 batches of 50 systems
+    Approving 1 minions, batch: 1
+    [{'action': 'ACCEPT',
+    'id': 'Precision_5570--SN-GPP8WT3--MAC-15-75-5B-DC-5F-BB',
+    'key': None,
+    'workspace': '846e294a-a007-47ac-9fc2-fac07eab240a'}]
+    Manage Systems Keys Result: {} {'error': None}
+    ```
+
+1. Validate that the selected system is now connected (green).
+
+    **Note** Misconfiguration of the RSA keys for the salt master may cause this operation to fail. Review the new secrets and [logs on the client system](https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z000000kGcSSAU&l=en-US)
+1. After connecting this single system, you can now uncomment the override of `managed_systems_to_approve` and run the notebook again.
+
+    ```python
+    # managed_systems_to_approve = [{"id":"Precision_5570--SN-GPP8WT3--MAC-15-75-5B-DC-5F-BB","workspace":"846e294a-a007-47ac-9fc2-fac07eab240a"}]
+    ```
+
+    **Note** The new run will re-approve all the systems in batches of 50 (this can also be modified in the notebook)
+1. Ensure all the previously connected clients are connected (green).
+1. [Optional] Delete the PVCs and volumes used by the old implementation. PVC names: `saltmaster-pillar-saltmaster-0` and `saltmaster-pki-saltmaster-0`.