Initial draft refresh of machine policies doc

src/pages/docs/infrastructure/deployment-targets/machine-policies.mdx

@@ -6,7 +6,6 @@ title: Machine policies
 description: Machine Policies allow you to customize the behavior of Tentacle and SSH endpoints like health check settings, machine connectivity, updates and more.
 navOrder: 110
 ---
-import HealthChecks from 'src/shared-content/infrastructure/health-checks.include.md';
 
 Machine policies are groups of settings that can be applied to Tentacle and SSH endpoints to modify their behavior. They can be used to:
 
@@ -21,9 +20,33 @@ You can access the machine policies by navigating to **Infrastructure ➜ Machin
 
 [Getting Started - Machine Policies](https://www.youtube.com/watch?v=uEjXuPttRO4)
 
+## Default machine policy
+By default Octopus Deploy comes with a default configured machine policy with the following rules:
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-default-policy.png)
+:::
+
+This machine policy cannot be deleted however it can be edited and any new deployment targets without an explicitly assigned machine policy will use the default one. Other machine policies will also inherit the bash/powershell scripts used for health checking from the default policy unless they have one explicitly set.
+
 ## Health check
 
-<HealthChecks />
+Octopus periodically runs health checks on deployment targets and workers to ensure that they are available and running the latest version of Calamari.
+
+The health status of a deployment target can be set by custom health check scripts. Deployment targets can have four health statuses:
+
+- Healthy
+- Healthy with warnings
+- Unhealthy
+- Unavailable
+
+A *healthy* deployment target completes a health check without any errors or warnings.  A deployment target that is *healthy with warnings* completes a health check but encounters a non-critical failure during the health check.  An *unhealthy* deployment target completes a health check but encounters a critical failure while running the health check script.  An *unavailable* deployment target is not contactable by Octopus during a health check.
+
+You can review the health status of you deployment targets by navigating to **Infrastructure ➜ Deployment Targets**.
+
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-deployment-target-overview.png)
+:::
+
 
 ### Health check step
 
@@ -33,12 +56,24 @@ This step allows a deployment target that was created in the currently executing
 
 Similarly, it allows you to confirm that the Tentacle service on a deployment target is running prior to attempting to perform an action against it.
 
-## Initial health check
+### Initial health check
 
-After installing and configuring a new Tentacle, you need to run a health check and can upgrade the version of Calamari.
+After creating a new Tentacle or SSH Target, Octopus Deploy will automatically schedule an initial health check, however if you desire you can manually start a health check by doing the following:
 
 1. From the **Infrastructure** tab, select **deployment targets**.
-2. Click the ... overflow menu and select **Check Health**. If you've installed multiple Tentacles, it will check all of your Tentacles (if you'd rather check only one Tentacle, select that Tentacle from the deployment targets section, click **Connectivity** and then **Check health**).
+2. Click the ... overflow menu and select **Check Health**. If you have multiple deployment targets it will start health checks for each of them.
+
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-check-health.png)
+:::
+
+However if you would like to check the health of a single deployment target you can do the following:
+1. Select the deployment target you would like to health check.
+2. Click on the **Connectivity** tab then click on the **Check Health** button on the top right of the page.
+
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-deployment-target-connectivity.png)
+:::
 
 The first time you complete a health check on a Tentacle or SSH Target, you will see health warnings and that Calamari needs to be installed.
 
@@ -49,24 +84,64 @@ Octopus will automatically push the latest version of Calamari with your first d
 1. From the Infrastructure tab, select deployment targets.
 2. Click the ... overflow menu and select **Upgrade Calamari on Deployment Targets**.
 
-### Health check interval
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-update-calamari.png)
+:::
+
+Like with the health check if you would rather just update Calamari on a single deployment target you can achieve this by:
+1. Select the deployment target you would like to update Calamari on.
+2. Click on the **Connectivity** tab then click on the **Update Calamari** button.
+
+## Health check configuration
+### Health check schedule type
+A health check can be scheduled in the following ways:
+
+**Never**
+
+No automatic health checks will be scheduled, when this option is selected. Manual health checks can still be run when desired.
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-health-check-never.png)
+:::
+
+**Interval**
+
+A health check will be scheduled after every X duration(X = days/hours/minutes)
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-health-check-interval.png)
+:::
 
-You can set the "Time between checks" to control how frequently automatic health checks run.
+**Cron expression**
 
+A health check will be scheduled in accordance to the valid cron expression. You can also choose the timezone the cron expression should adhere to.
 :::figure
-![](/docs/infrastructure/deployment-targets/images/5865585.png "width=500")
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-health-check-cron.png)
 :::
 
 ### Health check type
+You can configure health checks to run scripts on Tentacle or SSH deployment targets, or just check that a connection can be established with the Tentacle or SSH deployment targets.
+
+**Run health check script**
+
+Machine policies allow the configuration of custom health check scripts for Tentacle and SSH deployment targets. While we do not expose the full underlying script that runs during health checks, we give you an entry point to inject your own custom scripts.
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-health-check-type-script.png)
+:::
+By default machine policies will inherit the script from the default machine policy if one has not been defined. For more details on providing a custom script see the section [Custom health check script](#custom-health-check-script)
 
-You can configure health checks to run scripts on deployment targets, or just check that a connection can be established with the deployment target. When the "Run health check scripts" option is selected you will also have the opportunity to customize the PowerShell and Bash scripts that will be executed during the health check. The "Only perform connection test" option is recommended if you are using the raw scripting feature.
+**Connection test**
 
-### Custom health check scripts \{#custom-health-check-scripts}
+The health check will only perform a basic connection test to see if the deployment target is reachable. This option is recommended if you are using the raw scripting feature.
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-health-check-type-connection.png)
+:::
 
-Machine policies allow the configuration of custom health check scripts for Tentacle and SSH deployment targets. While we do not expose the full underlying script that runs during health checks, we give you an entry point to inject your own custom scripts. For example, here is the default custom health check script for PowerShell that checks disk space:
+### Custom health check script
+When the selected health check type is "Run health check scripts" you can provide script(s) that will execute during the health check.
 
 **Default PowerShell health check script**
 
+Here is the default custom health check script for PowerShell that checks disk space:
+
 ```powershell
 $freeDiskSpaceThreshold = 5GB
 Try {
@@ -76,10 +151,9 @@ Try {
 	Get-WmiObject Win32_MappedLogicalDisk | ? { ($_.FreeSpace -ne $null) -and ($_.DeviceId -ne $null)} | % { CheckDriveCapacity @{Name =$_.DeviceId; FreeSpace=$_.FreeSpace} }
 }
 ```
-
 The function *CheckDriveCapacity* informs you about how much space is available on your deployment target's local hard disk and will write a warning if the free disk space is less than this threshold. You can add additional PowerShell to this script to customize your health checks as you wish, modify or remove the disk space checking altogether. It's entirely up to you! Just remember, you can copy and paste the original script above *back* into your machine policy if you run into any problems and wish to get back to the default behavior.
 
-## Set the status
+**Set the status**
 
 A health check script can set the status of a target by returning a non-zero exit code or by writing a service message during the health check. PowerShell based deployment targets can use *Write-Warning*, *Write-Error* and *Fail-HealthCheck* to convey a healthy with warnings or unhealthy status:
 
@@ -113,16 +187,62 @@ fail_healthcheck "This is an error"
 When using a custom health check script, the script execution through Calamari is bypassed. This results in some behavioral differences compared with the normal scripting in Octopus that you would be accustomed to. You can still use the standard `#` variable substitution syntax, however since this is replaced on the server, environment variables from your target will not be available through Octopus variables.
 :::
 
-## Ignore machines that are unavailable during health checks
+## Machine connectivity configuration
+
+### Machine behavior during health checks
+This setting controls what the expected behavior of machines should be during a health check.
+
+**Unavailable causes health checks to fail**
+
+If a machine is unavailable during an attempted health check it will cause the check to be failed.
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-connectivity-health-check-default.png)
+:::
+
+**Ignore machines that are unavailable during health checks**
 
-By default, health checks fail if any deployment targets are unavailable during the health check.  Machine policies offer an option to ignore machines if they are unavailable during a health check:
+By default, health checks fail if any deployment targets are unavailable during the health check. Machine policies offer an option to ignore machines if they are unavailable during a health check:
 
 :::figure
-![](/docs/infrastructure/deployment-targets/images/5865592.png "width=500")
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-connectivity-health-check-ignore.png)
 :::
 
 By selecting **Unavailable machines will not cause health checks to fail,** any deployment targets that Octopus cannot contact during a health check will be skipped and the health check marked as successful. If the target is contactable but encounters an error or warning, the usual health check behavior will proceed (i.e., a warning will be reported or the health check will fail with an error).
 
+### Connection timeout and retries
+Machine policies define rules for how long should be waited for a connection to be established to a Tentacle or SSH target as well as how it should retry the connection if the initial attempts do not succeed.
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-connectivity-timeout-retries.png)
+:::
+
+**Connect timeout**
+
+How long in hours/minutes/seconds to wait for a Listening Tentacle or SSH target respond to a connection before timing out. Depending on what operating system the machine is running could potentially mean the timeout is shorter than expected.
+
+**Retry attempts**
+
+How many times the connection to the Listening Tentacle or SSH target should be retried before failing the health check.
+
+**Retry wait interval**
+
+How long to wait between connection retries.
+
+**Retry time limit**
+
+Maximum time limit for how long retries can be attempted for.
+For example you have the following configuration:
+- Retry time limit: 1 minute.
+- Retry attempts: 4.
+- Retry interval: 30 seconds.
+
+In this scenario only a maximum of 2 retries could be attempted before reaching the time limit.
+
+### Polling request queue timeout
+This setting controls how long Octopus Deploy will wait before cancelling a task in a polling Tentacle's queue. For more details on what the Polling Tentacle queue is see the doc on [Tentacle communication modes](/docs/infrastructure/deployment-targets/tentacle/tentacle-communication#polling-tentacles)
+:::figure
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-connectivity-polling-queue.png)
+:::
+
 ## Configure how Calamari, Tentacle and Kubernetes agents are updated \{#configure-machine-updates}
 
 Brand new Tentacle, Kubernetes agent and SSH endpoints require the installation of Calamari to perform a deployment.  Also, if Calamari is updated, the Octopus Server will push the update to Tentacle, Kubernetes agent and SSH endpoints. 
@@ -193,10 +313,10 @@ Machine policies can be configured to automatically remove unavailable machines
 
 ## Assign machine policies to machines
 
-Assign a machine policy to a machine by selecting a machine from the *Environments* screen and using the *Policy* drop down to select the machine policy:
+Assign a machine policy to a machine by selecting a machine from the *Deployment Targets* or *Workers* screen and using the *Policy* drop down to select the machine policy:
 
 :::figure
-![](/docs/infrastructure/deployment-targets/images/5865599.png "width=500")
+![](/docs/infrastructure/deployment-targets/machine-policies/machine-policies-connectivity-assign-policy.png)
 :::
 
 Machine policy can also be set from the command line by using the --policy argument: