Upgrade failure because opensearch is slow to start

[tomhughes]

Chef Server Version

15.9.38

Platform Details

Ubuntu 22.04

Configuration

Standalone

Scenario

Complete upgrade of chef server.

Steps to Reproduce

Run chef-server-ctl upgrade.

Expected Result

Upgrade completes.

Actual Result

Upgrade fails on "add internal user to opensearch security plugin" because opensearch is not responsive.

Analysis

There is already an attempt to address the problem in the form of

chef-server/omnibus/files/server-ctl-cookbooks/infra-server/recipes/opensearch.rb

Line 185 in 7940d04

chef_sleep 10

but at least on the machine we're using 10 seconds is nothing like enough for opensearch to finish starting and be ready to answer requests.

Bumping the sleep to 30 seconds while clearly not ideal as a solution seemed to fix things for us.