DAS-2034 - Remove custom retry logic from HOSS in favour of harmony-service-lib. #4
Conversation
| @@ -58,6 +58,7 @@ def get_request_shape_file(message: Message, working_dir: str, | |||
| raise UnsupportedShapeFileFormat(message.subset.shape.type) | |||
|
|
|||
| shape_file_url = message.subset.shape.process('href') | |||
| adapter_logger.info('Downloading request shape file') | |||
There was a problem hiding this comment.
Initially I included the path to the shape file, but this is somewhat meaningless as Harmony creates a temporary file, which obfuscates any original user naming convention (the file name is essentially a generated ID).
|
@lyonthefrog - I couldn't add you to the HOSS reviewers (you may need to tweak your NAMS request to include this repo). But I thought you would want to track this as someone also working on HOSS now. @jgallagher59701 and @ndp-opendap - for your awareness. (I'm getting a little ahead of ticket process here, but wanted to start getting this change propagated out as a quick win to help reduce load on OPeNDAP) |
|
|
||
|
|
||
| def rgetattr(input_object, requested_attribute, *args): | ||
| """ This is a recursive version of the inbuilt `getattr` method, such that | ||
| it can be called to retrieve nested attributes. For example: | ||
| the Message.subset.shape within the input Harmony message. |
There was a problem hiding this comment.
Nice, I saw this yesterday browsing the code.
| @@ -92,10 +91,18 @@ def test_download_url(self, mock_util_download): | |||
| mock_util_download.side_effect = [self.harmony_500_error, | |||
There was a problem hiding this comment.
This subtest name is incorrect now.
There was a problem hiding this comment.
Good catch. I updated that test name (and also the doc string for the overall test that mentioned retries would be tested)
Description
This ticket removes custom retry logic for downloading a file from OPeNDAP. The
harmony-service-libalso has retry logic, as does each step in a Harmony workflow. All three of these retry types currently can lead to 150 attempts to download a file, which is currently causing issues for OPeNDAP with the GHRSST MUR data.Jira Issue ID
DAS-2034.
Local Test Steps
run_notebooks.shscript). This will also mean settingharmony_host_url = 'http://localhost:3000'in the notebook. The tests should pass.http://localhost:3000/workflow-uiendpoint, and the test to check uses the collection concept ID 'C1245618475-EEDTEST' and has three steps in the workflow: query-cmr, harmony-opendap-subsetter and maskfill. The loggin will contain a line similar to:PR Acceptance Checklist
CHANGELOG.mdupdated to include high level summary of PR changes.VERSIONupdated if publishing a release.Documentation updated (if needed).