Replies: 1 comment 1 reply
-
If the engine being used is miniwdl then it doesn't use the ebs auto expansion. That is largely used by the engines that don't use EFS and read data directly into the container (Cromwell and Nextflow). It might be that The location of the Batch worker logs is defined by the launch template in the cloudwatch agent config section https://github.com/aws/amazon-genomics-cli/blob/main/packages/cdk/lib/constructs/launch-template-data.ts#L28-L72 The provisioning (or not) of the ebs-auto-expansion is controlled in the provision script at https://github.com/aws/amazon-genomics-cli/blob/main/packages/cdk/lib/artifacts/batch-artifacts/ecs-additions/provision.sh#L100-L110 Mainly we don't provision the auto expansion for engines that use EFS as it attaches an additional EBS volume per worker EC2 which is typically not used. |
Beta Was this translation helpful? Give feedback.
-
We have some WDL tasks that rapidly extract tar archives to local storage, and seeing them sporadically fail with out-of-disk space errors. Saw the docs' blurb alluding to the use of amazon-ebs-autoscale on the one hand, but also how tasks might be able to "outrun" it on the other hand; but it doesn't go into solutions/workarounds.
Where should we start looking in logs etc. to troubleshoot this? For example, where might we find evidence to distinguish whether the EBS autoscaling logic maybe wasn't working at all versus whether it merely didn't act quickly enough? If indeed the latter, are there tuning options (as an AGC user) to make it expand earlier, or at least increase the initial size?
Beta Was this translation helpful? Give feedback.
All reactions