feat(experimental-ec2-pattern): Pattern to deploy ASG updates w/CFN

[akash1810]

What does this change?

Reduce the “time spent broken” after failed deploys, making these manual steps redundant, by adding a new experimental pattern GuEc2AppExperimental to provision an EC2 based service with an AutoScalingRollingUpdate update policy. The configuration is based on a number of tests we ran; we've documented some various observations.

This is an alternative to #2395. It improves on the solution proposed in #2395 by providing continuity of ASG metrics and ASG activity history.

AutoScalingRollingUpdate offers an improved DX over the autoscaling Riff-Raff deployment type when a deployment fails. On deployment failure, the autoscaling Riff-Raff deployment type leaves the ASG over-capacity, requiring manual intervention to resolve, whereas AutoScalingRollingUpdate will perform a rollback automatically. This means deployments are more atomic.

Note

Unfortunately testing has shown unpredictable behaviour if a scale-in event occurs during deployment. Therefore we suspend the AlarmNotification process during a rolling update.

This means scaling events cannot occur during deployment and guardian/riff-raff#1342 is not fixed.

With AutoScalingRollingUpdate CloudFormation will deploy application updates to an ASG in a rolling fashion. The rate at which the update rolls out is dependant on how scaled-out an ASG is:

• If an ASG does not have any scaling policies, deployment will behave similar to Riff-Raff; the DesiredCapacity will be doubled, once new instances are healthy the capacity will be halved removing the old instances.

• For an ASG that has a scaling policy, the rate will be dynamic:

  • If it is running normally, DesiredCapacity will be doubled, once new instances are healthy the capacity will be halved removing the old instances (testing observations).
  • If the service is partially scaled, deployment will happen in batches relative to the remaining capacity (MinCapacity / (MaxCapacity - DesiredCapacity)) (testing observations).
  • If the service is fully scaled, deployment will happen in a "one out, one in" fashion (testing observations).

  This will be achieved by Riff-Raff setting MinInstancesInService at deploy time via a CloudFormation Parameter.

  This functionality is yet to be added to Riff-Raff. It doesn't block shipping this change, however services that use scaling policies cannot use this experimental pattern until then.

As application updates are performed entirely by CloudFormation, it is not necessary to use the autoscaling RIff-Raff deployment type¹. If it is used, it shouldn't break anything. However it does mean deployments will take a long time, as the instances will be rotated twice; once by CloudFormation and once by Riff-Raff's custom logic.

Changes to UserData

With AutoScalingRollingUpdate CloudFormation will remain in the UPDATE_IN_PROGRESS state whilst waiting for a SUCCESS signal to be received from the instance. This signal is sent manually.

The GuEc2AppExperimental pattern will add this signal emission to the UserData. A SUCCESS signal will be sent once the instance has successfully registered with the Target Group.

Why experimental?

There are a few requirements for this approach:

• The AWS CLI, and cfn-signal binaries need to be available and on the PATH (this is added via the AMIgo role aws-tools).
• The service's artifact should include a build number, as this is the most reliable way to create a difference with the running CloudFormation template, and thus trigger an update.

We're not (yet) validating these are met, as its tricky.

Proposed rollout

The rollout plan looks something like this:

1. Dogfood on some DevX services.
2. Test on a service within the department.
3. Move the pattern to stable, as a breaking change, with communication on the migration path, etc.

How to test

These changes have been informed by "real-world" testing that has been documented in https://github.com/guardian/testing-asg-rolling-update.

This branch is being successfully used in guardian/amiable, guardian/cdk-playground and guardian/security-hq.

How can we measure success?

A DX improvement when a deployment fails.

Once fully adopted, we will be able to remove the autoscaling deployment type in Riff-Raff, reducing it's complexity.

Have we considered potential risks?

As this new pattern is experimental there risk levels are fairly small. We should monitor usage and gather feedback before promoting to stable.

Checklist

• [ - ] I have listed any breaking changes, along with a migration path ²
• I have updated the documentation as required for the described changes ³

Footnotes

1. We still need the uploadArtifacts action of the autoscaling deployment type. The cloud-formation deployment type should depend on this step too. ↩

2. Consider whether this is something that will mean changes to projects that have already been migrated, or to the CDK CLI tool. If changes are required, consider adding a checklist here and/or linking to related PRs. ↩

3. If you are adding a new construct or pattern, has new documentation been added? If you are amending defaults or changing behaviour, are the existing docs still valid? ↩

[changeset-bot]

🦋 Changeset detected

Latest commit: 7a12f60

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@guardian/cdk	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[akash1810]

If the app contains hyphens (-), or other illegal characters for a CFN Parameter, they'll get stripped out. This means Riff-Raff cannot reliably just find CFN Parameters that start with MinInstancesInServiceFor. Therefore Riff-Raff's logic would be something like this:

1. Find all AWS::AutoScaling::ScalingPolicys in the CloudFormation template that has MinInstancesInService set via a CloudFormation Parameter
2. Find current DesiredCapacity of ASG that relates to the Scaling Policy
3. Set the CFN Parameter identified in step 1 to the value identified in step 2

[jacobwinch]

I wonder if our Aspect could add Outputs for Riff-Raff to read.

This would mean that step 1 can be simplified, as Riff-Raff would not need to parse the full template or lookup the associated ASG name (which will not be included in the template unless this property is set).

For example:

Outputs:
  MinInstancesInServiceForAppA:
    Description: The AutoScaling group name associated with the MinInstancesInServiceForAppA parameter.    
    Value: !Ref AsgForAppA 
  MinInstancesInServiceForAppB:
    Description: The AutoScaling group name associated with the MinInstancesInServiceForAppB parameter.    
    Value: !Ref AsgForAppB

I think we can obtain these outputs pretty easily via the existing describeStack function and IIUC using !Ref here will mean that we can avoid looking up the ASG name ourselves.

[akash1810]

Oh interesting idea! Tempted to make these changes in a separate PR once we've know more about what the changes in Riff-Raff would look like. WDYT?

[jacobwinch]

Tempted to make these changes in a separate PR once we've know more about what the changes in Riff-Raff would look like. WDYT?

Yes, that sounds good to me 👍

[jacobwinch]

Great work 💯

[akash1810]

Usage of this experimental pattern can be found via Service Catalogue.