Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(experimental-ec2-pattern): Pattern to deploy ASG updates w/CFN #2417

Merged
merged 23 commits into from
Sep 17, 2024
Merged

feat(experimental-ec2-pattern): Pattern to deploy ASG updates w/CFN #2417

merged 23 commits into from
Sep 17, 2024

Commits on Sep 17, 2024

  1. fix(riff-raff.yaml): Do not deploy ASG w/update policy 

    ASGs with an update policy will get deployed via CloudFormation,
instead of Riff-Raff's `autoscaling` deployment type.
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    0ccfcef View commit details
    Browse the repository at this point in the history

  2. feat(experimental-ec2-pattern): Add pattern to deploy ASGs updates vi… 

    …a CloudFormation (`AutoScalingRollingUpdate`)
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    24df0b2 View commit details
    Browse the repository at this point in the history

  3. test(riff-raff.yaml): Test GuEc2AppExperimental behaviour

    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    10701fd View commit details
    Browse the repository at this point in the history

  4. feat(experimental-ec2-pattern): Decorate added user data commands w/m… 

    …arkers

This should make it easier to parse a user data string if ever one is debugging.
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c83449c View commit details
    Browse the repository at this point in the history

  5. docs(experimental-ec2-pattern): Add doc string to class

    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    de842f6 View commit details
    Browse the repository at this point in the history

  6. fix(experimental-ec2-pattern): Set ASG DesiredCapacity 

    During a deployment, CloudFormation updates the min and desired.

During a rollback (e.g. if the healthcheck failed), CloudFormation only resets the min.
The desired is still elevated, meaning the service is over provisioned.

Explicitly setting the desired property of the ASG ensures CloudFormation rollback puts the ASG back to the initial state,
e.g. correctly provisioned.
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    3fe10c6 View commit details
    Browse the repository at this point in the history

  7. refactor(experimental-ec2-pattern): Use the same duration when creati… 

    …ng and updating ASG
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    6efd2dc View commit details
    Browse the repository at this point in the history

  8. fix(experimental-ec2-pattern): Suspend alarm notifications 

    A scale-in event fires during a rolling update can cause service disruption.
Suspend scaling events during a rolling update for safety.
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    e089a84 View commit details
    Browse the repository at this point in the history

  9. feat(experimental-ec2-pattern): Adjust ASG rolling update properties … 

    …where scaling policy present

Some practical testing of `AutoScalingRollingUpdate` has demonstrated that
when an ASG has a scaling policy, it is safest to dynamically set the `MinInstancesInService` property.
Add an aspect to do that.

See also https://github.com/guardian/testing-asg-rolling-update.
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    b2d7782 View commit details
    Browse the repository at this point in the history

  10. test(experimental-ec2-pattern): Ensure only a horizontally scaling AS… 

    …G is adjusted
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    22c9e48 View commit details
    Browse the repository at this point in the history

  11. docs(experimental-ec2-pattern): Update doc strings

    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    5ce18cd View commit details
    Browse the repository at this point in the history

  12. feat(experimental-ec2-pattern): Support an ASG w/multiple scaling pol… 

    …icies
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    ddb3a1e View commit details
    Browse the repository at this point in the history

  13. docs(experimental-ec2-pattern): Document thrown error

    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    767ec0f View commit details
    Browse the repository at this point in the history

  14. refactor(experimental-ec2-pattern): Simplify by reusing variable

    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    b8f6ff5 View commit details
    Browse the repository at this point in the history

  15. fix(experimental-ec2-pattern): Obtain instance id more reliably 

    The `ec2metadata` command was failing with a 401 with AMIable CODE in deployTools account:

```console
root@ip-10-248-51-213:/var/lib/cloud/instance# ec2metadata --instance-id
Traceback (most recent call last):
  File "/usr/bin/ec2metadata", line 249, in <module>
    main()
  File "/usr/bin/ec2metadata", line 245, in main
    display(metaopts, burl, prefix)
  File "/usr/bin/ec2metadata", line 192, in display
    value = m.get(metaopt)
  File "/usr/bin/ec2metadata", line 177, in get
    return self._get('meta-data/' + metaopt)
  File "/usr/bin/ec2metadata", line 137, in _get
    resp = urllib_request.urlopen(urllib_request.Request(url))
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unautho
```

This service uses IMDSv2. A 401 response usually happens when a request is made without a token.
However `ec2metadata` does exchange a token.

Switch to a more reliable mechanism.

See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html.
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    b0718b4 View commit details
    Browse the repository at this point in the history

  16. docs(experimental-ec2-pattern): Add changeset

    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    f4e2a7c View commit details
    Browse the repository at this point in the history

  17. refactor(experimental-ec2-pattern): Simplify tests by checking minima… 

    …l properties
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    98f8f9f View commit details
    Browse the repository at this point in the history

  18. refactor(experimental-ec2-pattern): Remove duplicated test 

    The behaviour being tested is already covered by `should only adjust properties of a horizontally scaling service`.
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    37c9533 View commit details
    Browse the repository at this point in the history

  19. refactor(experimental-ec2-pattern): Wrap code example comment in doc … 

    …string
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    a259531 View commit details
    Browse the repository at this point in the history

  20. refactor(riff-raff.yaml): Simplify handling of ASG update policy

    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    7b330b6 View commit details
    Browse the repository at this point in the history

  21. refactor: Extract repeated logic into function

    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    3ebd343 View commit details
    Browse the repository at this point in the history

  22. refactor(experimental-ec2-pattern): Simplify tests by checking minima… 

    …l properties
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    4d96a1a View commit details
    Browse the repository at this point in the history

  23. fix(experimental-ec2-pattern): Set PauseTime from healthcheck grace… 

    … period

Matching these properties allows rollbacks to happen as quickly as possible.
    @akash1810
    akash1810 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    7a12f60 View commit details
    Browse the repository at this point in the history