Deployments subsystem

[gabrielmougard]

This is based on #11577

Deployment subsystem

(Work In Progress) This is the initial work to introduce the so-called Deployment API for LXD.

Forum specification : https://discourse.ubuntu.com/t/introducing-the-deployment-api/38854

Database schemas

These are the new SQL tables:

erDiagram
    deployments ||--o{ deployment_configs : "0:N"
    deployments ||--o{ deployment_shapes : "0:N"
    deployments ||--o{ deployment_keys : "0:N"
    deployment_shapes ||--o{ deployment_shape_instances : "0:N"
    deployment_shapes ||--o{ deployment_shape_configs : "0:N"
    deployments {
        INT id PK
        TEXT_NOT_NULL name
        TEXT_NOT_NULL description
        TEXT scaling_webhook_url
        INT_NOT_NULL project_id
    }
    deployment_configs {
        INT id PK
        INT deployment_id FK
        TEXT_NOT_NULL key
        TEXT_NOT_NULL value
    }
    deployment_shapes {
        INT id PK
        INT deployment_id FK
        TEXT_NOT_NULL name
        TEXT_NOT_NULL description
        INT_NOT_NULL scaling_minimum
        INT_NOT_NULL scaling_maximum
        TEXT_NOT_NULL instance_template 
    }
    deployment_shape_configs {
        INT id PK
        INT shape_id FK
        TEXT_NOT_NULL key
        TEXT_NOT_NULL value
    }
    deployment_shape_instances {
        INT instance_id FK
        INT shape_id FK
    }
    deployment_keys {
        INT_NOT_NULL id PK
        INT_NOT_NULL deployment_id FK
        TEXT_NOT_NULL name
        TEXT_NOT_NULL description
        INT_NOT_NULL certificate_id
        TEXT_NOT_NULL role
    }

Loading

Important deployment lifecycle scenarios

Creating a Deployment

sequenceDiagram
    participant User as User
    participant LXD as Platform (LXD)
    participant DB as Database
    
    User->>LXD: Create Deployment (k8s-cluster01)
    LXD->>DB: Store Deployment Configuration
    Note over LXD, DB: Stores k8s-cluster01 deployment config

Loading

Adding Shapes to a Deployment

sequenceDiagram
    participant User as User
    participant LXD as Platform (LXD)
    participant DB as Database
    
    User->>LXD: Add Shape (k8s-api)
    LXD->>DB: Store Shape Configuration
    Note over LXD, DB: Stores min:3, max:5, config...
    
    User->>LXD: Add Shape (k8s-kubelet)
    LXD->>DB: Store Shape Configuration
    Note over LXD, DB: Stores min:1, max:15, config...

Loading

Governor reconciling with a deployment size requirements

sequenceDiagram
    participant Governor as MicroK8s/Governor
    participant LXD as Platform (LXD)
    participant DB as Database
    
    par Maintain minimum deployment size
        loop Deployment shapes are not scaled to their minimum
            Governor->>LXD: Query Deployment Info
            LXD->>DB: Retrieve Deployment and Shape Info
            LXD-->>Governor: Return Deployment and Shape Info

            Governor->>LXD: trigger platform to spawn instances in desired deployment shapes
            LXD->>DB: Spawn instances and create new deployment shape instance records 
        end
    and Look out for scale-up opportunities
        loop scale-up opportunities detected
            Governor->>LXD: Query Deployment Info
            LXD->>DB: Retrieve Deployment and Shape Info
            LXD-->>Governor: Return Deployment and Shape Info
            Note over Governor: Governor decides to scale the instances up
    
            Governor->>LXD: POST API Call to Scale Instances
            Note over LXD: Validates and Processes Scaling Request
            LXD->>DB: Create deployment shape instance record and launch the instance
            Note over LXD, DB: Reflects the newly scaled instances and their configurations
        end
    and Look out for scale-down opportunities
        loop scale-down opportunities detected
            Governor->>LXD: Query Deployment Info
            LXD->>DB: Retrieve Deployment and Shape Info
            LXD-->>Governor: Return Deployment and Shape Info
            Note over Governor: Governor decides to scale the instances down

            Governor->>LXD: DELETE API Call to Remove Instances
            Note over LXD: Validates and Processes Delete Request
            LXD->>DB: Delete deployment shape instance record a delete the instance
            Note over LXD, DB: Reflects the deleted instances
        end 
    end 

Loading

Deployment key lifecycle

sequenceDiagram
    participant Admin as Admin
    participant LXD as Platform (LXD)
    participant DB as Database
    participant Governor as MicroK8s/Governor
    
    Admin->>LXD: Create deployment key for an existing deployment
    Note over Admin, LXD: Admin uses a deployment_key for securing the deployment
    
    LXD->>DB: Generate and store the deployment key for the deployment
    Note over DB: Deployment is stored securely with the deployment_key
    
    Admin->>LXD: Assign Roles and Permissions
    Note over Admin, LXD: Each deployment has a role and admins can manage permissions for deployment end users 
    
    LXD->>DB: Store Roles and Permissions
    Note over DB: Stores the assigned roles and permissions for the deployment
    
    Governor->>LXD: Request Access with deployment key
    Note over Governor, LXD: Governor uses the deployment key to request access
    
    LXD->>DB: Validate deployment key and retrieve permissions
    Note over DB: Validates the deployment key and retrieves associated permissions
    
    LXD-->>Governor: Grant/Deny Access with Assigned Permissions
    Note over Governor: Receives access with permissions based on the valid deployment key

Loading

API routes

deployments related:

• GET /1.0/deployments
  • List the deployments (filtering with projects and deployment names available). No recursion URL parameter will return a list of the deployment URLs, recursion=1 will return the list of the deployment API objects.
• POST /1.0/deployments
  • Add a deployment
• DELETE /1.0/deployments/{deploymentName}
  • Delete a deployment
• GET /1.0/deployments/{deploymentName}
  • Get the specific details of a deployment
• PUT /1.0/deployments/{deploymentName}
  • Update a deployment (the update replace and existing deployment with the new one)
• PATCH /1.0/deployments/{deploymentName}
  • Update a deployment (the update merge the existing deployment's config with the new one)
• POST /1.0/deployments/{deploymentName}
  • Rename a deployment

shapes related:

• GET /1.0/deployments/{deploymentName}/shapes
  • List the shapes for a given deployment. No recursion URL parameter will return a list of the shape URLs for a given deployment, recursion=1 will return the list of the shape API objects for a given deployment.
• POST /1.0/deployments/{deploymentName}/shapes
  • Creates a new shape within a deployment.
• DELETE /1.0/deployments/{deploymentName}/shapes/{shapeName}
  • Delete a shape.
• GET /1.0/deployments/{deploymentName}/shapes/{shapeName}
  • Gets a specific shape.
• PUT /1.0/deployments/{deploymentName}/shapes/{shapeName}
  • Updates a specific shape.
• POST /1.0/deployments/{deploymentName}/shapes/{shapeName}
  • Rename a specific shape.

shape_instances related:

• GET /1.0/deployments/{deploymentName}/shapes/{shapeName}/instances
  • List instances in a shape. No recursion URL parameter will return a list of the instance URLs, recursion=1 will return the list of the instance API objects.
• POST /1.0/deployments/{deploymentName}/shapes/{shapeName}/instances/{instanceName}
  • Creates a new instance (with the name instanceName) within this shape. If the shape current size allow a one unit add-up, then the to-be-created instance will be launched using the shape instance_template.
• DELETE /1.0/deployments/{deploymentName}/shapes/{shapeName}/instances/{instanceName}
  • Deletes an instance in the given shape.
• PUT /1.0/deployments/{deploymentName}/shapes/{shapeName}/instances/{instanceName}/state
  • Update the state of an instance (start / stop / etc.)

CLI

deployment
├─ list [--format]
├─ show [<remote>:]<deployment_name>
├─ get [<remote>:]<deployment_name> <key/property> [--property]
├─ set [<remote>:]<deployment_name> (<key1>=<value1> <key2>=<value2> ...) [--property]
├─ unset [<remote>:]<deployment_name> (<key1/property1> <key2/property2> ...) [--property]
├─ create [<remote>:]<new_deployment_name> 
├─ edit [<remote>:]<deployment_name>
├─ rename [<remote>:]<deployment_name> <new_deployment_name>
├─ delete [<remote>:]<deployment_name>
├─ key
     ├─ list [<remote>:]<deployment_name>
     ├─ show [<remote>:]<deployment_name> <key_name>
     ├─ get [<remote>:]<deployment_name> <key_name> <key/property> [--property]
     ├─ create [<remote>:]<deployment_name> <new_key_name> <certificate_full_fingerprint> [--role]
     ├─ rename [<remote>:]<deployment_name> <key_name> <new_key_name>
     ├─ delete [<remote>:]<deployment_name> <key_name>
├─ shape
     ├─ list [--format]
     ├─ show [<remote>:]<deployment_name> <shape_name>
     ├─ create [<remote>:]<deployment_name> <new_shape_name> [--from-profile <instance_profile>, --from-image <remote:alias>, --scaling-min <min>, --scaling-max <max>, --vm]
     ├─ get [<remote>:]<deployment_name> <shape_name> <key/property> [--property]
     ├─ set [<remote>:]<deployment_name> <shape_name> (<key1>=<value1> <key2>=<value2> ...) [--property]
     ├─ unset [<remote>:]<deployment_name> <shape_name> (<key1/property1> <key2/property2> ...) [--property]
     ├─ edit [<remote>:]<deployment_name> <shape_name>
     ├─ rename [<remote>:]<deployment_name> <shape_name> <new_shape_name>
     ├─ delete [<remote>:]<deployment_name> <shape_name>
     ├─ instance
           ├─ launch [<remote>:]<deployment_name> <shape_name> <instance_name>
           ├─ delete [<remote>:]<deployment_name> <shape_name> <instance_name>
           ├─ list [<remote>:]<deployment_name> <shape_name> [--format]

TODO - features

• Fix // TODO
• Continue validation logic in lxd/deployments/deployment.go for shape and dep key and remove useless DB methods (e.g: DeleteDeployedInstance, because of the cascade relationship with the instances table)
• Each time there is a API call related to a deployment or a deployment shape, read and update tthe value of scaling_current (for one or multiple deployment shapes)
• shapes -> deployment_shapes , shape_instances -> deployment_shape_instances
• Add Config to shapes -> add DB logic
• Implement the REST endpoints for deployment_keys in this PR as well
• Make sure the governor facing calls are protected by JWT auth (making use of the deployment key).
• Effectively create / delete the instances within a deployment shape (if authenticated)
• Allow instance profiles to be used for a deployment shape instance template. (maybe a --from-profile <profile> flag would be nice when creating the deployment shape)

TODO - integration tests

deployment

• deployment creation:
  • empty creation
  • creation with --description flag
  • creation with --governor-webhook-url flag
  • creation from stdin :
    • with basic fields,
    • basic fields + config
• deployment list: with csv, json,yaml format
  • See if UsedBy is populated
• deployment show: we need to have a complete deployment with config, shapes(+config and instance template), deployment keys
  • See if UsedBy is populated
• deployment get: on config key and properties - with good and bad cases
• deployment set/unset: on config key and properties - with good and bad cases
• deployment edit : edit a good config and then get+show to know its effective.
• deployment rename: good and bad names
• deployment delete:
  • simple existing (no running instances), simple non existing
  • existing (with running instances)

deployment key

• deployment key creation:
  • empty creation: check acces key / secret key display
  • creation with --description flag
  • creation with --role flag
  • creation with stdin: no complex fields so should be easy
• deployment key list: with csv, json,yaml format (match the access keys)
• deployment key show: check that we have access key but no secret key
• deployment key get/set/unset/edit: only easy fields (properties not config) and should be impossible to set/unset access key
• deployment key rename: good and bad names
• deployment key delete: existing and non existing cases

deployment shape

• deployment_shape creation:
  • empty creation
  • creation with --description flag
  • creation with --scaling-min, --scaling-max, --from-profile flag
  • creation from stdin :
    • with basic fields,
    • basic fields + config
    • basic fields + config + instance template
• deployment_shape list: with csv, json,yaml format
• deployment_shape show: we need to have a complete deployment with config, instance template
• deployment_shape get: on config key and properties - with good and bad cases
• deployment_shape set/unset: on config key and properties - with good and bad cases
• deployment_shape edit : edit a good config and then get+show to know its effective.
• [NOT WORKING - old name still there and config erased]deployment_shape rename: good and bad names
• deployment_shape delete:
  • simple existing (no running instances), simple non existing
  • existing (with running instances)

deployment shape instance

• deployment shape instance launch: launch on bad deployment name, bad shape name, bad instance name, launch with bad scaling check, launch good case
• deployment shape instance delete: delete on bad deployment name, bad shape name, bad instance name, delete with bad scaling check, delete good case
• deployment shape instace list: list deployed instances on bad deployment name, bad shape name, bad instance name, list good case

Real user cases

• Create a deployment, 2 shapes (simple instance template) in it, 2 deployment keys (one with rw, other with ro access), spin up a minimal VM with LXD installed in it, go in it and add the host LXD as a remote, in this same VM try to call the JWT protected route with our private / public keys (how to do that in bash ? we can install this go-jwt cmd tool maybe) and play with governor facing calls to scale up and down.

[tomponline]

@gabrielmougard havent look at the code yet, but one thing I can say is that the top-level spec https://docs.google.com/document/d/1o4IH8mZgoRgD04buNYrXUB-ScPoiy3Nl2EbiTLu9hpU/edit#heading=h.vvl3wabtfsfn replaced "instance sets" with "shapes", so we should update the LXD specific spec and the implementation to reference shapes rather than instance sets.

[gabrielmougard]

@gabrielmougard havent look at the code yet, but one thing I can say is that the top-level spec https://docs.google.com/document/d/1o4IH8mZgoRgD04buNYrXUB-ScPoiy3Nl2EbiTLu9hpU/edit#heading=h.vvl3wabtfsfn replaced "instance sets" with "shapes", so we should update the LXD specific spec and the implementation to reference shapes rather than instance sets.

I'll change that thanks. And I'll also update the spec for the future work

[markylaing]

@gabrielmougard You can use this commit to get tests to pass 77fd6a6

There are a few changes:

• Checking for deployment type certificates in the TLS authorization driver.
• Changing all deployment API access handlers to allowAuthenticated (we can discuss how to integrate these with the fine-grained authorization when you are back).
• A hacky fix for renaming deployment keys. These we're being renamed against the ID of the deployment, not the ID of the key.
• Some test fixes for invalid config keys and/or invalid scaling values.

[tomponline]

thanks @markylaing !

[markylaing]

thanks @markylaing !

No worries 👍 @gabrielmougard we'll have to have a discussion about authorization before this is merged as we'll need to perform a migration (in this case a simple one thankfully, as we're just adding new types to the model)

[tomponline]

Yep agreed!

[tomponline]

Closing for now but please keep the branch you have.