Resources as a Map Text Resource for more powerful manipulations.

[IamfromSpace]

First off, this library is awesome. This is so much leg-work done for creating templates, and it's very cool.

Issue

However, in making more complex templates, I find that the distinct types per resource to be troublesome in making more complex abstractions. I think if there was a generic Resource type, where Properties was always JSON, and then there were smart constructors per resource type, then that would allow for constructing Resources as Map Text Resource which would overall be more flexible--and wouldn't preclude anyone who wanted to still use the Record based approach.

The specific use cases that I've run into are around dynamic resources--ones that may or may not be present in the template based on some other parameters. What I found was that Optional gets me pretty far, but doesn't quite get me all the way. It works for a case like this:

let SpecificResourceType = ...

in \(includeResource: Bool) ->
  { Resources =
    { MyResource =
      if   includeResource
      then Some
             SpecificResourceType.Resource::{
             , Properties = SpecificResourceType::Properties::{=}
             }
      else None SpecificResourceType.Resource.Type

However, we may have a case where we want to map over a list, and then each Resource needs a different Logical ID. An example might be if you were to make a couple different S3 buckets with slightly different function, or Lambdas, or Subnets, or etc. While you could break this out into multiple templates, there may be reason that you want them to all be deployed together, etc.

The Record approach not allowing us to use dynamic names (as far as I can tell, I'm fairly new to Dhall), also prevents us from getting some nice reuse when using Ref or GetAtt to ensure that we keep things DRY and reduce the chance of error.

Idea

What I've been doing is by hand making new smart constructors per resource that accept both a generic ResourceOptions type and the resource specific Properties type (exactly the same one defined in this lib currently). Then it converts the Properties into the JSON type and combines it all into a Resource type (which more or less matches the current Resource types, just Properties is now JSON). This lets me do all three approaches: Records, Records with Optionals, fully dynamic Maps. I personally think the syntax comes out pretty clear, no real compromises there.

Record Approach

let ResourceOptions = ...

let SpecificResourceType = ...

in { Resources = 
  { MyResource =
    SpecificResourceType.Constructor
      ResourceOptions::{=}
      SpecificResourceType.Properties::{=}
   }
}

Record with Optionals for Dynamically Present Resources

let Resource = ...

let ResourceOptions = ...

let SpecificResourceType = ...

in \(useResource : Bool) ->
  { Resources = 
    { MyResource =
        if   useResource
        then Some (
               SpecificResourceType.Constructor
                 ResourceOptions::{=}
                 SpecificResourceType.Properties::{=}
             )
        else None Resource
     }
  }

Map For Fully Dynamic Resources

let Resource = ...

let ResourceOptions = ...

let SpecificResourceType = ...

in \(names : List Text) ->
  { Resources =
    List/map
      Text
      Resource
      \(name : Text) ->
        { mapKey = name
        , mapValue = 
            SpecificResourceType.Constructor
              ResourceOptions::{=}
              SpecificResourceType.Properties::{=}
        }
      names
  }

Note that since we now use Text for our Logical IDs, names here could also go directly into a DependsOn ResourceOption in another resource, and would always stay synced as we modify our template.

Discussion

The hardest part I think here would be to generate the code that maps the specific resource Properties into JSON, but it is of course much harder to do by hand; I've been doing it a fair amount to try this out! However, this library seems pretty primed to be able to accomplish something like this, and the transforms seem fairly consistent. It would be much preferred over doing it by hand as well, since that's quite error prone when going into a generic type like JSON. Also, I'd be happy to contribute to such an effort.

If a breaking change was a concern, this could also be done in a non-breaking way. The Properties object already doesn't change. Each currently defined resource could return an additional key of Constructor (specific name isn't so important) that does the conversion from resource configuration to the common Resource type. Then users can migrate from one strategy to the next on their own time.

Curious to get your thoughts on this approach, before I moved forward with any sort of attempt at something like this. Cheers!

[jcouyang]

Hi @IamfromSpace
IIUC, you are after something similar to the union types in dhall-k8s https://github.com/dhall-lang/dhall-kubernetes/blob/master/1.22/typesUnion.dhall

so in the example, you can do this?

let Resource =  ./typesUnion.dhall
let Bucket = ./AWS::S3::Bucket.dhall
let Function = ./AWS::Lambda::Function.dhall
in \(names : List Text) ->
  { Resources =
    List/map
      Text
      Resource
      \(name : Text) ->
        { mapKey = name
        , mapValue = 
            merge {
               condA = Resource.AWS::S3::Bucket Bucket::{...},
               condB = Resource.AWS::Lambda::Function Function::{}
           } cond
        }
      names
  }

[IamfromSpace]

Yep! That would also work.

Notably, it would be a pretty huge union type (like the kubernetes one), so I hadn't really considered it. Stronger typing guarantees over JSON is also a plus. Defining my own union type is probably what I should be doing in my current templates, because it's a lot easier to add resources as needed, rather than define the JSON transform myself.

However, I think the downside of the union approach in this library is that it's a bit clunky as far as interfaces go, if I'm thinking about it correctly. It's also not necessarily intuitive, because you can and would rather omit it if you have a simple template. An example in the README could help though (ex: "If you have mixed type conditional resources....").

To me, the clunkiness comes from adding yet another layer of nesting, just to appease the type system. In this example, the Resource.AWS::S3::Bucket sure seems like it shouldn't be needed, given that we're using Bucket all over the place:

let Resource = ./typesUnion.dhall

let Bucket = ./AWS::S3::Bucket.dhall

in \(names : List Text) ->
  { Resources =
    List/map
      Text
      Resource
      \(name : Text) ->
        { mapKey = name
        , mapValue =
             Resource.AWS::S3::Bucket
               Bucket.Resource::{
               , Properties = Bucket::Properties::{=}
               }
        }
      names
  }

vs

let Resource = ...

let ResourceOptions = ...

let Bucket = ...

in \(names : List Text) ->
  { Resources =
    List/map
      Text
      Resource
      \(name : Text) ->
        { mapKey = name
        , mapValue = 
            Bucket.Constructor
              ResourceOptions::{=}
              Bucket.Properties::{=}
        }
      names
  }

Notably, it would be easy to make a basic constructor and advanced constructor (and possibly even default) so we could simplify the above to:

let Resource = ...

let ResourceOptions = ...

let Bucket = ...

in \(names : List Text) ->
  { Resources =
    List/map
      Text
      Resource
      \(name : Text) ->
        { mapKey = name
        , mapValue = 
            Bucket.Basic
              Bucket.Properties::{=}
        }
      names
  }

This approach remains consistent no matter which structure you're using. So if you have to switch from a Record to a Map one day, the migration is fairly simple.

Ultimately, I still prefer the Constructor (or some other name) to JSON pattern from a user interface perspective, but the union type is undoubtedly easier from an implementation standpoint. Curious to hear your thoughts.

Cheers!

[jcouyang]

it would be a pretty huge union type ... Defining my own union type

yes, but they can be easily generated, user don't need to define them, this also keeping the pattern consistence with dhall-k8s, which make it less pain working in a repo with both k8s and cf templates(most of my repos use both actually)

the clunkiness comes from adding yet another layer of nesting

union type is optional, so if all of your resource type is Bucket, you can just omit it still compiles

- let Resource = ./typesUnion.dhall

let Bucket = ./AWS::S3::Bucket.dhall

in \(names : List Text) ->
  { Resources =
    List/map
      Text
-      Resource
+     Bucket
      \(name : Text) ->
        { mapKey = name
        , mapValue =
-             Resource.AWS::S3::Bucket
               Bucket.Resource::{
               , Properties = Bucket::Properties::{=}
               }
        }
      names
  }

my main concert of introducing json will make it loose type, if the constructor create a JSON type, there is nothing to prevent user create their own JSON, and everything still type check, suddenly lost all the point of using dhall

in \(names : List Text) ->
  { Resources =
    List/map
      Text
      Resource
      \(name : Text) ->
        { mapKey = name
        , mapValue =  JSON.object (tomap {not-a-valid-cloudformation-resource = JSON.string "still compiles"})
        }
      names
  }

[IamfromSpace]

I wrote out a whole reply justifying JSON, and then realized there was a way to sort of combine everything we've talked about into a best of both worlds approach without it, haha.

I think if we combine a large union type and then smart constructors, we can get it all: strong typing, intuitive interface, consistent pattern across outer structures. The union type would be on the Properties. The Resource type would look like this:

{ DeletionPolicy : DeletionPolicy
, UpdateReplacePolicy : UpdateReplacePolicy
, UpdatePolicy : UpdatePolicy
, CreationPolicy : CreationPolicy
, MetaData : Map Text Text
, Condition : Optional Text
, Type : Text
, Properties : PropertiesUnion.Type
}

Each type module would then expose a Basic and Advanced constructor. The first one only accepts the associate Properties type and the second one accepts a ResourceOptions type (which is just like like Resource but omits Type and Properties). The smart constructors wrap the properties in the correct union constructor and Resource automatically for you. No large JSON conversion needed, so substantially easier to implement.

The S3 Bucket module then looks something like this:

let ResourceOptions = ...

let PropertiesUnion = ...

let resourceBuilderUtil = ...

let Properties : Type = { Name : Optional Text, ... }

let Advanced =
    \(resourceOptions : ResourceOptions) ->
        \(properties : Properties) ->
            resourceBuilderUtil
                "AWS::S3::Bucket" 
                resourceOptions
                (PropertiesUnion.Bucket properties)

let Basic = Advanced ResourceOptions::{=}

in
    { Properties
    , Basic
    , Advanced
    }

I omitted it, but the current Resource export can be retained for backwards compatibility.

I think this both seems pretty doable and appealing. And it clarifies again how I should be doing this on my end in the short term and saves a lot of work, haha, so I appreciate the discussion for that alone! :) Thoughts on this?

[IamfromSpace]

So I think we can basically do everything and stay backwards compatible (something I'm a firm believer in!).

Basically the new Resource, ResourceOption, PropertyUnion, and ResourceUnion types would live at the root level next to Fn (or somewhere similar). Each current CF Resource module stays exactly how it is now, but it adds in the Basic and Advanced smart constructors.

So fleshing out something like a Role module (because it's simpler than a Bucket), it could look something like this:

let ResourceOptions = (./../Resource.dhall).Options

-- Parameterized by code gen
let Properties = ./AWS::IAM::Role/Properties.dhall

-- Parameterized by code gen
let Tag = (./../PropertiesUnion.dhall).Role

-- Parameterized by code gen
let cfType = "AWS::S3::Bucket"

-- Generation is always identical
let Advanced =
    \(resourceOptions : ResourceOptions) ->
        \(properties : Properties) ->
            resourceOptions // { Type = cfType, Properties = Tag properties }

-- Generation is always identical
let Basic = Advanced ResourceOptions::{=}

in
    { Advanced
    , Basic
    , Properties
    , Resources = ./AWS::IAM::Role/Resources.dhall
    , Policy = ./AWS::IAM::Role/Policy.dhall
    , GetAttr =
        { Arn = (./../Fn.dhall).GetAttOf "Arn"
        , RoleId = (./../Fn.dhall).GetAttOf "RoleId"
        }
    }

So it's ultimately just a net add of new Record keys, union modules, and a new generic Resource type. Code gen is also quite simple, you can see it's just a couple parameterized lines beyond what's already there.

I've tried out something like this on my side and it's working quite well! The resulting template construction is nice and straight forward (typically I can just use the Basic constructor, which lets me jump right into specifying the resource specific properties), and I can mix and match or move seamlessly between Maps and Records without any alterations.

[jcouyang]

I mean backward compat for user, not module, for current user, at least for me, i don't want to rewrite all my existing CF to wrap a Basic just for upgrade

I'm still leaning towards the uniontypes pattern dhall-k8s using, for

1. consistent pattern, it is usually the same group of users who uses k8s and aws at the same time, keeping the same pattern reduce their learning and context switching effort
2. backward compat, the chance of the advance usecase you mention here is valid, but the not most common(at least myself write a lot of CF and yet haven't encountered any, i reckon it is not the most common usecase, 99% of time I may just need Basic). changing design and pattern dramatically for minor usecase isn't ideal
3. gen union types is minor effort, maybe less than ten lines of code will just do it.

[IamfromSpace]

So I also mean backwards compatibility for the user. I'm not suggesting anything where a user must change code to upgrade. Everything could be opt-in. I'm instead saying that all patterns can be simultaneously present, and it would be easy to support them all for substantial time.

So everyone doing something like:

{ Resources =
  { MyRole = 
     Role.Resources::{
       Properties = Role.Properties::{
         ...
       }
     }
   }
}

Would need no changes.

You could also optionally use or upgrade to the pattern I'm suggesting:

{ Resources =
   { MyRole =
     Role.Basic
       Role.Properties::{
         ...
       }
   }
}

And if you wanted to enable the k8s pattern (personally, I find the interface much clunkier, so I would recommend against it), you could make that available also simultaneously to all users:

{ Resources =
  { MyRole =
     ResourcesUnion.Role
       Role.Resources::{
         Properties = Role.Properties::{
           ...
         }
       }
   }
}

Totally possible to make all three patterns work all at the same time.

My take is:

1. It's not worth looking like another lib if you can do it better (and if you really want to, you can support it all).
2. Current users can upgrade without forced changes.
3. Low effort, and I'm happy to do it; I just didn't want to start if it doesn't have a chance of merge (but if you're undecided I'm also happy to "prove it" and go for a PR--don't need you to commit to it here).

[jcouyang]

all three patterns work all at the same time

This is exactly what I am worried about

Look, I'm standing in user's foot here, and my first two points are also from user's perspective. 3 ways of doing thing is not user friendly and totally unnecessary.

Imagine this release, and there will be some config using the traditional way, some config using Basic constructor, some using Advanced constructor, some using k8s union type wrapper. I wouldn't call this compat, this will quickly become a mess

when new members come in the team, who only knows json, previously i just need to tell them to write cf in dhall you just need to wrap them in the correct type, 99% of chance they dont need to know how to construct an advanced resource, even if they do we can just point the k8s folder and say just copy what those k8s config were doing

After the release they will then need to learn 4 ways of creating resources, and have to keep in mind when to use what patterns

When writing k8s, union pattern
When maintaining old aws dhall, traditional patten
When writing new aws dhall choose from Basic or Advanced constructor

[IamfromSpace]

Ah, well, in that case it seems like it's not an issue with backwards compatibility, and instead a preference for the current interface.

I'm not suggesting to keep all patterns forever; I would suggest deprecating and discouraging the current while promoting the new. Simultaneous support is only to offer users the ability to transition gradually at their own pace.

It sounds as though you prefer the CF mirroring interface (which I do see the benefit of). Personally, I think the smart constructor approach leverages the strengths of Dhall more and lets the user jump right into specifying the parts they care about. I also think the user that doesn't really understand Dhall will only be frustrated by additional wrappers required should they ever need a Map, since that doesn't look like CF.

[IamfromSpace]

Thinking this through further, I think there's yet another approach worth considering, which preserves the more CF-like interface.

In many ways, my preference for a smart constructor of type Options -> Properties -> Resource is really just stylistic--and style is not worth an interface change.

Ultimately the user's syntax of:

X
  Options::{=}
  Properties::{=}

Just isn't noticeably different from:

X
  Resources::{
    Properties::{=}
  }

So I agree that's not worth it.

I do think though that there's value in a helper that allows you to just specify Properties directly (I find this is all I need in most cases). If that could return a generic type that fits in a Map, well that would be nice too. This thought lets us merge the two idea.

So consider still adding a generic Resource (which uses a PropertiesUnion), but omit the Options suggestion. We can leverage SomeResourceType.Resources that exists today. Each resource type module adds a FromResources and FromProperties, where the latter is just a convenience. The former converts the modules specific resource type to the generic one.

So the Role module would look like this:

let Tag = (./../PropertiesUnion.dhall).Role

let Resources = ./AWS::IAM::Role/Resources.dhall

let Properties = ./AWS::IAM::Role/Properties.dhall

let FromResources =
    \(resource : Resources) ->
        resource // { Properties = Tag resources.Properties }

let FromProperties =
    \(properties : Properties) ->
        FromResources
            Resources::{ Properties = properties }

in
    { Properties,
    , Resources,
    , FromProperties
    , FromResources
    , Policy = ./AWS::IAM::Role/Policy.dhall
    , GetAttr =
        { Arn = (./../Fn.dhall).GetAttOf "Arn"
        , RoleId = (./../Fn.dhall).GetAttOf "RoleId"
        }
    }

Everything here permanently co-exists (and that's desirable, because now they work together). Current users need no changes today or tomorrow. If users need to move to a Map, they just need to wrap each resource in the FromResources constructor. This is a pretty close parallel to the k8s approach, so it should feel the same to users who know that pattern. For new users that only require Properties, these users now have a more convenient way to do so, and it just so happens be Map-ready.

This is also easy to implement; Tag is the only parameterized part of the code gen.

Notably, you could get basically the exact same interface with ResourcesUnion instead, but a PropertiesUnion feels like a much more "true" model of CF. The apparent downside of a PropertiesUnion is seemingly that you can't make it look like the k8s approach, but with the smart constructor style, you can get both.

Hopefully you've not already grown too tired of me, haha. I think this checks a lot of boxes.

[jcouyang]

co-exists is the issue

from users perspective

FromResources Bucket.Resource::{...}

vs

AWS::S3::Bucket Bucket.Resource::{...}

there isn't huge improvement, to me tbh they are pretty much the same thing

the later introduce nothing new, but FromResources and FromProperties bring little or no benefit but

After the release they will then need to learn 4 ways of creating resources, and have to keep in mind when to use what patterns

[IamfromSpace]

Ah, I am recommending against the ResourceUnion, as it’s not necessary. Instead of FromResources, you can name the smart constructor after the type.

[jcouyang]

sorry i still don't really get where this against union coming from, even inside FromResource union is used anyway, on contrary i found it is unnecessary to wrap the union inside a lambda and give it a custom name, besides in your example PropertiesUnion.dhall will be exposed anyway, there is nothing preventing user to use it directly in the union pattern, hence this level of encapsulation seems meaningless to me

as a lib it is impossible to accommodate all users' preference, it is more important to keep it basic, idiomatic to cloudformation and beginner friendly. usually beginner has less preference, but if there are too many ways of doing one thing they will get confused. for advanced user with strong preference(i really doubt there are many, though), i don't think it will be difficult for them to create their own favour way of constructing resource in their own project, like your example, it is just a standalone lambda they can put anywhere in their project and name with their favourite name

also for api design can't find any benefit of using custom lambda constructor wrapping union vs union, later is language native feature, the former is custom while not easier to use than later, instead it introduces more ways of doing one thing, which will bring inconsistency and unnecessary complexity to existing codebase

[IamfromSpace]

It is a good question, because what I'm suggesting at this point is actually quite close to just adding a ResourcesUnion.

The only differences (where the last one could be independent):

1. Use a union on Properties instead of Resources
2. Use a smart constructor to mimic the Resource's union tag (which is only a meaningful difference if it has a generic name)
3. Add a smart constructor for a Properties-only use case.

The foundation of the recommendation, I suppose, is really just that a Properties union feels like a much more idiomatic model of the data. That the Resource type is in fact common, and it's the Properties that vary. This also partly informs my preference of a ResourceOptions type, in that it drops the Type, which sort of must be defaulted to be useful, as it could otherwise mismatch the type of Properties supplied. I'm not suggesting a ResourceOptions type at this point, simply illustrating my preference for an idiomatic model rather than a model that looks like what CF looks like. If a PropertiesUnion is introduced, it's possible to be both idiomatic and similar to the CF.

The FromProperties to me is the kind of thing that users like when they can do things multiple ways. It's just the easier way with less boilerplate. The way that one moves from the easy way to the more configurable way seems fairly natural to me. As an external user, I think FromProperties is quite a bit work to add, or if a generic version is added, then type annotations start to defeat its purpose.

I have appreciated the conversation btw, it's certainly helped elucidate the ways to do things on my side. I think at this point I've made my case as fully as I can, and it is of course simply my own opinion, which has been incorrect from time to time, haha. If I haven't convinced you at this point, I'm okay with that :)