Environmental restrictions on Role creation

[kingdonb]

My environment has a restriction on role creation, so I'm unable to proceed past eksp-up.sh which tries to create several roles, in the eksp-stack.yaml

I've reached the point in the process where I'm seeing The following resource(s) failed to create: [DestroyClusterFuncRole, ProlongFuncRole, StatusFuncRole, CreateClusterFuncRole]. . Rollback requested by user. and the reason is API: iam:CreateRole User: arn:aws:sts::209773529123:assumed-role/cluster-api-role/cluster-api-role is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::209773529123:role/eksp-ProlongFuncRole-1O2WDWDVOYYNB

In summary my environment requires us to attach a permissions boundary to any roles that are created for our use; there are several permissions boundaries available for us and we just have to choose one off the shelf.

I have assumed an admin role which grants unlimited permissions except for several small things (that might or might not turn out to be a problem later on in the install process) like changing VPC structures is forbidden, deleting logs or creating access keys, limiting the size of instances that I may create, and in what regions, generally for safety of our environment. (I'm also forbidden from changing the boundaries themselves, or creating other roles which do not attach the boundary.)

I was going to try to just attach the permissions boundary myself but I'm unable to find the place in the eksphemeral code where this role is actually created. I think it might be something abstracted behind the SAM tools which were required to install first too, (but are not mentioned anywhere in the docs that I could find.)

Right now this is just a drive-by issue report as I am interested in deploying EKSphemeral for my team to use, alongside several other methods we have for getting our hands on a Kubernetes cluster to use for development and training purposes. I have some difficult requirements that might not be on your roadmap, but I hope that we can use this tool! I've developed something similar for another team, but much more rudimentary, and it has proved valuable to be able to provision disposable (auto-disposed) clusters with as little friction as possible.

I like your approach in particular because it is so lightweight. Also that it allows us to spin up disposable instances of potentially expensive resources like Kubernetes clusters without actually forcing us to worry about turning the lights off when we're done!

This is a great tool, and I think I will try it in my less-restricted personal account environment instead to see if I can find any other blockers that might prevent me from actually using it.

[mhausenblas]

Thanks a lot for your feedback and super excited to learn you looking into using EKSphemeral! Yeah, the roles are managed through SAM/CF and AFAICT SAM does support permission boundaries. I'll leave this issue open as a reminder, either I'll add an option myself or maybe you want to PR it ;)

[kingdonb]

Thank you for the pointer to the right docs! I will spend some time digging into that and how these functions are called from eksphemeral; I did find SAM mentioned in the developer docs, but it looks like it's not only a developer requirement, also needed for runtime admin user who doesn't build anything from source. Much appreciated, I will follow up soon!

[kingdonb]

I am tracking my progress here, so far I added permissions boundaries to the template.yml and eksp-stack.yml by hand, froze my ~/eksp in place, and it worked! I got through to a state where the stack is up and responds to eksp list at least.

Now I've realized that my aws cli is too old to manage and list EKS clusters, and this wasn't obvious from the error message that basically just said this, so I upgraded it.

$ eksp create cluster-spec.json
Trying to create a new ephemeral cluster ...
... using cluster spec cluster-spec.json
Seems you've set 'us-east-1' as the target region, using this for all following operations
Pre-flight check failed: EKS is not available in
Something bad happened after command completed:: exit status 1

Next is another issue related to permissions boundaries,

$ eksp create cluster-spec.json
Trying to create a new ephemeral cluster ...
... using cluster spec cluster-spec.json
Seems you've set 'us-east-2' as the target region, using this for all following operations
I will now provision the EKS cluster mh9-eksp using AWS Fargate:

AccessDenied: User: arn:aws:sts::209773529123:assumed-role/cluster-api-role/cluster-api-role is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::209773529123:role/ecsTaskExecutionRole
	status code: 403, request id: 221efa44-bf79-4f0c-b71c-d970a7f8dc3b
Something bad happened after command completed:: exit status 1

I will try to track down the place in the source where this happens, and add the role boundary there as well. Thanks very much for your help with this!

[kingdonb]

So, I found eksctl-io/eksctl#1221 it looks like this is an open issue already on EKSctl too.