aws_kinesisanalytics: Configuring a vpc

[JoeKodsi-Idealworks]

Description

Hey,

I have looked thoroughly through aws cdk documentation but couldn't find any way to configure a Kinesis Data Analytics Application with a VPC.

Is there a way to configure the networking of a Kinesis Data Analytics Application (adding vpc + subnets + security group)?

Use Case

This feature is needed in our use case to allow access for the Flink java application to access resources within our VPC.
Say we needed to communicate with AmazonMQ that resides in a private subnet, how would our Kinesis Data Analytics Application reach the queue server?

Proposed Solution

I am not sure how this could be implemented but as an end goal I could see a parameter within the aws_kinesisanalytics.CfnApplicationV2.ApplicationConfigurationProperty Object as such:

networking_configuration_property of type:

aws_kinesisanalytics.CfnApplicationV2.NetworkingConfigurationProperty

that accepts the follwing:

vpc:ec2.VPC,
subnets:[ec2.Subnet],
security_group:ec2.SecurityGroup

Other information

I am not certain this is relevant but:
I am using Python: 3.6.8
cdk: 1.129.0 (build fb43f89)
the issue I based my work upon: #12407
The PR that addressed the above issue: #12464

related to the above but on another note:
I found a bug in a generated example under the following construct:

aws_cdk.aws_kinesisanalytics_flink
_# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
import aws_cdk.aws_kinesisanalytics_flink as flink

flink_app = flink.Application(self, "Application",
    # ...
    property_groups=PropertyGroups(
        FlinkApplicationProperties={
            "input_stream_name": "my-input-kinesis-stream",
            "output_stream_name": "my-output-kinesis-stream"
        }
    )
)_

In Python 'PropertyGroups' is an abstract class that doesn't accept any params:
https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_kinesisanalytics_flink/PropertyGroups.html

Acknowledge

• I may be able to implement this feature request
• This feature might incur a breaking change

[peterwoodworth]

Hey @JoeKodsi-Idealworks,

Our kinesisanalytics module consists of only L1 constructs. In other words - the module only consists of what CloudFormation directly gives us to work with. So, what we have is identical to here and here

Any feature requests for these constructs would best go directly to CloudFormation's coverage roadmap. Thanks!

[peterwoodworth]

This is actually a documented issue already - aws-cloudformation/cloudformation-coverage-roadmap#547 be sure to give a thumbs up!

[peterwoodworth]

Also, PropertyGroups is defined as an interface in the source code

export interface PropertyGroups {
  readonly [propertyId: string]: {[mapKey: string]: string};
}

I don't usually use Python so I don't know off the top of my head how you would use this in Python, but I can look into that

[JoeKodsi-Idealworks]

Hey @JoeKodsi-Idealworks,

Our kinesisanalytics module consists of only L1 constructs. In other words - the module only consists of what CloudFormation directly gives us to work with. So, what we have is identical to here and here

Any feature requests for these constructs would best go directly to CloudFormation's coverage roadmap. Thanks!

Hello @peterwoodworth I appreciate the quick reply!
Since it is a Cfn construct it would make most sense to follow it under the link you have provide it. thanks for the tip!

Concerning the bug I mentioned, I guess to make sure that it is actually a bug one would need to create a aws-cdk/aws-kinesisanalytics-flink.Application.
I followed the generated example here: https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_kinesisanalytics_flink/README.html
While I was doing what I am suggesting(creating a aws-cdk/aws-kinesisanalytics-flink.Application) everything was going well until I needed to add propertyGroups which also has some documentation in the generated example.
-first thing I tried was to mimic the example that failed with error: __init__() got an unexpected keyword argument 'FlinkApplicationProperties'
-next I tried to configure the porpertyGroups as I would do for an aws_kinesisanalytics.CfnApplicationV2 by supplying property_group_id and property_map params. It also failed with error __init__() got an unexpected keyword argument 'property_group_id'.
This error is referring to the following class (the __init__ method/constructor doesn't accept any properties):

class PropertyGroups:
    def __init__(self) -> None:
        '''(experimental) Interface for building AWS::KinesisAnalyticsV2::Application PropertyGroup configuration.

        :stability: experimental
        '''
        self._values: typing.Dict[str, typing.Any] = {}

    def __eq__(self, rhs: typing.Any) -> builtins.bool:
        return isinstance(rhs, self.__class__) and rhs._values == self._values

    def __ne__(self, rhs: typing.Any) -> builtins.bool:
        return not (rhs == self)

    def __repr__(self) -> str:
        return "PropertyGroups(%s)" % ", ".join(
            k + "=" + repr(v) for k, v in self._values.items()
        )

I hope this helps you figure out my problem. After I faced this roadblock I resorted then to the Cfn class aws_kinesisanalytics.CfnApplicationV2 to create my flink application and configure the propertyGroups property which worked like a charm.

Thanks a lot for the support!

[JoeKodsi-Idealworks]

Hey @peterwoodworth , hope all is good on your end. Do you have any input for me regarding my last comment?
A simple acknowledgement if this is a bug or not would suffice me to know how to proceed with regards to creating a aws-cdk/aws-kinesisanalytics-flink.Application.

Thanks again and I hope this doesn't turn out to be a waste of time for you!

[peterwoodworth]

Hey @JoeKodsi-Idealworks, thanks for reminding me to look at this again

I was able to reproduce what you found - and wasn't able to figure out how to use the PropertyGroups interface in python. I'm not entirely sure if it's a bug, since I don't typically use Python, but it does seem to me like something weird is going on here.

We should document this in a separate issue. I'll do that by the end of today

[peterwoodworth]

I've created a new issue for this in the jsii repo, let's see what they say

[JoeKodsi-Idealworks]

I've created a new issue for this in the jsii repo, let's see what they say

Hey @peterwoodworth great to hear! I guess then this feature-request/issue can be closed and one could follow the outcome here: aws/jsii#3154

If you agree, this ticket can be closed when you see fit.

Thanks again for all the support
Cheers!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.