AVI Controller Deployment on AWS Terraform module

This Terraform module creates and configures an AVI (NSX Advanced Load-Balancer) Controller on AWS

Module Functions

The module is meant to be modular and can create all or none of the prerequiste resources needed for the AVI AWS Deployment including:

• VPC and Subnets for the Controller and SEs (configured with create_networking variable)
• IAM Roles, Policy, and Instance Profile (configured with create_iam variable)
• Security Groups for AVI Controller and SE communication
• AWS EC2 Instance using an official AVI AMI
• High Availability AVI Controller Deployment (configured with controller_ha variable)

During the creation of the Controller instance the following initialization steps are performed:

• Copy Ansible playbook to controller using the assigned public IP
• Run Ansible playbook to configure initial settings and AWS Full Access Cloud

The Ansible playbook can optionally add these configurations:

• Create Avi DNS Profile (configured with the configure_dns_profile variable)
• Create Avi DNS Virtual Service (configured with the configure_dns_vs variable)
• Configure GSLB (configured with the configure_gslb variable)

Usage

This is an example of a controller deployment that leverages an existing VPC (with a cidr_block of 10.154.0.0/16) and 3 subnets. The public key is already created in EC2 and the private key found in the "/home/user/.ssh/id_rsa" will be used to copy and run the Ansible playbook to configure the Controller.

terraform {
  backend "local" {
  }
}
provider "aws" {
  region = "us-west-1"
}
module "avi_controller_aws" {
  source  = "vmware/avi-alb-deployment-aws/aws"
  version = "1.0.x"

  region = "us-west-1"
  create_networking = "false"
  create_iam = "true"
  avi_version = "22.1.1"
  custom_vpc_id = "vpc-<id>"
  custom_subnet_ids = ["subnet-<id>","subnet-<id>","subnet-<id>"]
  avi_cidr_block = "10.154.0.0/16"
  controller_password = "<newpassword>"
  key_pair_name = "<key>"
  private_key_path = "/home/<user>/.ssh/id_rsa"
  name_prefix = "<name>"
  custom_tags = { "Role" : "Avi-Controller", "Owner" : "admin", "Department" : "IT", "shutdown_policy" : "noshut" }
}
output "controller_info" {
  value = module.avi_controller_aws.controllers
}

GSLB Deployment

For GSLB to be configured successfully the configure_gslb and configure_dns_vs variables must be configured. By default a new Service Engine Group (g-dns) and user (gslb-admin) will be created for the configuration.

The following is a description of the configure_gslb variable parameters and their usage:

Parameter	Description	Type
enabled	Must be set to "true" for Active GSLB sites	bool
leader	Must be set to "true" for only one GSLB site that will be the leader	bool
site_name	Name of the GSLB site	string
domains	List of GSLB domains that will be configured	list(string)
create_se_group	Determines whether a g-dns SE group will be created	bool
se_size	The instance type used for the Avi Service Engines	string
additional_sites	Additional sites that will be configured. This parameter should only be set for the primary GSLB site	string

The example below shows a GSLB deployment with 2 regions utilized.

terraform {
  backend "local" {
  }
}
provider "aws" {
  alias  = "west2"
  region = "us-west-2"
}
provider "aws" {
  alias  = "east1"
  region = "us-east-1"
}
module "avi_controller_aws_west2" {
  source                = "vmware/avi-alb-deployment-aws/aws"
  providers             = { aws = aws.west2 }
  version               = "1.0.x"

  region                = "us-west-2"
  create_networking     = "false"
  create_iam            = "true"
  controller_ha         = true
  avi_version           = "22.1.2"
  custom_vpc_id         = "vpc-<id>"
  custom_subnet_ids     = ["subnet-<id>","subnet-<id>","subnet-<id>"]
  avi_cidr_block        = "10.154.0.0/16"
  controller_password   = "<newpassword>"
  key_pair_name         = "<key>"
  private_key_path      = "/home/<user>/.ssh/id_rsa"
  name_prefix           = "<name>"
  custom_tags           = { "Role" : "Avi-Controller", "Owner" : "admin", "Department" : "IT" }
  se_ha_mode            = "active/active"
  configure_dns_profile = { enabled = "true", type = "AVI", usable_domains = ["west1.avidemo.net"] }
  configure_dns_vs      = { enabled = "true", allocate_public_ip = "true", subnet_name = "companyname-avi-subnet" }
  configure_gslb        = { enabled = "true", site_name = "West2"}
}
module "avi_controller_aws_east1" {
  source                = "vmware/avi-alb-deployment-aws/aws"
  providers             = { aws = aws.east1 }
  version               = "1.0.x"

  region                = "us-east-1"
  create_networking     = "false"
  create_iam            = "true"
  controller_ha         = true
  avi_version           = "22.1.2"
  custom_vpc_id         = "vpc-<id>"
  custom_subnet_ids     = ["subnet-<id>","subnet-<id>","subnet-<id>"]
  avi_cidr_block        = "10.155.0.0/16"
  controller_password   = "<newpassword>"
  key_pair_name         = "<key>"
  private_key_path      = "/home/<user>/.ssh/id_rsa"
  name_prefix           = "<name>"
  custom_tags           = { "Role" : "Avi-Controller", "Owner" : "admin", "Department" : "IT", "shutdown_policy" : "noshut" }
  se_ha_mode            = "active/active"
  configure_dns_profile = { enabled = "true", type = "AVI", usable_domains = ["east1.avidemo.net"] }
  configure_dns_vs      = { enabled = "true", allocate_public_ip = "true", subnet_name = "companyname-avi-subnet" }
  configure_gslb        = { enabled = "true", leader = "true", site_name = "East1", domains = ["gslb.avidemo.net"], additional_sites = [{name = "West2", ip_address_list = module.avi_controller_aws_west2.controllers[*].private_ip_address}] }
}
output "east1_controller_info" {
  value = module.avi_controller_aws_east1.controllers
}
output "westus2_controller_info" {
  value = module.avi_controller_aws_west2.controllers
}

Day 1 Ansible Configuration and Avi Resource Cleanup

The module copies and runs an Ansible play for configuring the initial day 1 Avi config. The plays listed below can be reviewed by connecting to the Avi Controller by SSH and changing to the ansible folder. In an HA setup the first controller will have these files.

avi-controller-aws-all-in-one-play.yml

This play will configure the Avi Cloud, Network, IPAM/DNS profiles, DNS Virtual Service, GSLB depending on the variables used. The initial run of this play will output into the ansible-playbook.log file which can be reviewed to determine what tasks were ran.

Example run (appropriate variable values should be used):

~$ ansible-playbook avi-controller-aws-all-in-one-play.yml -e password=${var.controller_password} -e aws_access_key_id=${var.aws_access_key} -e aws_secret_access_key=${var.aws_secret_key} > ansible-playbook-run.log

avi-upgrade.yml

This play will upgrade or patch the Avi Controller and SEs depending on the variables used. When ran this play will output into the ansible-playbook.log file which can be reviewed to determine what tasks were ran. This play can be ran during the initial Terraform deployment with the avi_upgrade variable as shown in the example below:

avi_upgrade = { enabled = "true", upgrade_type = "patch", upgrade_file_uri = "URL Copied From portal.avipulse.vmware.com"}

An full version upgrade can be done by changing changing the upgrade_type to "system". It is recommended to run this play in a lower environment before running in a production environment and is not recommended for a GSLB setup at this time.

Example run (appropriate variable values should be used):

~$ ansible-playbook avi-upgrade.yml -e password=${var.controller_password} -e upgrade_type=${var.avi_upgrade.upgrade_type} -e upgrade_file_uri=${var.avi_upgrade.upgrade_file_uri} > ansible-playbook-run.log

avi-cloud-services-registration.yml

This play will register the Controller with Avi Cloud Services. This can be done to enable centralized licensing, live security threat updates, and proactive support. When ran this play will output into the ansible-playbook.log file which can be reviewed to determine what tasks were ran. This play can be ran during the initial Terraform deployment with the register_controller variable as shown in the example below:

register_controller = { enabled = "true", jwt_token = "TOKEN", email = "EMAIL", organization_id = "LONG_ORG_ID" }

The organization_id can be found as the Long Organization ID field from https://console.cloud.vmware.com/csp/gateway/portal/#/organization/info.

The jwt_token can be retrieved at https://portal.avipulse.vmware.com/portal/controller/auth/cspctrllogin.

Example run (appropriate variable values should be used):

~$ ansible-playbook avi-cloud-services-registration.yml -e password=${var.controller_password} -e register_controller.jwt_token=${var.register_controller.jwt_token} > ansible-playbook-run.log

avi-cleanup.yml

This play will disable all Virtual Services, delete all existing Avi service engines, and de-register the controller from Cloud Services. This playbook should be ran before deleting the controller with terraform destroy to clean up the resources created by the Avi Controller. Note that additional items created by the controller may be created and need to be manually removed.

Example run (appropriate variable values should be used and -e register_controller.jwt_token is only needed when register_controller.enabled is set to true):

~$ ansible-playbook avi-cleanup.yml -e password=${var.controller_password} -e register_controller.jwt_token=${var.register_controller.jwt_token}

Contributing

The terraform-aws-avi-alb-deployment-aws project team welcomes contributions from the community. Before you start working with this project please read and sign our Contributor License Agreement (https://cla.vmware.com/cla/1/preview). If you wish to contribute code and you have not signed our Contributor Licence Agreement (CLA), our bot will prompt you to do so when you open a Pull Request. For any questions about the CLA process, please refer to our FAQ. For more detailed information, refer to CONTRIBUTING.md.

Requirements

Name	Version
terraform	>= 1.3.0
aws	>= 5.12.0
null	>= 3.2.0

Providers

Name	Version
aws	5.12.0
null	3.2.1

Modules

No modules.

Resources

Name	Type
aws_ec2_tag.custom_controller_1	resource
aws_ec2_tag.custom_controller_2	resource
aws_ec2_tag.custom_controller_3	resource
aws_eip.avi	resource
aws_iam_instance_profile.avi	resource
aws_iam_role.avi	resource
aws_iam_role.vmimport	resource
aws_iam_role_policy.avi_autoscaling	resource
aws_iam_role_policy.avi_ec2	resource
aws_iam_role_policy.avi_iam	resource
aws_iam_role_policy.avi_kms	resource
aws_iam_role_policy.avi_r53	resource
aws_iam_role_policy.avi_s3	resource
aws_iam_role_policy.avi_s3_backup	resource
aws_iam_role_policy.avi_sqs_sns	resource
aws_iam_role_policy.avi_vmimport_kms_policy	resource
aws_iam_role_policy.avi_vmimport_policy	resource
aws_instance.avi_controller	resource
aws_internet_gateway.avi	resource
aws_route.default_route	resource
aws_s3_bucket.s3_nsxalb_backups	resource
aws_s3_bucket_lifecycle_configuration.s3_nsxalb_backups	resource
aws_s3_bucket_ownership_controls.s3_nsxalb_backups	resource
aws_s3_bucket_public_access_block.s3_nsxalb_backups	resource
aws_s3_bucket_server_side_encryption_configuration.s3_nsxalb_backups	resource
aws_security_group.avi_controller_sg	resource
aws_security_group.avi_data_sg	resource
aws_security_group.avi_se_mgmt_sg	resource
aws_subnet.avi	resource
aws_vpc.avi	resource
null_resource.ansible_provisioner	resource
null_resource.changepassword_provisioner	resource
aws_ami.avi	data source
aws_availability_zones.azs	data source
aws_iam_role.vmimport	data source
aws_iam_roles.vmimport	data source
aws_kms_alias.ebs	data source
aws_kms_alias.s3	data source
aws_partition.current	data source
aws_subnet.custom	data source

Inputs

Name	Description	Type	Default	Required
avi_cidr_block	This CIDR that will be used for creating a subnet in the AVI VPC - a /16 should be provided. This range is also used for security group rules source IP range for internal communication between the Controllers and SEs	string	"10.255.0.0/16"	no
avi_upgrade	This variable determines if a patch upgrade is performed after install. The enabled key should be set to true and the url from the Avi Cloud Services portal for the should be set for the upgrade_file_uri key. Valid upgrade_type values are patch or system	object({ enabled = bool, upgrade_type = string, upgrade_file_uri = string })	{ "enabled": "false", "upgrade_file_uri": "", "upgrade_type": "patch" }	no
avi_version	The AVI Controller version that will be deployed	string	n/a	yes
aws_access_key	The Access Key that will be used to deploy AWS resources	string	""	no
aws_secret_key	The Secret Key that will be used to deploy AWS resources	string	""	no
boot_disk_size	The boot disk size for the Avi controller	number	128	no
ca_certificates	Import one or more Root or Intermediate Certificate Authority SSL certificates for the controller. The certificate must be in the PEM format and base64 encoded without line breaks. An example command for generating the proper format is 'base64 -w 0 ca.pem > ca.base64'	list(object({ name = string, certificate = string }))	[ { "certificate": "", "name": "" } ]	no
configure_controller	Configure the Avi Cloud via Ansible after controller deployment. If not set to true this must be done manually with the desired config	bool	"true"	no
configure_dns_profile	Configure a DNS Profile for DNS Record Creation for Virtual Services. The usable_domains is a list of domains that Avi will be the Authoritative Nameserver for and NS records may need to be created pointing to the Avi Service Engine addresses. Supported profiles for the type parameter are AWS or AVI. The AWS DNS Profile is only needed when the AWS Account used for Route53 is different than the Avi Controller and the configure_dns_route_53 variable can be used otherwise	object({ enabled = bool, type = optional(string, "AVI"), usable_domains = list(string), ttl = optional(string, "30"), aws_profile = optional(object({ iam_assume_role = string, region = string, vpc_id = string, access_key_id = string, secret_access_key = string })) })	{ "enabled": false, "type": "AVI", "usable_domains": [] }	no
configure_dns_route_53	Configures Route53 DNS integration in the AWS Cloud configuration. The following variables must be set to false if enabled: configure_dns_profile, configure_dns_vs, configure_gslb	bool	"false"	no
configure_dns_vs	Create Avi DNS Virtual Service. The subnet_name parameter must be an existing AWS Subnet. If the allocate_public_ip parameter is set to true a EIP will be allocated for the VS. The VS IP address will automatically be allocated via the AWS IPAM	object({ enabled = bool, subnet_name = string, allocate_public_ip = bool })	{ "allocate_public_ip": "false", "enabled": "false", "subnet_name": "" }	no
configure_gslb	Configures GSLB. In addition the configure_dns_vs variable must also be set for GSLB to be configured. See the GSLB Deployment README section for more information.	object({ enabled = bool, leader = optional(bool, false), site_name = string, domains = optional(list(string)), create_se_group = optional(bool, true), se_size = optional(string, "c5.xlarge"), additional_sites = optional(list(object({ name = string, ip_address_list = list(string) }))), })	{ "domains": [ "" ], "enabled": "false", "site_name": "" }	no
controller_ebs_encryption	Enable encryption on the Controller EBS Root Volume. The AWS Managed EBS KMS key will be used if no key is provided with the controller_ebs_encryption_key_arn variable	bool	"true"	no
controller_ebs_encryption_key_arn	AWS Resource Name of an existing KMS key for the Controller EBS (controller_ebs_encryption must be set to true)	string	null	no
controller_ha	If true a HA controller cluster is deployed and configured	bool	"false"	no
controller_password	The password that will be used authenticating with the AVI Controller. This password be a minimum of 8 characters and contain at least one each of uppercase, lowercase, numbers, and special characters	string	n/a	yes
controller_public_address	This variable controls if the Controller has a Public IP Address. When set to false the Ansible provisioner will connect to the private IP of the Controller.	bool	"false"	no
create_firewall_rules	This variable controls the Security Group creation for the Avi deployment. When set to false the necessary security group rules must be in place before the deployment and set with the firewall_custom_security_group_ids variable	bool	"true"	no
create_iam	Create IAM policy, roles, and instance profile for Avi AWS Full Access Cloud. If set to false the aws_access_key and aws_secret_key variables will be used for the Cloud configuration and all policy must be created as found in https://avinetworks.com/docs/latest/iam-role-setup-for-installation-into-aws/	bool	"true"	no
create_networking	This variable controls the VPC and subnet creation for the AVI Controller. When set to false the custom-vpc-name and custom-subnetwork-name must be set.	bool	"true"	no
custom_ami	The AMI ID of a custom controller AMI. For internal use.	string	null	no
custom_controller_name	This field can be used to specify a custom controller name to replace the (prefix-avi-controller) standard name. A numeric iterator will still be appended to the custom name (1,2,3)	string	null	no
custom_controller_subnet_ids	This field can be used to specify a list of existing VPC Subnets for the Controllers. The create-networking variable must also be set to false for this network to be used.	list(string)	null	no
custom_subnet_ids	This field can be used to specify a list of existing VPC Subnets for the SEs. The create-networking variable must also be set to false for this network to be used.	list(string)	null	no
custom_tags	Custom tags added to AWS Resources created by the module	map(string)	{}	no
custom_vpc_id	This field can be used to specify an existing VPC for the SEs. The create-networking variable must also be set to false for this network to be used.	string	null	no
dns_search_domain	The optional DNS search domain that will be used by the controller	string	null	no
dns_servers	The optional DNS servers that will be used for local DNS resolution by the controller. Example ["8.8.4.4", "8.8.8.8"]	list(string)	null	no
email_config	The Email settings that will be used for sending password reset information or for trigged alerts. The default setting will send emails directly from the Avi Controller	object({ smtp_type = string, from_email = string, mail_server_name = string, mail_server_port = string, auth_username = string, auth_password = string })	{ "auth_password": "", "auth_username": "", "from_email": "admin@avicontroller.net", "mail_server_name": "localhost", "mail_server_port": "25", "smtp_type": "SMTP_LOCAL_HOST" }	no
firewall_controller_allow_source_range	The IP range allowed to connect to the Avi Controller. Access from all IP ranges will be allowed by default. DEPRECATED in favor of firewall_controller_allow_source_ranges	string	null	no
firewall_controller_allow_source_ranges	The IP ranges allowed to connect to the Avi Controller. Access from all IP ranges will be allowed by default	list(string)	[ "0.0.0.0/0" ]	no
firewall_controller_security_group_ids	List of security group IDs that will be assigned to the controller. This variable must be set if the create_firewall_rules variable is set to false	list(string)	null	no
firewall_se_data_rules	The data plane traffic allowed for Virtual Services hosted on Services Engines. The configure_firewall_rules variable must be set to true for these rules to be created	list(object({ protocol = string, port = string, allow_ip_range = string, description = string }))	[ { "allow_ip_range": "0.0.0.0/0", "description": "https", "port": "443", "protocol": "tcp" }, { "allow_ip_range": "10.0.0.0/8", "description": "DNS", "port": "53", "protocol": "udp" } ]	no
instance_type	The EC2 instance type for the Avi Controller	string	"m5.2xlarge"	no
key_pair_name	The name of the existing EC2 Key pair that will be used to authenticate to the Avi Controller	string	n/a	yes
license_key	The license key that will be applied when the tier is set to ENTERPRISE with the license_tier variable	string	""	no
license_tier	The license tier to use for Avi. Possible values are ENTERPRISE_WITH_CLOUD_SERVICES or ENTERPRISE	string	"ENTERPRISE_WITH_CLOUD_SERVICES"	no
name_prefix	This prefix is appended to the names of the Controller and SEs	string	n/a	yes
ntp_servers	The NTP Servers that the Avi Controllers will use. The server should be a valid IP address (v4 or v6) or a DNS name. Valid options for type are V4, DNS, or V6	list(object({ addr = string, type = string }))	[ { "addr": "0.us.pool.ntp.org", "type": "DNS" }, { "addr": "1.us.pool.ntp.org", "type": "DNS" }, { "addr": "2.us.pool.ntp.org", "type": "DNS" }, { "addr": "3.us.pool.ntp.org", "type": "DNS" } ]	no
portal_certificate	Import a SSL certificate for the controller's web portal. The key and certificate must be in the PEM format and base64 encoded without line breaks. An example command for generating the proper format is 'base64 -w 0 certificate.pem > cert.base64'	object({ key = string, certificate = string, key_passphrase = optional(string) })	{ "certificate": "", "key": "" }	no
private_key_contents	The contents of the private key for the EC2 Key pair used for authenticating to the Avi Controller. Either private_key_path or private_key_contents must be supplied.	string	null	no
private_key_path	The local private key path for the EC2 Key pair used for authenticating to the Avi Controller. Either private_key_path or private_key_contents must be supplied.	string	null	no
region	The Region that the AVI controller and SEs will be deployed to	string	n/a	yes
register_controller	If enabled is set to true the controller will be registered and licensed with Avi Cloud Services. The Long Organization ID (organization_id) can be found from https://console.cloud.vmware.com/csp/gateway/portal/#/organization/info. The jwt_token can be retrieved at https://portal.avipulse.vmware.com/portal/controller/auth/cspctrllogin. Optionally the controller name and description used during the registration can be set; otherwise, the name_prefix and configure_gslb.site_name variables will be used.	object({ enabled = bool, jwt_token = string, email = string, organization_id = string, name = optional(string), description = optional(string) })	{ "email": "", "enabled": "false", "jwt_token": "", "organization_id": "" }	no
s3_backup_bucket	Name of the S3 bucket for Controller configuration backups	string	null	no
s3_backup_retention	Number of days to keep backups in S3 bucket	number	4	no
se_ebs_encryption	Enable encryption on SE AMI / EBS Volumes. The AWS Managed EBS KMS key will be used if no key is provided with se_ebs_encryption_key_arn variable	bool	"true"	no
se_ebs_encryption_key_arn	AWS Resource Name of an existing KMS key for SE AMI/EBS (se_ebs_encryption must be set to true)	string	null	no
se_ha_mode	The HA mode of the default Service Engine Group. Possible values active/active, n+m, or active/standby	string	"active/active"	no
se_instance_type	The instance type of the default Service Engine Group. Possible values can be found at https://aws.amazon.com/ec2/instance-types/	string	"c5.large"	no
se_s3_encryption	Enable encryption on SE S3 Bucket. The AWS Managed S3 KMS key will be used if no key is provided with se_s3_encryption_key_arn variable	bool	"true"	no
se_s3_encryption_key_arn	AWS Resource Name of an existing KMS key for SE S3 Bucket (se_s3_encryption must be set to true)	string	null	no
securechannel_certificate	Import a SSL certificate for the controller's secure channel communication. Only if there is strict policy that requires all SSL certificates to be signed a specific CA should this variable be used otherwise the default generated certificate is recommended. The full cert chain is necessary and can be provided within the certificate PEM file or separately with the ca_certificates variable. The key and certificate must be in the PEM format and base64 encoded without line breaks. An example command for generating the proper format is 'base64 -w 0 certificate.pem > cert.base64'	object({ key = string, certificate = string, key_passphrase = optional(string) })	{ "certificate": "", "key": "" }	no

Outputs

Name	Description
controller_private_addresses	The Private IP Addresses allocated for the Avi Controller(s)
controller_public_addresses	Public IP Addresses for the AVI Controller(s)
controller_security_group_id	Security Group associated with Avi Controller(s)
controllers	The AVI Controller(s) Information
service_engine_avi_data_security_group_id	Security Group associated with Avi Service Engines for data traffic
service_engine_avi_mgmt_security_group_id	Security Group associated with Avi Service Engines for management traffic