NOTE: Application Gateway for Containers has been released, which introduces numerous performance, resilience, and feature changes. Please consider leveraging Application Gateway for Containers for your next deployment. A list of corresponding translations from AGIC to Application Gateway for Containers may be found here.
The Kubernetes Ingress resource can be annotated with arbitrary key/value pairs. AGIC relies on annotations to program Application Gateway features, which are not configurable via the Ingress YAML. Ingress annotations are applied to all HTTP setting, backend pools and listeners derived from an ingress resource.
For an Ingress resource to be observed by AGIC it must be annotated with kubernetes.io/ingress.class: azure/application-gateway
. Only then AGIC will work with the Ingress resource in question.
The annotation allows to configure frontend listener to use different ports other than 80/443 for http/https.
If the port is within the App Gw authorized range (1 - 64999), this listener will be created on this specific port. If an invalid port or no port is set in the annotation, the configuration will fallback on default 80 or 443.
appgw.ingress.kubernetes.io/override-frontend-port: "port"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-overridefrontendport
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/override-frontend-port: "8080"
spec:
rules:
- http:
paths:
- path: /hello/
backend:
service:
name: store-service
port:
number: 80
pathType: Exact
External request will need to target http://somehost:8080 instead of http://somehost.
This annotation allows the backend path specified in an ingress resource to be re-written with prefix specified in this annotation. This allows users to expose services whose endpoints are different than endpoint names used to expose a service in an ingress resource.
appgw.ingress.kubernetes.io/backend-path-prefix: <path prefix>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-path-prefix: "/test/"
spec:
rules:
- http:
paths:
- path: /hello/
backend:
service:
name: store-service
port:
number: 80
pathType: Exact
In the example above we have defined an ingress resource named go-server-ingress-bkprefix
with an annotation appgw.ingress.kubernetes.io/backend-path-prefix: "/test/"
. The annotation tells application gateway to create an HTTP setting which will have a path prefix override for the path /hello
to /test/
.
NOTE: In the above example we have only one rule defined. However, the annotations is applicable to the entire ingress resource so if a user had defined multiple rules the backend path prefix would be setup for each of the paths specified. Thus, if a user wants different rules with different path prefixes (even for the same service) they would need to define different ingress resources.
If your incoming path is /hello/test/health but your backend requires /health you will want to ensure you have /* on your path
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
rules:
- http:
paths:
- path: /hello/test/*
pathType: Prefix
backend:
service:
name: store-service
This annotations allows us to specify the host name that Application Gateway should use while talking to the Pods.
appgw.ingress.kubernetes.io/backend-hostname: "internal.example.com"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-timeout
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-hostname: "internal.example.com"
spec:
rules:
- http:
paths:
- path: /hello/
backend:
service:
name: store-service
port:
number: 80
pathType: Exact
This annotation allows us to specify the protocol that Application Gateway should use while talking to the Pods. Supported Protocols: http
, https
Note
- Make sure to not use port 80 with HTTPS and port 443 with HTTP on the Pods.
appgw.ingress.kubernetes.io/backend-protocol: "https"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-timeout
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-protocol: "https"
spec:
rules:
- http:
paths:
- path: /hello/
backend:
service:
name: store-service
port:
number: 443
pathType: Exact
Application Gateway can be configured
to automatically redirect HTTP URLs to their HTTPS counterparts. When this
annotation is present and TLS is properly configured, Kubernetes Ingress
controller will create a routing rule with a redirection configuration
and apply the changes to your App Gateway. The redirect created will be HTTP 301 Moved Permanently
.
appgw.ingress.kubernetes.io/ssl-redirect: "true"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-redirect
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- hosts:
- www.contoso.com
secretName: testsecret-tls
rules:
- host: www.contoso.com
http:
paths:
- backend:
service:
name: websocket-repeater
port:
number: 80
The SSL certificate can be configured to Application Gateway either from a local PFX certificate file or a reference to a Azure Key Vault unversioned secret Id.
When the annotation is present with a certificate name and the certificate is pre-installed in Application Gateway, Kubernetes Ingress controller will create a routing rule with a HTTPS listener and apply the changes to your App Gateway.
appgw-ssl-certificate
annotation can also be used together with ssl-redirect
annotation in case of SSL redirect.
Please refer to appgw-ssl-certificate feature for more details.
Note
- Annotation "appgw-ssl-certificate" will be ignored when TLS Spec is defined in ingress at the same time.
- If a user wants different certs with different hosts(multi tls certificate termination), they would need to define different ingress resources.
- Configure from a local PFX certificate file
az network application-gateway ssl-cert create -g $resgp --gateway-name $appgwName -n mysslcert --cert-file \path\to\cert\file --cert-password Abc123
- Configure from a reference to a Key Vault unversioned secret id
az keyvault certificate create --vault-name $vaultName -n cert1 -p "$(az keyvault certificate get-default-policy)"
versionedSecretId=$(az keyvault certificate show -n cert --vault-name $vaultName --query "sid" -o tsv)
unversionedSecretId=$(echo $versionedSecretId | cut -d'/' -f-5) # remove the version from the url
az network application-gateway ssl-cert create -n mysslcert --gateway-name $appgwName --resource-group $resgp --key-vault-secret-id $unversionedSecretId
To use PowerShell, please refer to Configure Key Vault - PowerShell.
appgw.ingress.kubernetes.io/appgw-ssl-certificate: "name-of-appgw-installed-certificate"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-certificate
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: "name-of-appgw-installed-certificate"
spec:
rules:
- host: www.contoso.com
http:
paths:
- backend:
service:
name: websocket-repeater
port:
number: 80
Users now can configure their own root certificates to Application Gateway to be trusted via AGIC.
The annotaton appgw-trusted-root-certificate
shall be used together with annotation backend-protocol
to indicate end-to-end ssl encryption, multiple root certificates, separated by comma, if specified, e.g. "name-of-my-root-cert1,name-of-my-root-certificate2".
- Create your public root certificate for testing
openssl ecparam -out test.key -name prime256v1 -genkey
openssl req -new -sha256 -key test.key -out test.csr
openssl x509 -req -sha256 -days 365 -in test.csr -signkey test.key -out test.crt
- Configure your root certificate to Application Gateway
# Rename test.crt to test.cer
mv test.crt test.cer
# Configure the root certificate to your Application Gateway
az network application-gateway root-cert create --cert-file test.cer --gateway-name $appgwName --name name-of-my-root-cert1 --resource-group $resgp
- Repeat the steps above if you want to configure multiple trusted root certificates
appgw.ingress.kubernetes.io/backend-protocol: "https"
appgw.ingress.kubernetes.io/appgw-trusted-root-certificate: "name-of-my-root-cert1"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-certificate
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-protocol: "https"
appgw.ingress.kubernetes.io/appgw-trusted-root-certificate: "name-of-my-root-cert1"
spec:
rules:
- host: www.contoso.com
http:
paths:
- backend:
service:
name: websocket-repeater
port:
number: 80
Note: This annotation is supported since 1.6.0-rc1.
Users can configure a ssl profile on the Application Gateway per listener. When the annotation is present with a profile name and the profile is pre-installed in the Application Gateway, Kubernetes Ingress controller will create a routing rule with a HTTPS listener and apply the changes to your App Gateway.
connection-draining
: This annotation allows to specify whether to enable connection draining.
connection-draining-timeout
: This annotation allows to specify a timeout after which Application Gateway will terminate the requests to the draining backend endpoint.
appgw.ingress.kubernetes.io/connection-draining: "true"
appgw.ingress.kubernetes.io/connection-draining-timeout: "60"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-drain
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/connection-draining: "true"
appgw.ingress.kubernetes.io/connection-draining-timeout: "60"
spec:
rules:
- http:
paths:
- path: /hello/
backend:
service:
name: store-service
port:
number: 80
pathType: Exact
This annotation allows to specify whether to enable cookie based affinity.
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-affinity
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
spec:
rules:
- http:
paths:
- path: /hello/
backend:
service:
name: store-service
port:
number: 80
pathType: Exact
In addition to cookie-based-affinity, you can set cookie-based-affinity-distinct-name: "true"
to ensure a different affinity cookie is set per backend.
appgw.ingress.kubernetes.io/cookie-based-affinity-distinct-name: "true"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-affinity
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
appgw.ingress.kubernetes.io/cookie-based-affinity-distinct-name: "true"
spec:
rules:
- http:
paths:
- path: /affinity1/
pathType: Exact
backend:
service:
name: affinity-service
port:
number: 80
- path: /affinity2/
pathType: Exact
backend:
service:
name: affinity-service
port:
number: 80
This annotation allows to specify the request timeout in seconds after which Application Gateway will fail the request if response is not received.
appgw.ingress.kubernetes.io/request-timeout: "20"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-timeout
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/request-timeout: "20"
spec:
rules:
- http:
paths:
- path: /hello/
backend:
service:
name: store-service
port:
number: 80
pathType: Exact
This annotation allows us to specify whether to expose this endpoint on Private IP of Application Gateway.
Note
-
App Gateway doesn't support multiple IPs on the same port (example: 80/443). Ingress with annotation
appgw.ingress.kubernetes.io/use-private-ip: "false"
and another withappgw.ingress.kubernetes.io/use-private-ip: "true"
onHTTP
will cause AGIC to fail in updating the App Gateway. -
For App Gateway that doesn't have a private IP, Ingresses with
appgw.ingress.kubernetes.io/use-private-ip: "true"
will be ignored. This will reflected in the controller logs and ingress events for those ingresses withNoPrivateIP
warning.
appgw.ingress.kubernetes.io/use-private-ip: "true"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-timeout
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/use-private-ip: "true"
spec:
rules:
- http:
paths:
- path: /hello/
backend:
service:
name: store-service
port:
number: 80
pathType: Exact
This annotation allows you to attach an already created WAF policy to the list paths for a host within a Kubernetes Ingress resource being annotated.
The WAF policy must be created in advance. Example of using Azure Portal to create a policy:
Once the policy is created, copy the URI of the policy from the address bar of Azure Portal:
The URI would have the following format:
/subscriptions/<YOUR-SUBSCRIPTION>/resourceGroups/<YOUR-RESOURCE-GROUP>/providers/Microsoft.Network/applicationGatewayWebApplicationFirewallPolicies/<YOUR-POLICY-NAME>
Note
- Waf policy will only be applied to a listener if ingress rule path is not set or set to "/" or "/*"
appgw.ingress.kubernetes.io/waf-policy-for-path: "/subscriptions/abcd/resourceGroups/rg/providers/Microsoft.Network/applicationGatewayWebApplicationFirewallPolicies/adserver"
The example below will apply the WAF policy
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ad-server-ingress
namespace: commerce
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/waf-policy-for-path: "/subscriptions/abcd/resourceGroups/rg/providers/Microsoft.Network/applicationGatewayWebApplicationFirewallPolicies/adserver"
spec:
rules:
- http:
paths:
- path: /ad-server
backend:
service:
name: ad-server
port:
number: 80
pathType: Exact
- path: /auth
backend:
service:
name: auth-server
port:
number: 80
pathType: Exact
Note that the WAF policy will be applied to both /ad-server
and /auth
URLs.
This annotation allows specifically define a target host to be used for AGW health probe. By default, if backend container running service with liveliness probe of type HTTP GET
defined, host used in liveliness probe definition is also used as a target host for health probe. However if annotation appgw.ingress.kubernetes.io/health-probe-hostname
is defined it overrides it with its own value.
appgw.ingress.kubernetes.io/health-probe-hostname: <hostname>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-hostname: "my-backend-host.custom.app"
spec:
rules:
- http:
paths:
- path: /hello/
backend:
service:
name: store-service
port:
number: 80
pathType: Exact
Health probe port annotation allows specifically define target TCP port to be used for AGW health probe. By default, if backend container running service has liveliness probe of type HTTP GET
defined, port used in liveliness probe definition is also used as a port for health probe. Annotation appgw.ingress.kubernetes.io/health-probe-port
has precedence over such default value.
appgw.ingress.kubernetes.io/health-probe-port: <port number>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-hostname: "my-backend-host.custom.app"
appgw.ingress.kubernetes.io/health-probe-port: "443"
appgw.ingress.kubernetes.io/health-probe-path: "/healthz"
appgw.ingress.kubernetes.io/backend-protocol: https
spec:
tls:
- secretName: "my-backend-host.custom.app-ssl-certificate"
rules:
- http:
paths:
- path: /
backend:
service:
name: store-service
port:
number: 443
pathType: Exact
This annotation allows specifically define target URI path to be used for AGW health probe. By default, if backend container running service with liveliness probe of type HTTP GET
defined , path defined in liveliness probe definition is also used as a path for health probe. However annotation appgw.ingress.kubernetes.io/health-probe-path
overrides it with its own value.
appgw.ingress.kubernetes.io/health-probe-path: <URI path>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-hostname: "my-backend-host.custom.app"
appgw.ingress.kubernetes.io/health-probe-port: "8080"
appgw.ingress.kubernetes.io/health-probe-path: "/healthz"
spec:
rules:
- http:
paths:
- path: /
backend:
service:
name: store-service
port:
number: 8080
This annotation defines healthy status codes returned by the health probe. The values are comma separated list of individual status codes or ranges defined as <start of the range>-<end of the range>
.
appgw.ingress.kubernetes.io/health-probe-status-codes: <status codes>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-status-codes: "200-399, 401"
spec:
rules:
- http:
paths:
- path: /
backend:
service:
name: store-service
port:
number: 8080
pathType: Exact
This annotation sets AGW health probe interval. By default, if backend container running service with liveliness probe of type HTTP GET
defined, interval in liveliness probe definition is also used as a interval for health probe. However annotation appgw.ingress.kubernetes.io/health-probe-interval
overrides it with its value.
appgw.ingress.kubernetes.io/health-probe-interval: <interval seconds>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-interval: "20"
spec:
rules:
- http:
paths:
- path: /
backend:
service:
name: store-service
port:
number: 8080
pathType: Exact
This annotation allows specifically define timeout for AGW health probe. By default, if backend container running service with liveliness probe of type HTTP GET
defined, timeout defined in liveliness probe definition is also used for health probe. However annotation appgw.ingress.kubernetes.io/health-probe-timeout
overrides it with its value.
appgw.ingress.kubernetes.io/health-probe-timeout: <timeout seconds>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-timeout: "15"
spec:
rules:
- http:
paths:
- path: /
backend:
service:
name: store-service
port:
number: 8080
pathType: Exact
This annotation allows specifically define target unhealthy thresold for AGW health probe. By default, if backend container running service with liveliness probe of type HTTP GET
defined , threshold defined in liveliness probe definition is also used for health probe. However annotation appgw.ingress.kubernetes.io/health-probe-unhealthy-threshold
overrides it with its value.
appgw.ingress.kubernetes.io/health-probe-unhealthy-threshold: <unhealthy threshold>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-unhealthy-threshold: "5"
spec:
rules:
- http:
paths:
- path: /
backend:
service:
name: store-service
port:
number: 8080
pathType: Exact
This annotation allows to assign an existing rewrite rule set to the corresponding request routing rule(s). Rewrite rule set is managed via Azure Portal / CLI / PS.
appgw.ingress.kubernetes.io/rewrite-rule-set: <rewrite rule set>
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/rewrite-rule-set: add-custom-response-header
spec:
rules:
- http:
paths:
- path: /
pathType: Exact
backend:
service:
name: store-service
port:
number: 8080
Note: This annotation is supported since 1.6.0-rc1.
This annotation allows to assign a header/URL rewrite rule set created via the AzureApplicationGatewayRewrite CR to be associated to all rules in an ingress resource. AzureApplicationGatewayRewrite CR should be present in the same namespace as the ingress.
appgw.ingress.kubernetes.io/rewrite-rule-set-custom-resource: <name of rewrite rule set custom resource>
apiVersion: appgw.ingress.azure.io/v1beta1
kind: AzureApplicationGatewayRewrite
metadata:
name: my-rewrite-rule-set
spec:
rewriteRules:
- name: rule1
ruleSequence: 21
conditions:
- ignoreCase: false
negate: false
variable: http_req_Host
pattern: example.com
actions:
requestHeaderConfigurations:
- actionType: set
headerName: incoming-test-header
headerValue: incoming-test-value
responseHeaderConfigurations:
- actionType: set
headerName: outgoing-test-header
headerValue: outgoing-test-value
urlConfiguration:
modifiedPath: "/api/"
modifiedQueryString: "query=test-value"
reroute: false
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-server-ingress-bkprefix
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/rewrite-rule-set-custom-resource: my-rewrite-rule-set
spec:
rules:
- http:
paths:
- path: /
pathType: Exact
backend:
service:
name: store-service
port:
number: 8080
This annotation allows to append additional hostnames to the host
specified in the ingress resource. This applies to all the rules in the ingress resource.
appgw.ingress.kubernetes.io/hostname-extension: "hostname1, hostname2"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: store-app-ingress
namespace: test-ag
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/hostname-extension: "prod-store.app.com"
spec:
rules:
- host: "store.app.com"
http:
paths:
- path: /
pathType: Exact
backend:
service:
name: store-service
port:
number: 8080