v1.15.0

• Kubernetes v1.15.0
• Migrate from Terraform v0.11 to v0.12.x (action required!)
  • Migration instructions for Terraform v0.12
• Require terraform-provider-ct v0.3.2+ to support Terraform v0.12 (action required)
• Update Calico from v3.7.2 to v3.7.3
• Remove Fedora Atomic modules (deprecated in March) (#501)

AWS

• Require terraform-provider-aws v2.7+ to support Terraform v0.12 (action required)
• Allow using Flatcar Linux Edge by setting os_image to "flatcar-edge"

Azure

• Require terraform-provider-azurerm v1.27+ to support Terraform v0.12 (action required)
• Avoid unneeded rotations of Regular priority virtual machine scale sets
  • Azure only allows eviction_policy to be set for Low priority VMs. Supporting Low priority VMs meant when Regular VMs were used, each terraform apply rolled workers, to set eviction_policy to null.
  • Terraform v0.12 nullable variables fix the issue so plan does not produce a diff.

Bare-Metal

• Require terraform-provider-matchbox v0.3.0+ to support Terraform v0.12 (action required)
• Allow using Flatcar Linux Edge by setting os_channel to "flatcar-edge"

DigitalOcean

• Require terraform-provider-digitalocean v1.3+ to support Terraform v0.12 (action required)
• Change the default worker_type from s-1vcpu1-1gb to s-1vcpu-2gb

Google Cloud

• Require terraform-provider-google v2.5+ to support Terraform v0.12 (action required)

Addons

• Update Grafana from v6.2.1 to v6.2.4
• Update node-exporter from v0.18.0 to v0.18.1

v1.14.3

v1.14.3

• Kubernetes v1.14.3
• Update CoreDNS from v1.3.1 to v1.5.0
  • Add ready plugin to improve readinessProbe
• Fix trailing slash in terraform-render-bootkube version (#479)
• Recommend updating terraform-provider-ct plugin from v0.3.1 to v0.3.2 (#487)

AWS

• Rename worker pool module count variable to worker_count (#485) (action maybe)
  • count will become a reserved variable name in Terraform v0.12

Azure

• Replace azurerm_autoscale_setting with azurerm_monitor_autoscale_setting (#482)
  • Require terraform-provider-azurerm v1.22+ (action required)
• Rename worker pool module count variable to worker_count (#485) (action maybe)
  • count will become a reserved variable name in Terraform v0.12

Bare-Metal

• Recommend updating terraform-provider-matchbox plugin from v0.2.3 to v0.3.0 (#487)

Google Cloud

• Rename worker pool module count variable to worker_count (#485) (action maybe)
  • count will become a reserved variable name in Terraform v0.12

Addons

• Update Prometheus from v2.9.2 to v2.10.0
• Update Grafana from v6.1.6 to v6.2.1

v1.14.2

• Kubernetes v1.14.2
• Update etcd from v3.3.12 to v3.3.13
• Upgrade Calico from v3.6.1 to v3.7.2
• Change VXLAN port from 8472 (kernel default) to 4789 (IANA)

AWS

• Only set internal VXLAN rules when networking is "flannel" (default: calico)

Azure

• Allow choosing Calico as the network provider (experimental) (#472)
  • Add a networking variable accepting "flannel" (default) or "calico"
  • Use VXLAN encapsulation since Azure doesn't support IPIP

DigitalOcean

• Allow choosing Calico as the network provider (experimental) (#472)
  • Add a networking variable accepting "flannel" (default) or "calico"
  • Use VXLAN encapsulation since DigitalOcean doesn't support IPIP
• Add explicit ordering between firewall rule creation and secure copying Kubelet credentials (#469)
  • Fix race scenario if copies to nodes were before rule creation, blocking cluster creation

Addons

• Update Prometheus from v2.8.1 to v2.9.2
  • Update kube-state-metrics from v1.5.0 to v1.6.0
• Update node-exporter from v0.17.0 to v0.18.0
• Update Grafana from v6.1.3 to v6.1.6
• Reduce nginx-ingress Role RBAC permissions (#458)

v1.14.1

• Kubernetes v1.14.1

Addons

• Update Grafana from v6.1.1 to v6.1.3
• Update nginx-ingress from v0.23.0 to v0.24.1

v1.14.0

• Kubernetes v1.14.0
• Update Calico from v3.6.0 to v3.6.1
• Add enable_aggregation option for CNCF conformance (#436)
  • Aggregation is disabled by default to retain our security stance. Extensions should be considered part of the control plane and scrutinized carefully. Favor leaving aggregation disabled.

AWS

• Add ability to load balance TCP applications (#443)
  • Output the network load balancer ARN as nlb_id
  • Accept a worker_target_groups (ARN) list to which worker instances should be added

Azure

• Add ability to load balance TCP/UDP applications (#447)
  • Output the load balancer ID as loadbalancer_id
• Output worker_security_group_name and worker_address_prefix for extending firewall rules (#447)

DigitalOcean

• Harden internal (node-to-node) firewall rules to align with other platforms (#444)
• Add ability to load balance TCP applications (#444)
  • Output controller_tag and worker_tag to simplify extending firewall rules

Google Cloud

• Add ability to load balance TCP/UDP applications (#442)
  • Add worker instances to a target pool, output as worker_target_pool
  • Health check for workers with Ingress controllers. Forward rules don't support differing internal/external ports, but some Ingress controllers support TCP/UDP proxy as a workaround
• Remove Haswell minimum CPU platform requirement (#439)
  • Google Cloud API implements min_cpu_platform to mean "use exactly this CPU". Revert #405 added in v1.13.4.
  • Fix error creating clusters in new regions without Haswell (e.g. europe-west2) (#438)

Addons

• Update Prometheus from v2.8.0 to v2.8.1
• Update Grafana from v6.0.2 to v6.1.1
  • Add dashboard for pods in a workload (deployment/daemonset/statefulset) (#446)
  • Add dashboard for workloads by namespace

v1.13.5

• Kubernetes v1.13.5
• Resolve in-addr.arpa reverse DNS lookups (PTR) for pod IPv4 addresses (#415)
  • Reverse DNS lookups for service IPv4 addresses unchanged
• Upgrade Calico from v3.5.2 to v3.6.0 (#430)
  • Change pod IPAM from host-local to calico-ipam. pod_cidr is still divided into /24 subnets per node, but managed as ippools and ipamblocks
• Suggest updating terraform-provider-ct from v0.3.0 to v0.3.1 (#434)
• Announce: Fedora Atomic modules will be not be updated beyond Kubernetes v1.13.x (#437)
  • Thank you Project Atomic team and users, please see the deprecation notice

AWS

• Support terraform-provider-aws v2.0+ (#419)

Bare-Metal

• Change the default iPXE kernel and initrd download protocol from HTTP to HTTPS (#420)
  • Require an iPXE-enabled network boot environment with support for TLS downloads. PXE clients must chainload to iPXE firmware compiled with DOWNLOAD_PROTO_HTTPS enabled. (action required)
  • Only affects Container Linux and Flatcar Linux install profiles that pull public images (default)
  • Add download_protocol variable. Recognizing boot firmware TLS support is difficult in some environments, set the protocol to "http" for the old behavior (discouraged)

DigitalOcean

• Fix kubelet hostname-override to set node metadata InternalIP correctly (#424)
  • Uniquely, DigitalOcean does not resolve hostnames to instance private IPs. Kubelet auto-detect mechanisms require the internal IP be set directly.
  • Regressed in v1.12.3 (#337) which aimed to provide friendly hostname-based node names on DigitalOcean

Addons

• Update Prometheus from v2.7.1 to v2.8.0
  • Refresh rules based on upstreams (#426)
  • Define NetworkPolicy to allow only traffic from the Grafana addon
• Update Grafana from v6.0.0 to v6.0.2
  • Add liveness and readiness probes
  • Refresh dashboards and organize to stay below ConfigMap size limit (#426)
• Remove heapster manifests from addons (#427)
  • Heapster addon powers kubectl top (in early Kubernetes, running the addon was expected). Today, there are better monitoring options.
  • kubectl top reliance on a non-core extension means its not in-scope for minimal Kubernetes
  • Look to prior releases if you still wish to apply heapster

v1.13.4

• Kubernetes v1.13.4
• Update etcd from v3.3.11 to v3.3.12
• Update Calico from v3.5.0 to v3.5.2
• Assign priorityClassNames to critical cluster and node components (#406)
  • Inform node out-of-resource eviction and scheduler preemption and ordering
• Add CoreDNS readiness probe (#410)

Bare-Metal

• Recommend updating terraform-provider-matchbox plugin from v0.2.2 to v0.2.3 (#402)
• Improve docs on using Ubiquiti EdgeOS with bare-metal clusters (#413)

Google Cloud

• Support terraform-provider-google v2.0+ (#407)
  • Require terraform-provider-google v1.19+ (action required)
• Set the minimum CPU platform to Intel Haswell (#405)
  • Haswell or better is available in every zone (no price change)
  • A few zones still default to Sandy/Ivy Bridge (shifts in April 2019)

Addons

• Modernize Prometheus rules and alerts (#404)
  • Drop extraneous metrics (#397)
  • Add pod name label to metrics discovered via service endpoints
  • Rename kubernetes_namespace label to namespace
• Modernize Grafana and dashboards, see docs (#403, #404)
  • Upgrade Grafana from v5.4.3 to v6.0.0!
  • Enable Grafana Explore UI as a Viewer (inspect/edit without saving)
• Update nginx-ingress from v0.22.0 to v0.23.0
  • Raise nginx-ingress liveness/readiness timeout to 5 seconds
  • Remove nginx-ingess default-backend (#401)

Fedora Atomic

• Build Kubelet system container with buildah. The image is an OCI format and slightly larger.

v1.13.3

• Kubernetes v1.13.3
• Update etcd from v3.3.10 to v3.3.11
• Update CoreDNS from v1.3.0 to v1.3.1
  • Switch from the proxy plugin to the faster forward plugin for upsteam resolvers
• Update Calico from v3.4.0 to v3.5.0
• Update flannel from v0.10.0 to v0.11.0
• Reduce pod eviction timeout for deleting pods on unready nodes to 1 minute
  • Respond more quickly to node preemption (previously 5 minutes)
• Fix automatic worker deletion on shutdown for cloud platforms
  • Lowering Kubelet privileges in #372 dropped a needed node deletion authorization. Scale-in due to manual terraform apply (any cloud), AWS spot termination, or Azure low priority deletion left old nodes registered, requiring manual deletion (kubectl delete node name)

AWS

• Add ingress_zone_id output with the NLB DNS name's Route53 zone for use in alias records (#380)

Azure

• Fix azure provider warning, public_ip allocation_method replaces public_ip_address_allocation
  • Require terraform-provider-azurerm v1.21+ (action required)

Addons

• Update nginx-ingress from v0.21.0 to v0.22.0
• Update Prometheus from v2.6.0 to v2.7.1
• Update kube-state-metrics from v1.4.0 to v1.5.0
  • Fix ClusterRole to collect and export PodDisruptionBudget metrics (#383)
• Update node-exporter from v0.15.2 to v0.17.0
• Update Grafana from v5.4.2 to v5.4.3

v1.13.2

• Kubernetes v1.13.2
• Add ServiceAccounts for kube-apiserver and kube-scheduler (#370)
• Use lower-privilege TLS client certificates for Kubelets (#372)
• Use HTTPS liveness probes for kube-scheduler and kube-controller-manager (#377)
• Update CoreDNS from v1.2.6 to v1.3.0
• Allow the certificates.k8s.io API to issue certificates signed by the cluster CA (#376)
  • Configure controller manager to sign CSRs that are manually approved by an administrator

AWS

• Change controller_type and worker_type default from t2.small to t3.small (#365)
  • t3.small is cheaper, provides 2 vCPU (instead of 1), and 5 Gbps of pod-to-pod bandwidth!

Bare-Metal

• Remove the kubeconfig output variable

Addons

• Update Prometheus from v2.5.0 to v2.6.0

v1.13.1

• Kubernetes v1.13.1
• Update Calico from v3.3.2 to v3.4.0 (#362)
  • Install CNI plugins with an init container rather than a sidecar
  • Improve the calico-node ClusterRole
• Recommend updating terraform-provider-ct plugin from v0.2.1 to v0.3.0 (#363)
  • Migration instructions for upgrading terraform-provider-ct in-place for v1.12.2+ clusters (action required)
  • Require switching from ~/.terraformrc to the Terraform third-party plugins directory ~/.terraform.d/plugins/
  • Require Container Linux 1688.5.3 or newer

Google Cloud

• Increase TCP proxy apiserver backend service timeout from 1 minute to 5 minutes (#361)
  • Align port-forward behavior closer to AWS/Azure (no timeout)

Addons

• Update Grafana from v5.4.0 to v5.4.2