Skip to content

Latest commit

 

History

History
3396 lines (2718 loc) · 372 KB

CHANGELOG-1.23.md

File metadata and controls

3396 lines (2718 loc) · 372 KB
Raw

v1.23.13

Downloads for v1.23.13

Source Code

filename sha512 hash
kubernetes.tar.gz 0995ad8ee7e2a8733db5287f9dedcfa565cc7782f83a633459718a155ae60b18dd8a6978e134abdecb05bdfd28c85744e402c53e54fd1ec7332f627b2e76c2fa
kubernetes-src.tar.gz 0e3129af29a28c96528f4842caa9234dcd198d9b1712cdf262581a535fa851013ce31c6543014145bb984c8994673bc73f88bbc83ea7142bc92737b81df8bae3

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 1f94745e78d8ee50725d5145d4eac928a7ae481e51758ed259232013f245961a9a54f106b556330d710a7d8c45e7c13a304fec6f56f8c5ce8e959e204efb6b27
kubernetes-client-darwin-arm64.tar.gz 255843cdf86ba0a18b60d8dd5b515fa9a3225119875e9c6a3f7a128d1a0545d92c28abf8d1e2c02194f3f047417ade6752c791bfbc1469e566d305122522f60e
kubernetes-client-linux-386.tar.gz 4076aa1723fff8d1d9a455159994ee002855bec419c4e56d894ee95dbf527a222236bbd65e711aa967df185bdcd71d9ac3bd6dd0143cdc18c80a28ec09e675e2
kubernetes-client-linux-amd64.tar.gz 5dee83bd9ef3c8ce3923c7f8fa3a6cbbaab81158ae065f0721117debdbd6412cb2f091eda9e9cbcc7637533c0f328228a9714f56de303bbe61c93b94398a91f7
kubernetes-client-linux-arm.tar.gz 2c5026cb1204e4a3a78280c837193bed7cd1287c5053ab83f3660762e18168c5b3896709d1a3f18cf66f92b359b258b1fcc4bc9acea651531bd7fca5fa991e9d
kubernetes-client-linux-arm64.tar.gz f0f9505bfdf6e9b89c995797cec522dedfec34b1880a52be042c81a797ddcf080e27f6af366944247ce5edc0757aaa9f67425840bedad276e9e18180c4363334
kubernetes-client-linux-ppc64le.tar.gz ebd37318e83e42d8a1833598b6f31ee28d358987e5727e0bb0e9ce4e4de7c26c299a26beb4f857d852e04db569c92750f44061239493bcf040b403b41b3e865b
kubernetes-client-linux-s390x.tar.gz 037d6b51e4b0e7011048087940637b3653034ad22219c281b49ee76a2e7194b79502a9f4a299349772b68b3b256570a6ce7e44896746e761c606ef04e104453b
kubernetes-client-windows-386.tar.gz 1e938c7b44a4093d84b5370c51590105d64b5283c3543bcb579252733b062db613c5203fd336de621442963a1dd4f51bf4799289e98fe5e09f953065e503255f
kubernetes-client-windows-amd64.tar.gz 0f2ed66cc2c2471c3574320654b19e8894a7ee42e7e1cb9b579119abb3f9dc82087acf41b060627e3ddf0aaae6a0ed04cbbe740cbbef26eaa754cce715ae32de
kubernetes-client-windows-arm64.tar.gz 02032b45f01c2b2a1ebbe48aace0f44297c677f64e122c2933f0b66ecb0ca48c13795097cce9007efa6f70f2dbdac42b6a08219ecebe29404a728202ebdabddc

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 59b8bb9a28799c2f07c961bceac6b600c443df30e0a5051950f784f031c69dd7deb46555a734696a262a95d7d1d0b5e1623440ca4017806896d06758a3a08d8b
kubernetes-server-linux-arm.tar.gz 2061a03b768b9e24055dd7280e1695717c5eba32cecbdd8c96be0755f95907505106bbd0e24d8cb9098c5d9cd1aa6f70529550a411317cf749fd89457923b33f
kubernetes-server-linux-arm64.tar.gz b4ca99506f8efa6232e7a4a7d6d37973b317fb053f21b73a5bebe2f2e685cafca75a29345a67ffe41b631e5af9a7c5e2625e6113b0eebdd2781547728c976a70
kubernetes-server-linux-ppc64le.tar.gz fb3d036fde212363bbcb5c8e04cca76eeea6ec90e3513a9f2f4cbef4655548b4e061143be9c6e0e17b2dc496ae1990847a11c948b7d72d9e9fd0555d7a4df5a7
kubernetes-server-linux-s390x.tar.gz 62f09c403cb97ddb536a67ec4816846e9b7fa136faa8958831ec980d4755b2b88483292f705ca12c0c7ced0e3e3f0daa811835ff578c8d31b24c25389c09033e

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 97167211801a737b52f766a2fd7b0afddacac22dbf4b03cbbeae734b2a7e1243f8e5d715c84f4f8dfa862c62892f01a42d2561c96386bf2283dd21e64bb91a72
kubernetes-node-linux-arm.tar.gz f1e75a62c8360296cd4bae498c46721e44f2219322f7361ca9d32a0026a5c9f797c99c3b34df2f985466aa11eaacda425ec6aab9f6368c426860efcd0496af5c
kubernetes-node-linux-arm64.tar.gz 86fb0f9cea876a7e8331ac82de1feebadca56accc3495c30c558b5851e4242a9c9fc8420c7e23c2db734b6ef8134cc0455dfb7a67798838e921c3120936386c9
kubernetes-node-linux-ppc64le.tar.gz 997190a1d39a0a2684d65922ef583e72eff9fe5c77011f3d1edd837ee7ff4ce4aa1f898b9e2f1acba4d239b2f3bf33f874ace28105b2b51825b57f699309622d
kubernetes-node-linux-s390x.tar.gz 3210ae564dc1066c70ea2bfbb6b961290fea748db8169d3ac9a00cd3082b24edbebe0b5fe0fbb276f794360ff82c502209c7c155cb2d6e58eb82a95d5fd829cb
kubernetes-node-windows-amd64.tar.gz 2cdac85dc652cc0dfde4556343e96747aefae678b14324b4eef7adfc9a9bfc2d6e7b3c9801da3ccdf9d18a52282edab054c2f0e1f6a7a06e3845c9915100195f

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.13 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.13 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.13 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.13 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.13 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.12

Changes by Kind

Bug or Regression

  • Azure: Skip "instance not found" error for LB backend address pools (#111428, @lzhecheng) [SIG Cloud Provider]
  • Fix list cost estimation in Priority and Fairness for list requests with metadata.name specified. (#112557, @marseel) [SIG API Machinery]
  • Fixes an issue affecting log rotation with CRI runtimes where kubelet tries to compress Windows logs before closing file handles (#112483, @ibabou) [SIG Node and Windows]
  • For raw block CSI volumes on Kubernetes, kubelet was incorrectly calling CSI NodeStageVolume for every single "map" (i.e. raw block "mount") operation for a volume already attached to the node. This PR ensures it is only called once per volume per node. (#112403, @akankshakumari393) [SIG Storage]
  • Make sure auto-mounted subpath mount source is already mounted (#112499, @andyzhangx) [SIG Storage]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.23.12

Downloads for v1.23.12

Source Code

filename sha512 hash
kubernetes.tar.gz c6ef86c6510a6aaff54c36190b22161d910b6ae6d9a1f0304094112ead3ee25bdbf6349cff9d5f926b287b4f75aba7befb8898d31c87f1ba19a687183e4ada09
kubernetes-src.tar.gz 5736949e8ea5dbea4c7a3ff332d4d69dca6db558c4337f189fcb0e0d712e5bddc18d580d1b61f979acd4262369a22d9e6c6c5ca36e62e1d0f5a3f51225832e9c

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz eacd137e88441b56b827460eb949a01f72dbc975a62b941b3d8093d168c191f0c59a6eddc7d27165071ba827b87d1d6ae80c372f1b1bf3a3462fe55fe6946136
kubernetes-client-darwin-arm64.tar.gz 6c4ef41a6c324d28aef169b320552acd40d3cd64165c99b213b5834b64977cc6b2d153c50be576fc9d196303431b2a67cf53f0bb812ad7f50c06f0f6ec873abd
kubernetes-client-linux-386.tar.gz 1d045d962c97bc064eae749981f60ecab48c38966dd4fbb69dc1fb861d5112360b5deae6e5ddab586cf11c574f1f1ba16c00d98557dc835a8d0521ac79265354
kubernetes-client-linux-amd64.tar.gz 55478ab38a23ffdeca0b4e9a45f75431d8bf18f6ad08f7f2a1151be2b92479abab590e526ccbc7196f737d52b85382a01c7b8fae0af186c5a6ce4baf2e4e03f4
kubernetes-client-linux-arm.tar.gz c80b390cbd3039b90ac956268eac321448264b3ab8e753e6651933b7a26bfb4c0f50717f2ca3d8298b175165bb56133c5d7305c34bfa7838e4c029cb05868168
kubernetes-client-linux-arm64.tar.gz 1abda0aa71ecddd6a4128bf1985070490cb83d16d2c5ebed257c312a764bccd71f54ea67a21c728a4cf8675ff5dfb8525709ab4e553aaca9ddcd448c26265a31
kubernetes-client-linux-ppc64le.tar.gz 5747070f19780cfe1b2fc9b38f004ce542136cf637cd4ab70de57ae87c6576f5d5efe58d4d8fe6870a224f26775f0865760b03cf06773c9c11d6503bbfa5ee64
kubernetes-client-linux-s390x.tar.gz fff1e69c75b64be5269afc1f801d8bf7e33343f386c9d932c4d96ec3271040793df0ce185260429c45f6cf9fbc025a2604cbe12ce01c9fe917a1684ad32398ce
kubernetes-client-windows-386.tar.gz 02851aaf61e8ffc2834d9baf825b7411ff25663a94778029e629142d296da97d1ab42e4deb6e12bd23f0bc09313ae477b192491d0d369d9136ad03c8f690d4ff
kubernetes-client-windows-amd64.tar.gz 95d47f3888f803a88f759c01f25d9b410a8534f745c79f925337b1d636734de71822859a427f1b5bc0d361ea391d397132031951602c984f5d4d56c61da483cf
kubernetes-client-windows-arm64.tar.gz ae333672d0caee1b58d71514e4734ee7a2ef8851d2fa674c6af6947c7caadace3ab1d12d4852f377ac2e23a0ec95c28fb95b1e46601aa88f5e1ead2d9e19a1dc

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 6e80857d208a72387f66b548a067268042b465712df6c2babc4fc5e2ac9349a3754a32cc44af5c244d23e34b0ceb9c973fe9a59fe51f558bb786c993a03a8a47
kubernetes-server-linux-arm.tar.gz 6dc1d09f58e4c615eea96e754449932ddffd2deac540dc0c1dc657fa1650dddce5c7364ca67b68288afa1176a76ead53baecf95abd4dbcaff93e8e4b09c1a0c2
kubernetes-server-linux-arm64.tar.gz 506303e92dca80821259a1842131b536cacafd0fe63d48bab0348c4fb6f5ca86456605bfb30a0f2993b7ab4a5d48b25eb1dabf8cc5eb4be91743343808db833c
kubernetes-server-linux-ppc64le.tar.gz 0aad929cdd11e2494baa46bd69256086281e98f5f76beb50b21f3494b3a24da607a6f1bd87fa960c36e6ff2ffd39a23a7cf342bffa337b12ca492c1c6db5b61c
kubernetes-server-linux-s390x.tar.gz 1a02255daddfd3c510f27bb480062493d8b0336136ec52e74b130362d7abe006193219cba115ec8296fc79f80a2c544e0fcc8a655cd7f38c3736af1f28b1c1e5

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz e35f9f226a214ee977deb467e60f6724cb626eed6a806c2e1c707f23f8bd6aac59d315dd3dab6d706889c4fb5f48af5919bfc606e01e6281ffa8c63b29c8bd62
kubernetes-node-linux-arm.tar.gz b808f29dbfeb825adc0eaf799e933286c47f0978d642b240737bfd9cbb36cb40f0499a31d3b7c2201f8e1cbafd29f4d5edda02c79df1952b0079354839532ff1
kubernetes-node-linux-arm64.tar.gz 3f7e085d8b0ed71128752beb2422438256443329a9a60812657149751b6a7e68382701472f2994fa23d28d7d00be3de7991080b9928576a72aefe0dcc29644aa
kubernetes-node-linux-ppc64le.tar.gz 4e4c00160811404bba2c3581cc61351ab79e90d697c886ca5b0215eec54353c474915c2b4274b48326f563b14c01ef5c50249a921335b5171c4d7582eff1658c
kubernetes-node-linux-s390x.tar.gz 701c8d8bf175cd89b670082d88db186d85c9cb68774c297e8dc3ed576475562f02e8e2bce7cb440d35d274139762d1814045b35f021e805bf0d1ad3ecbee50d9
kubernetes-node-windows-amd64.tar.gz 3f8f84063803add6281b5afe883c4b7ff594111dea89156a4fd9ad66f8e9e6c089cf167f3f57e4903d58d7c5b18f7c4ad5684a590c66581fbabfba6ef44232aa

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.12 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.12 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.12 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.12 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.12 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.11

Changes by Kind

Bug or Regression

  • Kube-apiserver: gzip compression switched from level 4 to level 1 to improve large list call latencies in exchange for higher network bandwidth usage (10-50% higher). This increases the headroom before very large unpaged list calls exceed request timeout limits. (#112400, @shyamjvs) [SIG API Machinery]
  • Kube-apiserver: resolved a regression that treated 304 Not Modified responses from aggregated API servers as internal errors (#112529, @liggitt) [SIG API Machinery]
  • Kubeadm: allow RSA and ECDSA format keys in preflight check (#112536, @SataQiu) [SIG Cluster Lifecycle]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.23.11

Downloads for v1.23.11

Source Code

filename sha512 hash
kubernetes.tar.gz 90fb263651c00d4b801feeed82446e0bfd69a42e804fdc98a08cf83c8f6fab24712d88b812bc8f70f6ffd0d33fc03099e889ec1cb141ef423c1455a59905a51a
kubernetes-src.tar.gz e25ba52f86f8fc1afc88e4ed814a75761b2ba85f5df27c96a4fca5c26358dee8e53cde914197d2884e69155f5b45bb09e7d1a1c340f35ba44e15bc958bf5cea3

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 9991740791a6ff7f1ca7de7869cdc3d9c3ba24a2ed57ed38213a17d714954db3d746b6ca004edd8ec869ead0423716b52fd97c24177a2284ff293fbdc2c50913
kubernetes-client-darwin-arm64.tar.gz d986a0ade1a9ef3200319d1b7b5445a7980e54e9febe7d3ad35d3b1fb3c5c1bd04e6d69727de7d1303cda9ac8b47f154270d63edf323d9a9065b0d4f0deda8bb
kubernetes-client-linux-386.tar.gz 291bd7e7c63618d48a58f17efc07f815d554efc61167a908f44e494d58928d408bc1772c1c231d4ecb41a73e1a2fa34e836691ced24ff16b5614c621dddc3bca
kubernetes-client-linux-amd64.tar.gz 7f4e290d5057b5d8376690d6ffa79e45217228df94fb23061b1be0adc71a9a573009058654dba6b58c5deaa457080e3a31121ea860e47f480f1a4c042e55f979
kubernetes-client-linux-arm.tar.gz c625bc6d1c1f7a454116fc7397ec7a65674aff022166eb06319e45c4bfd1297d1a218e7581bea180201e6bc6232cf6110c5a211201517108cd3164151869eac8
kubernetes-client-linux-arm64.tar.gz eb3c01b02b5ddc5d8f40ec401cdb81916711bed8ec49578ad1ff32f994638c9f5786032d24107b41879a4242415685174e3c94d0ee7496087b64c12f654e66e7
kubernetes-client-linux-ppc64le.tar.gz e3f2993fca11589eaf27d1ee2ba513dd349e577677fc3d8b357361ecb3fa1a99e911865ff88caccaa75c2340e3535bbcaee20592aedbe53da540e79c2e2ab845
kubernetes-client-linux-s390x.tar.gz 63a3dcd8f24754df3c5a2f5c970ce22d75753d027ac32da1c5a9d6bf60033e9093bf233df4d7594536e3419a6673cbf8ee361896adbea8078e895bd37254f74f
kubernetes-client-windows-386.tar.gz 1ff486a460ce6d5526c7f73f8fb6e87525c139aaf935787b6decaa00095dd1e49a5ec1126007926f737394c57b19f10de3480c9c513d728ee677f688c8998066
kubernetes-client-windows-amd64.tar.gz 9406aece47869024872b912d3ace880e016491ec709e28dc0e5019999782a93ec736b8af8cc453f5b5bb92cc801492056f60494be4a4d75314a73eb9dccd261c
kubernetes-client-windows-arm64.tar.gz 9e8493a3fca2d1003092a9d8c078543dfa4e6492c98daf6b84c7ec7cb7bf8e5bdc88083f815a42903c1c2cdc8a0ef4c2220aa217c330597ca1e9c8171314846f

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 767eec5ae28f8c0151379236e3f46870573964af4bf68ff38da28fc900c057d521bd6e2ca79458a95acdc6062069928f5a246ad69aff2e751f772ae9baec548a
kubernetes-server-linux-arm.tar.gz 4cbba22196d0002494d785f03138a6b1f747c4d00c686c9a52b61fe73e57fb28ac9314edd44d6136b3c82a457b5059f267c7dfc0270e55c7ff3ed8305cee850b
kubernetes-server-linux-arm64.tar.gz d69a5a95a05532b19cef65a020e6b8594f63b5d0fa2d29768c3c633a4170ac69f2b41e98aba1a4039b5f500d7719ad84f76010e229b296fd89ded9ac21298735
kubernetes-server-linux-ppc64le.tar.gz 9c9ba9a27567b5ca253bee66a65e978783ce02a56d13312e933ce829dba1c7d6c00648e552ea647dccbd1f858fb0d90842a7acabecea47be8248f3e46af5d043
kubernetes-server-linux-s390x.tar.gz b0b32c62216b18a8456788c6763eeac2c443e7c3781b29fcb356025d1c836c248c3f157ff65a94412b3b02306196259411e3f2e49a888b5f233d176b9e69541b

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz aace59e2477138f50799f5d536ae2370b1cb76425f0411d5022fc9ee438715be6ce27b8e6c400ede704ed2d6b599d13fb12b15eed272a4c73cec60ff08857747
kubernetes-node-linux-arm.tar.gz 044216a04fb7c995038d1f36237fb268396621aabbee7a9c0ac492ab6e70f697b80ff833b1a4aad7f32cdbd35dd90e6931028747c73e04adec4eb786a658140c
kubernetes-node-linux-arm64.tar.gz b31cdddeaffec6618cb1826b28f73e657bbd6b272801e4d0399ef6a5db1ea0afcd9fea050ca0a6aaa2714eeda573d0cf9441c462f0492777f07ac94d46aef8b9
kubernetes-node-linux-ppc64le.tar.gz bdb3ddde36fc29374d5f16bf7ee078aab94f20b0b4b356eb6ceba5088d5de694f26acf79c82a7758ed0976ae754a4a7b6122e31cb0cf66a0cf3a3d1e971bc88c
kubernetes-node-linux-s390x.tar.gz 170f42796052cb451793c42757e3e8fe69d200e20372edcfaecd42403b8a0efd7571c2456d6a21bc2c3a35c7b0c6ebcab6f83d21a772b6522a36f3d4b1c8d0ab
kubernetes-node-windows-amd64.tar.gz 3bb5daf6403bcc88a32470feac9a248d2a0a39f21375f5b0cc5fe081062061872d4269b4d2d977eb74bf6259cc1908e1bd138bb7eeb95d373e337f85979da1b0

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.11 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.11 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.11 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.11 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.11 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.10

Important Security Information

This release contains changes that address the following vulnerabilities:

CVE-2022-3172: Aggregated API server can cause clients to be redirected (SSRF)

A security issue was discovered in kube-apiserver that could allow an attacker controlled aggregated API server to redirect client traffic to any URL. This could lead to the client performing unexpected actions as well as leaking the client's credentials to third parties.

There is no mitigation from this issue. Cluster admins should take care to secure aggregated API servers and should not grant access to mutate APIServices to untrusted parties.

Affected Versions:

  • kube-apiserver v1.25.0
  • kube-apiserver v1.24.0 - v1.24.4
  • kube-apiserver v1.23.0 - v1.23.10
  • kube-apiserver v1.22.0 - v1.22.14
  • kube-apiserver <= v1.21.?

Fixed Versions:

  • kube-apiserver v1.25.1
  • kube-apiserver v1.24.5
  • kube-apiserver v1.23.11
  • kube-apiserver v1.22.14

This vulnerability was reported by Nicolas Joly & Weinong Wang from Microsoft

CVSS Rating: Medium (5.1) CVSS:3.1/AV:N/AC:H/PR:H/UI:R/S:C/C:L/I:L/A:L

Changes by Kind

Bug or Regression

  • Fix an ephemeral port exhaustion bug caused by improper connection management that occurred when a large number of objects were handled by kubectl while exec auth was in use. (#112338, @enj) [SIG API Machinery and Auth]
  • Fix problem in updating VolumeAttached in node status (#112303, @xing-yang) [SIG Apps]
  • Kube-apiserver: redirect responses are no longer returned from backends by default. Set --aggregator-reject-forwarding-redirect=false to continue forwarding redirect responses. (#112358, @enj) [SIG API Machinery]
  • UserName check for 'ContainerAdministrator' is now case-insensitive if runAsNonRoot is set to true on Windows. (#112212, @PushkarJ) [SIG Node, Testing and Windows]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.23.10

Downloads for v1.23.10

Source Code

filename sha512 hash
kubernetes.tar.gz 0e986db4a54b5d07773b17fae599680bf221c87d3c7ec185feba7aa23fe990f999af6ea28d8f5631b30f797104cbca8ac9a15bc029452d48b2eb9c6536c8abb3
kubernetes-src.tar.gz 4e44687dc3594ede040199eea620ca330c41d323c70a69e1cf0d4b9ead28afca5b477d55e89eaf9ee05462bdda1aecb6c2382e2fdc0d156c1177d5f703e7c2ad

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 90d6164c0ae119eb187c7d878ea41fc825f3d9cdf466bba3e0758613fce67736131f28fc16ca9c26754b4d117c5ffa436e144b59df84da753b6066c67f92c453
kubernetes-client-darwin-arm64.tar.gz 72fb1c734a6f0392ac98db78b365c0816fdf3c995fbfce35bd46034959c8cb5939b09affdcbc8e57866f754866456646c84dccc7a098909bfd57ec559329e64c
kubernetes-client-linux-386.tar.gz 59dcae5a3cdf9e240017cba82ec07fa97ad5fbd8a17d09388aabc38529027fb4c9c67c55741157cc888e514929bcd5277fda5fe7eedfe7d63c3374b62dcb0e0d
kubernetes-client-linux-amd64.tar.gz fae98e5e61dbf869428d14e05a2daa60d221a41688faf016157c5a09b400ed2a4a49c90681fea0bbacc75e0e158f75da62d026a6fedd37f9e52d56e3112f25c9
kubernetes-client-linux-arm.tar.gz 681728f0ab837b1fb679e881fb0c9e728dc93bdfff9dbe3e333234166192d79b907a991d21e66a25ac56b16bc446332cb4bdc9891bb5334826f07c159617ab49
kubernetes-client-linux-arm64.tar.gz 58339517a0a89204f357354173df493bea41149a100dd91f73f290f19ad8903b2e7733bf1c2acd21be9266b50a55f1dd0fc039af1f42ef98dc10ce0f86185ab3
kubernetes-client-linux-ppc64le.tar.gz a3c703613749e61f4bb618fce2091fc5c10eb85fc4507838ca0c79765139948cfbc46ee60a2218d95863e0b2ff3369650a4ce0a59fdfd548a921fe6a3cea54f4
kubernetes-client-linux-s390x.tar.gz 8f4d343205b3d2b0ba61f1639e76edf7934da3c7316095af510c9adc9a3a623065b4a62da4bbfe6ea95809020143751cf21ba774ba6b5ece4dd2de11efeefb70
kubernetes-client-windows-386.tar.gz 292f13a099c04dd74af026f67a13d5f7ff85f215e2b5fbb784f07e2408ddf2f0041cac4205043415a5e8fdf4183e9589c8c68bdef90e426df0747f743fb3123b
kubernetes-client-windows-amd64.tar.gz f89a6ecef52164f5f77274ea4240cb5000a9a3b200af85187d7194850ff2e5b8fc89fd41ad4311452ed80c8785b2a5f8cafe3c3de727eb68aa1a0b688f198647
kubernetes-client-windows-arm64.tar.gz 16893e512a991180ca8e0d7c052e74b5e35cffb3a37ad92623ff376e2fc9d4b77edcdb6d7fb147c4de00a7b419ec04043aeb4f33a1e7114179885494c36c2948

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz d587c2106ef717e6c32d4480ff9cab768a4c58c40b65c8da61446b18aa071ac41a09d5d854e1577f3550b2df84fdf5685a3b9f5766acb9eaf51e2ae587af3b08
kubernetes-server-linux-arm.tar.gz 244d57c42709c3f45aba508eb800ff228d091d474b8985689d455f916bbf15a93c081fdc6c1e1dfc7fac1905ea411cef085b88ebde41e0296d15146d2b49490b
kubernetes-server-linux-arm64.tar.gz 4a512a1440d0cbdc2dd6a4e43e4aa6dd23835be576b6aa58b4cc9c5c502f607d3a7aebefbbedaf55c5778f3acf102d5c19a8822203e9638b3bd4752d23a46d38
kubernetes-server-linux-ppc64le.tar.gz 1198ab579e87585f335c310fae173d344897fdc0a503c560612d7961b0cb737cced2a1a9408b144b15eeddbb37298dc997a75c984242e5b97b7f58009d0fc71a
kubernetes-server-linux-s390x.tar.gz 3f655435f01106d64de55a2c688e086456ec2265c6719ab0c80064c49666c9b27b7664e69d76df32540873f78480d6f8751566c018d06c4018aa08f51c9e15c3

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz fc0a7e46e0b3611f7c3e3d50af377ab24b154590c50d75bf17572df71c5fb5fa2de50db246dba114f0714e8f0b5e7da70f01065bda6a9be2ada958af61d9c0a9
kubernetes-node-linux-arm.tar.gz 6b00d621d862899aa858ee838b3767bf094b36ff44cd9f414b5d429a40f26f410942f4325652908fd92dd1117208a1163d246a309a6b3344750f6dece847ca9e
kubernetes-node-linux-arm64.tar.gz 4fa47526aab15142e88d3954d68299583427511154076bcc849826671185621b9f2ddc9fdd31f958f5a257234d9def2f0795497cf4fdfaf5b6901204f1044862
kubernetes-node-linux-ppc64le.tar.gz 56522b8eecdc8a85e337dab05957277fe1f086e7029d649ce234423d014ad48557d4aba160fc48f0df72c108a43b0862f7fea24ebd7e80a643a2e4a041181de1
kubernetes-node-linux-s390x.tar.gz 3bded271505dc71328ba9e16917929e0b83c9a89615fc4683bf5dde565a9e4c5f54e85618bfb70457ecf3218317754efe2d599cc46efb9f2ffa6ea355c6336ec
kubernetes-node-windows-amd64.tar.gz 5408c4da6e6866cc8a962b43e1f91981d7b95e530c3783d4f9849b9f848237569f061e8739f7cc134e6e5807627cc46e52c9a38ce9908eeacf319d09df7b738b

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.10 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.10 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.10 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.10 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.10 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.9

Changes by Kind

Feature

  • Kubernetes is now built with Golang 1.17.12 (#111465, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
  • Kubernetes is now built with Golang 1.17.13 (#111640, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

Bug or Regression

  • Fix JobTrackingWithFinalizers when a pod succeeds after the job is considered failed, which led to API conflicts that blocked finishing the job. (#111665, @alculquicondor) [SIG Apps and Testing]
  • Fix memory leak in the job controller related to JobTrackingWithFinalizers (#111723, @alculquicondor) [SIG Apps]
  • Fix memory leak on kube-scheduler preemption (#111804, @amewayne) [SIG Scheduling]
  • Fixing issue on Windows nodes where HostProcess containers may not be created as expected. (#110965, @marosset) [SIG Node and Windows]
  • If the parent directory of the file specified in the --audit-log-path argument does not exist, Kubernetes now creates it. (#111226, @vpnachev) [SIG Auth]
  • Reduce API server memory when many CRDs are loaded by sharing a single etcd3 client logger across all clients (#111649, @negz) [SIG API Machinery]
  • Updating kubelet permissions check for Windows nodes to see if process is elevated instead of checking if process owner is in Administrators group (#111079, @marosset) [SIG Node and Windows]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.23.9

Downloads for v1.23.9

Source Code

filename sha512 hash
kubernetes.tar.gz 78d2929a7582e3b875bf00f921c580de42538aa05dd27c9cc16b64109c504aa11961408e07363d77b8ea0177ff17ebc4b37dc729dff32dddb713e864425d7494
kubernetes-src.tar.gz 91e861a195735695a41035a694863b0483b7f4f7956b719c07ee0b31dd1a2f16b3e5e91643c0e35ad3f5ddedadabc307a728ede03ca94c901698353c65622431

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 1ef3d70708efc694932ba94b860044dcb0ad3e1513f25f88140f8391faa142a2160b35e479feee5612d6ff3f156dae8d9fd4b23e3f35471fe4932fb64ca8a188
kubernetes-client-darwin-arm64.tar.gz ad087e869b0b9b79fb8a0f31ecb4145ee7fac338ad73dd3f64c904518ef04bc0e8a7f3b7ef403c1f3633980b368e2c9739dc7797ce2aa99df47d58766f169ee2
kubernetes-client-linux-386.tar.gz c8c1111988d918f385f97ba1563369d33fef8b52204f621919e61980be0e6ee79242face11b46841f17dac8a0d5164bcf54de3668655debd90b831be0b7033df
kubernetes-client-linux-amd64.tar.gz 3128800d337cc9be19494bcc724b9ea112444be6194136ef0e49b969394522533262a3f92554cf3543ae9b1e0bf063de7694f73ba230154bf2e59b74599f40ee
kubernetes-client-linux-arm.tar.gz b5e125472055b35344b2ccfe34ecef545d8e3501a150479a1adb9c9218a96011de3ef36c9bfbb352ded41fdbcbd460bf4c600648fb586fec3bf9685eaac405b8
kubernetes-client-linux-arm64.tar.gz 0bdde0bca2208de991ff3196ca73886f7129cd8037608ca8079973fe73d2742adb162fb693db6f30e0f4e4d15f7591e19e95ac1ce70d02cae77ded8e746f7b35
kubernetes-client-linux-ppc64le.tar.gz 325a42eefc77305015ae23d61185ab728055aa622576c79ef5264120d0c00e118a2fe9c1bcbcc3250ee0088f380b9ea7f81755331ad552fe16e841fee55072e9
kubernetes-client-linux-s390x.tar.gz 7586d553bee0a49226e973f0851b8f03394154849904e4b88f632d80924906c02321d78ba3ddfbd2be1c6328b7675a568fbc8ca8973bacfb6d11b6252d996479
kubernetes-client-windows-386.tar.gz 68eb204d5bb4411d63c87838088db05d97349c10e6fae7f2e9f55ee934a347246f305045cd756195f20702937ab9c5082adaa84264f72841d81dbd8cbbcef6c2
kubernetes-client-windows-amd64.tar.gz e21c10e5fe4f1b39f3c69e5c48030661ffc8e5fdfb20c8274ea61ad4adb4bccca557fa403a6250aa5ce17daeec11c330f9f29153748f5114b7275c2e87e10693
kubernetes-client-windows-arm64.tar.gz 142a19090b3a169fc43982bd8b9a5d1d67855e27fe9ce2a9e863d83ba948d2e880c3b2ee3561a5d821f27dbdbc2b15af3eb2152d3244f1dff2829cf83b48e793

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 90d5ac43e6e2794ade95b74328f68d1b87a6034b3ccfbb73ca8fadf14b2a98fd1700fe2790b4c963375516972505774e30c13a7cd4f10099f80dc4dafd357c0c
kubernetes-server-linux-arm.tar.gz 5dad2818d5457572afd0a8b6c74004d9c36c94614601c237c72585eca214330fb3aefdf3cb778c17cf0fa3a062f6d9416b11ae32229479ed0ae603c148a3f28b
kubernetes-server-linux-arm64.tar.gz 2cb1ae6078250a26dd405ad98d9440cc390f87b56543497299c0d51338cb563c38c57308fbea3e96f8c2ef56102d48b8bdd4ae6c22093ae884edebe14b0f766c
kubernetes-server-linux-ppc64le.tar.gz f1905fa13f308aabe453a35bdc2d194d6a55e5db221e61ddc3436a628d850f708da19240801210ae4804c8786d82d3005d0f252dc969bedeccfc5141a409fb68
kubernetes-server-linux-s390x.tar.gz e05f415e64424fb5c2ce17dc6008c9340e8a69964cc397a542286ed2a5970c19d415cfa13b2122958411b24893cbb3f817a627a1ac7f79fefa39ee0de204fed6

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 44140e5ebe0441e4cf6a155250b08c6e321cd32208ac672ff46af97c01b40de42142ca019a892453f024db7894acb77e0421c3fc6f6058243d6c0ae55d212cd3
kubernetes-node-linux-arm.tar.gz 797905b2ae3b07934ef6a3c546e1343d82ad3f45555d9963a1c4997c9bfaca79226b6825db8b36078289ec350864d4759dbe1e84548ebdc1a91439c9f0e4fdc1
kubernetes-node-linux-arm64.tar.gz 4e9487951ed430e307686500ff44d371aac184d02b453495f6b643f24943d347739a9441322e94173afdb801ca47954c99d8e2744c7bbc6707b79a9733fb3903
kubernetes-node-linux-ppc64le.tar.gz a1d74ef1d83aa1b095537a729acc61995ca1b18467febcc5ccf25399579d69aea05b0bdcab3c97e04a172edc61f54fca2ccebfbe9461a72079fbb800095375e6
kubernetes-node-linux-s390x.tar.gz 4b4c5441f5d3d9ac7347a4da2db5e453986a2e0cb53cd17b77dfca57edb543ec6a96b7a9ca2e3732063dffe84cb1873c883dcc0562db336b367f8c902ee7f9b5
kubernetes-node-windows-amd64.tar.gz 4710f1b88761e735f59ca4828c4e60fb9a125a86dca826355d10c6c14e9d195a300693e1a26c86b75fd9790ccf4fa803bb77969f20a64e096037b7a3dbb02294

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.9 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.9 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.9 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.9 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.9 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.8

Changes by Kind

Bug or Regression

  • Fix a bug that caused the wrong result length when using --chunk-size and --selector together (#110757, @Abirdcfly) [SIG API Machinery and Testing]
  • Fix bug that prevented the job controller from enforcing activeDeadlineSeconds when set (#110545, @harshanarayana) [SIG Apps]
  • Fix image pulling failure when IMDS is unavailable in kubelet startup (#110523, @andyzhangx) [SIG Cloud Provider]
  • Fix printing resources with int64 fields (#110602, @sanchezl) [SIG API Machinery]
  • Fixed a regression introduced in 1.23.0 where Azure load balancers were not kept up to date with the state of cluster nodes. In particular, nodes that are not in the ready state and are not newly created (i.e. not having the node.cloudprovider.kubernetes.io/uninitialized taint) now get removed from Azure load balancers. (#109932, @ricky-rav) [SIG Cloud Provider]
  • Fixed potential scheduler crash when scheduling with unsatisfied nodes in PodTopologySpread. (#110853, @kerthcet) [SIG Scheduling]
  • Kubeadm: fix the bug that configurable KubernetesVersion not respected during kubeadm join (#111022, @SataQiu) [SIG Cluster Lifecycle]
  • Reduced time taken to sync proxy rules on Windows kube-proxy with kernelspace mode (#110702, @daschott) [SIG Network and Windows]
  • Updated cAdvisor to v0.43.1 to pick up a kubelet fix where network metrics can be missing in some cases when used with containerd (#111013, @bobbypage) [SIG Node]

Dependencies

Added

Nothing has changed.

Changed

Removed

Nothing has changed.

v1.23.8

Downloads for v1.23.8

Source Code

filename sha512 hash
kubernetes.tar.gz de69d27776c066cdc450b65d075fff9efe39175823aa3cd65cf8e431ba397c38add39ef53fc4d21b74c900d9bb5ed055a0f05d7a25a429c2f1f3967c2c5ff01c
kubernetes-src.tar.gz 25386d6c7c0fc9829bc178c8a24b3d0fb65895495c0c3f25e6a3072728ee39e4b55855e1587a6dbf2fe75bd8a192d908dde7843c118f00fd71c9f76b14a9180a

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz e4a95fd15f417b51baf73f096a940adbe4426609f0fde2077c93b50e7b0cd5da8086e753c47112bc71575b00f8efaa030f92abc5f100a7113b9375ba4b9f58e9
kubernetes-client-darwin-arm64.tar.gz 4296920ddfea2fee14b680a231a72688771030387112f51f502c52662b3c72f51d254d579ca0f24ed6d6327056f1b047e52d43f2ff2bb6bf4267b3b30f1cbdbc
kubernetes-client-linux-386.tar.gz d600fb12e7fa00374875efb6d2ede65790cb64b149bca6db2eecdba1fc8845a0a46126041cef3bb3489180ce2825ed116257098f6009912635f1bb6560977c21
kubernetes-client-linux-amd64.tar.gz 071de5987abc6295c3f9843a31b3fc0bea106f1ca3498bf089f867c3914a20a8fc563fc0763640c589691fbd9ab7dc4280d8f8de2b2904bde13f8c4f059580c9
kubernetes-client-linux-arm.tar.gz 4677affdf764a5983ab64af3876c120919ce319e0eb67f922439a93f13a3f62de8b81c652f9d1947572c1f051f49a54157717d6a1416b49cde7f76c49b8e342e
kubernetes-client-linux-arm64.tar.gz 3ac6eb2dd77a71ea608777429a8efa0c26aa3bf835f125a114c3d2312337a00c5b96eaf32edaef319c055be023d9862ff6f7fc06c0056d4b2491eaafe34315b0
kubernetes-client-linux-ppc64le.tar.gz 233efa4ace4c9ac36b9eae75c32d477adc01c0690ecc069b4daecbbe2aabb50514255ef659bc9a2ffd2ceeeadd33965aa743299bf02814d200b11dfd2061d170
kubernetes-client-linux-s390x.tar.gz 841d6a53cae230bcdf58e65ab6d780737c41c99f0144443905caad62b508b94f738640e2fb7a43ea39bb82387ba537389fb86d555572de1eb6adcf8d80fe31b4
kubernetes-client-windows-386.tar.gz eefcb9290ac80c6d0e50fa77d4fabd8b71949412c37272ad33468bc31251d031b877cb26e17ee56d51616347331f5f9d17fa170a44d1f0f9530be9c3605981dc
kubernetes-client-windows-amd64.tar.gz 30b82098d3dc3fbc045174dddc273ba38ade9a64e6fb5a4a70646343752ed882f7342bc78c70b6af02fe9b42a20ef84763ea89418aa98618a19c39945aa34eb3
kubernetes-client-windows-arm64.tar.gz 76c7f0e5ae9158d52a3bf76e638fc750bda8a4f207a9cfe3d99f2f1035b85d4feb8443de01d2daae2bf2414814286929b523341d896c144b6383b7a98b482cf1

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz ca0c4023c1bbeb1aafafb073348e42a016f6793b64a84b33f9619971950ee0698cc6f008418c707cf5126ca82bff936508e63e02ecf89f38997d2aeb2f5facf0
kubernetes-server-linux-arm.tar.gz e33be64370b1418aa3ef935f1a01e4aff3285c0f3dbd8b0af1be60af5bb245f291671b557cda6c41e4af82da49b42bbe1562e568011099195e24f8add036227d
kubernetes-server-linux-arm64.tar.gz 86350da9262bdad99d997bd4a6f1f20067a77e2a8d0b4359d7681dfe0a272b42fa30bc4de47920aa8aadcfb5ab0d9b626c8fe3c268db0ff9c35d0f32f94e8cb0
kubernetes-server-linux-ppc64le.tar.gz d0954bd675eae099195f8986a9c2ada9eadf804794f8bd91371c53423173398457987ef9ca73664f097f8301486f6416f31a7540bbbcbc844fe3ca7fb58529ee
kubernetes-server-linux-s390x.tar.gz 017d4026ff30bb10460d22571a592c50f6c9496b49acecb8a333174b33a8a98cd3f234d49546d45947d9af1c3ab5a1e337b42015968e9572b482bf7a4a58d181

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 3003ba3faa6d175c2e0b898b3a07b3bb22e6509a9fc7948582d92009a648b83bbe329a8c2f843a466f94089fd6a26529c89abef8c769f406a55f39a09bc55017
kubernetes-node-linux-arm.tar.gz 3938d4d33cbf021108ea1dc03534286c58ad68ee059027a239548cd2bce3708eb288f6893368014f885b20ebe26c4d556d27cf8fdf29bf478e5ed23bdbfe5325
kubernetes-node-linux-arm64.tar.gz badf58767eae4a20f81200f609fd5966479b455a3bd9d3833d5cbb0e78c31e6d624c7301cc366b5b07da6299d2d5e0228db2aaf78c43e947a3a80a4db210965b
kubernetes-node-linux-ppc64le.tar.gz 301c234325e9a588315ff128469c49b64f471c6a40ce0567b745f8315bbe8847b8ba9e6e098c317bf20f0613e65e3d604afba12edcb84154e636427139344cc5
kubernetes-node-linux-s390x.tar.gz f862838e1f7294de4b58359f0ff4e4644ff6229f83bd615f35aa5b625a1754a4c6e538fb2f911c13ed434fd81d2e2feefdbd4fd0148149ea24d3f6df8c01a7ec
kubernetes-node-windows-amd64.tar.gz cecb68bad4aea5790afc72c895d20d93bac53985437dbddb0917c7b9b8f876831758a66595c70fd8c142140a2eecc8d2fc2a52885ca4d062ea34586a2871f133

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.8 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.8 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.8 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.8 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.8 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.7

Changes by Kind

Feature

  • Kubernetes is now built with Golang 1.17.11 (#110423, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

Bug or Regression

  • EndpointSlices marked for deletion are now ignored during reconciliation. (#110483, @aryan9600) [SIG Apps and Network]
  • Fixed a kubelet issue that could result in invalid pod status updates to be sent to the api-server where pods would be reported in a terminal phase but also report a ready condition of true in some cases. (#110480, @bobbypage) [SIG Node and Testing]
  • Pods will now post their readiness during termination. (#110417, @aojea) [SIG Network, Node and Testing]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.23.7

Downloads for v1.23.7

Source Code

filename sha512 hash
kubernetes.tar.gz 3dcd42ec4fb7fbc3b00dc18414bcab7438c22b8040e9d66998e844b289fba31b3b951800cc9e6d88722d1868c27e7dd3873c6d761d43c247570f83e8e6e5ac97
kubernetes-src.tar.gz 80482eef8ea1c07c0c5e272d8b0fb56c22998d20f40c275d19d3d33197f4a7a1c4a71783f3cbb71ceb21c5ac990ee253d28888eae68e8dbfbee3c39d0f047af6

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 15ed16d69f01e585e5272e0284f7c9112eaee22d73afd781de40f8f6d40e71ecc46cd6951beb3a171b539c8b343ad695ed67ba67779b18d8c668a20ef07e1c35
kubernetes-client-darwin-arm64.tar.gz 0f8eecfef713c2eea9793f36c90150c2d7dc444c69504f303f025c74b1f74a4d92fd61af379765c01eebdfb696812f98f256225d5cdbbfc34c26d0d9c4874ced
kubernetes-client-linux-386.tar.gz 16a86a9f6bf6c28c541d8b73e6704693d31fed4a09afa16d8b5414fd091133e2746c5f477f6571eec9a7540be88a9ab18ce644e99e6c8ad6e79506e73b0a8211
kubernetes-client-linux-amd64.tar.gz 971d2bb7fb2ac098829a91a5451687bf016f8cccda80fa605b19a55e99b5a411ff838d8ec25b5abd2e3984b6807c7ecb9dd1ba744279be65e45495443ecb1726
kubernetes-client-linux-arm.tar.gz 6c003e494b26f7ff03653667db2abce5533817d733c95c64b5782374a4ca40eaa54682a5f7d0e6bb5043c3e43676fed868f6bcbb233b410bc999b276bf65a8e7
kubernetes-client-linux-arm64.tar.gz 9a07bc8671a398c01aa31c5c3facc1469b708a7f388081c4e4110ee852af8a6d74cc700c925b3c867852e259477a232a992eda4c1211a9ac91233dc3de5f7ee1
kubernetes-client-linux-ppc64le.tar.gz c60592be14d218799dd15360d004adbabd26baf73137f3ada526fc2d6e6b7e76dc8b2dcba939fc2c92d23f1a837fb5ae788ddc1ff06b5274f5aee503e27cefd4
kubernetes-client-linux-s390x.tar.gz 6958d77d1f126a41ddeb4f5e7b5dea953c8174924165a5ded0c5b518349ae526cfcc869f8d9aa6f4cf078480877b89a9dd6a5b4c2f40b556e16f1618cf74e32b
kubernetes-client-windows-386.tar.gz 5ec708da681544ee7c8729f5c3afc44c869c3be4b8fdaaed4668f13c2b9a40f8a138dd01b93595f84ec4b3982027344a3146b7aea1617f60ff752f8182b2f792
kubernetes-client-windows-amd64.tar.gz 60e52a530a4dba6d79abe21c55a3214b070840dab97f6da193cbe4f41b0a5a0707378ae7d118a018d5aec16b8a149784064d37f1f29cad635b9c876e0df744d1
kubernetes-client-windows-arm64.tar.gz d5d35a538ba553c92601bd487b309269b0cd8c26c108d6ea4d107326d31ca58c44292bf0fc12612a5a2357f64053cb042a09cddb6fea6aac6a4d4e62019fe53d

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz de3dccf8a7e47542094528f6738b1d34073977c6031318bacb1656656860054e6cfe21280b0f314a1f1c73d393929341ad4866604ee73eff37af5087825a766b
kubernetes-server-linux-arm.tar.gz 5ccdc82f5289bb7177b81e909c742a43f6dc8c5cc80abeb97808a7097ab08019eea0ae5c959f7f938457cc684eb1ed8c90e5f436a1b8e85d91a7dbe9a34e331b
kubernetes-server-linux-arm64.tar.gz a463d637fe8becb4e1b5b99b1cec44538ff31a386c7d331ba7791904773f931630840657f723212af65e519a7830b47bf50063616ee8c906212a6751bf426333
kubernetes-server-linux-ppc64le.tar.gz 09ce40a6a684e0060ba41a9dd5eeb6da9aad9098f284b4b082ac386fe12fdb55514c4987455cb6aaf80acb74c637b77828ce7909da82bcf3f2afc094f3864bee
kubernetes-server-linux-s390x.tar.gz 3d3f904f935915aa12b36ac80c43275250c7a4b9a169aa641e225592bbdbc9da28c923056458211e5d3aec6f7344477d5fa9a9239d59f47599836082926f5de6

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 9dd2ec04902bb6dc0eb0a75735c8f2958ab7dc106487ac962564e6fc514c90581ac32e15e8fc658c598f0b6c3d199a938c4824bda23157f379510a8cdecc4a7f
kubernetes-node-linux-arm.tar.gz d73d8cee8bac5b977ec0b11917a4d4eb8d5d33c8a2548210a806a3882e449d242504934ec199bcac80c31c53542f883f69d48c599eab3fe2cd2e3d95760ec966
kubernetes-node-linux-arm64.tar.gz 4498a1b60813f0e1461c00d5cb2351a522f3e7c9d9d81ffcc99bf925378885aa5dbb64fae00dda3afdbe2f9517eb30ea28cc18c11158a9a46f69c0bc3244f2c2
kubernetes-node-linux-ppc64le.tar.gz 50353816f8772d830d8ae05e1a535553fbcc2d552d00ba822b777c17b8492ba4ef5acd03b1d9f56583064893f173e3cf7b67faa4bdd77c11261a21c32bac8e4a
kubernetes-node-linux-s390x.tar.gz c2c4b35a42ae84d498c5e27175bcbc641e74aa42056df2f36c6bdb35f6330b5aca26c2d5b8656750f13b75d02c092d0f0ac5d4ed287d13033f696f243af9ca39
kubernetes-node-windows-amd64.tar.gz dde3429d945c4b8c22a551e8663678e76d7a1136b9f9012ead3af534582f738092b0d5e629a983ad446d0396db143188a440edf471ee8c102a331006fcd00ab8

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.7 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.7 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.7 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.7 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.7 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.6

Changes by Kind

API Change

  • Sets JobTrackingWithFinalizers, beta feature, as disabled by default, due to unresolved bug kubernetes#109485 (#109491, @alculquicondor) [SIG Apps, Auth, CLI, Network, Node, Scheduling, Storage and Testing]

Feature

  • Kubernetes is now built with Golang 1.17.10 (#110045, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

Failing Test

  • Allow KUBE_TEST_REPO_LIST to be a remote url (#109512, @eddiezane) [SIG Cloud Provider and Testing]

Bug or Regression

  • Correct event registration for multiple scheduler plugins; this fixes a potential significant delay in re-queueing unschedulable pods. (#109446, @ahg-g) [SIG Scheduling and Testing]

  • Existing InTree AzureFile PVs which don't have a secret namespace defined will now work properly after enabling CSI migration - the namespace will be obtained from ClaimRef. (#108000, @RomanBednar) [SIG Cloud Provider and Storage]

  • Fix JobTrackingWithFinalizers that:

    • was declaring a job finished before counting all the created pods in the status
    • was leaving pods with finalizers, blocking pod and job deletions

    JobTrackingWithFinalizers is still disabled by default. (#109486, @alculquicondor) [SIG Apps and Testing]

  • Fix a bug that out-of-tree plugin is misplaced when using scheduler v1beta3 config (#108890, @Huang-Wei) [SIG Scheduling]

  • Fix kubectl completion zsh to use any command name rather than hardcoded kubectl (#109235, @soltysh) [SIG CLI]

  • Kubeadm: add the flag "--experimental-initial-corrupt-check" to etcd static Pod manifests to ensure etcd member data consistency (#109075, @neolit123) [SIG Cluster Lifecycle]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.23.6

Downloads for v1.23.6

Source Code

filename sha512 hash
kubernetes.tar.gz 6cf7b3b00ab86a74cbaff599b121189ae73f7506b5041ee5351def4c2b9ee689f0fe23ff5a6ee682d994d3fdc47e54ba32640fd915a384cd33dfe7213f0e001b
kubernetes-src.tar.gz 3a1acc0889add88f72bdfdababb54da8ef61e2d82176e2b102a5b73c3f3110123a9732bc5a8c657c75ce1bf379d1b13b3cc1941042c9f67644f39cf55891d24b

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 96faea4fb5df5c34b4f9affcf1612a70370af718d87587ed1f1e28d9df8c57e469d5902c09444413fed5f5dd578aff88a9efec9c2df201b1cbc461d701469c7b
kubernetes-client-darwin-arm64.tar.gz c30428e83089d38f47fd39a583324b122fc19a9f39ee4454f218f61352a4f7638e94f9bc1d223ce76732115999434ce039e4762df954629126460566238c0027
kubernetes-client-linux-386.tar.gz a0da9c6f41c0164c81b7e805533020221969fb68f0816c58e1e982a2a744350eb552aaec2c5e4a9410568dbb6b72225c901140cc308f132843cc53a058e6f47f
kubernetes-client-linux-amd64.tar.gz 626620f7943575f4568b625b27e4df5e43073cf395f5666b7e44424a5964f913a6453e8fdd13990a52ca3187eb15c0e73a5f8d6ca2f297ca842afa0d111a7254
kubernetes-client-linux-arm.tar.gz d8ab4acff263761d89ca95cfc207d17d51fbc8c6b79f1cfa9782ea2afa15cad8341265b74a834b0bad808f24bc40bfeed1e12e4dd4a31cf82aaf103a5ce6c4a1
kubernetes-client-linux-arm64.tar.gz 28006d52582842479f91094506696957a99a74fefaf3dd1e44ded910468f616ef918b40935027cd431026c9a8ac70dfb5adb57fd6f2ec96e717258d1326dbca6
kubernetes-client-linux-ppc64le.tar.gz db8e85c87fb12beb12798e66a49eccd72f590eec1167be226960786276c8d31f9aed694d4a0e96efccdd29886b587f5e4ff6ccddaf354867dabba910ef7396f2
kubernetes-client-linux-s390x.tar.gz 9e3e11398fb5f980517ef4eb9e732db6fb0301e820a50d141e6bf90775728c131c0f0eeae183c0b9314533089e7f44643abb0b55b1c2a1e38a7cea4f739acbae
kubernetes-client-windows-386.tar.gz b0afd0e85826fc061978e753682926b61a7b4a8618a8e0033185f69774afc7e7bd0de459d7cfb2bdc42f34398959953a8e57752766b9c283f517d6c6ce1c6ce2
kubernetes-client-windows-amd64.tar.gz 76c7564d4910a03836eb43d5c18692b69d54862e30dea11994f5c4656c8911c08f083a2f0b0ee1e4756a4dd659b1a549bdeece8aa571e4e2ca7beaffb8523c44
kubernetes-client-windows-arm64.tar.gz 1f9974ebeba3b8f751c561a64cc51c7ea109fb595c5177dfae87e9786804b19302fa96d7bac90fc8d99c04ef48d7400a0b11636a0264034749c7931bd01cf687

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz a33fd6a01b9a10e00fd6cea33c0e621fc96c83ed434564ec3c36d1c2589d5466413b3c2993dacf5d4d40560ea3f99afff03e126b4707f1aa0816324d3595af24
kubernetes-server-linux-arm.tar.gz d6d7a7d16538a3484aff2c25358db81f1bbeb01c8a41930712e086a407fe9099e0100b7e4981b86e85824478aa1091d71deab211dd87582fb2869190cb8bfbeb
kubernetes-server-linux-arm64.tar.gz c5f57460c375f7eaae7fcb14781c8131a1ae20392319648191daddd2397246831f1c5d2b8d3e4d859df5e7c17e23056279553a4b3e72835a9fe08a55da500140
kubernetes-server-linux-ppc64le.tar.gz 219bb24fc0a483e325bde0b982ba10a561588dc39b48eb5fd0e1cc31d1b6c2cbb6217a424a2eb5c64f4210fc3d9f692308171341b075a6b07c10c7932037c0ae
kubernetes-server-linux-s390x.tar.gz 90d0cca8028df8bfa260f4c9abba70747a5aeedcd59a92459bb5e94c2f0caefa39c68f11f38f916994d13fb5553134faa92ed735ed329f588b2c10874f95202b

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 8083afd4dd05659e364d3592f71b0694f9c3d4c20d2260efbb2814ff8b914be3ced6d4690f9cf546522168b1ad6aab2ea70d9092cff0c7207db61514294b452d
kubernetes-node-linux-arm.tar.gz 1c46b5cf9a2baf9d4c117a57efa64447701d1e55e249ab8a367008630e05036607fa0afc6c0e3eedec3efecd1a4d5c13316ac1034ae97e9d4971fba0f0c2e2ff
kubernetes-node-linux-arm64.tar.gz 5e36c75eeede07c31ac29961460ee5cc32e6ffa7f53cf6f674e2087d11668f6bf27013daf72a4137e041f17c114d263f5c7b6c891ba084ecc76029f63bba1011
kubernetes-node-linux-ppc64le.tar.gz c4eba116db85a94e5e647d8ca7f5d334a9aa8d42c27e10077eebce28e7948adb06d6cefcf10db8abd762960dcd7d4b6ab2c1caf7b079fc84bfef356cb822eb07
kubernetes-node-linux-s390x.tar.gz 50905829928ac041d95243bac78879aa607d7a5bdadb92495e022f856e1f332204ec4e0cc2b9c848b1afddf1cb8b0fe249944537005a97967237581bc42e48b9
kubernetes-node-windows-amd64.tar.gz ff71310b0453efb8ba364572eb5ade35a9826f3a6b1d5d215764b5e165054c3cd77e24b7f75a443c9ab577f39899212fea0fe849e3d0a8bbf786ac568f2c63be

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.6 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.6 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.6 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.6 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.6 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.5

Changes by Kind

API Change

  • Omits alpha-level enums from the static openapi file captured in api/openapi-spec (#109179, @liggitt) [SIG Apps and Auth]

Feature

  • Kubernetes is now built with Golang 1.17.9 (#109462, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

Bug or Regression

  • Bug: client-go clientset was not defaulting the user agent, using the default golang agent for all the requests. (#108791, @aojea) [SIG API Machinery and Instrumentation]
  • E2e tests wait for kube-root-ca.crt to be populated in namespaces for use with projected service account tokens, reducing delays starting those test pods and errors in the logs. (#108860, @eddiezane) [SIG Testing]
  • Failure to start a container cannot accidentally result in the pod being considered "Succeeded" in the presence of deletion. (#108882, @rphillips) [SIG Node]
  • Fix indexer bug that resulted in incorrect index updates if number of index values for a given object was changing during update (#109137, @wojtek-t) [SIG API Machinery]
  • Fix the overestimated cost of delegated API requests in kube-apiserver API priority&fairness (#109216, @wojtek-t) [SIG API Machinery]
  • Fixed a regression that could incorrectly reject pods with OutOfCpu errors if they were rapidly scheduled after other pods were reported as complete in the API. The Kubelet now waits to report the phase of a pod as terminal in the API until all running containers are guaranteed to have stopped and no new containers can be started. Short-lived pods may take slightly longer (~1s) to report Succeeded or Failed after this change. (#108723, @bobbypage) [SIG Apps, Node and Testing]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.23.5

Downloads for v1.23.5

Source Code

filename sha512 hash
kubernetes.tar.gz ced23c737deb52c9e84af8d2937b19d13822c05d588c29bde120a125c97276ea1f008c649e9a2117b6f026f5fc057afe0089d8ee1154c75e67e054e8ec73b847
kubernetes-src.tar.gz 7fe3cf10bb534a26ee74fa1eedad0c98d874d8de0b8e2c91ed10ebf880e74968bd39d4b6e7170d694ea68bbdc775b3ca70ab78ec5f4f2d15bcb20d6e869f4b2c

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 0c5dfa5cd898aba6599273da7c16aa81f11d0e3a031d92820c98daabb2d2372c8a1dcaf2cb203d88ff172066239c4c783810356840a0b19603366f7760f41ae4
kubernetes-client-darwin-arm64.tar.gz f41e58a9118378685ff42499d082f34bed72a23ab55c1e94cd6ec209aad8e5de943c9385d9b449bb70325b4a01f58f63f0784c43230a681bd13603ce034d85e0
kubernetes-client-linux-386.tar.gz 795f9f68447b36ed912cad4b3f874780bbc28ef7dd34881927467ee42961563a8e6cb1d6d150f79a579ebfeaca2631bd99703625770b851e30dacfec5fbdde81
kubernetes-client-linux-amd64.tar.gz 51f5679a0cb11a65f25c3479bbfdfd21c4d0acd8814d3cbaf5aaeea7682178a3820c3555b17ea6ee24470ac67ebfd0f78cc98513e5b526436494350be64bda69
kubernetes-client-linux-arm.tar.gz 056d62df16a9725c7ae8b072cadb4b713bbd2230fff95d777721a951b6a1443c480920dd000039f1ef4da1df4ea8fabef096d7166a6dace5e6e6fe3b3e67da53
kubernetes-client-linux-arm64.tar.gz a6b5b19a71e971cdfb0b4819add8f7ef3c24a99b6201b67f52ef6c65787a4d9008ba69b6f950bd162d05e7620c34971346a6b4b7cbeefe2ffe4a84a32661f208
kubernetes-client-linux-ppc64le.tar.gz 969d000f87e991755f91f9b16c114c8606d342f669625111609c1991537e7085eef6c20a815c0c0890e2a405e015f9c5daacaab526b444fa5e0c075f0ac4d017
kubernetes-client-linux-s390x.tar.gz 64a1bb89a47a37e7de1bc8835963d8ef3253c86306e5c4ef4d6b2609699fa5dce1d8611406c1d8bf22b3a7ec6194f61e526c8276381f79bad0fd173d6cdaf50f
kubernetes-client-windows-386.tar.gz 381d1602538fd7926758fad59a64dba6fa560ecc48593cb55f7b3bcb494ac221dab84be1f9e945036a8cb6336c8a0ecdfc33c0e94f8613112dcda2e020e2646f
kubernetes-client-windows-amd64.tar.gz a794fc29d9d2de0d5550a05dd7712b91dd39dc6c75bd9f291c25fb4acd3a5b6fcfaea07c768e291bd62b10c9b27d3d0f57af450ab489d872a3af7b89852c5878
kubernetes-client-windows-arm64.tar.gz f4225d21f12a270019c7a96330f14159fe8a34b869370deeff3920b8dee3cd78e9173605033b9e138ccdef7304bbc39ac41ed582c0360b580a1b0cd67091e6b9

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 063ad74fb1463ee7a7bf4fb746eef1e02980c170cfa89c93444bee0841a84133bdbe91035f6608ece15096fec3e0c9aa50ebdc2b15ce589d86e2f07d10a1d747
kubernetes-server-linux-arm.tar.gz baa1a310236fc5baad609285fc4717913791cbad920dca17d8736e0052af9cad07ee454323e8d8c03cad70456bd69d79cececb7019b3f3a48978552bed3d2b68
kubernetes-server-linux-arm64.tar.gz b3388b6da8fcbbaa30ac881f0f0dbf6ca501bd5fc52aca33174025fe6234af2872ab1a25a5a74049cfc627359a748c7cbc0f129ad3b800c4707d44f98ac69d52
kubernetes-server-linux-ppc64le.tar.gz 1eab49c6ad3bc7f368b2756239f60b59cb946f6bf56974a5c62688f5bfc5175e2d4ae33453bf8b1e1baedfcd9cba23cad913ca282a57be87fed18ddb6c28e754
kubernetes-server-linux-s390x.tar.gz 3abcaa6fda41b19a0b5e2627e93b0004759d291ea22c8698008f0924a7c8a5c2aece81c508e69e37a60980f94adb6873a68fe79a7c659641a453d706c90b26c4

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 57eeae81081e06e35484b353be04f18bf5c2556175a0355d63cbe3eea80d51decfae28eb42cb5fc8907492a70e4e9bae54bd86956caea7c3a51b1fc909feaea6
kubernetes-node-linux-arm.tar.gz 26510f1f342efa97d3a340db699f8d18d3b8430e082ff32c4596fc3efb629cdc0d427af6f749c4ed845ea2c0f2594ece7e23bc875970e5433e585812e8eef84c
kubernetes-node-linux-arm64.tar.gz 70a73dc630df6fe60682e5379624116607a6fef571f17423de309d1ce20895eb358472809e736743740d4a2df1f708799975013d4cdc47499c5f1df3d4b2d630
kubernetes-node-linux-ppc64le.tar.gz ebf98d17dc9ffba9ab895bf4877099563cc01ae930b9ec0342936ec53dd6b5335cb2ff01baaf620dbb9d7270e2f6831e12201c3da6c0ccbdccba30288bfb1317
kubernetes-node-linux-s390x.tar.gz 708a9101cf73f5c78cd7d7199833b5f2f74a7cca8a4f1e0db629e3a47250a3e5338269de4b3fba703d9c084ccb4f3f30e57a88f5dedc09fe47964a0751275460
kubernetes-node-windows-amd64.tar.gz 8aa1ad4a60edc6b677f21509eb6120dae4bd396317f28ef2d73a49986e3aafb899a5e07c7944cf5337a9b9bd514e14c3d49757c9041d5aaf80bd93f493288101

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.5 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.5 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.5 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.5 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.5 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.4

Changes by Kind

API Change

  • Fixes a regression in v1beta1 PodDisruptionBudget handling of "strategic merge patch"-type API requests for the selector field. Prior to 1.21, these requests would merge matchLabels content and replace matchExpressions content. In 1.21, patch requests touching the selector field started replacing the entire selector. This is consistent with server-side apply and the v1 PodDisruptionBudget behavior, but should not have been changed for v1beta1. (#108139, @liggitt) [SIG Auth and Testing]

Feature

  • Kubernetes is now built with Golang 1.17.8 (#108559, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

Bug or Regression

  • Bump sigs.k8s.io/apiserver-network-proxy/konnectivity-client to v0.0.30, fixing goroutine leaks in kube-apiserver. (#108438, @andrewsykim) [SIG API Machinery, Auth and Cloud Provider]
  • Fix kubectl config flags incorrectly setting burst and discovery limits (#108401, @ulucinar) [SIG CLI]
  • Fix static pod restarts in cases where the container is not present. (#108164, @rphillips) [SIG Node]
  • Fixes a bug where a partial EndpointSlice update could cause node name information to be dropped from endpoints that were not updated. (#108201, @robscott) [SIG Network]
  • Fixes a regression in the kubelet restarting static pods. (#107931, @rphillips) [SIG Node and Testing]
  • Fixes error handling in a kubectl method used in downstream packages. (#107938, @heybronson) [SIG CLI]
  • Increase Azure ACR credential provider timeout (#108209, @andyzhangx) [SIG Cloud Provider]
  • Kube-apiserver: removed apf_fd from server logs (added in 1.23.0) which could contain data identifying the requesting user (#108634, @jupblb) [SIG API Machinery and Scalability]

Dependencies

Added

Nothing has changed.

Changed

  • sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.27 → v0.0.30

Removed

Nothing has changed.

v1.23.4

Downloads for v1.23.4

Source Code

filename sha512 hash
kubernetes.tar.gz c88f633b0b418469aa381cd39da1581236ce3e7df6f983434d4ce95fbd810a63005a78745f20f16661fdc6280d83be2cb52c8777c0ab5a8c526dea30669b463c
kubernetes-src.tar.gz ae34f80b5a13f717179954a99bb5d0481b3b9bb1ea27e805341f6911d0cd1b3f2d58e5cd375b8b84efb0325fc13d7536f9642790aad129e8361d60f36169e2e5

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 6cdbaffea1ed917cddf6c4d7630eb33dddc36c7194421bb291ea7c7a8acdc235ea061cb3a8d72d84e5d314f61497594b9a5b40dbf4a69baaac88cfb48eadccdf
kubernetes-client-darwin-arm64.tar.gz 200025cd65155ef8d6a854a96f8fc28f819dc3d4f7417af4c0da141d31036d48eebeb3fbfb4efd5d1adf974de54eb549f6c76f26c585b689d46484ca12902edb
kubernetes-client-linux-386.tar.gz 111157e8a37ddf5d746018a4c8c8b16e86f6c2e18b228d9935b5eb2da630959e22362e3c328435ce0b5976eaba420f331c8ac0eb217d81577f0a5a956b529397
kubernetes-client-linux-amd64.tar.gz 280c0b62d7b19c23b30e52e2b6d3aad676de003cbe711ee20bf03227362133782ab6c40f9bf4ac2ae264d0522a863084050e0283eb7b46d70393091994aca30c
kubernetes-client-linux-arm.tar.gz 0d2c1d7091fcfa37b439e3c8ae5643385860e1cf6578fe58df85913b607f356c5e6e2b348e8f156fd293d53c8b4b520527640c5b585990397f42699c3ba0e146
kubernetes-client-linux-arm64.tar.gz fd995845ebd87195b8de662097f423f6c4c71addeefd95387303be16814374eaf044894e28e84cf1906cf33569c5b486919fff97e7aa32e7528fe591c100f3c1
kubernetes-client-linux-ppc64le.tar.gz f88356c7f1bf84eba54ad83de924f35f893b3f8dd6ed78518907f8f69cc85048c3b1eb176e945e8b3ecf5b2cbdd0f95985e4f5ffe91eccd3cb24050d6a3c89ed
kubernetes-client-linux-s390x.tar.gz 899ae660288a6dee79e3e9b64ca8c3c37c8bcdf290748074172f546da785d805f9ed281b3f9a443a0dd1e86b8e112c8b98c0bc85592526e10ab8420dee90ca59
kubernetes-client-windows-386.tar.gz 0bd0219cf5653204ac89cf246521f4ce56f89218ce4ace979504a70947a7aad9c7e783fbcf2e090996bedc90ffc3084683527ba5df76858f31727a58e41b2740
kubernetes-client-windows-amd64.tar.gz aad2ae58858017484683347849d04baf3bfc7eee6984548383b3bb0150fc8cc25b1aa73bf4e1fcb9878e05cc2f71c44811f5b50b3bdf9662b79627f3ef3d3d9f
kubernetes-client-windows-arm64.tar.gz c5d6a22f8264b38b67c316a44619095c1826e46996ffd93e4a78f6f7cf8d85f6e9835270659dbe01b0c495873e40d9373fdfafa26e5fc6d8e078407c763b22b6

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz cd5e90d25bd48dbbd1755eb5d328676c6424d191c2936d7477c9dde72cd374389f888bfee6ffd02836bbe136e8bf9a2cb8adcad2c650714dc6112075949c06c7
kubernetes-server-linux-arm.tar.gz cef6eaea9e1f65cbcf8cbfd254d61e6faacd1550bb691ee5cf7f157efc61a4cce6a6abdb26c42c628706f55b062f7892ab5d40665703e22d6c1dc16dfa8e8ad0
kubernetes-server-linux-arm64.tar.gz a7762d0b380fb06675bad6d4b987e3ddfe0c2b54ce1592c9d2c853d3a8a4d85bbfa77013e92985f3d47269010ff03fcef1166f91aa691169e920b367b3babdb0
kubernetes-server-linux-ppc64le.tar.gz e8390ca2a4cf7d2e4b1ab5a42da4a47a0761fea200ee83626c8e81b2790cc6a20b25b090ae2b67a0a81a946a35f3468b3a09dbfabf195fe359a1a159260c424f
kubernetes-server-linux-s390x.tar.gz 1501c640e22acbe03ec06b76ccad8bece1038039f2a7d71d9504843c8173551b04b2d144ec437d3ea1998e8ee9f25df3739e85788c738f6efca274dac919a947

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 9f75ec2e71469bf5d53f0ac305128d7b685f7d2daf4dd3218a8c89b36bd3f3e73f69696ece76bb4ed1254e1fcd89abc1fe49896db73222a7ce8c0795afd37e93
kubernetes-node-linux-arm.tar.gz f9085bf9b750dbebcf713aa4b5166e65aa8a313be468e1e8014e79a516018b930a9dd4e093fbb632a1538e59c6b42017c0c15050fe67407c92e7141ae073af49
kubernetes-node-linux-arm64.tar.gz 5e6f30f07f1f49092c2201303d5f3843343c1453cbcb1603617df9aa43bf4549581afd9332d634127320fa543747f730c57c00f437cde3f912b6476b571e5bb8
kubernetes-node-linux-ppc64le.tar.gz a63da0682d139a9ccc261fef8e9b944b31d8cf020416f5fe3216ea5da6e2b76c65778064df51009e8a4630f41c811e9c082682a5731766647b372de2b04f1035
kubernetes-node-linux-s390x.tar.gz 3c7e7295f1b133a0469f9a70a56891cef9d990e7959c75f4c5dfa1dbe8f2e6bc689743b32cf5d5ba1bf854c5dac812328f4fcff6606a97c77ae06c24848de2e7
kubernetes-node-windows-amd64.tar.gz 434c0a397a10c06bbed9228944741605a97eee3c0fad6f4de341a5ac882439879433f84a5e269019ad599d440218a1277bac0dbc38e01ff294ec7044a1283076

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.4 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.4 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.4 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.4 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.4 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.3

Changes by Kind

API Change

  • Fix OpenAPI serialization of the x-kubernetes-validations field (#108030, @liggitt) [SIG API Machinery]

Feature

  • Kubernetes is now built with Golang 1.17.7 (#108100, @xmudrii) [SIG Cloud Provider, Instrumentation, Release and Testing]

Bug or Regression

  • Fix Azurefile volumeid collision issue in csi migration (#107575, @andyzhangx) [SIG Cloud Provider and Storage]
  • Fix e2e test "Services should respect internalTrafficPolicy=Local Pod and Node, to Pod (hostNetwork: true)" (#107902, @xueqzhan) [SIG Network and Testing]
  • Fixes a regression in 1.23 where update requests to previously persisted Service objects that have not been modified since 1.19 can be rejected with an incorrect spec.clusterIPs: Required value error (#107875, @liggitt) [SIG Network and Testing]
  • Fixes static pod add and removes restarts in certain cases. (#107761, @rphillips) [SIG Node]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.23.3

Downloads for v1.23.3

Source Code

filename sha512 hash
kubernetes.tar.gz 339d208b86206272494d4f31a384fd8430911a1f8205d4a73605f412b4653fd816e79653bd0a0dacf52c9b9f6a3194279cc1059683c2ffd560c1ad3fa185e20f
kubernetes-src.tar.gz 9530d46878aff36b26b6e8f8bb04c53eb402bd822851f5aa65a2ac6e46064f67820a63a2153ffd317a0b6fa3383cdbc58fe55484a0dd3197862f3133fede5a5b

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 96810fc0f294bfe412f06125b257ae4e7d4ddc7d11247435a3f500830af334560b61399ba8ac9753c63857b42a7fd13c8f7d55a927cb7981aecd886a84ae8a90
kubernetes-client-darwin-arm64.tar.gz c9d1bad58e46908d190760fb1615188180ca1c734ab4137437e62baa76d92e3b79acf8c39e693c5b947bf650bf894f5fb96651d4b617d35992cd82c1d1f64709
kubernetes-client-linux-386.tar.gz cbc63934798cb57f0be188346e852fb9ca8da071c38c5b75c8199b5637bc8df7df9019855607c72ac634d7944e86db0b9485b29b36365d93f8b85f2703b0f3c9
kubernetes-client-linux-amd64.tar.gz 7ee6292a77d7042ed3589f998231985e82abd90143496a65e29b8141dd39dced5f9cd87a7eeba1efa4dbf61e5ddec9e7929c14b7afcdf01d83af322ddf839efb
kubernetes-client-linux-arm.tar.gz 36147a76eb16869bc07608f947b78ff15c3d60b73deef208e5c135d93e1a48d17e5f8a447894a450a6cf1a1d5f058b07463dcbf1d0120d17133d2d98cbce2444
kubernetes-client-linux-arm64.tar.gz fb66a9735f40e2df40388df1f8e17aedd1ac87f7190d76e3eb2a5dd1a11494be56ae312be5aea0f7613b826f3b9b3ee923eef9736b689415605de351eb8861df
kubernetes-client-linux-ppc64le.tar.gz 6b3103ade6e0d7d918461ae33978636343b3d122dda5a68287c21eaba7c6abc2de49277828640e4d25904134aa4311665615ac78783b5fec48314bb3ce09a3ee
kubernetes-client-linux-s390x.tar.gz 13edeefe00b9d9c151ba27a4190a0dfe5fbb7fdf409a83787eb90a0f38be1cfffa7be50b180d5fe96acbdb6ff335ef9ba8c90fb828f4e9953e4798aa3b20963c
kubernetes-client-windows-386.tar.gz 4c761bf7ddf59a980cc800602c5ae1379d9b39b3a15fc35a0a9b2b34fe18150dfea5e6c84b2548477df4a563ffd3290fe84cac80e5f9bc6f07863a2efdfe049e
kubernetes-client-windows-amd64.tar.gz 92cc39b07c62ce5c436f167bb8929f678e362484d7c40b7ef76562ea61db93a38e811325e0992b42349edfc5136b110aff169a6423a29408e65266cbd204c8f8
kubernetes-client-windows-arm64.tar.gz 71a136733fc032af8c825c402355b73e1049f3706c8b88fc3c7c78da0d3e0e6e7eb1fea455a833092ec7d9e12e27230c1097fad1129f487fcdcc4e54b81cceb2

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 667bc04778070685e5fb5b6281fe78263c5081af0613adfe9a68df0695210cb2273e89a1d37a27e4cbf947b9e565ef7697d8b90ddfba23aeeb4c9f8474a373c5
kubernetes-server-linux-arm.tar.gz dd3dbb478185819a4783ac5ed923282ce5be6d9595228226dd8c8c447ac78e5ab7b4558c452c39767df571dd0669552ce096a6bc4d0ae993511e5c3274759961
kubernetes-server-linux-arm64.tar.gz 1eab0b0102cf6635a3e92249d95c08ac36baaccc14a612f512009e0aff28d563b4fd818deb4aaa471aa8399a3a5bb67b10b484dcf0d3e54615d906a1bd861cdd
kubernetes-server-linux-ppc64le.tar.gz e648010752a1db8d23119ffd44b672f67040e4f841b014df699bfa328c32cf97bb928a1c17d1719098a229d83844a593a9cfb1d0833f562a7d64e71ff20120d9
kubernetes-server-linux-s390x.tar.gz 0da7c96e2360f8272dd6cab9da7a3a6b516760b39a9fcd38e86365c2b6f7bff0e839892b58910921aa25d8dce0c94b46cee58bd686b9369995641b6771fa0191

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 9fd17ed04dc8e13ba5b4d67ec657b8afba721c344bd9785669af3def481dcbd8a2eecb02e54e5eebd0559645c6e819f757c49de731e53073f06a12d871e569eb
kubernetes-node-linux-arm.tar.gz 8c934bc5b3a545a8a5c2fa9f6df7358127125509ebd9ddf1b074121f78c6520ff0e4af9217b9930dd21b28c4960946673914f35060283b83606073d308ef28e7
kubernetes-node-linux-arm64.tar.gz 6319071775767b4eab400f3068ec4c0901756ccb79db63fa9ed6754047bacbaec55cbef366a92e057ca610eb6f20d6e4341b9a1f7a36ced44435d07bdb0af0f3
kubernetes-node-linux-ppc64le.tar.gz 8bf110dac7e4e61ca9a2a513a6e296bf36bfd8dee85e7c2c46f831e4eacdeaf6b238b361b044b5a7de0f14f1d735f41e8e169cdb676ae4f4c109da457593918f
kubernetes-node-linux-s390x.tar.gz 35afc0d3f31b6795a280cb7005cc7c5253e897758aba36f4d4558a0ffc2b34ac4f0e7a1e7a42ffc3c67ca1a52365bd3052ea554e43b8cca4bcf2ab38c6ea7929
kubernetes-node-windows-amd64.tar.gz 8d687018bf4b70065d4871406702d57f0ef14abb6c8e8bd7635d2d94f8a56aead9a641ede4477e34534bc705e76bb94cec10dbb9414c5885ad0a5d07d1105401

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name architectures
k8s.gcr.io/conformance:v1.23.3 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-apiserver:v1.23.3 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-controller-manager:v1.23.3 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-proxy:v1.23.3 amd64, arm, arm64, ppc64le, s390x
k8s.gcr.io/kube-scheduler:v1.23.3 amd64, arm, arm64, ppc64le, s390x

Changelog since v1.23.2

Changes by Kind

Feature

  • Kubernetes is now built with Golang 1.17.6 (#107613, @palnabarun) [SIG Cloud Provider, Instrumentation, Release and Testing]

Bug or Regression

  • Fix: delete non existing Azure disk issue (#107406, @andyzhangx) [SIG Cloud Provider]
  • Fixes a regression in 1.23 that incorrectly pruned data from array items of a custom resource that set x-kubernetes-preserve-unknown-fields: true (#107689, @liggitt) [SIG API Machinery]

Dependencies

Added

Nothing has changed.

Changed

  • k8s.io/utils: cb0fa31 → 6203023

Removed

Nothing has changed.

v1.23.2

Downloads for v1.23.2

Source Code

filename sha512 hash
kubernetes.tar.gz f30d444bd0fc62bd8f7d352dacbdc2fe8904707f3c4f6d719e62f6c9509d5d544a1b26964228c3ff29b9c451534d9f85fe25a60b09b332fa5291e542720cfa05
kubernetes-src.tar.gz 6a54e73972672415c9d1472764f6f266700da807a6ee9cd530e28a5158d33280702f3d94948e914347e32df24d9bebdc0d4627adef5221fc7bd3b12e2f8d2a93

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 7371cbf87768e49cbc7ef2776fe037eae2809eefefc5242733da119328d49facf714cb04113c15cfbda99b6b5f62fb82b66951c04011fcb642bd056adb11acb3
kubernetes-client-darwin-arm64.tar.gz a5c8aa760e1cd94e469beb1e73c5abc786bc294278b2e92082dc1afc7ce6e3f8c7f157752331c45579738642d3c159c23d0ac303e0838157cf6fa06d649ef800
kubernetes-client-linux-386.tar.gz 84af3cd829626a8737e38650dd231e158edce11706612f357e19c2e8ef316b31239d6ab82c5ba565c1bbf2975556eeefb3ca6c757d17d27303ff6cf1ad4b9f0a
kubernetes-client-linux-amd64.tar.gz c8653aa2bce09a29041b7347ce2d45710abd8bc3cfe79265e0aa04a24c2028344f0b280f52b2858e869bf997cd0e71b6ac8f22ec8a2b4b39328e73339746f565
kubernetes-client-linux-arm.tar.gz 35a00f6296ab70d0af2838915b848b9f0df5b778935ab8089b4b180c7406b958a1c909adc1bbdf12857d0c75b18be3c236c617f74d392d3eb2f8cb85eb862eca
kubernetes-client-linux-arm64.tar.gz b00539ec1d993e272b77d4ed3a46be743af645cecc6320c9a017c0c5f4f48dd0272377f67e993c2f06aae0fdeb6a174da901be9ff1e9f6c8cd311e16c5cf60ce
kubernetes-client-linux-ppc64le.tar.gz 135e2aa8ae000ac6fe88ec8afa0f671147b9d8936def510a53d2a456191805daceaef8152d22d91a57bcf4bc7e8b47e587440ed260d3f8b2d5013c90d125fcbe
kubernetes-client-linux-s390x.tar.gz 860ad0d3eb064e1ca3b2ce74a296fde1fffe3e620ddfd579f7d022032419bb8f0c7300565bd94d6681cc50ba2268bb2d64a94c1cc3b9b999f0c031de1b92999b
kubernetes-client-windows-386.tar.gz 907c3043a1f06912238ee5d91f7d76d9bbc5417363deb3d9f2cef86bd79e72ad4f7d9cbeaaa7003afa5570bf2a689a8d619ef60e5cb89e61d53eb5ef373102e7
kubernetes-client-windows-amd64.tar.gz 1abd5aeeaaff5884238ee39022ef18b91518026c93e4305f25de9d2cc2136fbacb9799deaf138c691ad82928ac22f7156046793b04b7240625b2e5043b65e9ee
kubernetes-client-windows-arm64.tar.gz ab5617a9a6b154af6a2329523c2fd356583b8266cf9dd512dccdab5a21a9f82ab5eab4802d65e0bc191cc492215e2d5fb850f6f8c02c403635cd0efbe8349450

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 7f4cc97250be176a9af0136c25a549fc491caa0ef6300ccca798c599c5ed5182c08ebba1d8f669ddf3011633cd7b12523d7e37abc65faa369defdde9e351eb22
kubernetes-server-linux-arm.tar.gz 572fbad7f0edcb1e1294d81e222f4add78d302d80ea28d7456b8020b53f01fbe54bee424d4807ed05d6eadaaab46a323faf36b3a3da7427bdc075749c3365d9e
kubernetes-server-linux-arm64.tar.gz 971ce29019cf248c167c27fd081458fde613f7f92c5fe4ad3816eb12ab157c30eb78105b999144d0b31e7093c0f826b1df9cc275b120d0ac269062726669d0c8
kubernetes-server-linux-ppc64le.tar.gz e1cc8146a2c2a0b5774a3516548323aed948dd03e93545ec11c23be8bc5b23e3395b593a656dafb36711e7b368b3cf1c8dd2b55bf5a486fc3ac4875bd011dd22
kubernetes-server-linux-s390x.tar.gz c3f766166e64f878c9077c1d070326da0ce71881b40204d60596921d62d568282f6527137e0f362ff3895ad156ab92dc026ed19f305120872aa385386033e6cd

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 2fab5c395fbf5f88e56430c51979670db0be5119d79a8a74846900601a313917b91615efabe49a92b3b8ad2f11d2dc5892f701549b0debeb16012c3705612d01
kubernetes-node-linux-arm.tar.gz cfe8e40981aec67fbe9cfcba4af18b1ea4f5b4f34e5b9b11cdb6dee076ad8be2b351d0871b1e9d715cec4ca629c87a5cb598a70b6f8e1dcd15486c1f387a5a42
kubernetes-node-linux-arm64.tar.gz b75d8d1e0b9ba1b06b3f08d67ffbff01784eee8973797d0f1565efda4d61bdb89fbea45c385ca3224fc75237ffe6a41920fd1f51b73d5021007ec9e4ef88af73
kubernetes-node-linux-ppc64le.tar.gz fc47a39b6cbe9d4237740d060234c065c7bcf33fbb10b3cffc670b6f7eacfb9f44c2695c620a42fd37a2e30e375f29d206ef4d4fc8a1ded7f57bb9894c85b178
kubernetes-node-linux-s390x.tar.gz 1e9b0d5197b436a14a1e772027c951ba580c5a047acf201102c585190eefc8a8871fee8e20a40efcc01c0475776540ec8cb8f09aef43c149e335a722670cd855
kubernetes-node-windows-amd64.tar.gz 37d6edc06bb5a555c0594875f917a80f42486e59252c0a8b813b3a935352306a19178c4b5c0af4251b40be58fbe3b99377387d1c861e32f4599ef0e275bd4299

Changelog since v1.23.1

Changes by Kind

Feature

  • Kube-apiserver: when merging lists, Server Side Apply now prefers the order of the submitted request instead of the existing persisted object (#107567, @jiahuif) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Storage and Testing]

Bug or Regression

  • An inefficient lock in EndpointSlice controller metrics cache has been reworked. Network programming latency may be significantly reduced in certain scenarios, especially in clusters with a large number of Services. (#107167, @robscott) [SIG Apps and Network]

  • Client-go: fix that paged list calls with ResourceVersionMatch set would fail once paging kicked in. (#107334, @fasaxc) [SIG API Machinery]

  • Fix a panic when using invalid output format in kubectl create secret command (#107347, @rikatz) [SIG CLI]

  • Fix: azuredisk parameter lowercase translation issue (#107429, @andyzhangx) [SIG Cloud Provider and Storage]

  • Fixed a bug that a pod's .status.nominatedNodeName is not cleared properly, and thus over-occupied system resources. (#107109, @Huang-Wei) [SIG Scheduling and Testing]

  • Fixes a rare race condition handling requests that timeout (#107458, @liggitt) [SIG API Machinery]

  • Mount-utils: Detect potential stale file handle (#106988, @andyzhangx) [SIG Storage]

  • The feature gate was mentioned as csiMigrationRBD where it should have been CSIMigrationRBD to be in parity with other migration plugins. This release correct the same and keep it as CSIMigrationRBD.

    users who have configured this feature gate as csiMigrationRBD has to reconfigure the same to CSIMigrationRBD from this release. (#107554, @humblec) [SIG Storage]

Other (Cleanup or Flake)

  • Updates konnectivity-network-proxy to v0.0.27. This includes a memory leak fix for the network proxy (#107037, @jdnurme) [SIG API Machinery, Auth and Cloud Provider]

Dependencies

Added

Nothing has changed.

Changed

  • sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.25 → v0.0.27
  • sigs.k8s.io/structured-merge-diff/v4: v4.1.2 → v4.2.1

Removed

Nothing has changed.

v1.23.1

Downloads for v1.23.1

Source Code

filename sha512 hash
kubernetes.tar.gz d7b53be1a9695143b780fb9ff1271c65dd1584e09ef77fe5aa3db4f965a9a7a8b59af8981b3dfeba1f89dd48a81e30f1cd4d443b7b9bee9f1695b3346b41c8fc
kubernetes-src.tar.gz 00e07c8f2b42bda04f780f74fb5625f52d7a16b99424a0f7a4c67101923eabe495f440f66317ef151b9ae48253c0dd2fa4730b8a4e28b86f13c7741e96cdaaa8

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz b7e858cdb049e710d3961a791d0ae4b0c2309b024f98a80dd470e4e6a2ab30bd2aae72eed57561af27dafdb317617d60c8b7f454c2a0873bd2801751d8cc0260
kubernetes-client-darwin-arm64.tar.gz df441b4757c7ccc3e13145ad264c1de8101e442cf267b3537d2506e9c62f5da4738fca14967367c58933554fb5deaa47cfed146bfb03e0b81b0fac656696b4cd
kubernetes-client-linux-386.tar.gz 3dcc223be7562a78d01e491fe101196d9c644490e4d9a11f5a2314cd25c0f1870ed1dc0c2b872c24d1802348a3d209d0e1304ed16a5cac76d121090e914f4191
kubernetes-client-linux-amd64.tar.gz aaf6c4f2f65b27902ce02069aa5a7c5b195099deb522fbe7638c5458fc1ba6c2fbe2eb00fa18edc952989dfc27c7a252b37792987c55dc044b88d4e350569891
kubernetes-client-linux-arm.tar.gz 5091b2731725a53a9d0db7e45dcd3f534978f6f0361dff13f36e8c22259506df622e25d0c3b81867328a3314d98d9545b351bbf49ff45f9fc5444f2f67359546
kubernetes-client-linux-arm64.tar.gz dbd3ce1ec9cc3e89a0510ce3809966b38fbd95e538b9b9426b9c303e1dbb71eb44c1446bf0b4c70a58d9f1c29a75bff71543fe0e8c724a5d5a03082eabdd9f02
kubernetes-client-linux-ppc64le.tar.gz 677e733f714b148478b9576bd8fe56e78e7517b2fcc5b1d276e92c958933002b9565bc8bdc8f22c2c41857c9b1e92a008baf48477d31d485cfca6a68a82f32ee
kubernetes-client-linux-s390x.tar.gz 8019fae5c10b146594e300fb5591eba45df7a637db4bcbea8943bbef37d06f2d349f338c4b48500ad92e477a869c29e980dfc80e42d2a759ad62236757c53718
kubernetes-client-windows-386.tar.gz b9d3831e78220abc15b0384ab3739f4ba105e324270012a2519ffae4b22c963f2eb602963b68f6353807ab42ae1a1699ea82114edb5e2650af59bd7b6c700288
kubernetes-client-windows-amd64.tar.gz d661b48b7d5dedadcd644a9dd4eb99b756784e43026a39cd837194f8eae5d4356fe0c4de7f4b85ef7ef6fb27a5bac8b9cc99ab92e8bb5bc092f38d3d781e9921
kubernetes-client-windows-arm64.tar.gz 833ba3c0afafc1fa5fd40d20e0450eeb591fe3eb2b73cbcea74fcc029cdf17caabc3a2ae27d6f3227b30c90c31c0d9a2c57656ed3f8fd1173d38938b98e74374

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz be1b895d3b6e7d36d6e8b9415cb9b52bb47d34a3cb289029a8bce8a26933c6cc5c0fe2e9760c3e02b20942aa8b0633b543715ccf629119946c3f2970245f6446
kubernetes-server-linux-arm.tar.gz e5518962fc1543cfba096693c9ca3fb026a11395dd6eabdf50fc577c598e6c546860a96def681681dbc84025e8f06e3bbb13b28a1ba829bee359b779e884f626
kubernetes-server-linux-arm64.tar.gz c546831cca738c3178ff464891d15f84c10d754c1c9b70742b1fa638d108afabf320aabae9dcfd1ec2e6e77439e5151c561ca6a2cb8989cb533035f48509bda8
kubernetes-server-linux-ppc64le.tar.gz 6e72592a8ab51d6e7875c327159918a737deca88e168574b0dae77c08e0325acc575b62bf3c166e3c6da438fdfc2f996911e134de0c57a1c5001ce29d3b7d97e
kubernetes-server-linux-s390x.tar.gz 1092c009c518f28b089c962e33c73f97af2226e5d100856a3ac996f47da4519a4c450dd98ac5d78519fe12e47b955df98675bf4ecbf0d8a722d994d5777b4107

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 77bdd46d93d6dcdb2e5a66ecf0e5f9b6fffe49c53920a36c2e0c4ceb3625631a4a09662dbdf4c170bc5ee35968da257b80048d21d4d1f7302123a549fe99429d
kubernetes-node-linux-arm.tar.gz fb619f27a72cb014e5e97c287d768fbd23bcd21dd8f5d2cc1a4e4d7a21727781ba914a1af9fc33436795e7e7c835bca3e9afa4cbc5d691691f39fcb5108617c7
kubernetes-node-linux-arm64.tar.gz 6f3a98d8929f1e7088ad8751dea6a9fe588ff6a7c40d29b5723ebc47f1badc2130ff4a72c6b2fd3bc0c9dce2b93329ee3c7cef3be93fdfcaf82162bbe0314adc
kubernetes-node-linux-ppc64le.tar.gz 3c8b14d8680612af73f56dbf8b1de4c7c45addf98cda1aa7cdf0b8a0e8f12a95dffe336fd3aa4cd93e072cda5dcde1a88da3034ce8cf5bd11718800596342986
kubernetes-node-linux-s390x.tar.gz 6f8a30f6ea0114156e636208ab09de9763ed08844d1278f7d034575ce0b4b486aa3fd3cec8992651d6de8263b66aa8285a1180ff2f6011a2904eabcd872eaebb
kubernetes-node-windows-amd64.tar.gz a0cf768d92b51d370842dd7b819c20d6fac2bee955763ead4700f095ac0bead030a9a4b2347987e5ee862402f3d5a7a306343229777011be21544286a2ad9448

Changelog since v1.23.0

Changes by Kind

Feature

  • Kubernetes is now built with Golang 1.17.5
    • golang.org/x/net to v0.0.0-20211209124913-491a49abca63 (#106835, @cpanato) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release, Storage and Testing]

Bug or Regression

  • Kubeadm: allow the "certs check-expiration" command to not require the existence of the cluster CA key (ca.key file) when checking the expiration of managed certificates in kubeconfig files. (#106931, @neolit123) [SIG Cluster Lifecycle]
  • Kubeadm: during execution of the "check expiration" command, treat the etcd CA as external if there is a missing etcd CA key file (etcd/ca.key) and perform the proper validation on certificates signed by the etcd CA. Additionally, make sure that the CA for all entries in the output table is included - for both certificates on disk and in kubeconfig files. (#106926, @neolit123) [SIG Cluster Lifecycle]
  • Kubectl: restores --dry-run, --dry-run=true, and --dry-run=false for compatibility with pre-1.23 invocations. (#107021, @liggitt) [SIG CLI and Testing]
  • Reverts graceful node shutdown to match 1.21 behavior of setting pods that have not yet successfully completed to "Failed" phase if the GracefulNodeShutdown feature is enabled in kubelet. The GracefulNodeShutdown feature is beta and must be explicitly configured via kubelet config to be enabled in 1.21+. This changes 1.22 and 1.23 behavior on node shutdown to match 1.21. If you do not want pods to be marked terminated on node shutdown in 1.22 and 1.23, disable the GracefulNodeShutdown feature. (#106900, @bobbypage) [SIG Node and Testing]

Dependencies

Added

Nothing has changed.

Changed

  • golang.org/x/net: e898025 → 491a49a

Removed

Nothing has changed.

v1.23.0

Documentation

Downloads for v1.23.0

Source Code

filename sha512 hash
kubernetes.tar.gz 850f92f4a4f397773ceabdacdb0513fa3cd2eb8867f7e3697f42bc595c3c710f81a8b9b34679d783ca2e1900dd272e0af209126cf55719e321af8da04a4b1c2b
kubernetes-src.tar.gz ee53eb3b32bc4745a3f58dd0af1a8f4157e74b71b896eb39ae0658f6f3d0497b8a1334205cc19a3fe1cc7056b7606b0c745e89860f73b14ffaf4ed7bf9b6ef11

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz ec7acc668cf32ec2ecf9ff515096cfe0d421c096522f4d4f6dd5504046051d6b4a15a130aab67e0d545078b26c1a0d27f2f567b1f4ac68b76324e15a216799f5
kubernetes-client-darwin-arm64.tar.gz 5c8dcc6c847d44bef1d739015627369b8b7f0b27d96bb0264bac1ea029d3e247565639cdfb9041e28d74f27c7650661ec275e452ede448181c454d77916fc432
kubernetes-client-linux-386.tar.gz a94aa935b348dfeee09b47f3a34e8cb7b2d7213dc28e8df189a4b8438a317cfd575d97ae09651fb6a46a528cd00f0808c6e6c95d9e176837b5be463620593acd
kubernetes-client-linux-amd64.tar.gz 82574d81696693510d8becd2a2319d517dda2b424b63c5525299ed24c9f4abe1b1803fc799b2a75c650175a5a70ef03eab6068e5bf2a286a43fcaabee9681747
kubernetes-client-linux-arm.tar.gz 8dd15ed487883f76ed869458df3b5e859518491ac5b3aedb7ec95fd6c8ba1bde081859e0a7c171d9674a773b655db67975dbab9b842d1c0864a300341e58035a
kubernetes-client-linux-arm64.tar.gz 125f51a712fa4cad241e84a57baa4bc7950b4977bb4f7275ec21e82758ea90137d00d39841061cd9c4665144a2cf8ab87b99e185a64152f9682ef524266c45a3
kubernetes-client-linux-ppc64le.tar.gz ebf7130485c33a59fc81c2a6b8d19b847470f57f4be49ead06e0405a9b34891de456199a87ef105b0428f3d9767ac39f83b8199171ba60ab7b7c538b0558d1de
kubernetes-client-linux-s390x.tar.gz dc0a122755d096f18de5d33a7596bae3c8cce058d22999c6754377cd074b8d69f597c62b5468cc80eda4144edea026b9b946b522b144e8d9ad66874de2d0bd9b
kubernetes-client-windows-386.tar.gz d64f72835dca883e666fc44f80db6b75467b5f952bf40f241b8b1034a8b7565aa101b3d8d9be4b47523cf22a1bb51db3924376405fdf40cd3cab688c5e9db21c
kubernetes-client-windows-amd64.tar.gz 915b7e23517dba67db9aa8b20f18f3451897fe7ab2bae1cd64bc22810e38efa3f2f62483ba377c6d1fed738de24874c2c14ef772f02f201f41a35d80eb2f872b
kubernetes-client-windows-arm64.tar.gz 2702663c0bc4316e83573c6c262040e72c13c6ea50b9dd042dd8d375e719e8729db759db3bffa8bd6a838661278e02a3047ce402b0348ae082393b37643047cb

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 633cba8102648735b93d91a9840a39597b3242d2489081e71b0131a9bf246e2f52060bc8f8abcc2154f39c1e2410dec5e8b189d220743b55f861062c86edd2f8
kubernetes-server-linux-arm.tar.gz 39eb3f85a46f6c71ba70ffa391e706b6c57c8f9fb7eea95960f944a3fee7883da56e3dca9eec7f3121800911e45681b2bcc78c4585ce7081720a8658e1837f47
kubernetes-server-linux-arm64.tar.gz 91236a70b0ff67b54c939215ac71a7e03e4202e71d8b11f687fc6406eb54da6271e8f095a4e812b3aaad23275e0a04b4cfa7bfa8f455a98ccf8c5a22b8b5e59f
kubernetes-server-linux-ppc64le.tar.gz a41f590fcc271861d73cae14032c51d7674efba48f160550da8be7095240be39a59bdd8012886eb94afa8062576e43643a76c2796ce847d6df33d7147a807e9f
kubernetes-server-linux-s390x.tar.gz 2bb2d3087e911e5c296ae194697190470606c1ac761fb3e69533492109d012e756428daac93e89665f6bbac7c4b4fd1ab9e554718244234f43c6e7219c396eff

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 4b0d9188914f3b8dbb7cab384b9f2a63b5d0f53b78c7c9d4920613e780d17afffd95809eb828753556048481634b5a5bd0b89b50360e4d3f2a37d21bb88d5a36
kubernetes-node-linux-arm.tar.gz 039c77a9a5f7e12d826ded7f8590b9f63c79863fcc463e150848ab6304de171aebbdc948f9398478b96a763f982d02f955f8327f651a04e8caced4c6fd2b8e12
kubernetes-node-linux-arm64.tar.gz d60c740f2f5b2ffe95bc9d1ea0e26ca6af1aef718d5c498339eabe6a9cc1cdff8cb69f9cc0f2389adb1eaa3dacdbcaacae3e19fd27ce035f561813d8f29b5f90
kubernetes-node-linux-ppc64le.tar.gz 163b33fcab0226e950c2d2415ece74ac840c592ede2e7276babab466ef3ef1cdbee95102ca4cbefeb836ce1b2897d19b0534967dbd0dcb6706753b453adb27eb
kubernetes-node-linux-s390x.tar.gz c01db605cc3b744a6a13962e318f94ae66eb221d4ed3758a868452bc610cdcf79297332e158cf9e88002e3ac72c9489b7c27b001ba40df1852c6ba098afb9586
kubernetes-node-windows-amd64.tar.gz d5431cd60990bd56649ce11e3c5a72b92a733e6abafaee6517cec333a3dcabd6469ebd4e3d9b052d250ecb35a0c1dcac16cd841612d73886438731f0f817e2d1

Changelog since v1.22.0

What's New (Major Themes)

Deprecation of FlexVolume

FlexVolume is deprecated. Out-of-tree CSI driver is the recommended way to write volume drivers in Kubernetes. See this doc for more information. Maintainers of FlexVolume drivers should implement a CSI driver and move users of FlexVolume to CSI. Users of FlexVolume should move their workloads to CSI driver.

Deprecation of klog specific flags

To simplify the code base, several logging flags got marked as deprecated in Kubernetes 1.23. The code which implements them will be removed in a future release, so users of those need to start replacing the deprecated flags with some alternative solutions.

Software Supply Chain SLSA Level 1 Compliance in the Kubernetes Release Process

Kubernetes releases are now generating provenance attestation files describing the staging and release phases of the release process and artifacts are verified as they are handed over from one phase to the next. This final piece completes the work needed to comply with Level 1 of the SLSA security framework (Supply-chain Levels for Software Artifacts).

IPv4/IPv6 Dual-stack Networking graduates to GA

IPv4/IPv6 dual-stack networking graduates to GA. Since 1.21, Kubernetes clusters are enabled to support dual-stack networking by default. In 1.23, the IPv6DualStack feature gate is removed. The use of dual-stack networking is not mandatory. Although clusters are enabled to support dual-stack networking, Pods and Services continue to default to single-stack. To use dual-stack networking: Kubernetes nodes have routable IPv4/IPv6 network interfaces, a dual-stack capable CNI network plugin is used, Pods are configured to be dual-stack and Services have their .spec.ipFamilyPolicy field set to either PreferDualStack or RequireDualStack.

HorizontalPodAutoscaler v2 graduates to GA

Version 2 of the HorizontalPodAutoscaler API graduates to stable in the 1.23 release. The HorizontalPodAutoscaler autoscaling/v2beta2 API is deprecated in favor of the new autoscaling/v2 API, which the Kubernetes project recommends for all use cases.

This release does not deprecate the v1 HorizontalPodAutoscaler API.

Generic Ephemeral Volume feature graduates to GA

The generic ephemeral volume feature moved to GA in 1.23. This feature allows any existing storage driver that supports dynamic provisioning to be used as an ephemeral volume with the volume’s lifecycle bound to the Pod. All StorageClass parameters for volume provisioning and all features supported with PersistentVolumeClaims are supported.

Skip Volume Ownership change graduates to GA

The feature to configure volume permission and ownership change policy for Pods moved to GA in 1.23. This allows users to skip recursive permission changes on mount and speeds up the pod start up time.

Allow CSI drivers to opt-in to volume ownership and permission change graduates to GA

The feature to allow CSI Drivers to declare support for fsGroup based permissions graduates to GA in 1.23.

PodSecurity graduates to Beta

PodSecurity moves to Beta. PodSecurity replaces the deprecated PodSecurityPolicy admission controller. PodSecurity is an admission controller that enforces Pod Security Standards on Pods in a Namespace based on specific namespace labels that set the enforcement level. In 1.23, the PodSecurity feature gate is enabled by default.

Container Runtime Interface (CRI) v1 is default

The Kubelet now supports the CRI v1 API, which is now the project-wide default. If a container runtime does not support the v1 API, Kubernetes will fall back to the v1alpha2 implementation. There is no intermediate action required by end-users, because v1 and v1alpha2 do not differ in their implementation. It is likely that v1alpha2 will be removed in one of the future Kubernetes releases to be able to develop v1.

Structured logging graduate to Beta

Structured logging reached its Beta milestone. Most log messages from kubelet and kube-scheduler have been converted. Users are encouraged to try out JSON output or parsing of the structured text format and provide feedback on possible solutions for the open issues, such as handling of multi-line strings in log values.

Simplified Multi-point plugin configuration for scheduler

The kube-scheduler is adding a new, simplified config field for Plugins to allow multiple extension points to be enabled in one spot. The new multiPoint plugin field is intended to simplify most scheduler setups for administrators. Plugins that are enabled via multiPoint will automatically be registered for each individual extension point that they implement. For example, a plugin that implements Score and Filter extensions can be simultaneously enabled for both. This means entire plugins can be enabled and disabled without having to manually edit individual extension point settings. These extension points can now be abstracted away due to their irrelevance for most users.

CSI Migration updates

CSI Migration enables the replacement of existing in-tree storage plugins such as kubernetes.io/gce-pd or kubernetes.io/aws-ebs with a corresponding CSI driver. If CSI Migration is working properly, Kubernetes end users shouldn’t notice a difference. After migration, Kubernetes users may continue to rely on all the functionality of in-tree storage plugins using the existing interface.

  • CSI Migration feature is turned on by default but stays in Beta for GCE PD, AWS EBS, and Azure Disk in 1.23.
  • CSI Migration is introduced as an Alpha feature for Ceph RBD and Portworx in 1.23.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

  • Kubeadm: remove the deprecated flag --experimental-patches for the init|join|upgrade commands. The flag --patches is no longer allowed in a mixture with the flag --config. Please use the kubeadm configuration for setting patches for a node using {Init|Join}Configuration.patches. (#104065, @pacoxu)
  • Log messages in JSON format are written to stderr by default now (same as text format) instead of stdout. Users who expected JSON output on stdout must now capture stderr instead or in addition to stdout. (#106146, @pohly) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
  • Support for the seccomp annotations seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io/[name] has been deprecated since 1.19, will be dropped in 1.25. Transition to using the seccompProfile API field. (#104389, @saschagrunert)
  • kube-log-runner is included in release tar balls. It can be used to replace the deprecated --log-file parameter. (#106123, @pohly) [SIG API Machinery, Architecture, Cloud Provider, Cluster Lifecycle and Instrumentation]
  • Kubernetes is built using golang 1.17. This version of go removes the ability to use a GODEBUG=x509ignoreCN=0 environment setting to re-enable deprecated legacy behavior of treating the CommonName of X.509 serving certificates as a host name. This behavior has been disabled by default since Kubernetes 1.19 / go 1.15. Serving certificates used by admission webhooks, custom resource conversion webhooks, and aggregated API servers must now include valid Subject Alternative Names. If you are running Kubernetes 1.22 with GODEBUG=x509ignoreCN=0 set, check the apiserver_kube_aggregator_x509_missing_san_total and apiserver_webhooks_x509_missing_san_total metrics for non-zero values to see if the API server is connecting to webhooks or aggregated API servers using certificates that will be considered invalid in Kubernetes 1.23+.

Known Issues

Etcd v3.5.[0-2] data corruption

Data corruption issue was found in etcd v3.5.0 release that was shipped with 1.22 Kubernetes release. Please read up-to-date production recommendations for etcd.

Changes by Kind

Deprecation

  • A deprecation notice has been added when using the kube-proxy userspace proxier, which will be removed in v1.25. (#103860) (#104631, @perithompson)
  • Added apiserver_longrunning_requests metric to replace the soon to be deprecated apiserver_longrunning_gauge metric. (#103799, @jyz0309)
  • Controller-manager: the following flags have no effect and would be removed in v1.24:
    • --port
    • --address The insecure port flags --port may only be set to 0 now.
  • Kube-scheduler: the --port and --address flags have no effect and would be removed in v1.24. The insecure port flags --port may only be set to 0 now. Also metricsBindAddress and healthzBindAddress fields from kubescheduler.config.k8s.io/v1beta1 are no-op and expected to be empty. Removed in kubescheduler.config.k8s.io/v1beta2 completely. (#96345, @ingvagabund) In addition, please be careful that:
    • kube-scheduler MUST start with --authorization-kubeconfig and --authentication-kubeconfig correctly set to get authentication/authorization working.
    • liveness/readiness probes to kube-scheduler MUST use HTTPS now, and the default port has been changed to 10259.
    • Applications that fetch metrics from kube-scheduler should use a dedicated service account which is allowed to access nonResourceURLs /metrics. (#96345, @ingvagabund) [SIG Cloud Provider, Scheduling and Testing]
  • Feature-gate VolumeSubpath has been deprecated and cannot be disabled. It will be completely removed in 1.25 (#105474, @mauriciopoppe)
  • Kubeadm: add a new output/v1alpha2 API that is identical to the output/v1alpha1, but attempts to resolve some internal dependencies with the kubeadm/v1beta2 API. The output/v1alpha1 API is now deprecated and will be removed in a future release. (#105295, @neolit123)
  • Kubeadm: add the kubeadm specific, Alpha (disabled by default) feature gate UnversionedKubeletConfigMap. When this feature is enabled kubeadm will start using a new naming format for the ConfigMap where it stores the KubeletConfiguration structure. The old format included the Kubernetes version - "kube-system/kubelet-config-1.22", while the new format does not - "kube-system/kubelet-config". A similar formatting change is done for the related RBAC rules. The old format is now DEPRECATED and will be removed after the feature graduates to GA. When writing the ConfigMap kubeadm (init, upgrade apply) will respect the value of UnversionedKubeletConfigMap, while when reading it (join, reset, upgrade), it would attempt to use new format first and fallback to the legacy format if needed. (#105741, @neolit123) [SIG Cluster Lifecycle and Testing]
  • Kubeadm: remove the deprecated / NO-OP phase update-cluster-status in kubeadm reset (#105888, @neolit123)
  • Remove 'master' as a valid EgressSelection type in the EgressSelectorConfiguration API. (#102242, @pacoxu)
  • Removed kubectl --dry-run empty default value and boolean values. kubectl --dry-run usage must be specified with --dry-run=(server|client|none). (#105327, @julianvmodesto)
  • Removed deprecated metric scheduler_volume_scheduling_duration_seconds. (#104518, @dntosas)
  • The deprecated --experimental-bootstrap-kubeconfig flag has been removed. This can be set via --bootstrap-kubeconfig. (#103172, @niulechuan)

API Change

  • A new field omitManagedFields has been added to both audit.Policy and audit.PolicyRule so cluster operators can opt in to omit managed fields of the request and response bodies from being written to the API audit log. (#94986, @tkashem) [SIG API Machinery, Auth, Cloud Provider and Testing]
  • A small regression in Service updates was fixed. The circumstances are so unlikely that probably nobody would ever hit it. (#104601, @thockin)
  • Added a feature gate StatefulSetAutoDeletePVC, which allows PVCs automatically created for StatefulSet pods to be automatically deleted. (#99728, @mattcary)
  • Client-go impersonation config can specify a UID to pass impersonated uid information through in requests. (#104483, @margocrawf)
  • Create HPA v2 from v2beta2 with some fields changed. (#102534, @wangyysde) [SIG API Machinery, Apps, Auth, Autoscaling and Testing]
  • Ephemeral containers graduated to beta and are now available by default. (#105405, @verb)
  • Fix kube-proxy regression on UDP services because the logic to detect stale connections was not considering if the endpoint was ready. (#106163, @aojea) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Contributor Experience, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]
  • If a conflict occurs when creating an object with generateName, the server now returns an "AlreadyExists" error with a retry option. (#104699, @vincepri)
  • Implement support for recovering from volume expansion failures (#106154, @gnufied) [SIG API Machinery, Apps and Storage]
  • In kubelet, log verbosity and flush frequency can also be configured via the configuration file and not just via command line flags. In other commands (kube-apiserver, kube-controller-manager), the flags are listed in the "Logs flags" group and not under "Global" or "Misc". The type for -vmodule was made a bit more descriptive (pattern=N,... instead of moduleSpec). (#106090, @pohly) [SIG API Machinery, Architecture, CLI, Cluster Lifecycle, Instrumentation, Node and Scheduling]
  • Introduce OS field in the PodSpec (#104693, @ravisantoshgudimetla)
  • Introduce v1beta3 API for scheduler. This version

    • increases the weight of user specifiable priorities. The weights of following priority plugins are increased

      • TaintTolerations to 3 - as leveraging node tainting to group nodes in the cluster is becoming a widely-adopted practice
      • NodeAffinity to 2
      • InterPodAffinity to 2

    • Won't have HealthzBindAddress, MetricsBindAddress fields (#104251, @ravisantoshgudimetla)

  • Introduce v1beta2 for Priority and Fairness with no changes in API spec. (#104399, @tkashem)
  • JSON log output is configurable and now supports writing info messages to stdout and error messages to stderr. Info messages can be buffered in memory. The default is to write both to stdout without buffering, as before. (#104873, @pohly)
  • JobTrackingWithFinalizers graduates to beta. Feature is enabled by default. (#105687, @alculquicondor)
  • Kube-apiserver: Fixes handling of CRD schemas containing literal null values in enums. (#104969, @liggitt)
  • Kube-apiserver: The rbac.authorization.k8s.io/v1alpha1 API version is removed; use the rbac.authorization.k8s.io/v1 API, available since v1.8. The scheduling.k8s.io/v1alpha1 API version is removed; use the scheduling.k8s.io/v1 API, available since v1.14. (#104248, @liggitt)
  • Kube-scheduler: support for configuration file version v1beta1 is removed. Update configuration files to v1beta2(xref: kubernetes/enhancements#2901) or v1beta3 before upgrading to 1.23. (#104782, @kerthcet)
  • KubeSchedulerConfiguration provides a new field MultiPoint which will register a plugin for all valid extension points (#105611, @damemi) [SIG Scheduling and Testing]
  • Kubelet should reject pods whose OS doesn't match the node's OS label. (#105292, @ravisantoshgudimetla) [SIG Apps and Node]
  • Kubelet: turn the KubeletConfiguration v1beta1 ResolverConfig field from a string to *string. (#104624, @Haleygo)
  • Kubernetes is now built using go 1.17. (#103692, @justaugustus)
  • Performs strict server side schema validation requests via the fieldValidation=[Strict,Warn,Ignore]. (#105916, @kevindelgado)
  • Promote IPv6DualStack feature to stable. Controller Manager flags for the node IPAM controller have slightly changed:
    1. When configuring a dual-stack cluster, the user must specify both --node-cidr-mask-size-ipv4 and --node-cidr-mask-size-ipv6 to set the per-node IP mask sizes, instead of the previous --node-cidr-mask-size flag.
    2. The --node-cidr-mask-size flag is mutually exclusive with --node-cidr-mask-size-ipv4 and --node-cidr-mask-size-ipv6.
    3. Single-stack clusters do not need to change, but may choose to use the more specific flags. Users can use either the older --node-cidr-mask-size flag or one of the newer --node-cidr-mask-size-ipv4 or --node-cidr-mask-size-ipv6 flags to configure the per-node IP mask size, provided that the flag's IP family matches the cluster's IP family (--cluster-cidr). (#104691, @khenidak)
  • Remove NodeLease feature gate that was graduated and locked to stable in 1.17 release. (#105222, @cyclinder)
  • Removed deprecated --seccomp-profile-root/seccompProfileRoot config. (#103941, @saschagrunert)
  • Since golang 1.17 both net.ParseIP and net.ParseCIDR rejects leading zeros in the dot-decimal notation of IPv4 addresses, Kubernetes will keep allowing leading zeros on IPv4 address to not break the compatibility. IMPORTANT: Kubernetes interprets leading zeros on IPv4 addresses as decimal, users must not rely on parser alignment to not being impacted by the associated security advisory: CVE-2021-29923 golang standard library "net" - Improper Input Validation of octal literals in golang 1.16.2 and below standard library "net" results in indeterminate SSRF & RFI vulnerabilities. Reference: https://nvd.nist.gov/vuln/detail/CVE-2021-29923 (#104368, @aojea)
  • StatefulSet minReadySeconds is promoted to beta. (#104045, @ravisantoshgudimetla)
  • Support pod priority based node graceful shutdown. (#102915, @wzshiming)
  • The "Generic Ephemeral Volume" feature graduates to GA. It is now enabled unconditionally. (#105609, @pohly)
  • The Kubelet's --register-with-taints option is now available via the Kubelet config file field registerWithTaints (#105437, @cmssczy) [SIG Node and Scalability]
  • The CSIDriver.Spec.StorageCapacity can now be modified. (#101789, @pohly)
  • The CSIVolumeFSGroupPolicy feature has moved from beta to GA. (#105940, @dobsonj)
  • The IngressClass.Spec.Parameters.Namespace field is now GA. (#104636, @hbagdi)
  • The Service.spec.ipFamilyPolicy field is now required in order to create or update a Service as dual-stack. This is a breaking change from the beta behavior. Previously the server would try to infer the value of that field from either ipFamilies or clusterIPs, but that caused ambiguity on updates. Users who want a dual-stack Service MUST specify ipFamilyPolicy as either "PreferDualStack" or "RequireDualStack". (#96684, @thockin)
  • The TTLAfterFinished feature gate is now GA and enabled by default. (#105219, @sahilvv)
  • The kube-controller-manager supports --concurrent-ephemeralvolume-syncs flag to set the number of ephemeral volume controller workers. (#102981, @SataQiu)
  • The legacy scheduler policy config is removed in v1.23, the associated flags policy-config-file, policy-configmap, policy-configmap-namespace and use-legacy-policy-config are also removed. Migrate to Component Config instead, see https://kubernetes.io/docs/reference/scheduling/config/ for details. (#105424, @kerthcet)
  • Track the number of Pods with a Ready condition in Job status. The feature is alpha and needs the feature gate JobReadyPods to be enabled. (#104915, @alculquicondor)
  • Users of LogFormatRegistry in component-base must update their code to use the logr v1.0.0 API. The JSON log output now uses the format from go-logr/zapr (no v field for error messages, additional information for invalid calls) and has some fixes (correct source code location for warnings about invalid log calls). (#104103, @pohly)
  • Validation rules for Custom Resource Definitions can be written in the CEL expression language using the x-kubernetes-validations extension in OpenAPIv3 schemas (alpha). This is gated by the alpha "CustomResourceValidationExpressions" feature gate. (#106051, @jpbetz) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing]

Feature

  • (beta feature) If the CSI driver supports the NodeServiceCapability VOLUME_MOUNT_GROUP and the DelegateFSGroupToCSIDriver feature gate is enabled, kubelet will delegate applying FSGroup to the driver by passing it to NodeStageVolume and NodePublishVolume, regardless of what other FSGroup policies are set. (#106330, @verult) [SIG Storage]

  • Add a new distribute-cpus-across-numa option to the static CPUManager policy. When enabled, this will trigger the CPUManager to evenly distribute CPUs across NUMA nodes in cases where more than one NUMA node is required to satisfy the allocation. (#105631, @klueska)

  • Add fish shell completion to kubectl. (#92989, @WLun001)

  • Add mechanism to load simple sniffer class into fluentd-elasticsearch image (#92853, @cosmo0920)

  • Add support for Portworx plugin to csi-translation-lib. Alpha release

    Portworx CSI driver is required to enable migration. This PR adds support of the CSIMigrationPortworx feature gate, which can be enabled by:

    1. Adding the feature flag to the kube-controller-manager --feature-gates=CSIMigrationPortworx=true
    2. Adding the feature flag to the kubelet config:

    featureGates: CSIMigrationPortworx: true (#103447, @trierra) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]

  • Add support to generate client-side binaries for windows/arm64 platform (#104894, @pacoxu)

  • Added PowerShell completion generation by running kubectl completion powershell. (#103758, @zikhan)

  • Added a Processing condition for the workqueue API. Changed Shutdown for the workqueue API to wait until the work queue finishes processing all in-flight items. (#101928, @alexanderConstantinescu)

  • Added a new feature gate CustomResourceValidationExpressions to enable expression validation for Custom Resource. (#105107, @cici37)

  • Added a new flag --append-server-path to kubectl proxy that will automatically append the kube context server path to each request. (#97350, @FabianKramm)

  • Added ability for kubectl wait to wait on arbitary JSON path (#105776, @lauchokyip)

  • Added support for PodAndContainerStatsFromCRI feature gate, which allows a user to specify their pod stats must also come from the CRI, not cAdvisor. (#103095, @haircommander)

  • Added support for setting controller-manager log level online. (#104571, @h4ghhh)

  • Added the ability to specify whether to use an RFC7396 JSON Merge Patch, an RFC6902 JSON Patch, or a Strategic Merge Patch to perform an override of the resources created by kubectl run and kubectl expose. (#105140, @brianpursley)

  • Adding option for kubectl cp to resume on network errors until completion, requires tar in addition to tail inside the container image (#104792, @matthyx)

  • Adding support for multiple --from-env-file flags. (#104232, @lauchokyip)

  • Adding support for multiple --from-env-file flags. (#101646, @lauchokyip)

  • Adds --as-uid flag to kubectl to allow uid impersonation in the same way as user and group impersonation. (#105794, @margocrawf)

  • Adds new [alpha] command 'kubectl events' (#99557, @bboreham)

  • Allow node expansion of local volumes. (#102886, @gnufied)

  • Allow to build kubernetes with a custom kube-cross image. (#104185, @dims)

  • Allows users to prevent garbage collection on pinned images (#103299, @wgahnagl) [SIG Node]

  • CRI v1 is now the project default. If a container runtime does not support the v1 API, Kubernetes will fall back to the v1alpha2 implementation. (#106501, @ehashman)

  • Changed feature CSIMigrationAWS to on by default. This feature requires the AWS EBS CSI driver to be installed. (#106098, @wongma7)

  • Client-go: pass DeleteOptions down to the fake client Reactor (#102945, @chenchun)

  • Cloud providers can set service account names for cloud controllers. (#103178, @nckturner)

  • Display Labels when kubectl describe ingress. (#103894, @kabab)

  • Enhance scheduler VolumeBinding plugin to handle Lost PVC as UnschedulableAndUnresolvable (#105245, @yibozhuang)

  • Ensures that volume is deleted from the storage backend when the user tries to delete the PV object manually and the PV ReclaimPolicy is set to Delete. (#105773, @deepakkinni)

  • Expose a NewUnstructuredExtractor from apply configurations meta/v1 package that enables extracting objects into unstructured apply configurations. (#103564, @kevindelgado)

  • Feature gate StorageObjectInUseProtection has been deprecated and cannot be disabled. It will be completely removed in 1.25 (#105495, @ikeeip)

  • Graduating controller_admission_duration_seconds, step_admission_duration_seconds, webhook_admission_duration_seconds, apiserver_current_inflight_requests and apiserver_response_sizes metrics to stable. (#106122, @rezakrimi) [SIG API Machinery, Instrumentation and Testing]

  • Graduating pending_pods, preemption_attempts_total, preemption_victims and schedule_attempts_total metrics to stable. Also e2e_scheduling_duration_seconds is renamed to scheduling_attempt_duration_seconds and the latter is graduated to stable. (#105941, @rezakrimi) [SIG Instrumentation, Scheduling and Testing]

  • Health check of kube-controller-manager now includes each controller. (#104667, @jiahuif)

  • Integration testing now takes periodic Prometheus scrapes from the etcd server. There is a new script ,hack/run-prometheus-on-etcd-scrapes.sh, that runs a containerized Prometheus server against an archive of such scrapes. (#106190, @MikeSpreitzer) [SIG API Machinery and Testing]

  • Introduce a feature gate DisableKubeletCloudCredentialProviders which allows disabling the in-tree kubelet credential providers.

    The feature gate DisableKubeletCloudCredentialProviders is currently in Alpha, which means is currently disabled by default. Once this feature gate moves to beta, in-tree credential providers will be disabled by default, and users will need to migrate to use external credential providers. (#102507, @ostrain)

  • Introduces a new metric: admission_webhook_request_total with the following labels: name (string) - the webhook name, type (string) - the admission type, operation (string) - the requested verb, code (int) - the HTTP status code, rejected (bool) - whether the request was rejected, namespace (string) - the namespace of the requested resource. (#103162, @rmoriar1)

  • Kubeadm: add support for dry running kubeadm join. The new flag kubeadm join --dry-run is similar to the existing flag for kubeadm init/upgrade and allows you to see what changes would be applied. (#103027, @Haleygo)

  • Kubeadm: do not check if the /etc/kubernetes/manifests folder is empty on joining worker nodes during preflight (#104942, @SataQiu)

  • Kubectl will now provide shell completion choices for the --output/-o flag (#105851, @marckhouzam)

  • Kubelet should reconcile kubernetes.io/os and kubernetes.io/arch labels on the node object. The side-effect of this is kubelet would deny admission to pod which has nodeSelector with label kubernetes.io/os or kubernetes.io/arch which doesn't match the underlying OS or arch on the host OS.

    • The label reconciliation happens as part of periodic status update which can be configured via flag --node-status-update-frequency (#104613, @ravisantoshgudimetla) [SIG Node, Testing and Windows]

  • Kubernetes is now built with Golang 1.16.7. (#104199, @cpanato)

  • Kubernetes is now built with Golang 1.17.1. (#104904, @cpanato)

  • Kubernetes is now built with Golang 1.17.2 (#105563, @mengjiao-liu)

  • Kubernetes is now built with Golang 1.17.3 (#106209, @cpanato) [SIG API Machinery, Cloud Provider, Instrumentation, Release and Testing]

  • Move ConfigurableFSGroupPolicy to GA and rename metric volume_fsgroup_recursive_apply to volume_apply_access_control (#105885, @gnufied)

  • Move the GetAllocatableResources Endpoint in PodResource API to the beta that will make it enabled by default. (#105003, @swatisehgal)

  • Moving WindowsHostProcessContainers feature to beta (#106058, @marosset)

  • Node affinity, Node selectors, and tolerations are now mutable for Jobs that are suspended and have never been started (#105479, @ahg-g)

  • Pod template annotations and labels are now mutable for Jobs that are suspended and have never been started (#105980, @ahg-g)

  • PodSecurity: in 1.23+ restricted policy levels, Pods and containers which set runAsUser=0 are forbidden at admission-time; previously, they would be rejected at runtime (#105857, @liggitt)

  • Shell completion now knows to continue suggesting resource names when the command supports it. For example kubectl get pod pod1 <TAB> will suggest more Pod names. (#105711, @marckhouzam)

  • Support to enable Hyper-V in GCE Windows Nodes created with kube-up (#105999, @mauriciopoppe)

  • The CPUManager policy options are now enabled, and we introduce a graduation path for the new CPU Manager policy options. (#105012, @fromanirh)

  • The Pods and Pod controllers that are exempted from the PodSecurity admission process are now marked with the pod-security.kubernetes.io/exempt: user/namespace/runtimeClass annotation, based on what caused the exemption.

    The enforcement level that allowed or denied a Pod during PodSecurity admission is now marked by the pod-security.kubernetes.io/enforce-policy annotation.

    The annotation that informs about audit policy violations changed from pod-security.kubernetes.io/audit to pod-security.kubernetes.io/audit-violation. (#105908, @stlaz)

  • The /openapi/v3 endpoint will be populated with OpenAPI v3 if the feature flag is enabled (#105945, @Jefftree)

  • The CSIMigrationGCE feature flag is turned ON by default (#104722, @leiyiz)

  • The DownwardAPIHugePages feature is now enabled by default. (#106271, @mysunshine92)

  • The PodSecurity admission plugin has graduated to beta and is enabled by default. The admission configuration version has been promoted to pod-security.admission.config.k8s.io/v1beta1. See https://kubernetes.io/docs/concepts/security/pod-security-admission/ for usage guidelines. (#106089, @liggitt)

  • The ServiceAccountIssuerDiscovery feature gate is removed. It reached GA in Kubernetes 1.21. (#103685, @mengjiao-liu)

  • The constants/variables from k8s.io for STABLE metrics is now supported. (#103654, @coffeepac)

  • The kubectl describe namespace now shows Conditions (#106219, @dlipovetsky)

  • The etcd container image now supports Windows. (#92433, @claudiubelu)

  • The kube-apiserver's Prometheus metrics have been extended with some that describe the costs of handling LIST requests. They are as follows.

    • apiserver_cache_list_total: Counter of LIST requests served from watch cache, broken down by resource_prefix and index_name
    • apiserver_cache_list_fetched_objects_total: Counter of objects read from watch cache in the course of serving a LIST request, broken down by resource_prefix and index_name
    • apiserver_cache_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from watch cache, broken down by resource_prefix
    • apiserver_cache_list_returned_objects_total: Counter of objects returned for a LIST request from watch cache, broken down by resource_prefix
    • apiserver_storage_list_total: Counter of LIST requests served from etcd, broken down by resource
    • apiserver_storage_list_fetched_objects_total: Counter of objects read from etcd in the course of serving a LIST request, broken down by resource
    • apiserver_storage_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from etcd, broken down by resource
    • apiserver_storage_list_returned_objects_total: Counter of objects returned for a LIST request from etcd, broken down by resource (#104983, @MikeSpreitzer)

  • The pause image list now contains Windows Server 2022. (#104438, @nick5616)

  • The script kube-up.sh installs csi-proxy v1.0.1-gke.0. (#104426, @mauriciopoppe)

  • This PR adds the following metrics for API Priority and Fairness.

    • apiserver_flowcontrol_priority_level_seat_count_samples: histograms of seats occupied by executing requests (both regular and final-delay phases included), broken down by priority_level; the observations are taken once per millisecond.
    • apiserver_flowcontrol_priority_level_seat_count_watermarks: histograms of high and low watermarks of number of seats occupied by executing requests (both regular and final-delay phases included), broken down by priority_level.
    • apiserver_flowcontrol_watch_count_samples: histograms of number of watches relevant to a given mutating request, broken down by that request's priority_level and flow_schema. (#105873, @MikeSpreitzer) [SIG API Machinery, Instrumentation and Testing]

  • Topology Aware Hints have graduated to beta. (#106433, @robscott) [SIG Network]

  • Turn on CSIMigrationAzureDisk by default on 1.23 (#104670, @andyzhangx)

  • Update the system-validators library to v1.6.0 (#106323, @neolit123) [SIG Cluster Lifecycle and Node]

  • Updated Cluster Autosaler to version 1.22.0. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.22.0. (#104293, @x13n)

  • Updates debian-iptables to v1.6.7 to pick up CVE fixes. (#104970, @PushkarJ)

  • Updates the following images to pick up CVE fixes:

  • Upgrade etcd to 3.5.1 (#105706, @uthark) [SIG Cloud Provider, Cluster Lifecycle and Testing]

  • When feature gate JobTrackingWithFinalizers is enabled:

    • Limit the number of Pods tracked in a single Job sync to avoid starvation of small Jobs.
    • The metric job_pod_finished_total counts the number of finished Pods tracked by the Job controller. (#105197, @alculquicondor)

  • When using RequestedToCapacityRatio ScoringStrategy, empty shape will cause error. (#106169, @kerthcet) [SIG Scheduling]

  • client-go event library allows customizing spam filtering function. It is now possible to override SpamKeyFunc, which is used by event filtering to detect spam in the events. (#103918, @olagacek)

  • client-go, using log level 9, traces the following events of a HTTP request: - DNS lookup - TCP dialing - TLS handshake - Time to get a connection from the pool - Time to process a request (#105156, @aojea)

Documentation

  • Graduating pod_scheduling_duration_seconds, pod_scheduling_attempts, framework_extension_point_duration_seconds, plugin_execution_duration_seconds and queue_incoming_pods_total metrics to stable. (#106266, @ahg-g) [SIG Instrumentation, Scheduling and Testing]
  • The test "[sig-network] EndpointSlice should have Endpoints and EndpointSlices pointing to API Server [Conformance]" only requires that there is an EndpointSlice that references the "kubernetes.default" service, it no longer requires that its named "kubernetes". (#104664, @aojea)
  • Update description of --audit-log-maxbackup to describe behavior when value = 0. (#103843, @Arkessler)
  • Users should not rely on unsupported CRON_TZ variable when specifying schedule, both the API server and cronjob controller will emit warnings pointing to https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ containing explanation (#106455, @soltysh) [SIG Apps]

Failing Test

  • Fixes hostPath storage E2E tests within SELinux enabled env (#104551, @Elbehery)

Bug or Regression

  • (PodSecurity admission) errors validating workload resources (deployment, replicaset, etc.) no longer block admission. (#106017, @tallclair) [SIG Auth]

  • A pod that the Kubelet rejects was still considered as being accepted for a brief period of time after rejection, which might cause some pods to be rejected briefly that could fit on the node. A pod that is still terminating (but has status indicating it has failed) may also still be consuming resources and so should also be considered. (#104817, @smarterclayton)

  • Add Kubernetes Events to the Kubelet Graceful Shutdown feature. (#101081, @rphillips)

  • Add Pod Security admission metrics: pod_security_evaluations_total, pod_security_exemptions_total, pod_security_errors_total (#105898, @tallclair)

  • Add support for Windows Network stats in Containerd (#105744, @jsturtevant) [SIG Node, Testing and Windows]

  • Added show-capacity option to kubectl top node to show Capacity resource usage (#102917, @bysnupy) [SIG CLI]

  • Apimachinery: Pretty printed JSON and YAML output is now indented consistently. (#105466, @liggitt)

  • Be able to create a Pod with Generic Ephemeral Volumes as raw block devices. (#105682, @pohly)

  • CA, certificate and key bundles for the generic-apiserver based servers will be reloaded immediately after the files are changed. (#104102, @tnqn)

  • Change kubectl diff --invalid-arg status code from 1 to 2 to match docs (#105445, @ardaguclu)

  • Changed kubectl describe to compute age of an event using the EventSeries.count and EventSeries.lastObservedTime. (#104482, @harjas27)

  • Changes behaviour of kube-proxy start; does not attempt to set specific sysctl values (which does not work in recent Kernel versions anymore in non-init namespaces), when the current sysctl values are already set higher. (#103174, @Napsty)

  • Client-go uses the same HTTP client for all the generated groups and versions, allowing to share customized transports for multiple groups versions. (#105490, @aojea)

  • Disable aufs module for gce clusters. (#103831, @lizhuqi)

  • Do not unmount and mount subpath bind mounts during container creation unless bind mount changes (#105512, @gnufied) [SIG Storage]

  • Don't prematurely close reflectors in case of slow initialization in watch based manager to fix issues with inability to properly mount secrets/configmaps. (#104604, @wojtek-t)

  • Don't use a custom dialer for the kubelet if is not rotating certificates, so we can reuse TCP connections and have only one between the apiserver and the kubelet. If users experiment problems with stale connections using HTTP1.1, they can force the previous behavior of the kubelet by setting the environment variable DISABLE_HTTP2. (#104844, @aojea) [SIG API Machinery, Auth and Node]

  • EndpointSlice Mirroring controller now cleans up managed EndpointSlices when a Service selector is added (#105997, @robscott) [SIG Apps, Network and Testing]

  • Enhanced event messages for pod failed for exec probe timeout (#106201, @yxxhero) [SIG Node]

  • Ensure Pods are removed from the scheduler cache when the scheduler misses deletion events due to transient errors (#106102, @alculquicondor) [SIG Scheduling]

  • Ensure InstanceShutdownByProviderID return false for creating Azure VMs. (#104382, @feiskyer)

  • Evicted and other terminated Pods will no longer revert to the Running phase. (#105462, @ehashman)

  • Fix kube-apiserver metric reporting for the deprecated watch path of /api/<version>/watch/.... (#104161, @wojtek-t)

  • Fix a regression where the Kubelet failed to exclude already completed pods from calculations about how many resources it was currently using when deciding whether to allow more pods. (#104577, @smarterclayton)

  • Fix detach disk issue on deleting vmss node. (#104572, @andyzhangx)

  • Fix job controller syncs: In case of conflicts, ensure that the sync happens with the most up to date information. Improves reliability of JobTrackingWithFinalizers. (#105214, @alculquicondor)

  • Fix job tracking with finalizers for more than 500 pods, ensuring all finalizers are removed before counting the Pod. (#104666, @alculquicondor)

  • Fix pod name of NonIndexed Jobs to not include rogue -1 substring (#105676, @alculquicondor)

  • Fix scoring for NodeResourcesBalancedAllocation plugins when nodes have containers with no requests. (#105845, @ahmad-diaa)

  • Fix system default topology spreading when nodes don't have zone labels. Pods correctly spread by default now. (#105046, @alculquicondor)

  • Fix: do not try to delete a LoadBalancer that does not exist (#105777, @nilo19)

  • Fix: ignore non-VMSS error for VMAS nodes in reconcileBackendPools. (#103997, @nilo19)

  • Fix: leave the probe path empty for TCP probes (#105253, @nilo19)

  • Fix: remove VMSS and VMSS instances from SLB backend pool only when necessary (#105839, @nilo19)

  • Fix: skip instance not found when decoupling VMSSs from LB (#105666, @nilo19)

  • Fix: skip case sensitivity when checking Azure NSG rules. (#104384, @feiskyer)

  • Fixed a bug that prevents a PersistentVolume that has a PersistentVolumeClaim UID which doesn't exist in local cache but exists in etcd from being updated to the Released phase. (#105211, @xiaopingrubyist)

  • Fixed a bug where using kubectl patch with $deleteFromPrimitiveList on a nonexistent or empty list would add the item to the list (#105421, @brianpursley)

  • Fixed a bug which could cause webhooks to have an incorrect copy of the old object after an Apply or Update (#106195, @alexzielenski) [SIG API Machinery]

  • Fixed a bug which kubectl would emit duplicate warning messages for flag names that contain an underscore and recommend using a nonexistent flag in some cases. (#103852, @brianpursley)

  • Fixed a panic in kubectl when creating secrets with an improper output type (#106317, @lauchokyip)

  • Fixed a regression setting --audit-log-path=- to log to stdout in 1.22 pre-release. (#103875, @andrewrynhard)

  • Fixed an issue which didn't append OS's environment variables with the one provided in Credential Provider Config file, which may fail execution of external credential provider binary. See kubernetes#102750. (#103231, @n4j)

  • Fixed applying of SELinux labels to CSI volumes on very busy systems (with "error checking for SELinux support: could not get consistent content of /proc/self/mountinfo after 3 attempts") (#105934, @jsafrane) [SIG Storage]

  • Fixed architecture within manifest for non amd64 etcd images. (#104116, @saschagrunert)

  • Fixed architecture within manifest for non amd64 etcd images. (#105484, @saschagrunert)

  • Fixed azure disk translation issue due to lower case managed kind. (#103439, @andyzhangx)

  • Fixed client IP preservation for NodePort service with protocol SCTP in ipvs. (#104756, @tnqn)

  • Fixed concurrent map access causing panics when logging timed-out API calls. (#105734, @marseel)

  • Fixed consolidate logs for instance not found error Fixed skip not found nodes when reconciling LB backend address pools (#105188, @nilo19)

  • Fixed occasional pod cgroup freeze when using cgroup v1 and systemd driver. (#104528, @kolyshkin)

  • Fixed the issue where logging output of kube-scheduler configuration files included line breaks and escape characters. The output also attempted to output the configuration file in one section without showing the user a more readable format. (#106228, @sanchayanghosh) [SIG Scheduling]

  • Fixes a bug that could result in the EndpointSlice controller unnecessarily updating EndpointSlices associated with a Service that had Topology Aware Hints enabled. (#105267, @llhuii)

  • Fixes a regression that could cause panics in LRU caches in controller-manager, kubelet, kube-apiserver, or client-go. (#104466, @stbenjam)

  • Fixes an issue where an admission webhook can observe a v1 Pod object that does not have the defaultMode field set in the injected service account token volume in kube-api-server. (#104523, @liggitt)

  • Fixes the should support building a client with a CSR E2E test to work with clusters configured with short certificate lifetimes (#105396, @liggitt)

  • Graceful node shutdown, allow the actual inhibit delay to be greater than the expected inhibit delay. (#103137, @wzshiming)

  • Handle Generic Ephemeral Volumes properly in the node limits scheduler filter and the kubelet hostPath check. (#100482, @pohly)

  • Headless Services with no selector which were created without dual-stack enabled will be defaulted to RequireDualStack instead of PreferDualStack. This is consistent with such Services which are created with dual-stack enabled. (#104986, @thockin)

  • Ignore not a vmss instance error for VMAS nodes in EnsureBackendPoolDeleted. (#105185, @ialidzhikov)

  • Ignore the case when comparing azure tags in service annotation. (#104705, @nilo19)

  • Ignore the case when updating Azure tags. (#104593, @nilo19)

  • Introduce a new server run option 'shutdown-send-retry-after'. If true the HTTP Server will continue listening until all non longrunning request(s) in flight have been drained, during this window all incoming requests will be rejected with a status code 429 and a 'Retry-After' response header. (#101257, @tkashem)

  • Kube-apiserver: Avoid unnecessary repeated calls to admission webhooks that reject an update or delete request. (#104182, @liggitt)

  • Kube-apiserver: Server Side Apply merge order is reverted to match v1.22 behavior until http://issue.k8s.io/104641 is resolved. (#106661, @liggitt)

  • Kube-apiserver: events created via the events.k8s.io API group for cluster-scoped objects are now permitted in the default namespace as well for compatibility with events clients and the v1 API (#100125, @h4ghhh)

  • Kube-apiserver: fix a memory leak when deleting multiple objects with a deletecollection. (#105606, @sxllwx)

  • Kube-proxy health check ports used to listen to :<port> for each of the services. This is not needed and opens ports in addresses the cluster user may not have intended. The PR limits listening to all node address which are controlled by --nodeport-addresses flag. if no addresses are provided then we default to existing behavior by listening to :<port> for each service (#104742, @khenidak)

  • Kube-proxy: delete stale conntrack UDP entries for loadbalancer ingress IP. (#104009, @aojea)

  • Kube-scheduler now doesn't print any usage message when unknown flag is specified. (#104503, @sanposhiho)

  • Kube-up now includes CoreDNS version v1.8.6 (#106091, @rajansandeep) [SIG Cloud Provider]

  • Kubeadm: When adding an etcd peer to an existing cluster, if an error is returned indicating the peer has already been added, this is accepted and a ListMembers call is used instead to return the existing cluster. This helps to diminish the exponential backoff when the first AddMember call times out, while still retaining a similar performance when the peer has already been added from a previous call. (#104134, @ihgann)

  • Kubeadm: do not allow empty --config paths to be passed to kubeadm kubeconfig user (#105649, @navist2020)

  • Kubeadm: fix a bug on Windows worker nodes, where the downloaded KubeletConfiguration from the cluster can contain Linux paths that do not work on Windows and can trip the kubelet binary. (#105992, @hwdef) [SIG Cluster Lifecycle and Windows]

  • Kubeadm: switch the preflight check (called 'Swap') that verifies if swap is enabled on Linux hosts to report a warning instead of an error. This is related to the graduation of the NodeSwap feature gate in the kubelet to Beta and being enabled by default in 1.23 - allows swap support on Linux hosts. In the next release of kubeadm (1.24) the preflight check will be removed, thus we recommend that you stop using it - e.g. via --ignore-preflight-errors or the kubeadm config. (#104854, @pacoxu)

  • Kubelet did not report kubelet_volume_stats_* metrics for Generic Ephemeral Volumes. (#105569, @pohly)

  • Kubelet's Node Grace Shutdown will terminate probes when shutting down (#105215, @rphillips)

  • Kubelet: fixes a file descriptor leak in log rotation (#106382, @rphillips) [SIG Node]

  • Kubelet: the printing of flags at the start of kubelet now uses the final logging configuration. (#106520, @pohly)

  • Make the etcd client (used by the API server) retry certain types of errors. The full list of retriable (codes.Unavailable) errors can be found at https://github.com/etcd-io/etcd/blob/main/api/v3rpc/rpctypes/error.go#L72 (#105069, @p0lyn0mial)

  • Metrics changes: Fix exposed buckets of scheduler_volume_scheduling_duration_seconds_bucket metric. (#100720, @dntosas)

  • Migrated kubernetes object references (= name + namespace) to structured logging when using JSON as log output format (#104877, @pohly)

  • Pass additional flags to subpath mount to avoid flakes in certain conditions. (#104253, @mauriciopoppe)

  • Pod SecurityContext sysctls name parameter for update requests where the existing object's sysctl contains slashes and kubelet sysctl whitelist support contains slashes. (#102393, @mengjiao-liu) [SIG Apps, Auth, Node, Storage and Testing]

  • Pod will not start when Init container was OOM killed. (#104650, @yxxhero) [SIG Node]

  • PodResources interface was changed, now it returns only isolated CPUs (#97415, @AlexeyPerevalov)

  • Provide IPv6 support for internal load balancer. (#103794, @nilo19)

  • Reduce the number of calls to docker for stats via dockershim. For Windows this reduces the latency when calling docker, for Linux this saves cpu cycles. (#104287, @jsturtevant) [SIG Node and Windows]

  • Removed the error message label from the kubelet_started_pods_errors_total metric (#105213, @yxxhero)

  • Resolves a potential issue with GC and NS controllers which may delete objects after getting a 404 response from the server during its startup. This PR ensures that requests to aggregated APIs will get 503, not 404 while the APIServiceRegistrationController hasn't finished its job. (#104748, @p0lyn0mial)

  • Respect grace period when updating static pods. (#104743, @gjkim42) [SIG Node and Testing]

  • Revert building binaries with PIE mode. (#105352, @ehashman)

  • Reverts adding namespace label to admission metrics (and histogram exansion) due to cardinality issues. (#104033, @s-urbaniak)

  • Reverts the CRI API version surfaced by dockershim to v1alpha2. (#106808, @saschagrunert)

  • Scheduler resource metrics over fractional binary quantities (2.5Gi, 1.1Ki) were incorrectly reported as very small values. (#103751, @y-tag)

  • Support more than 100 disk mounts on Windows (#105673, @andyzhangx)

  • Support using negative array index in JSON patch replace operations. (#105896, @zqzten)

  • The --leader-elect* CLI args are now honored in scheduler. (#105915, @Huang-Wei)

  • The --leader-elect* CLI args are now honored in the scheduler. (#105712, @Huang-Wei)

  • The client-go dynamic client sets the header Content-Type: application/json by default (#104327, @sxllwx)

  • The kube-Proxy now correctly filters out unready endpoints for Services with Topology. (#106507, @robscott)

  • The pods/binding subresource now honors metadata.uid and metadata.resourceVersion (#105913, @aholic)

  • The kube-proxy sync_proxy_rules_iptables_total metric now gives the correct number of rules, rather than being off by one.

    Fixed multiple iptables proxy regressions introduced in 1.22:

    • When using Services with SessionAffinity, client affinity for an endpoint now gets broken when that endpoint becomes non-ready (rather than continuing until the endpoint is fully deleted).

    • Traffic to a service IP now starts getting rejected (as opposed to merely dropped) as soon as there are no longer any usable endpoints, rather than waiting until all of the terminating endpoints have terminated even when those terminating endpoints were not being used.

    • Chains for endpoints that won't be used are no longer output to iptables, saving a bit of memory/time/cpu. (#106030, @danwinship) [SIG Network]

  • Topology Aware Hints now ignores unready endpoints when assigning hints. (#106510, @robscott)

  • Topology Hints now excludes control plane notes from capacity calculations. (#104744, @robscott)

  • Update Go used to build migrate script in etcd image to v1.16.7. (#104301, @serathius)

  • Updated json representation for a conflicted taint to Key=Effect when a conflicted taint occurs in kubectl taint. (#104011, @manugupt1)

  • Upgrades functionality of kubectl kustomize as described at https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.4.1 (#106389, @natasha41575) [SIG CLI]

  • Watch requests that are delegated to aggregated API servers no longer reserve concurrency units (seats) in the API Priority and Fairness dispatcher for their entire duration. (#105511, @benluddy)

  • When a static pod file is deleted and recreated while using a fixed UID, the pod was not properly restarted. (#104847, @smarterclayton)

  • XFS-filesystems are now force-formatted (option -f) in order to avoid problems being formatted due to detection of magic super-blocks. This aligns with the behaviour of formatting of ext3/4 filesystems. (#104923, @davidkarlsen)

  • --log-flush-frequency had no effect in several commands or was missing. Help and warning texts were not always using the right format for a command (add_dir_header instead of add-dir-header). Fixing this included cleaning up flag handling in component-base/logs: that package no longer adds flags to the global flag sets. Commands which want the klog and --log-flush-frequency flags must explicitly call logs.AddFlags; the new cli.Run does that for commands. That helper function also covers flag normalization and printing of usage and errors in a consistent way (print usage text first if parsing failed, then the error). (#105076, @pohly)

Other (Cleanup or Flake)

  • All klog flags except for -v and -vmodule are deprecated. Support for -vmodule is only guaranteed for the text log format. (#105042, @pohly)
  • Better pod events ("waiting for ephemeral volume controller to create the persistentvolumeclaim"" instead of "persistentvolumeclaim not found") when using generic ephemeral volumes. (#104605, @pohly)
  • Changed buckets in apiserver_request_duration_seconds metric from [0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,1.25, 1.5, 1.75, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 40, 50, 60] to [0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.25, 1.5, 2, 3, 4, 5, 6, 8, 10, 15, 20, 30, 45, 60] (#106306, @pawbana) [SIG API Machinery, Instrumentation and Testing]
  • Deprecate apiserver_longrunning_gauge and apiserver_register_watchers in 1.23.0. (#103793, @yan-lgtm)
  • Enhanced error message for nodes not selected by scheduler due to pod's PersistentVolumeClaim(s) bound to PersistentVolume(s) that do not exist. (#105196, @yibozhuang)
  • Fix an issue in cleaning up CertificateSigningRequest objects with an unparseable status.certificate field. (#103823, @liggitt)
  • Kube-apiserver: requests to node, Service, and Pod /proxy subresources with no additional URL path now only automatically redirect GET and HEAD requests. (#95128, @Riaankl)
  • Kube-apiserver: sets an upper-bound on the lifetime of idle keep-alive connections and the time to read the headers of incoming requests. (#103958, @liggitt)
  • Kubeadm: external etcd endpoints passed in the ClusterConfiguration that have Unicode characters are no longer IDNA encoded (converted to Punycode). They are now just URL encoded as per Go's implementation of RFC-3986, have duplicate "/" removed from the URL paths, and passed like that directly to the kube-apiserver --etcd-servers flag. If you have etcd endpoints that have Unicode characters, it is advisable to encode them in advance with tooling that is fully IDNA compliant. If you don't do that, the Go standard library (used in k8s and etcd) would do it for you when making requests to the endpoints. (#103801, @gkarthiks)
  • Kubeadm: remove the --port flag from the manifest for the kube-controller-manager since the flag has been a NO-OP since 1.22 and insecure serving was removed for the component. (#104157, @knight42)
  • Kubeadm: remove the --port flag from the manifest for the kube-scheduler since the flag has been a NO-OP since 1.23 and insecure serving was removed for the component. (#105034, @pacoxu)
  • Kubeadm: update references to legacy artifacts locations, the ci-cross prefix has been removed from the version match as it does not exist in the new gs://k8s-release-dev bucket. (#103813, @SataQiu)
  • Kubectl: deprecated command line flags (like several of the klog flags) now have a DEPRECATED: <explanation> comment. (#106172, @pohly) [SIG CLI]
  • Kubemark is now built as a portable, static binary. (#106150, @pohly) [SIG Scalability and Testing]
  • Migrate cmd/proxy/{config, healthcheck, winkernel} to structured logging (#104944, @jyz0309)
  • Migrate pkg/proxy to structured logging (#104908, @CIPHERTron)
  • Migrate pkg/scheduler/framework/plugins/interpodaffinity/filtering.go,pkg/scheduler/framework/plugins/podtopologyspread/filtering.go, pkg/scheduler/framework/plugins/volumezone/volume_zone.go to structured logging (#105931, @mengjiao-liu)
  • Migrate pkg/scheduler to structured logging. (#99273, @yangjunmyfm192085)
  • Migrate cmd/proxy/app and pkg/proxy/meta_proxier to structured logging (#104928, @jyz0309)
  • Migrated cmd/kube-scheduler/app/server.go, pkg/scheduler/framework/plugins/nodelabel/node_label.go, pkg/scheduler/framework/plugins/nodevolumelimits/csi.go, pkg/scheduler/framework/plugins/nodevolumelimits/non_csi.go to structured logging (#105855, @shivanshu1333)
  • Migrated pkg/proxy/ipvs to structured logging (#104932, @shivanshu1333)
  • Migrated pkg/proxy/userspace to structured logging. (#104931, @shivanshu1333)
  • Migrated pkg/proxy to structured logging (#104891, @shivanshu1333)
  • Migrated pkg/scheduler/framework/plugins/volumebinding/assume_cache.go to structured logging. (#105904, @mengjiao-liu) [SIG Instrumentation, Scheduling and Storage]
  • Migrated pkg/scheduler/framework/preemption/preemption.go, pkg/scheduler/framework/plugins/examples/stateful/stateful.go, and pkg/scheduler/framework/plugins/noderesources/resource_allocation.go to structured logging (#105967, @shivanshu1333) [SIG Instrumentation, Node and Scheduling]
  • Migrated pkg/proxy/winuserspace to structured logging (#105035, @shivanshu1333)
  • Migrated scheduler file cache.go to structured logging (#105969, @shivanshu1333) [SIG Instrumentation and Scheduling]
  • Migrated scheduler files comparer.go, dumper.go, node_tree.go to structured logging (#105968, @shivanshu1333) [SIG Instrumentation and Scheduling]
  • More detailed logging has been added to the EndpointSlice controller for Topology. (#104741, @robscott)
  • Remove deprecated and not supported old cronjob controller. (#106126, @soltysh) [SIG Apps]
  • Remove ignore error flag for drain, and set this feature as default (#105571, @yuzhiquan) [SIG CLI]
  • Remove the deprecated flags --csr-only and --csr-dir from kubeadm certs renew. Please use kubeadm certs generate-csr instead. (#104796, @RA489)
  • Support allocating whole NUMA nodes in the CPUManager when there is not a 1:1 mapping between socket and NUMA node (#102015, @klueska)
  • Support for Windows Server 2022 was added to the k8s.gcr.io/pause:3.6 image. (#104711, @claudiubelu)
  • Surface warning when users don't set propagationPolicy for jobs while deleting. (#104080, @ravisantoshgudimetla)
  • The AllowInsecureBackendProxy feature gate is removed. It reached GA in Kubernetes 1.21. (#103796, @mengjiao-liu)
  • The BoundServiceAccountTokenVolume feature gate that is GA since v1.22 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#104167, @ialidzhikov)
  • The StartupProbe feature gate that is GA since v1.20 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#104168, @ialidzhikov)
  • The SupportPodPidsLimit and SupportNodePidsLimit feature gates that are GA since v1.20 are unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#104163, @ialidzhikov)
  • The apiserver exposes 4 new metrics that allow to track the status of the Service CIDRs allocations: - current number of available IPs per Service CIDR - current number of used IPs per Service CIDR - total number of allocation per Service CIDR - total number of allocation errors per ServiceCIDR (#104119, @aojea)
  • The flag --deployment-controller-sync-period has been deprecated and will be removed in v1.24. (#103538, @Pingan2017)
  • The image gcr.io/kubernetes-e2e-test-images will no longer be used in E2E / CI testing, k8s.gcr.io/e2e-test-images will be used instead. (#103724, @claudiubelu)
  • The kube-proxy image contains /go-runner as a replacement for deprecated klog flags. (#106301, @pohly)
  • The maximum length of the CSINode id field has increased to 256 bytes to match the CSI spec. (#104160, @pacoxu)
  • Troubleshooting: informers log handlers that take more than 100 milliseconds to process an object if the DeltaFIFO queue starts to grow beyond 10 elements. (#103917, @aojea)
  • Update cri-tools dependency to v1.22.0. (#104430, @saschagrunert)
  • Update migratecmd/kube-proxy/app logs to structured logging. (#98913, @yxxhero)
  • Update build images to Debian 11 (Bullseye)
    • debian-base:bullseye-v1.0.0
    • debian-iptables:bullseye-v1.0.0
    • go-runner:v2.3.1-go1.17.1-bullseye.0
    • kube-cross:v1.23.0-go1.17.1-bullseye.1
    • setcap:bullseye-v1.0.0
    • cluster/images/etcd: Build 3.5.0-2 image
    • test/conformance/image: Update runner image to base-debian11 (#105158, @justaugustus)
  • Update conformance image to use debian-base:buster-v1.9.0. (#104696, @PushkarJ)
  • volume.kubernetes.io/storage-provisioner annotation will be added to dynamic provisioning required PVC. volume.beta.kubernetes.io/storage-provisioner annotation is deprecated. (#104590, @Jiawei0227)

Dependencies

Added

  • bazil.org/fuse: 371fbbd
  • github.com/OneOfOne/xxhash: v1.2.2
  • github.com/antlr/antlr4/runtime/Go/antlr: b48c857
  • github.com/cespare/xxhash: v1.1.0
  • github.com/cncf/xds/go: fbca930
  • github.com/getkin/kin-openapi: v0.76.0
  • github.com/go-logr/zapr: v1.2.0
  • github.com/google/cel-go: v0.9.0
  • github.com/google/cel-spec: v0.6.0
  • github.com/google/martian/v3: v3.1.0
  • github.com/kr/fs: v0.1.0
  • github.com/pkg/sftp: v1.10.1
  • github.com/spaolacci/murmur3: f09979e
  • sigs.k8s.io/json: c049b76

Changed

Removed

v1.23.0-rc.1

Downloads for v1.23.0-rc.1

Source Code

filename sha512 hash
kubernetes.tar.gz 4cf838ebcd3bb756cc604b194dd3b58716b41a34c35f636a7c23af4a501829c7ce23def2fae43547f86bff326bb9268a16b34058e83cada4626930b60a928d97
kubernetes-src.tar.gz 88af717a64f237de86f287c3715540fcaf6fd9e526a715832b395965ae27187319cc8955434f2511bcace46be455f56d22d569b083bf65f63bd0539fc6ce76a0

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 2aa8dbe6fe7926dd78083a243323dd0090f19acb22cc6ab6e7fb74ae3efd494f9927469b466a78eda1a210657935237ae235fbcf99dcc765908e590d8b5979ee
kubernetes-client-darwin-arm64.tar.gz 8aaaa11ecdefc349a64b12775656c9f1ed9bfdb72e3dac0f651c629f4f113dc701eb01d07e529c9633bbd5d0eca455a2bf0f654d52830b77917c28ec7bb84dd5
kubernetes-client-linux-386.tar.gz 03f482611ec7d12b30c03e382e53b1b00d7263882204d146a0fb74185d19eb90487ee87d898ec8b6c2ed388ac7b31455d79bafc3cf490efacea06825c1ff49c4
kubernetes-client-linux-amd64.tar.gz bac04a5d242ff32fbed1a7e756107afd68652ed489c405aee026132f8da57107ea9292ad776ec9f68c5b8a4bc5391a56d6a0ea538461d78778a4f09633fe3bba
kubernetes-client-linux-arm.tar.gz cc3ac01dd58f01e8d85b4c0ef70914331addc5b4abde05175d5194d3754c11b7f113a2e2ff79d15a17562565cd7572e643054b67457261563e9f316c3e20a473
kubernetes-client-linux-arm64.tar.gz 8fd0109f6ff07656dc0a265248c3fba81cf2faeb3a872dfb4423f14a92bff793e9ca8c330d83a1e957cdf9358287e325dd4f2ea7e52a0bf66ff1f2631b900a2c
kubernetes-client-linux-ppc64le.tar.gz 0610276ae835a9b151ad5ac9b192ef7d15b385279c9676ac4f2dfcb0600cb66dd4dcd39e07d65d637f8ae07f0f808f9eddf3312c45c6c435c2db0e0716cf3c29
kubernetes-client-linux-s390x.tar.gz bbba68df6895eb722ea7f8678e25e7be6e2d34a50e30c806445515e02b00e3d6982af8e2d54ee757c4fb661fea2f9fa61bd46ffe576897f0583598e3fb0be080
kubernetes-client-windows-386.tar.gz 1c41e825e83b0363967af94e759d6f8adb6f77f7f176c804922b089262df933d21528734dd8ebffd3d49fa7196a558a61d178d64f45444708688fe571dd2b4be
kubernetes-client-windows-amd64.tar.gz 2b39c66f4d69cdaeff9fa8f46331b2eece1714998df7a9300b6be85e9fec7ccc8cccdc02b3900ab7aa989816242b410234a1b9562b2dd51cff89ae19fd9be491
kubernetes-client-windows-arm64.tar.gz 77d00160870636703c9604613b6829c3727241616723de638653f5044373ed233608d8680357d16a37ee8fe88f0c158d93da644ddcddd27047f1d76da1a98a68

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz d54cbd2ed123ab30ad8662523a9f28af530bba0fe15b22738dbc2a5c3f6917c7ef4e0fcd03eaea36cee38420e968b91607e8d5022598e55030a21777448c57c0
kubernetes-server-linux-arm.tar.gz 835d94a3833e7b3b257f2174840e974f957d17e026dc2e1781f2d4590738afc947a27982cd06579a711c31dad5933beaa01c3629950dcb3e36f21b4f8e57aad8
kubernetes-server-linux-arm64.tar.gz 1fed95a3649d87390cb266918f5ed1b8fb00b4ef027cf004f6bffb3d05d1f6f84497c4ac11f34f3d8334d83cab929f46b62122e107ce6c0c6d628c01bb957fc7
kubernetes-server-linux-ppc64le.tar.gz cb563f6d5625eaa5db4f92b239508c0f30e2c7d9e6200651f23cf59f1962301c2113830e2718c515bcc05cce1a5c1b34d77281c3dc2253e4429e8425a852c238
kubernetes-server-linux-s390x.tar.gz a382396ddf43b6a4f5e95a95f58c2618cb63e5d5bbe3d8f8c74bdc1f23855eec14e4aa32e6ad6b2d2bfc0ecd2f9da602b2f410f17b0e680dc8f2e69d7f0f0127

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz fca9757fe2c67a01841810890fd62fb546c6a7ccb3f475d68a1960a054c0d4db44f71b2669e96d4c7bdb9ac727156448c9eeea64dbdb7442a04e91e5f88fdd76
kubernetes-node-linux-arm.tar.gz 30a29ba0a71d5cf29a359da78c0c42abc1c478d8cb4736b1e4dfb905770f537e170eba6b1079173867e6973e34b6d51f9d9f86c5573e1e6e7539de0bdd617c7b
kubernetes-node-linux-arm64.tar.gz 066a437d780b0c871e5635246f9434ba9e7652393546f8b3edba83a56e5a431b4e9aa443a82670d8c5f906bb630426f8a3d8c8b0d146426a20816f588182ea38
kubernetes-node-linux-ppc64le.tar.gz 8816dc1907a743cf8345acc41f5831d195b8531e313024060a5f322e5ba53061c2878f483e1d7253ed06bdb7edbbcf8784d2f5ef1d6c134e486ee8db9d1852b2
kubernetes-node-linux-s390x.tar.gz 671e9513d0aeace5352bf22a1522aa64e4f0b041110f6d11663d4fa3c97dc1c60a6197341dd000c8fd35e0cd51b279e748bea04b220dbecd9f589058632e59cc
kubernetes-node-windows-amd64.tar.gz cae457dde9c3bd813b97b2f57912f72cb08668b31700ceb356234ad78183fac45312e9a92e83e094ba7d991152bfe4375d098dfa6db6e3f2675630dd61441039

Changelog since v1.23.0-rc.0

Changes by Kind

Bug or Regression

  • Kube-apiserver: Server Side Apply merge order is reverted to match v1.22 behavior until http://issue.k8s.io/104641 is resolved. (#106661, @liggitt) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Storage and Testing]
  • Reverts the CRI API version surfaced by dockershim to v1alpha2 (#106808, @saschagrunert) [SIG Network and Node]

Dependencies

Added

Nothing has changed.

Changed

  • sigs.k8s.io/structured-merge-diff/v4: v4.2.0 → v4.1.2

Removed

Nothing has changed.

v1.23.0-rc.0

Downloads for v1.23.0-rc.0

Source Code

filename sha512 hash
kubernetes.tar.gz ede62f7d1bde6a11e60b8ff119366c42902090c5b005ca73590856645c16ff12c904cdf45528cb5f48d4ece31db62f8a2c6a2fc4f10d29052c660036f5a47b5b
kubernetes-src.tar.gz 6103bde6ceeb7b6c40e6e7391731acc4228cf799ee8b7cf612baa8327212a183f16fd560f25b1d608e7f629c230310c585e2e1551436f9569a9d7d5a8c3dbb38

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 0266edfb98cf69c62466c87caa1028510cdb0600dfee9f25ba13b6936935011f6b90e8fe6b008a4c8d060012066adff45b28f01cf1c7f9e24293800576c5dc74
kubernetes-client-darwin-arm64.tar.gz 8bebf2537a53670a8487ccb43faec62b00439124c99d67ae88c5f1360bb863f03648c430be836580b56dc4526609fd53ef35cabf7a4622c346c87731d5e94575
kubernetes-client-linux-386.tar.gz 7cb542707711b5c4cf1402f07d2102b8633b4b75c43be4424f3e15da047badf484b458f6c3e27ecc82343f58f99437c4ab92a0a17f4976f806636d7b070c3396
kubernetes-client-linux-amd64.tar.gz eec960213ccb94b1cf1dc47aaf508eb12c1d04c13474b66960db4e0379d23a59aa484bba06cc04e6e7aba22a6e0eb8afd11a35799829dd540767eaedd66b37d1
kubernetes-client-linux-arm.tar.gz 7db536134da64c586058603283d752bc0bd7c2ea63b312513fff95d65bc24f978b24c0145446efea6f0b6b8f87f3c74c9a4dd581fa5a186ac630e6d58d30ee9d
kubernetes-client-linux-arm64.tar.gz 93a33630fe6bd89fb06f739f7a4184c151c4ac5a8230798b5c3a9137f553f59495e8cc2231f155d6ba51517f70ee095752872c067a12e7bba69ba2b4d9724ee9
kubernetes-client-linux-ppc64le.tar.gz afc9ffb6632b4c85837f87d6764e54a8111d4df3a23320294ac0b942dd089789d018faf91a4d5a22ddc4496d9204470267fba09b2a1d6d6f8ad74353df060675
kubernetes-client-linux-s390x.tar.gz d5c28f9e65d6a910cf6478342c3e1bd968b16820b2aae6d7ab51d1a646a3b4e46dd7bc2f255f5a02e516d36c90320f4c7b0f836192aae37f45da283fbd76dc88
kubernetes-client-windows-386.tar.gz cccb34fd97fb3f05aaf900569bd07772e4ba95f723f7ea71f191926fe05f01b4e608ab49d876b65405051c504043c74b725d09c92321e17a35587703269a37a8
kubernetes-client-windows-amd64.tar.gz 61298763df834a36a1f10c0a45cc7c0b520ad38c2f013fd39e0ab11cb07b7f416b9aa695c4289cf21baf9281c3a9ec4e99e93b4b82b368658b91fa19f6486c69
kubernetes-client-windows-arm64.tar.gz 99d52ce6c6f620464239e4610711daa8d79e0551bab587d9d2342fe315a07fb94b19ffbfdc8f4a769ae51c848f2896517ac3abd256504492bc24389878749775

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz fff4bdfce528a16abce7d075570bad7e5fd3b64baa5bd154595b44d25945379e4ea6fb56869681cd2222ad55965214d7e96ea1d32ddd519629a1b943dfbceb2b
kubernetes-server-linux-arm.tar.gz dc777f74f6d6eef8d56d379cad36e535566993df3abf0be5e00cf22790f01f4336cd757b395aa724b5743db2ad74ce966746755d4970e04a599861f59ea8f12c
kubernetes-server-linux-arm64.tar.gz a4d935e6816e6bd14e037819949644626813885ef308c7e5ab0a680f71b155cd164c46b963dcdbdcd91cadbf1d0870c66934bb62c9f95698699d1ae3dcc25cb5
kubernetes-server-linux-ppc64le.tar.gz 9a69761c04556e246e046e18bd4b875e813aef1a1e01931ad0aef61ad11d741d4a1f416949dc9f5a8b7103823f6061557d075a35b4766a8e1192872a3fedb637
kubernetes-server-linux-s390x.tar.gz 6f64559358d05d659faf79476524101420b6b4e83e27bba11c407624723926adfd14045792a8a874100527d21cf334b56293a94ac1c3e8916b7ec3e9926b52e4

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz ed3972e5bb9550d0999b4f4b9da315607bfdb349fae9395e23bb36e72c6cc30aa6df42084b0faa0f5f890b983ed7132357d49d755cde081fe4c53bf78a935f83
kubernetes-node-linux-arm.tar.gz c87b42358cc75f362eb5a1b52780ee62fa7c18a8760cf7ee744e1910558ffbe5869fea75e49bf8822795c5255ddd935a9364aa067bffeca500b8fb61d70402c5
kubernetes-node-linux-arm64.tar.gz 34cf8aea703f279559765fdf5c0a079e4679e407134c666082a4ee56d1352c1b66291c847a983ee547a64209143a83a15d89f73d4f2fbb3473dfefb8f9aa630e
kubernetes-node-linux-ppc64le.tar.gz 86e8bb17b715aea6df3c4ddfd68e4423414ac9a9b86cd38e6272a2226849a456d86496d6154b468b97c22164dc22f80f1dd456a2011ca22f1b782791cf4d3c84
kubernetes-node-linux-s390x.tar.gz 61723328d454ceade01b1ad5e71bf96caae9badd58b02e877833eafae8465cbc9d09cfd1f414749c4040419bbc594a9256f02a44ce7e762a9753ae50511e57de
kubernetes-node-windows-amd64.tar.gz 3c22248012c2e301209832ded5ecd20d87eb4350a09592763c7de6b46c821b158abbad816c76e012e692ff21ba5da9fb0e861b914166f339f9a0d676c3d4b60a

Changelog since v1.23.0-beta.0

Changes by Kind

API Change

  • Add gRPC probe to Pod.Spec.Container.{Liveness,Readiness,Startup}Probe (#106463, @SergeyKanzhelev) [SIG API Machinery, Apps, CLI, Node and Testing]
  • Adds a feature gate StatefulSetAutoDeletePVC, which allows PVCs automatically created for StatefulSet pods to be automatically deleted. (#99728, @mattcary) [SIG API Machinery, Apps, Auth and Testing]
  • Performs strict server side schema validation requests via the fieldValidation=[Strict,Warn,Ignore] query parameter. (#105916, @kevindelgado) [SIG API Machinery, Apps, Auth, Cloud Provider and Testing]
  • Support pod priority based node graceful shutdown (#102915, @wzshiming) [SIG Node and Testing]

Feature

  • CRI v1 is now the project default. If a container runtime does not support the v1 API, Kubernetes will fall back to the v1alpha2 implementation. (#106501, @ehashman) [SIG Network, Node and Testing]

Bug or Regression

  • Kube-Proxy now correctly filters out unready endpoints for Services with Topology Aware Hints enabled. (#106507, @robscott) [SIG Network]
  • Kubelet: the printing of flags at the start of kubelet now uses the final logging configuration (#106520, @pohly) [SIG Node]
  • Topology Aware Hints now ignores unready endpoints when assigning hints. (#106510, @robscott) [SIG Apps and Network]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.23.0-beta.0

Downloads for v1.23.0-beta.0

Source Code

filename sha512 hash
kubernetes.tar.gz 048cc297840fd70dc571863bbed9da8176a479ca6b8ff17c9a2cc1b1dbf286377d85eb7fccc5d85e1d652658c393ea1eab7ab518631510e1e7462ea638a56b2b
kubernetes-src.tar.gz 1d3f6f5bb54b61312934169845417dffc428bed0f51342dc2b0eebf7f16899843b0f66f9fb2dcdb2a6e9f25bbdc930ea9adac552b0b011e656151c8cae2f4f71

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz e22ce7199acf369eacf8422c8ee417041289e927bfc03c238f45faec75c2dabd7f8201c77ed39f20ac311d1ba289766825b7b2f738cfc59b5652a20b98117180
kubernetes-client-darwin-arm64.tar.gz 22fa13ca86eb5837db3844b6b7fd134c3ffa3ba5a008635bfa83613a100fa48b3e2331cdf5d368cb267c3cd27e3947fe08ac2540342f1b221192e972695a2cd6
kubernetes-client-linux-386.tar.gz 8e239ce934d121b21b534a6d521ca02bf1c6709831e181d103c8d86cdab01b296546be25902162b1060876744f3b579de018b7c2d198e5d5efdd9c849b3ba7ef
kubernetes-client-linux-amd64.tar.gz e9355264e3ca91da833fe3c8c1dcc55c287a9b813aad91f26b09e6a75f48be57d12cb235c5f9c6fe2a0aceee09e2b5da84568d81d8002066c8e77d848a03f112
kubernetes-client-linux-arm.tar.gz 80e93b6c8cce8221f9a5aba8018fcd95b7ec57728a202fdd158b8df86a733e32d6bb60d8b7ea78da9556058074e9bb88c072b4207a43a4fd2f256cce2593a8df
kubernetes-client-linux-arm64.tar.gz 769a1aa41988bbf11a11ef40f42c76740fcbe7fe1fd5d6da948729e1a62bf9c4f28101f47fa9ccd12de50a378b3654e1e4c2d50afad59182c03b8d1e972341e7
kubernetes-client-linux-ppc64le.tar.gz 4a9346caef2714f03e65dc3e5e46ade1b311b91ef184b8a47466583e834f44dcdb21c3800793e87c20064b25c3eac2c34637ff6817f1752d52425cdfd5a912fb
kubernetes-client-linux-s390x.tar.gz f2129ea05e581a38bdc2771cfdd92ad990620fabf9655f7343c56541a544aa4c6c1e1a2e91a338d06dd0064f35fb5e3027259c317a0909badcbadc9e418c6ced
kubernetes-client-windows-386.tar.gz 2dc9459b02f4ed564a7d0e2062e3590c5240debc6a64449d1c714382ded197d5fcf99feecb80ba6483d265ab34126958737cd692783e675b39159be94729c018
kubernetes-client-windows-amd64.tar.gz e58cb2f87f619d34afbb2c2c0f2bab484970406216698b79129637cb27c5508b2ca4bd2a3a91847868631bd72947887317692a73fec0f8d67c26aa59868c9d8f
kubernetes-client-windows-arm64.tar.gz 515bd2e3c95afe613db998ed42ea5456771c488e0963c9fe0328816a6baba09ea4e915d22538e05d478556d17f1678d6a96b75cae25ba742be73da23d04f72ff

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz adc6c0e5c07c3e1d24ac4399ea725da5d72a043feaea0063f26188e469b4b8cf537df245015631f1efce9d5e457724858327da3c7c9763f6ca4538aaf77a5e67
kubernetes-server-linux-arm.tar.gz e6e673cb9baecc56ae03d716569769391cd6f8d38d85810f0199e71b20a4d4c3c92efe7b31a67af463fb01029d94cbcb0c6fe7a0918123055f3fa8f373e76c49
kubernetes-server-linux-arm64.tar.gz f91dc6e948b702784909ca0c4b8758ad9dbfbcd202ec4e329666b07d42488df00ad64de6a68405668ed881e62e0515271c8168e8316519cd95802239abde4951
kubernetes-server-linux-ppc64le.tar.gz fbbf3daff8caa89f8249122ba19d67a0d9298fb47d327c0bebd7a54adad4fe6e809164d8bf8e563c79b1f9c8b646f29d18789ec938cbc5746e30649b392c7121
kubernetes-server-linux-s390x.tar.gz a4ccda542f1b86667e6bf29afd091a2ce6f3a30165ff8b918585fc7794be26d00bd846acaa5b805b270a60df69fbe9827bab6ee472129996e28052bbbe1b0593

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 4d7dd2e50fe65fd1140c51deeb90d8d9f89bbba59502becf626757e2e9eb59fb781bbf3ecb899f1b8e391746329c5c017177287004195387151799e73887f05b
kubernetes-node-linux-arm.tar.gz d38cd4a06b983a7253d99a6d927c40cbacc636bd73d33172ee03cda502f806638d3cc6f096bc13a55a2faf11ab3e85d77dfd20559e2c880cf54f45ba0875c75c
kubernetes-node-linux-arm64.tar.gz fa1fa35f30ca589e031485affd2a1016ba5ca0efdf64b35d49c7738342acb55c40733e53fb3b477734bab68d97b00f9adcfb5954ab365169d8f00ac804cc60fb
kubernetes-node-linux-ppc64le.tar.gz 412b3a133a7711e32455e49d1aac4ce9ee0e44df89afca40dfa8ac52a8aa98649bd4dd7eff85addd8a525bb16b65966dbde1df0c62a994213b4cfa1a7a3b8128
kubernetes-node-linux-s390x.tar.gz 7e0e217893665a56406b6f1404d616da8578396890b04474fed12ea6b48f5fbf52432efd43c13f66a643284fd54c0fd3441940c777eb1cd0796443fd72d69b6f
kubernetes-node-windows-amd64.tar.gz 768dfe871a028ff7d972d9b59935c1ebdcc8ea0ccf990ee84060ef3bb995ddecb48a49d9fb2ff12dc44ed404d6d9362ee78af3492a4206bb23eb8a0ac8d63ca2

Changelog since v1.23.0-alpha.4

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

  • Log messages in JSON format are written to stderr by default now (same as text format) instead of stdout. Users who expected JSON output on stdout must now capture stderr instead or in addition to stdout. (#106146, @pohly) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
  • kube-log-runner is included in release tar balls. It can be used to replace the deprecated --log-file parameter. (#106123, @pohly) [SIG API Machinery, Architecture, Cloud Provider, Cluster Lifecycle and Instrumentation]

Changes by Kind

Deprecation

  • Kubeadm: add a new output/v1alpha2 API that is identical to the output/v1alpha1, but attempts to resolve some internal dependencies with the kubeadm/v1beta2 API. The output/v1alpha1 API is now deprecated and will be removed in a future release. (#105295, @neolit123) [SIG Cluster Lifecycle]
  • Kubeadm: add the kubeadm specific, Alpha (disabled by default) feature gate UnversionedKubeletConfigMap. When this feature is enabled kubeadm will start using a new naming format for the ConfigMap where it stores the KubeletConfiguration structure. The old format included the Kubernetes version - "kube-system/kubelet-config-1.22", while the new format does not - "kube-system/kubelet-config". A similar formatting change is done for the related RBAC rules. The old format is now DEPRECATED and will be removed after the feature graduates to GA. When writing the ConfigMap kubeadm (init, upgrade apply) will respect the value of UnversionedKubeletConfigMap, while when reading it (join, reset, upgrade), it would attempt to use new format first and fallback to the legacy format if needed. (#105741, @neolit123) [SIG Cluster Lifecycle and Testing]

API Change

  • A new field omitManagedFields has been added to both audit.Policy and audit.PolicyRule so cluster operators can opt in to omit managed fields of the request and response bodies from being written to the API audit log. (#94986, @tkashem) [SIG API Machinery, Auth, Cloud Provider and Testing]
  • Create HPA v2 from v2beta2 with some fields changed. (#102534, @wangyysde) [SIG API Machinery, Apps, Auth, Autoscaling and Testing]
  • Fix kube-proxy regression on UDP services because the logic to detect stale connections was not considering if the endpoint was ready. (#106163, @aojea) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Contributor Experience, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]
  • Implement support for recovering from volume expansion failures (#106154, @gnufied) [SIG API Machinery, Apps and Storage]
  • In kubelet, log verbosity and flush frequency can also be configured via the configuration file and not just via command line flags. In other commands (kube-apiserver, kube-controller-manager), the flags are listed in the "Logs flags" group and not under "Global" or "Misc". The type for -vmodule was made a bit more descriptive (pattern=N,... instead of moduleSpec). (#106090, @pohly) [SIG API Machinery, Architecture, CLI, Cluster Lifecycle, Instrumentation, Node and Scheduling]
  • IngressClass.Spec.Parameters.Namespace field is now GA. (#104636, @hbagdi) [SIG Network and Testing]
  • KubeSchedulerConfiguration provides a new field MultiPoint which will register a plugin for all valid extension points (#105611, @damemi) [SIG Scheduling and Testing]
  • Kubelet should reject pods whose OS doesn't match the node's OS label. (#105292, @ravisantoshgudimetla) [SIG Apps and Node]
  • The CSIVolumeFSGroupPolicy feature has moved from beta to GA. (#105940, @dobsonj) [SIG Storage]
  • The Kubelet's --register-with-taints option is now available via the Kubelet config file field registerWithTaints (#105437, @cmssczy) [SIG Node and Scalability]
  • Validation rules for Custom Resource Definitions can be written in the CEL expression language using the x-kubernetes-validations extension in OpenAPIv3 schemas (alpha). This is gated by the alpha "CustomResourceValidationExpressions" feature gate. (#106051, @jpbetz) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing]

Feature

  • (beta feature) If the CSI driver supports the NodeServiceCapability VOLUME_MOUNT_GROUP and the DelegateFSGroupToCSIDriver feature gate is enabled, kubelet will delegate applying FSGroup to the driver by passing it to NodeStageVolume and NodePublishVolume, regardless of what other FSGroup policies are set. (#106330, @verult) [SIG Storage]

  • /openapi/v3 endpoint will be populated with OpenAPI v3 if the feature flag is enabled (#105945, @Jefftree) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing]

  • Add support for PodAndContainerStatsFromCRI featuregate, which allows a user to specify their pod stats must also come from the CRI, not cAdvisor. (#103095, @haircommander) [SIG Node]

  • Add support for Portworx plugin to csi-translation-lib. Alpha release

    Portworx CSI driver is required to enable migration. This PR adds support of the CSIMigrationPortworx feature gate, which can be enabled by:

    1. Adding the feature flag to the kube-controller-manager --feature-gates=CSIMigrationPortworx=true
    2. Adding the feature flag to the kubelet config:

    featureGates: CSIMigrationPortworx: true (#103447, @trierra) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]

  • Added ability for kubectl wait to wait on arbitary JSON path (#105776, @lauchokyip) [SIG CLI]

  • Added the ability to specify whether to use an RFC7396 JSON Merge Patch, an RFC6902 JSON Patch, or a Strategic Merge Patch to perform an override of the resources created by kubectl run and kubectl expose. (#105140, @brianpursley) [SIG CLI]

  • Adding option for kubectl cp to resume on network errors until completion, requires tar in addition to tail inside the container image (#104792, @matthyx) [SIG CLI]

  • Adds --as-uid flag to kubectl to allow uid impersonation in the same way as user and group impersonation. (#105794, @margocrawf) [SIG API Machinery, Auth, CLI and Testing]

  • Allows users to prevent garbage collection on pinned images (#103299, @wgahnagl) [SIG Node]

  • CSIMigrationGCE feature flag is turned ON by default (#104722, @leiyiz) [SIG Apps, Cloud Provider, Node, Storage and Testing]

  • Changed feature CSIMigrationAWS to on by default. This feature requires the AWS EBS CSI driver to be installed. (#106098, @wongma7) [SIG Storage]

  • Ensures that volume is deleted from the storage backend when the user tries to delete the PV object manually and the PV ReclaimPolicy is Delete. (#105773, @deepakkinni) [SIG Apps and Storage]

  • Graduating controller_admission_duration_seconds, step_admission_duration_seconds, webhook_admission_duration_seconds, apiserver_current_inflight_requests and apiserver_response_sizes metrics to stable. (#106122, @rezakrimi) [SIG API Machinery, Instrumentation and Testing]

  • Graduating pending_pods, preemption_attempts_total, preemption_victims and schedule_attempts_total metrics to stable. Also e2e_scheduling_duration_seconds is renamed to scheduling_attempt_duration_seconds and the latter is graduated to stable. (#105941, @rezakrimi) [SIG Instrumentation, Scheduling and Testing]

  • Integration testing now takes periodic Prometheus scrapes from the etcd server. There is a new script ,hack/run-prometheus-on-etcd-scrapes.sh, that runs a containerized Prometheus server against an archive of such scrapes. (#106190, @MikeSpreitzer) [SIG API Machinery and Testing]

  • Kube-apiserver: when merging lists, Server Side Apply now prefers the order of the submitted request instead of the existing persisted object (#105983, @jiahuif) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Storage and Testing]

  • Kubectl describe namespace now shows Conditions (#106219, @dlipovetsky) [SIG CLI]

  • Kubelet should reconcile kubernetes.io/os and kubernetes.io/arch labels on the node object. The side-effect of this is kubelet would deny admission to pod which has nodeSelector with label kubernetes.io/os or kubernetes.io/arch which doesn't match the underlying OS or arch on the host OS.

    • The label reconciliation happens as part of periodic status update which can be configured via flag --node-status-update-frequency (#104613, @ravisantoshgudimetla) [SIG Node, Testing and Windows]

  • Kubernetes is now built with Golang 1.17.3 (#106209, @cpanato) [SIG API Machinery, Cloud Provider, Instrumentation, Release and Testing]

  • Move ConfigurableFSGroupPolicy to GA Rename metric volume_fsgroup_recursive_apply to volume_apply_access_control (#105885, @gnufied) [SIG Instrumentation and Storage]

  • Moving WindowsHostProcessContainers feature to beta (#106058, @marosset) [SIG Windows]

  • The DownwardAPIHugePages feature is now enabled by default. (#106271, @mysunshine92) [SIG Node]

  • The PodSecurity admission plugin has graduated to beta and is enabled by default. The admission configuration version has been promoted to pod-security.admission.config.k8s.io/v1beta1. See https://kubernetes.io/docs/concepts/security/pod-security-admission/ for usage guidelines. (#106089, @liggitt) [SIG Auth and Testing]

  • This PR adds the following metrics for API Priority and Fairness.

    • apiserver_flowcontrol_priority_level_seat_count_samples: histograms of seats occupied by executing requests (both regular and final-delay phases included), broken down by priority_level; the observations are taken once per millisecond.
    • apiserver_flowcontrol_priority_level_seat_count_watermarks: histograms of high and low watermarks of number of seats occupied by executing requests (both regular and final-delay phases included), broken down by priority_level.
    • apiserver_flowcontrol_watch_count_samples: histograms of number of watches relevant to a given mutating request, broken down by that request's priority_level and flow_schema. (#105873, @MikeSpreitzer) [SIG API Machinery, Instrumentation and Testing]

  • Topology Aware Hints have graduated to beta. (#106433, @robscott) [SIG Network]

  • Update the system-validators library to v1.6.0 (#106323, @neolit123) [SIG Cluster Lifecycle and Node]

  • Upgrade etcd to 3.5.1 (#105706, @uthark) [SIG Cloud Provider, Cluster Lifecycle and Testing]

  • When using RequestedToCapacityRatio ScoringStrategy, empty shape will cause error. (#106169, @kerthcet) [SIG Scheduling]

  • This release enables in-tree RBD migration to CSI driver with a couple of feature gates. These featuregates are alpha in this release.

    • CSIMigrationRBD: when enabled, it will redirect traffic from in tree rbd plugin ( kubernetes.io/rbd ) to CSI driver ( rbd.csi.ceph.com) , default to false now.
    • IntreePluginRBDUnregister: Disables the RBD in-tree driver

    The feature gates can be enabled by:

    1. Adding the feature flag to the kube-controller-manager --feature-gates=CSIMigrationRBD=true
    2. Adding the feature flag to the kubelet config: featureGates: CSIMigrationRBD: true

    As a Kubernetes cluster operator that administers storage, here are the prerequisites that you must complete before you attempt migration to the RBD CSI driver:

    • You must install the Ceph CSI driver (rbd.csi.ceph.com), v3.5.0 or above, into your Kubernetes cluster.
    • Considering the clusterID field is a required parameter for CSI driver for its operations, but in-tree StorageClass has monitors field as a required parameter, a Kubernetes storage admin has to create a clusterID based on the monitors hash ( ex:#echo -n '<monitors_string>' | md5sum) in the CSI config map and keep the monitors under this clusterID configuration.
    • Also, if the value of adminId in the in-tree Storageclass is different from admin, the adminSecretName mentioned in the in-tree Storageclass has to be patched with the base64 value of the adminId parameter value, otherwise this step can be skipped.(#95361, @humblec) [SIG API Machinery, Node, Scheduling, Storage]

Documentation

  • Graduating pod_scheduling_duration_seconds, pod_scheduling_attempts, framework_extension_point_duration_seconds, plugin_execution_duration_seconds and queue_incoming_pods_total metrics to stable. (#106266, @ahg-g) [SIG Instrumentation, Scheduling and Testing]
  • Users should not rely on unsupported CRON_TZ variable when specifying schedule, both the API server and cronjob controller will emit warnings pointing to https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ containing explanation (#106455, @soltysh) [SIG Apps]

Bug or Regression

  • (PodSecurity admission) errors validating workload resources (deployment, replicaset, etc.) no longer block admission. (#106017, @tallclair) [SIG Auth]

  • Add support for Windows Network stats in Containerd (#105744, @jsturtevant) [SIG Node, Testing and Windows]

  • Added show-capacity option to kubectl top node to show Capacity resource usage (#102917, @bysnupy) [SIG CLI]

  • Do not unmount and mount subpath bind mounts during container creation unless bind mount changes (#105512, @gnufied) [SIG Storage]

  • Don't use a custom dialer for the kubelet if is not rotating certificates, so we can reuse TCP connections and have only one between the apiserver and the kubelet. If users experiment problems with stale connections using HTTP1.1, they can force the previous behavior of the kubelet by setting the environment variable DISABLE_HTTP2. (#104844, @aojea) [SIG API Machinery, Auth and Node]

  • EndpointSlice Mirroring controller now cleans up managed EndpointSlices when a Service selector is added (#105997, @robscott) [SIG Apps, Network and Testing]

  • Enhanced event messages for pod failed for exec probe timeout (#106201, @yxxhero) [SIG Node]

  • Ensure Pods are removed from the scheduler cache when the scheduler misses deletion events due to transient errors (#106102, @alculquicondor) [SIG Scheduling]

  • Fix a panic in kubectl when creating secrets with an improper output type (#106317, @lauchokyip) [SIG CLI]

  • Fixed a bug which could cause webhooks to have an incorrect copy of the old object after an Apply or Update (#106195, @alexzielenski) [SIG API Machinery]

  • Fixed applying of SELinux labels to CSI volumes on very busy systems (with "error checking for SELinux support: could not get consistent content of /proc/self/mountinfo after 3 attempts") (#105934, @jsafrane) [SIG Storage]

  • Fixed bug where using kubectl patch with $deleteFromPrimitiveList on a nonexistent or empty list would add the item to the list (#105421, @brianpursley) [SIG API Machinery]

  • Fixed the issue where logging output of kube-scheduler configuration files included line breaks and escape characters. The output also attempted to output the configuration file in one section without showing the user a more readable format. (#106228, @sanchayanghosh) [SIG Scheduling]

  • Kube-up now includes CoreDNS version v1.8.6 (#106091, @rajansandeep) [SIG Cloud Provider]

  • Kubeadm: fix a bug on Windows worker nodes, where the downloaded KubeletConfiguration from the cluster can contain Linux paths that do not work on Windows and can trip the kubelet binary. (#105992, @hwdef) [SIG Cluster Lifecycle and Windows]

  • Kubectl port-forward service will now properly exit when the attached pod dies (#103526, @brianpursley) [SIG API Machinery]

  • Kubelet: fixes a file descriptor leak in log rotation (#106382, @rphillips) [SIG Node]

  • Pod SecurityContext sysctls name parameter for update requests where the existing object's sysctl contains slashes and kubelet sysctl whitelist support contains slashes. (#102393, @mengjiao-liu) [SIG Apps, Auth, Node, Storage and Testing]

  • Pod will not start when Init container was OOM killed. (#104650, @yxxhero) [SIG Node]

  • Reduce the number of calls to docker for stats via dockershim. For Windows this reduces the latency when calling docker, for Linux this saves cpu cycles. (#104287, @jsturtevant) [SIG Node and Windows]

  • Respect grace period when updating static pods. (#104743, @gjkim42) [SIG Node and Testing]

  • The kube-proxy sync_proxy_rules_iptables_total metric now gives the correct number of rules, rather than being off by one.

    Fixed multiple iptables proxy regressions introduced in 1.22:

    • When using Services with SessionAffinity, client affinity for an endpoint now gets broken when that endpoint becomes non-ready (rather than continuing until the endpoint is fully deleted).

    • Traffic to a service IP now starts getting rejected (as opposed to merely dropped) as soon as there are no longer any usable endpoints, rather than waiting until all of the terminating endpoints have terminated even when those terminating endpoints were not being used.

    • Chains for endpoints that won't be used are no longer output to iptables, saving a bit of memory/time/cpu. (#106030, @danwinship) [SIG Network]

  • Upgrades functionality of kubectl kustomize as described at https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.4.1 (#106389, @natasha41575) [SIG CLI]

Other (Cleanup or Flake)

  • Changed buckets in apiserver_request_duration_seconds metric from [0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,1.25, 1.5, 1.75, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 40, 50, 60] to [0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.25, 1.5, 2, 3, 4, 5, 6, 8, 10, 15, 20, 30, 45, 60] (#106306, @pawbana) [SIG API Machinery, Instrumentation and Testing]
  • Kubectl: deprecated command line flags (like several of the klog flags) now have a DEPRECATED: <explanation> comment. (#106172, @pohly) [SIG CLI]
  • Kubemark is now built as a portable, static binary. (#106150, @pohly) [SIG Scalability and Testing]
  • Migrated pkg/scheduler/framework/plugins/volumebinding/assume_cache.go to structured logging. (#105904, @mengjiao-liu) [SIG Instrumentation, Scheduling and Storage]
  • Migrated pkg/scheduler/framework/preemption/preemption.go, pkg/scheduler/framework/plugins/examples/stateful/stateful.go, and pkg/scheduler/framework/plugins/noderesources/resource_allocation.go to structured logging (#105967, @shivanshu1333) [SIG Instrumentation, Node and Scheduling]
  • Migrated scheduler file cache.go to structured logging (#105969, @shivanshu1333) [SIG Instrumentation and Scheduling]
  • Migrated scheduler files comparer.go, dumper.go, node_tree.go to structured logging (#105968, @shivanshu1333) [SIG Instrumentation and Scheduling]
  • Remove deprecated and not supported old cronjob controller. (#106126, @soltysh) [SIG Apps]
  • Remove ignore error flag for drain, and set this feature as default (#105571, @yuzhiquan) [SIG CLI]
  • The kube-proxy image contains /go-runner as a replacement for deprecated klog flags. (#106301, @pohly) [SIG Testing]

Dependencies

Added

  • github.com/OneOfOne/xxhash: v1.2.2
  • github.com/antlr/antlr4/runtime/Go/antlr: b48c857
  • github.com/cespare/xxhash: v1.1.0
  • github.com/cncf/xds/go: fbca930
  • github.com/getkin/kin-openapi: v0.76.0
  • github.com/google/cel-go: v0.9.0
  • github.com/google/cel-spec: v0.6.0
  • github.com/spaolacci/murmur3: f09979e

Changed

  • github.com/containerd/containerd: v1.4.9 → v1.4.11
  • github.com/coredns/corefile-migration: v1.0.12 → v1.0.14
  • github.com/docker/docker: v20.10.2+incompatible → v20.10.7+incompatible
  • github.com/envoyproxy/go-control-plane: 668b12f → 63b5d3c
  • github.com/golang/glog: 23def4e → v1.0.0
  • github.com/google/cadvisor: v0.39.2 → v0.43.0
  • golang.org/x/net: 60bc85c → e898025
  • golang.org/x/sys: 41cdb87 → f4d4317
  • golang.org/x/text: v0.3.6 → v0.3.7
  • google.golang.org/genproto: f16073e → fe13028
  • google.golang.org/grpc: v1.38.0 → v1.40.0
  • google.golang.org/protobuf: v1.26.0 → v1.27.1
  • k8s.io/kube-openapi: 7fbd8d5 → e816edb
  • k8s.io/system-validators: v1.5.0 → v1.6.0
  • sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.23 → v0.0.25
  • sigs.k8s.io/kustomize/api: v0.8.11 → v0.10.1
  • sigs.k8s.io/kustomize/cmd/config: v0.9.13 → v0.10.2
  • sigs.k8s.io/kustomize/kustomize/v4: v4.2.0 → v4.4.1
  • sigs.k8s.io/kustomize/kyaml: v0.11.0 → v0.13.0
  • sigs.k8s.io/structured-merge-diff/v4: v4.1.2 → v4.2.0

Removed

Nothing has changed.

v1.23.0-alpha.4

Downloads for v1.23.0-alpha.4

Source Code

filename sha512 hash
kubernetes.tar.gz aeb10a3fbb89694c52d47203cc958d3543b21426938a9664348163aacd41e20ea7670617a28d8ce6d8d51492980facd5fab062e8ad664dafd7b8dbff1c2bb54f
kubernetes-src.tar.gz b7a8999335ce15b68360478b22af4daaed10e9db50d597e077d731de194208355d1b2134f5635331d9049dc638d05f1f792d52c5890e521f0af3dc2f3e64fbb8

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 5654879ac03f4c7193a8df49cfd4b7253add031c197f50bada40942738bf5720d1c06e31a1d1a7bd1b1a540aa46897e4b34ad8a7e087bd206a7b69b9ffaf5edb
kubernetes-client-darwin-arm64.tar.gz 5dce9fee32436c971ef17595f88f3c74f5644ab3af0e3f854a79fb42f3c8d6d8f507fbb0d7b5bcba52ddf1a49ada2559c477278037cf2dadccba72f0398a1093
kubernetes-client-linux-386.tar.gz e9eb7dab22801c043da2833fde89d2cd721b9dd622df0ff42b25a6742cfab5cff8bfe3ebbda6cc584cf92db3940b95e25aff935863ed999374ee8923ca0b1215
kubernetes-client-linux-amd64.tar.gz 570eeaed029bb05235c58138a777cfd6a4b17d4d91aba346b1fc9a0e573781947599d31a8997e889165db561a18a7ab4d613c2b40a8b2dc0d0225f2411b0fd73
kubernetes-client-linux-arm.tar.gz 298923762745cc064a4489aa01d55f57076b84538aef3a6a3554b60257d9959b4eebbb8aeeecdaf14246fa4f1c17750e1b69c63d4940ae71f87010692e41675c
kubernetes-client-linux-arm64.tar.gz 498527f1cf2d16af576a6b6d27b5ddbb876e24bd85e34e2c91cf39ef467d366b2059e580fdcccb91e0b61a5f52795273b77ed94a1073b5c0bd574b8661afbe0e
kubernetes-client-linux-ppc64le.tar.gz 2632b0fb69565819ef1b6797a834e65f96629df4fd8bec01fce7370672a39afa181854d6ab44afc1c4a6b8143158cf170f5a8e61b75a48071ade2d5ab89d1b2c
kubernetes-client-linux-s390x.tar.gz b793a5a8fce9109343ada86f29cf356c6973cd80d81ca47af5c7e4fa11ffccc273f77aba52b1db42ee12abb94ee23677c21910f57c9385646e35742a1c60e17e
kubernetes-client-windows-386.tar.gz b92e34ee58e1247c1c444134dd9fa78033d0fda1f51509b43016543596cb211128f8aff730d9a3a9118dfeba139186db2a5dd45455427c7521776e63ee77218c
kubernetes-client-windows-amd64.tar.gz 0b5ea6a2de0ff6f71647f428fdbee67c7eb2b918d725cf236ce60daa02e94bd998d15ea0ebb20c4106453e220d11d31506161d5dee3cde6c616dfb5efd11c25e
kubernetes-client-windows-arm64.tar.gz a4e570be453d1df779bb85c62efb41e98209bb93b57b7655a94a737d552c90f9d3061df9088204c7787344dc6a3eb3f843c58651394c0436d2c90b55e499bfab

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz cf215ee7372edd7d5dbf07faee8ccf83de477c8cd431c0fac58357bb8e027349d8edf87364e7db5cec0936f991388f7b183e81e5f92cb6cdf6303efd8cd65d83
kubernetes-server-linux-arm.tar.gz d6281a6727fcdab956170dca7563fc5099ef79b06c96b2f6bc87fcd0b74f1dea0e14aca344cb41b5b6811919cf4a6d6f60cb08b7fb7034690fd0c4ead82e55ca
kubernetes-server-linux-arm64.tar.gz 4ec30cdfd8128ca405201c0c40750e10bac016e1e53a7662265328564b09e4feb831a259125bcdd64169d221145cbb166a463216e884dd76f4bb9a72a00e64e0
kubernetes-server-linux-ppc64le.tar.gz 42b31174a95d0999c78750a1d2c866918c91d11d6406df4e984913f64806708add35c27c0daf255b5d28e98eb815355d1913911f921d34e618dea4d2ebf91949
kubernetes-server-linux-s390x.tar.gz c4e2b38681c0858d560adc8a330f27e95a035cb0e426c6ff332dcd435cefe88441ea866badab5514c4055191324c48aac108d5d6934a9fd4697da179168b6632

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 708b40c9c0d2cfcb6f9874aa3f1b5a27796cbe2bfe7a2345f381e0d9062df8a6769b2bf29b8b641929ef1f5952897c5739e876e8315eb51cced460b13994c247
kubernetes-node-linux-arm.tar.gz f89448106af23d6658b9c2e7b43240fb82051d2f89a302ee61fc1cc78e593535993ba12f412c3df907f415e55c38f1783ce9141075198bb9f197b6fa26328d49
kubernetes-node-linux-arm64.tar.gz bcd8d9fbb244048a3ef3f79f1d4e8f2645bbd69caf353e67ee5c5a4ffd4443da420e5984422933cd4c622c58017a942e20af076f26bfa22f5f38f73a831370ca
kubernetes-node-linux-ppc64le.tar.gz c0724d053c601d4e80ea19957bd32005aeba0cf8f5e03e8e36412aed0777e860ae680302eef632c8e7d4ef1a8e789e48dc58489ad1a7bf7fd20cb0f755e797af
kubernetes-node-linux-s390x.tar.gz 0245f592b92d79ccd102961e5b23a9f5b275829e627254fe8ce5f0a7df53ec2c4a9436942686b9d31b696635ab88131cf92e1002869369fa1cf6f080f8073b5f
kubernetes-node-windows-amd64.tar.gz 2a5c6c79ea65f47a42d25b236709a00eafb793e5d87b5f56516da16b85b06e03020679f7cabfb7dc4bc252ff57da0afa52e357915cb1d3801dc3d5c32f096edf

Changelog since v1.23.0-alpha.3

Changes by Kind

Deprecation

  • A deprecation notice has been added when using the kube-proxy Userspace proxier, which will be removed in v1.25. (#103860) (#104631, @perithompson) [SIG Network]
  • Feature-gate VolumeSubpath has been deprecated and cannot be disabled. It will be completely removed in 1.25 (#105474, @mauriciopoppe) [SIG Storage]
  • Kubeadm: remove the deprecated / NO-OP phase "update-cluster-status" in "kubeadm reset" (#105888, @neolit123) [SIG Cluster Lifecycle]
  • Removed kubectl --dry-run empty default value and boolean values. kubectl --dry-run usage must be specified with --dry-run=(server|client|none). (#105327, @julianvmodesto) [SIG CLI and Testing]

API Change

  • Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

    (#104782, @kerthcet) [SIG Scheduling and Testing]

  • Ephemeral containers have reached beta maturity and are now available by default. (#105405, @verb) [SIG API Machinery, Apps, Node and Testing]

  • Introduce OS field in the Pod Spec (#104693, @ravisantoshgudimetla) [SIG API Machinery and Apps]

  • Introduce v1beta3 api for scheduler. This version

    • increases the weight of user specifiable priorities. The weights of following priority plugins are increased

      • TaintTolerations to 3 - as leveraging node tainting to group nodes in the cluster is becoming a widely-adopted practice
      • NodeAffinity to 2
      • InterPodAffinity to 2

    • Won't have HealthzBindAddress, MetricsBindAddress fields (#104251, @ravisantoshgudimetla) [SIG Scheduling and Testing]

  • JSON log output is configurable and now supports writing info messages to stdout and error messages to stderr. Info messages can be buffered in memory. The default is to write both to stdout without buffering, as before. (#104873, @pohly) [SIG API Machinery, Architecture, CLI, Cluster Lifecycle, Instrumentation, Node and Scheduling]

  • JobTrackingWithFinalizers graduates to beta. Feature is enabled by default. (#105687, @alculquicondor) [SIG Apps and Testing]

  • Remove NodeLease feature gate that was graduated and locked to stable in 1.17 release. (#105222, @cyclinder) [SIG Apps, Node and Testing]

  • TTLAfterFinished is now GA and enabled by default (#105219, @sahilvv) [SIG API Machinery, Apps, Auth and Testing]

  • The "Generic Ephemeral Volume" feature graduates to GA. It is now enabled unconditionally. (#105609, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling, Storage and Testing]

  • The legacy scheduler policy config is removed in v1.23, the associated flags policy-config-file, policy-configmap, policy-configmap-namespace and use-legacy-policy-config are also removed. Migrate to Component Config instead, see https://kubernetes.io/docs/reference/scheduling/config/ for details. (#105424, @kerthcet) [SIG Scheduling and Testing]

  • Track the number of Pods with a Ready condition in Job status. The feature is alpha and needs the feature gate JobReadyPods to be enabled. (#104915, @alculquicondor) [SIG API Machinery, Apps, CLI and Testing]

Feature

  • Add a new distribute-cpus-across-numa option to the static CPUManager policy. When enabled, this will trigger the CPUManager to evenly distribute CPUs across NUMA nodes in cases where more than one NUMA node is required to satisfy the allocation. (#105631, @klueska) [SIG Node]

  • Add support to generate client-side binaries for windows/arm64 platform (#104894, @pacoxu) [SIG CLI, Testing and Windows]

  • Added a new feature gate CustomResourceValidationExpressions to enable expression validation for Custom Resource. (#105107, @cici37) [SIG API Machinery]

  • Adds new [alpha] command 'kubectl events' (#99557, @bboreham) [SIG CLI]

  • Client-go, using log level 9, trace the following events of an http request: - dns lookup - tcp dialing - tls handshake - time to get a connection from the pool - time to process a request (#105156, @aojea) [SIG API Machinery]

  • Client-go: pass DeleteOptions down to the fake client Reactor (#102945, @chenchun) [SIG API Machinery, Apps and Auth]

  • Enhance scheduler volumebinding plugin to handle Lost PVC as UnschedulableAndUnresolvable during PreFilter stage (#105245, @yibozhuang) [SIG Scheduling and Storage]

  • Feature-gate StorageObjectInUseProtection has been deprecated and cannot be disabled. It will be completely removed in 1.25 (#105495, @ikeeip) [SIG Apps]

  • Kubectl will now provide shell completion choices for the --output/-o flag (#105851, @marckhouzam) [SIG CLI]

  • Kubernetes is now built with Golang 1.17.2 (#105563, @mengjiao-liu) [SIG API Machinery, Cloud Provider, Instrumentation, Release and Testing]

  • Move the getAllocatableResources endpoint in podresource-api to the beta that will make it enabled by default. (#105003, @swatisehgal) [SIG Node and Testing]

  • Node affinity, node selector and tolerations are now mutable for jobs that are suspended and have never been started (#105479, @ahg-g) [SIG Apps, Scheduling and Testing]

  • Pod template annotations and labels are now mutable for jobs that are suspended and have never been started (#105980, @ahg-g) [SIG Apps]

  • PodSecurity: add a container image and manifests for the PodSecurity validating admission webhook (#105923, @liggitt) [SIG Auth]

  • PodSecurity: in 1.23+ restricted policy levels, pods and containers which set runAsUser=0 are forbidden at admission-time; previously, they would be rejected at runtime (#105857, @liggitt) [SIG Auth]

  • Shell completion now knows to continue suggesting resource names when the command supports it. For example "kubectl get pod pod1 " will suggest more pod names. (#105711, @marckhouzam) [SIG CLI]

  • Support to enable Hyper-V in GCE Windows Nodes created with kube-up (#105999, @mauriciopoppe) [SIG Cloud Provider and Windows]

  • The CPUManager policy options are now enabled, and we introduce a graduation path for the new CPU Manager policy options. (#105012, @fromanirh) [SIG Node and Testing]

  • The etcd container image now supports Windows. (#92433, @claudiubelu) [SIG API Machinery and Windows]

  • The pods and pod controllers that are exempted from the PodSecurity admission process are now marked with the "pod-security.kubernetes.io/exempt: user/namespace/runtimeClass" annotation, based on what caused the exemption.

    The enforcement level that allowed or denied pod during PodSecurity admission is now marked by the "pod-security.kubernetes.io/enforce-policy" annotation.

    The annotation that informs about audit policy violations changed from ""pod-security.kubernetes.io/audit" to ""pod-security.kubernetes.io/audit-violation". (#105908, @stlaz) [SIG Auth]

  • When feature gate JobTrackingWithFinalizers is enabled:

    • Limit the number of pods tracked in a single job sync to avoid starvation of small jobs.
    • The metric job_pod_finished_total counts the number of finished pods tracked by the job controller (#105197, @alculquicondor) [SIG Apps, Instrumentation and Testing]

Failing Test

  • Fixes hostpath storage e2e tests within SELinux enabled env (#104551, @Elbehery) [SIG Testing]

Bug or Regression

  • (PodSecurity admission) errors validating workload resources (deployment, replicaset, etc.) no longer block admission. (#106017, @tallclair) [SIG Auth]
  • Add Pod Security admission metrics: pod_security_evaluations_total, pod_security_exemptions_total, pod_security_errors_total (#105898, @tallclair) [SIG Auth, Instrumentation and Testing]
  • Apimachinery: pretty-printed json and yaml output is now indented consistently (#105466, @liggitt) [SIG API Machinery]
  • Change kubectl diff --invalid-arg status code from 1 to 2 to match docs (#105445, @ardaguclu) [SIG CLI]
  • Client-go uses the same http client for all the generated groups and versions, allowing to share customized transports for multiple groups versions. (#105490, @aojea) [SIG API Machinery, Auth, Instrumentation and Testing]
  • Evicted and other terminated pods will no longer revert to Running phase (#105462, @ehashman) [SIG Node and Testing]
  • Fix pod name of NonIndexed jobs to not include rogue -1 substring (#105676, @alculquicondor) [SIG Apps]
  • Fix scoring for NodeResourcesBalancedAllocation plugins when nodes have containers with no requests. (#105845, @ahmad-diaa) [SIG Scheduling]
  • Fix: consolidate logs for instance not found error fix: skip not found nodes when reconciling LB backend address pools (#105188, @nilo19) [SIG Cloud Provider]
  • Fix: do not delete the lb that does not exist (#105777, @nilo19) [SIG Cloud Provider]
  • Fix: ignore not a VMSS error for VMAS nodes in EnsureBackendPoolDeleted. (#105185, @ialidzhikov) [SIG Cloud Provider]
  • Fix: leave the probe path empty for TCP probes (#105253, @nilo19) [SIG Cloud Provider]
  • Fix: remove VMSS and VMSS instances from SLB backend pool only when necessary (#105839, @nilo19) [SIG Cloud Provider]
  • Fix: skip instance not found when decoupling vmss from lb (#105666, @nilo19) [SIG Cloud Provider]
  • Fixed a bug that prevents PersistentVolume that has a Claim UID which doesn't exist in local cache but exists in ETCD from being updated to Released phase. (#105211, @xiaopingrubyist) [SIG Apps]
  • Fixed architecture within manifest for non amd64 etcd images. (#105484, @saschagrunert) [SIG API Machinery]
  • Fixes a bug that could result in the EndpointSlice controller unnecessarily updating EndpointSlices associated with a Service that had Topology Aware Hints enabled. (#105267, @llhuii) [SIG Apps and Network]
  • Fixes the should support building a client with a CSR e2e test to work with clusters configured with short certificate lifetimes (#105396, @liggitt) [SIG Auth and Testing]
  • Generic ephemeral volumes can be used also as raw block devices, but the Pod validation was refusing to create pods with that combination. (#105682, @pohly) [SIG Apps, Storage and Testing]
  • Generic ephemeral volumes were not considered properly by the node limits scheduler filter and the kubelet hostpath check. (#100482, @pohly) [SIG Node, Scheduling, Storage and Testing]
  • Kube-apiserver: fix a memory leak when deleting multiple objects with a deletecollection. (#105606, @sxllwx) [SIG API Machinery]
  • Kubeadm: do not allow empty "--config" paths to be passed to "kubeadm kubeconfig user" (#105649, @navist2020) [SIG Cluster Lifecycle]
  • Kubelet did not report kubelet_volume_stats_* metrics for generic ephemeral voiumes. (#105569, @pohly) [SIG Node]
  • Kubelet's Node Grace Shutdown will terminate probes when shutting down. (#105215, @rphillips) [SIG Node]
  • Kubernetes object references (= name + namespace) were not logged as struct when using JSON as log output format. (#104877, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
  • Podresources interface was changed, now it returns only isolated cpus (#97415, @AlexeyPerevalov) [SIG Node and Testing]
  • Release-note Removed error message label from kubelet_started_pods_errors_total metric (#105213, @yxxhero) [SIG Instrumentation and Node]
  • Resolves a potential issue with GC and NS controllers which may delete objects after getting a 404 response from the server during its startup. This PR ensures that requests to aggregated APIs will get 503, not 404 while the APIServiceRegistrationController hasn't finished its job. (#104748, @p0lyn0mial) [SIG API Machinery]
  • Revert building binaries with PIE mode. (#105352, @ehashman) [SIG Node, Release and Security]
  • Support more than 100 disk mounts on Windows (#105673, @andyzhangx) [SIG Storage and Windows]
  • Support using negative array index in json patch replace operations. (#105896, @zqzten) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
  • The --leader-elect* CLI args are now honored in scheduler. (#105712, @Huang-Wei) [SIG Scheduling]
  • The client-go dynamic client sets the header 'Content-Type: application/json' by default (#104327, @sxllwx) [SIG API Machinery]
  • The pods/binding subresource now honors metadata.uid and metadata.resourceVersion preconditions (#105913, @aholic) [SIG Scheduling]
  • Topology Hints now excludes control plane notes from capacity calculations. (#104744, @robscott) [SIG Apps and Network]
  • Watch requests that are delegated to aggregated apiservers no longer reserve concurrency units (seats) in the API Priority and Fairness dispatcher for their entire duration. (#105511, @benluddy) [SIG API Machinery]
  • --log-flush-frequency had no effect in several commands or was missing. Help and warning texts were not always using the right format for a command (add_dir_header instead of add-dir-header). Fixing this included cleaning up flag handling in component-base/logs: that package no longer adds flags to the global flag sets. Commands which want the klog and --log-flush-frequency flags must explicitly call logs.AddFlags; the new cli.Run does that for commands. That helper function also covers flag normalization and printing of usage and errors in a consistent way (print usage text first if parsing failed, then the error). (#105076, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling and Testing]

Other (Cleanup or Flake)

  • All klog flags except for -v and -vmodule are deprecated. Support for -vmodule is only guaranteed for the text log format. (#105042, @pohly) [SIG API Machinery, Architecture, CLI, Cluster Lifecycle and Instrumentation]
  • Kube-apiserver: requests to node, service, and pod /proxy subresources with no additional URL path now only automatically redirect GET and HEAD requests. (#95128, @Riaankl) [SIG API Machinery, Architecture and Testing]
  • Migrate pkg/scheduler/framework/plugins/interpodaffinity/filtering.go,pkg/scheduler/framework/plugins/podtopologyspread/filtering.go, pkg/scheduler/framework/plugins/volumezone/volume_zone.go to structured logging (#105931, @mengjiao-liu) [SIG Instrumentation and Scheduling]
  • Migrated cmd/kube-scheduler/app/server.go, pkg/scheduler/framework/plugins/nodelabel/node_label.go, pkg/scheduler/framework/plugins/nodevolumelimits/csi.go, pkg/scheduler/framework/plugins/nodevolumelimits/non_csi.go to structured logging (#105855, @shivanshu1333) [SIG Instrumentation and Scheduling]
  • Migrated pkg/proxy to structured logging (#104891, @shivanshu1333) [SIG Network]
  • Migrated pkg/proxy/ipvs to structured logging (#104932, @shivanshu1333) [SIG Network]
  • Support allocating whole NUMA nodes in the CPUManager when there is not a 1:1 mapping between socket and NUMA node (#102015, @klueska) [SIG Node]

Dependencies

Added

  • sigs.k8s.io/json: c049b76

Changed

Removed

Nothing has changed.

v1.23.0-alpha.3

Downloads for v1.23.0-alpha.3

Source Code

filename sha512 hash
kubernetes.tar.gz 083e6ca03c9d701768b1b5666f354223a3f7dca9fc6410ce45bbf5947152620e300b46df9b6019134e7d736ba44916537eb3bea8fa57e5f7bc3cc34898b4a5dd
kubernetes-src.tar.gz c3fc74d52e1b7e808c03b9caa30e3e73be30eb8330ce676000b93d5324bbdba93bd005d125b999ba937b79d4751af99b37986911365416f7175d223345f95914

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz 31d8adc657afbd305df18bfec397a825536357e23b241a19aa538b6ddefefc59743f737db98756e04deea89cc6f260d40a80f02b4d1dc34af1d19e8d796dcd8a
kubernetes-client-darwin-arm64.tar.gz b69c4d6cde1c476bafa2ca9916ce3e5bf7286be0ff6a08193bdd1a954ba89b64b1b14193d1acec17ccc141024ee3097971448017b5c9f1327e0961b1e92b2224
kubernetes-client-linux-386.tar.gz 059f25ee48aa4b0d1621d6ba87af8fb7e765634d723d98a4e9739f50d3703e7dd3973f4d1ed886c0f3ad6eba165ed81d4e63ecde3b39e66fcbec7d3aa2dfed2e
kubernetes-client-linux-amd64.tar.gz 291dba14160803065895799adcde39bdad7a5b0372403f283d6d5e9a094fe1fc79c70e7546f93ee692b9fd297e2667cb558e4209161ecb4bf89965df5746ed4d
kubernetes-client-linux-arm.tar.gz 988e12cd7466033578acc487447df376c409e4f79726a4721af1aedbe931e927b22a93d6224891b61b55c7a0ec12e42d8cfcd40e15a9a0cbbc1dbf0e59ab0341
kubernetes-client-linux-arm64.tar.gz b3f21dac41b38e671fa7a95892468e2c27fab51abf9c77b336550e5ec213af204e16cac11dd76262fedb0087cf5ad1950af7e36599a38d50cc270cf831cd4f0b
kubernetes-client-linux-ppc64le.tar.gz beebf01e2e4ff09bb711284bb9a5c7cc519e4ac8a826dc829394fa28bd9a3149ba73088eaf6712d39a8cab96b0a1c2859e9d5955fee892b759eaddcdeaa8b93c
kubernetes-client-linux-s390x.tar.gz 87e5d3d8ba01f9fefb2300e9f06146a254d39d72eaa10cad8c444428b738b3763483ee9eb82f0a13d2ff5aba35fdcb4320598fd5a6a2a07ea3fd00b4ac682d3c
kubernetes-client-windows-386.tar.gz 71bfc5a1df9c47735476af10225830212f68c83357ff7d443e18f9b7881524db910781a95d11ff6697cb587352059b5841f7b24fda40b5302ad252bfb6da7e51
kubernetes-client-windows-amd64.tar.gz 078b0c698f9535f3eee41ecf162d57e2ace67243da36067b78b30cfbb7b27cfcf97af4c5db48cdd592953e26b42b31794002eb96317476849e89e2126c6df99d

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 951b790158dadf46c32e1a1e9c12f2cc8f41e1645602ebff6b4130a08a377bc6d92549186b420332d620d67191123d98a5d717ac0f5ee9643bebe88947ead8fa
kubernetes-server-linux-arm.tar.gz 0e7a5b9f39b4f45c45bdb5a19dd3695d28f53e1039d76bc572421c707917944d28b1dbfc36e59214b5bc2b93a787900d8e6eb0b587aa801ea8a8faacdb814a4e
kubernetes-server-linux-arm64.tar.gz 921e060120b8651a0f80977360faca9f207189cee10bc61f669ceba4e540ef48c0ceff1a877ee4c7d31b01b88096bce93c577f68f93b2341c8542dfd89972b60
kubernetes-server-linux-ppc64le.tar.gz 292cde446b754a87f4ef5384fadbd30017e53ed2744d45a724be467c86ccd9837bfb490db6396642a869937f2f0d080d9655e89ca3345f8365d109a9bcdd18d9
kubernetes-server-linux-s390x.tar.gz e0ea667f828ce3b36ca4b2a05fb286da5eb321852c50caf0957694553caf2908b27bcc37a5a82277a2606cf6ff4d9e33617ad61628845d9c21f5cf68c960ca92

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz e13cd3f75628d354bd1544a5495600fb905741431eb4af4da3d980cc0b7565e3f9c1585d9686cc4e967e54fb854f05bbedfe0c60bb7b855fa027ac8ac45b26e0
kubernetes-node-linux-arm.tar.gz 6c91b42350528692ff558b667bffd41c5b967c7aa6101471274e4b16b0ac6f84afe01722881328fd4f6f8fe71c7852620fa000186c6f7e56e498fcc2c67ad793
kubernetes-node-linux-arm64.tar.gz 81728e1388e9cdb436d6847c868f28ab2771331e5e40cd5a7af13cb8dc80a7e4e66a215c12f8183b4884807a3962f913ef5343b889e3c4ecd0e410e8d53aaea9
kubernetes-node-linux-ppc64le.tar.gz 299649f1b25cc38f3a7543ef4d3ee6d42c85e24ac41b4eb61927bc5c5f0c533a39f9ddd4d5ad1df54c625d77aeb41f6c31b1ca7fd8983262f84fefdf1cb2cfd0
kubernetes-node-linux-s390x.tar.gz fd6cbc93f98abff9803b43215af6e75a4f7b91ca06969220a779468f34b5ec5ec69f20b529e0cd7b10ba8769bbe2507d46f84ce1d8cd0760380ab9264dd94672
kubernetes-node-windows-amd64.tar.gz a5bfaf2e3ad8d3d2127c3e3e0f131c615a03563253da6bf0e1fd793f6ef71287f341ce1bd0d35eb9a81e0721a5baf03e7c72863b5ed8eb45e8fe70573904ed54

Changelog since v1.23.0-alpha.2

Changes by Kind

Deprecation

  • Remove 'master' as a valid EgressSelection type in the EgressSelectorConfiguration API. (#102242, @pacoxu) [SIG API Machinery and Cloud Provider]
  • Remove VolumeSubpath feature gate (#105090, @saad-ali) [SIG Apps, Node and Storage]
  • The deprecated --experimental-bootstrap-kubeconfig flag has been removed. This can be set via --bootstrap-kubeconfig. (#103172, @niulechuan) [SIG Node]

API Change

  • Client-go impersonation config can specify a UID to pass impersonated uid information through in requests. (#104483, @margocrawf) [SIG API Machinery, Auth and Testing]
  • IPv6DualStack feature moved to stable. Controller Manager flags for the node IPAM controller have slightly changed:
    1. When configuring a dual-stack cluster, the user must specify both --node-cidr-mask-size-ipv4 and --node-cidr-mask-size-ipv6 to set the per-node IP mask sizes, instead of the previous --node-cidr-mask-size flag.
    2. The --node-cidr-mask-size flag is mutually exclusive with --node-cidr-mask-size-ipv4 and --node-cidr-mask-size-ipv6.
    3. Single-stack clusters do not need to change, but may choose to use the more specific flags. Users can use either the older --node-cidr-mask-size flag or one of the newer --node-cidr-mask-size-ipv4 or --node-cidr-mask-size-ipv6 flags to configure the per-node IP mask size, provided that the flag's IP family matches the cluster's IP family (--cluster-cidr). (#104691, @khenidak) [SIG API Machinery, Apps, Auth, Cloud Provider, Cluster Lifecycle, Network, Node and Testing]
  • Kubelet: turn the KubeletConfiguration v1beta1 ResolverConfig field from a string to *string. (#104624, @Haleygo) [SIG Cluster Lifecycle and Node]

Feature

  • Add mechanism to load simple sniffer class into fluentd-elasticsearch image (#92853, @cosmo0920) [SIG Cloud Provider and Instrumentation]
  • Kubeadm: do not check if the '/etc/kubernetes/manifests' folder is empty on joining worker nodes during preflight (#104942, @SataQiu) [SIG Cluster Lifecycle]
  • The kube-apiserver's Prometheus metrics have been extended with some that describe the costs of handling LIST requests. They are as follows.
    • apiserver_cache_list_total: Counter of LIST requests served from watch cache, broken down by resource_prefix and index_name
    • apiserver_cache_list_fetched_objects_total: Counter of objects read from watch cache in the course of serving a LIST request, broken down by resource_prefix and index_name
    • apiserver_cache_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from watch cache, broken down by resource_prefix
    • apiserver_cache_list_returned_objects_total: Counter of objects returned for a LIST request from watch cache, broken down by resource_prefix
    • apiserver_storage_list_total: Counter of LIST requests served from etcd, broken down by resource
    • apiserver_storage_list_fetched_objects_total: Counter of objects read from etcd in the course of serving a LIST request, broken down by resource
    • apiserver_storage_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from etcd, broken down by resource
    • apiserver_storage_list_returned_objects_total: Counter of objects returned for a LIST request from etcd, broken down by resource (#104983, @MikeSpreitzer) [SIG API Machinery and Instrumentation]
  • Turn on CSIMigrationAzureDisk by default on 1.23 (#104670, @andyzhangx) [SIG Cloud Provider]

Bug or Regression

  • Changes behaviour of kube-proxy start; does not attempt to set specific sysctl values (which does not work in recent Kernel versions anymore in non-init namespaces), when the current sysctl values are already set higher. (#103174, @Napsty) [SIG Network]
  • Fix job controller syncs: In case of conflicts, ensure that the sync happens with the most up to date information. Improves reliability of JobTrackingWithFinalizers. (#105214, @alculquicondor) [SIG Apps]
  • Fix system default topology spreading when nodes don't have zone labels. Pods correctly spread by default now. (#105046, @alculquicondor) [SIG Scheduling]
  • Headless Services with no selector which were created without dual-stack enabled will be defaulted to RequireDualStack instead of PreferDualStack. This is consistent with such Services which are created with dual-stack enabled. (#104986, @thockin) [SIG Network]
  • Kube-apiserver: events created via the events.k8s.io API group for cluster-scoped objects are now permitted in the default namespace as well for compatibility with events clients and the v1 API (#100125, @h4ghhh) [SIG API Machinery, Apps and Testing]
  • Kube-controller incorrectly enabled support for generic ephemeral inline volumes if the storage object in use protection feature was enabled. (#104913, @pohly) [SIG API Machinery]
  • Kubeadm: switch the preflight check (called 'Swap') that verifies if swap is enabled on Linux hosts to report a warning instead of an error. This is related to the graduation of the NodeSwap feature gate in the kubelet to Beta and being enabled by default in 1.23 - allows swap support on Linux hosts. In the next release of kubeadm (1.24) the preflight check will be removed, thus we recommend that you stop using it - e.g. via --ignore-preflight-errors or the kubeadm config. (#104854, @pacoxu) [SIG Cluster Lifecycle]
  • Makes the etcd client (used by the API server) retry certain types of errors. The full list of retriable (codes.Unavailable) errors can be found at https://github.com/etcd-io/etcd/blob/main/api/v3rpc/rpctypes/error.go#L72 (#105069, @p0lyn0mial) [SIG API Machinery]
  • When a static pod file is deleted and recreated while using a fixed UID, the pod was not properly restarted. (#104847, @smarterclayton) [SIG Node and Testing]
  • XFS-filesystems are now force-formatted (option -f) in order to avoid problems being formatted due to detection of magic super-blocks. This aligns with the behaviour of formatting of ext3/4 filesystems. (#104923, @davidkarlsen) [SIG Storage]

Other (Cleanup or Flake)

  • Enhanced error message for nodes not selected by scheduler due to pod's PersistentVolumeClaim(s) bound to PersistentVolume(s) that do not exist. (#105196, @yibozhuang) [SIG Scheduling and Storage]
  • Kubeadm: remove the --port flag from the manifest for the kube-scheduler since the flag has been a NO-OP since 1.23 and insecure serving was removed for the component. (#105034, @pacoxu) [SIG Cluster Lifecycle]
  • Migrate cmd/proxy/{config, healthcheck, winkernel} to structured logging (#104944, @jyz0309) [SIG Network]
  • Migrate cmd/proxy/app and pkg/proxy/meta_proxier to structured logging (#104928, @jyz0309) [SIG Apps, Cluster Lifecycle, Network, Node and Testing]
  • Migrate pkg/proxy to structured logs (#104908, @CIPHERTron) [SIG Network]
  • Migrated pkg/proxy/winuserspace to structured logging (#105035, @shivanshu1333) [SIG Network]
  • The BoundServiceAccountTokenVolume feature gate that is GA since v1.22 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#104167, @ialidzhikov) [SIG Auth]
  • The SupportPodPidsLimit and SupportNodePidsLimit feature gates that are GA since v1.20 are unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#104163, @ialidzhikov) [SIG Node]
  • Update build images to Debian 11 (Bullseye)
    • debian-base:bullseye-v1.0.0
    • debian-iptables:bullseye-v1.0.0
    • go-runner:v2.3.1-go1.17.1-bullseye.0
    • kube-cross:v1.23.0-go1.17.1-bullseye.1
    • setcap:bullseye-v1.0.0
    • cluster/images/etcd: Build 3.5.0-2 image
    • test/conformance/image: Update runner image to base-debian11 (#105158, @justaugustus) [SIG API Machinery, Architecture, Release and Testing]

Dependencies

Added

Nothing has changed.

Changed

Removed

Nothing has changed.

v1.23.0-alpha.2

Downloads for v1.23.0-alpha.2

Source Code

filename sha512 hash
kubernetes.tar.gz 121d51f42a52b28e27a4b2f914a4f80fa3fba6328e6a4a5c96dec39c5b28c05461fcc290ef35a49058e237091532b24db3cd8c61801bcb6736aee1dd7dbcffc3
kubernetes-src.tar.gz 641d47241acfadb3b13bccec57795749d2c9e3e07ffa7aa4b30df3a488643631eb8e5cd581bcfb764dff4ac5ed755f72d94e80746142123b09e1675e81421a91

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz f734cb514ee56adcb2d991a6f0550df907c72f8a61cc2a13117e61b8d5826ff942a582a2e9383deb1a61d5df2243362f1327942a3b4883490eb3296647ce3737
kubernetes-client-darwin-arm64.tar.gz 24d1f851cd5782f8f39054e37beda1554dadd8a28cb3272b00d50fc095d1fc3018768c1ea72a44eda61ff0f58f71b33dd28cbdc54467d620e87c3694ecf14cc2
kubernetes-client-linux-386.tar.gz 082ad4abea58de3b629fc2ed4560a836cdbeb1adefb0c4cf47044bf33c750d8fcd8a06e2c4ce365853e83a58d52e0129d510a698dd894bd1261f8184dd1cab42
kubernetes-client-linux-amd64.tar.gz b3b0b23479c05b57ca574cf17cdcde7e716033bc4f6a80532d1175d8e533e3202bece0dcf503731d5a60319c526ce1ce4a0bc900bf87536321208a59cf890e35
kubernetes-client-linux-arm.tar.gz f5dac2976ce04310f74bba6102080554309b851fbd966ff1220d3eb23089db8eb8da519a6bd8865c94f2f24346a4d27eb40fd0a3ff06ca9c6874e1fc6f356b67
kubernetes-client-linux-arm64.tar.gz 057b372150749b13a38e04802c7cf566765e0fbb27f1b5f7bf6d3cc3f71eb3020916ea7f8579ecc7fcc10e2db1b5c8caa31a1e8a3aac80da86e4e777f515d42f
kubernetes-client-linux-ppc64le.tar.gz 9a090d22aeba011c6d039bff59dbdc23ac4a112828db3cbba588d8b0ee1cd14d16e0eacefbb000e5a3ff26bcce4730824819f86a99b7a9826f35fa9964f9f27a
kubernetes-client-linux-s390x.tar.gz 435e20055badb619289dc7c572af300bd2f86068d0b8f326e8d9abfda5347f2449e316158c412e9b946a2541208c3e8cc6e5c823946e74ac4fc2d594d410179a
kubernetes-client-windows-386.tar.gz 55f192a4d095d494bb53af1b7133124b762a677eb46247b9dba71d10ea6830b37c30d603908e7a9c63f371baff508b19406e89b231ed5ece0497627f09753f68
kubernetes-client-windows-amd64.tar.gz 944059d1f1918a793490b95be8130d06189508ba8e79e79ca8cfd2ab98bf396ac551786514b093cc6afe4b3fd15736d728cfcdce18bb32fbee41bc0a97f5c4be

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz a76a4b86ee151ba027f7cf4a2072451ae4c829182bb14e00ce1967421744bfc1e58f141b6eaf2ab27ece67054ae307f8e0768477ab9c3c4749eaad397d495182
kubernetes-server-linux-arm.tar.gz 95aeb4eb473ab4920d81904bc89c6126732b9c6888f9e57493ee99d692042ca44f6844ac1dade1409565f4d9fbec59445402e1f7deac6cbf5b6df16ac814b58c
kubernetes-server-linux-arm64.tar.gz 3c56e906aafc2a1ac72300352a334662bec5d59e3e523c19b9d65bc52ad9075dc2631f259513efd0f654e220fe0e7d54dfa5028d7eaad81d5d87ca251653f75d
kubernetes-server-linux-ppc64le.tar.gz b74bacafe9bb6a7cf407747b03e78ae3873e50deec4eaa08758d5e1d5287ac23af59b3ef26f888fe4cd44ccb1455beafcd1384e700230eb445720e3acae5f2e3
kubernetes-server-linux-s390x.tar.gz d3f8f8d9c233b114129f615252d42782cd366978a49506393a40af3f8b5b1250ce99e9806881675e112a69270a0411fb2f00ea19b99ad7415b9e0074beb2726d

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 146e2f762c179178a57a8c7af7c26470c5d580b8ff8400615162ad1056625f87ce2b32598538d82652f88639e54afb782810529b074c36eb52cc6374414a6181
kubernetes-node-linux-arm.tar.gz 9357d1b387e1b049fb6cec06a7081afc2ce7e906484c9b061fb0449d147a6c4f9c9dc7a9219cdca5ed71df6c73784f360018d9e48d4fa2aa7eeabef60649d7a4
kubernetes-node-linux-arm64.tar.gz 8394f8f9d6ee823cb9a470ea67e15d4d0c6aca7065fe826788f50955905373fc3cdddd6db43901c07736588d8d6a3d3e2916bc8d45fd6bd06307583686137a0a
kubernetes-node-linux-ppc64le.tar.gz 7211cb426834484bff39f1ab3c9541203429039f8f5e522ca9e28c43da749e197128a3cae28db0467fc339305d2f23f85e8b4ed9ec116506c3d8076744a88d5e
kubernetes-node-linux-s390x.tar.gz a7c1a38250398171d3df5865749e9928867c4f44106ae66d44cf9f948ce4f4eed9d1f273a5d369996425b1e12482fceccde4c7652770a8c9fb3f161811323b69
kubernetes-node-windows-amd64.tar.gz 2007b3b16597cc06b486f87f35b6c637404f07c11d88b8c8e1c2c9bbea97f762bd7d4f9a31f42f78a917c595af5cb89e6885dd88f3766836dc6e4ec79cf084f2

Changelog since v1.23.0-alpha.1

Changes by Kind

Deprecation

  • Controller-manager: the following flags have no effect and would be removed in v1.24:

    • --port
    • --address The insecure port flags --port may only be set to 0 now. Also metricsBindAddress and healthzBindAddress fields from kubescheduler.config.k8s.io/v1beta1 are no-op and expected to be empty. Removed in kubescheduler.config.k8s.io/v1beta2 completely.

    In addition, please be careful that:

    • kube-scheduler MUST start with --authorization-kubeconfig and --authentication-kubeconfig correctly set to get authentication/authorization working.
    • liveness/readiness probes to kube-scheduler MUST use HTTPS now, and the default port has been changed to 10259.
    • Applications that fetch metrics from kube-scheduler should use a dedicated service account which is allowed to access nonResourceURLs /metrics. (#96345, @ingvagabund) [SIG Cloud Provider, Scheduling and Testing]

  • Removed deprecated metric scheduler_volume_scheduling_duration_seconds (#104518, @dntosas) [SIG Instrumentation, Scheduling and Storage]

API Change

  • A small regression in Service updates was fixed. The circumstances are so unlikely that probably nobody would ever hit it. (#104601, @thockin) [SIG Network]
  • Introduce v1beta2 for Priority and Fairness with no changes in API spec (#104399, @tkashem) [SIG API Machinery and Testing]
  • Kube-apiserver: Fixes handling of CRD schemas containing literal null values in enums. (#104969, @liggitt) [SIG API Machinery, Apps and Network]
  • Kubelet: turn the KubeletConfiguration v1beta1 ResolverConfig field from a string to *string. (#104624, @Haleygo) [SIG Cluster Lifecycle and Node]
  • Kubernetes is now built using go1.17 (#103692, @justaugustus) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]
  • Removed deprecated --seccomp-profile-root/seccompProfileRoot config (#103941, @saschagrunert) [SIG Node]
  • Since golang 1.17 both net.ParseIP and net.ParseCIDR rejects leading zeros in the dot-decimal notation of IPv4 addresses. Kubernetes will keep allowing leading zeros on IPv4 address to not break the compatibility. IMPORTANT: Kubernetes interprets leading zeros on IPv4 addresses as decimal, users must not rely on parser alignment to not being impacted by the associated security advisory: CVE-2021-29923 golang standard library "net" - Improper Input Validation of octal literals in golang 1.16.2 and below standard library "net" results in indeterminate SSRF & RFI vulnerabilities. Reference: https://nvd.nist.gov/vuln/detail/CVE-2021-29923 (#104368, @aojea) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage and Testing]
  • StatefulSet minReadySeconds is promoted to beta (#104045, @ravisantoshgudimetla) [SIG Apps and Testing]
  • The Service.spec.ipFamilyPolicy field is now required in order to create or update a Service as dual-stack. This is a breaking change from the beta behavior. Previously the server would try to infer the value of that field from either ipFamilies or clusterIPs, but that caused ambiguity on updates. Users who want a dual-stack Service MUST specify ipFamilyPolicy as either "PreferDualStack" or "RequireDualStack". (#96684, @thockin) [SIG API Machinery, Apps, Network and Testing]
  • Users of LogFormatRegistry in component-base must update their code to use the logr v1.0.0 API. The JSON log output now uses the format from go-logr/zapr (no v field for error messages, additional information for invalid calls) and has some fixes (correct source code location for warnings about invalid log calls). (#104103, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
  • When creating an object with generateName, if a conflict occurs the server now returns an AlreadyExists error with a retry option. (#104699, @vincepri) [SIG API Machinery]

Feature

  • Add fish shell completion to kubectl (#92989, @WLun001) [SIG CLI]
  • Added PowerShell completion generation by running kubectl completion powershell (#103758, @zikhan) [SIG CLI]
  • Added a Processing condition for the workqueue API Changed Shutdown for the workqueue API to wait until the work queue finishes processing all in-flight items. (#101928, @alexanderConstantinescu) [SIG API Machinery and Apps]
  • Added a new flag --append-server-path to kubectl proxy that will automatically append the kube context server path to each request. (#97350, @FabianKramm) [SIG API Machinery, CLI and Testing]
  • Added support for setting controller-manager log level online (#104571, @h4ghhh) [SIG API Machinery, Apps and Cloud Provider]
  • Adding support for multiple --from-env-file flags (#104232, @lauchokyip) [SIG CLI]
  • Cloud providers can set service account names for cloud controllers. (#103178, @nckturner) [SIG API Machinery and Cloud Provider]
  • Health check of kube-controller-manager now includes each controller. (#104667, @jiahuif) [SIG API Machinery and Cloud Provider]
  • Kubernetes is now built with Golang 1.17.1 (#104904, @cpanato) [SIG API Machinery, Cloud Provider, Instrumentation, Release and Testing]
  • The pause image list now contains Windows Server 2022 (#104438, @nick5616) [SIG Windows]
  • Updates debian-iptables to v1.6.7 to pick up CVE fixes (#104970, @PushkarJ) [SIG API Machinery, Network, Release, Security and Testing]

Documentation

  • Conformance: the test "[sig-network] EndpointSlice should have Endpoints and EndpointSlices pointing to API Server [Conformance]" only requires that there is an EndpointSlice that references the "kubernetes.default" service, it no longer requires that its named "kubernetes". (#104664, @aojea) [SIG Architecture, Network and Testing]

Bug or Regression

  • A pod that the Kubelet rejects was still considered as being accepted for a brief period of time after rejection, which might cause some pods to be rejected briefly that could fit on the node. A pod that is still terminating (but has status indicating it has failed) may also still be consuming resources and so should also be considered. (#104817, @smarterclayton) [SIG Node]
  • Changed kubectl describe to compute Age of an event using the count and lastObservedTime fields available in the event series (#104482, @harjas27) [SIG CLI]
  • Don't prematurely close reflectors in case of slow initialization in watch based manager to fix issues with inability to properly mount secrets/configmaps. (#104604, @wojtek-t) [SIG Node]
  • Fix Job tracking with finalizers for more than 500 pods, ensuring all finalizers are removed before counting the Pod. (#104666, @alculquicondor) [SIG Apps and Instrumentation]
  • Fix a regression where the Kubelet failed to exclude already completed pods from calculations about how many resources it was currently using when deciding whether to allow more pods. (#104577, @smarterclayton) [SIG Node]
  • Fix detach disk issue on deleting vmss node (#104572, @andyzhangx) [SIG Cloud Provider]
  • Fix: ensure InstanceShutdownByProviderID return false for creating Azure VMs (#104382, @feiskyer) [SIG Cloud Provider]
  • Fix: ignore the case when comparing azure tags in service annotation (#104705, @nilo19) [SIG Cloud Provider]
  • Fix: ignore the case when updating Azure tags (#104593, @nilo19) [SIG Cloud Provider]
  • Fixed bug where kubectl would emit duplicate warning messages for flag names that contain an underscore and recommend using a nonexistent flag in some cases (#103852, @brianpursley) [SIG CLI and Cluster Lifecycle]
  • Fixed client IP preservation for NodePort service with protocol SCTP in ipvs mode (#104756, @tnqn) [SIG Network]
  • Fixed occasional pod cgroup freeze when using cgroup v1 and systemd driver. (#104528, @kolyshkin) [SIG Node]
  • Fixes a regression that could cause panics in LRU caches in controller-manager, kubelet, kube-apiserver, or client-go (#104466, @stbenjam) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
  • Kube-apiserver: fixes an issue where an admission webhook can observe a v1 Pod object that does not have the defaultMode field set in the injected service account token volume (#104523, @liggitt) [SIG Auth]
  • Kube-proxy health check ports used to listen to : for each of the services. This is not needed and opens ports in addresses the cluster user may not have intended. The PR limits listening to all node address which are controlled by --nodeport-addresses flag. if no addresses are provided then we default to existing behavior by listening to : for each service (#104742, @khenidak) [SIG Network]
  • Kube-scheduler now doesn't print any usage message when unknown flag is specified (#104503, @sanposhiho) [SIG Scheduling]
  • Metrics changes: Fix exposed buckets of scheduler_volume_scheduling_duration_seconds_bucket metric (#100720, @dntosas) [SIG Apps, Instrumentation, Scheduling and Storage]
  • Scheduler resource metrics over fractional binary quantities (2.5Gi, 1.1Ki) were incorrectly reported as very small values. (#103751, @y-tag) [SIG API Machinery and Scheduling]

Other (Cleanup or Flake)

  • Generic ephemeral volumes: better pod events ("waiting for ephemeral volume controller to create the persistentvolumeclaim"" instead of "persistentvolumeclaim not found") (#104605, @pohly) [SIG Scheduling and Storage]
  • Kubeadm: remove the deprecated flags "--csr-only" and "--csr-dir" from "kubeadm certs renew". Please use "kubeadm certs generate-csr" instead. (#104796, @RA489) [SIG Cluster Lifecycle]
  • Migrate pkg/scheduler to structured logging (#99273, @yangjunmyfm192085) [SIG Scheduling]
  • Migrated pkg/proxy/userspace to structured logging (#104931, @shivanshu1333) [SIG Network]
  • More detailed logging has been added to the EndpointSlice controller for Topology Aware Hints. (#104741, @robscott) [SIG Apps and Network]
  • Support for Windows Server 2022 was added to the k8s.gcr.io/pause:3.6 image. (#104711, @claudiubelu) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release and Testing]
  • The maximum length of the CSINode id field has increased to 256 bytes to match the CSI spec (#104160, @pacoxu) [SIG Storage]
  • Update conformance image to use debian-base:buster-v1.9.0 (#104696, @PushkarJ) [SIG Architecture, Release, Security and Testing]
  • volume.kubernetes.io/storage-provisioner annotation will be added to dynamic provisioning required PVC. volume.beta.kubernetes.io/storage-provisioner annotation is deprecated. (#104590, @Jiawei0227) [SIG Apps and Storage]

Dependencies

Added

  • bazil.org/fuse: 371fbbd
  • github.com/go-logr/zapr: v1.1.0
  • github.com/kr/fs: v0.1.0
  • github.com/pkg/sftp: v1.10.1

Changed

Removed

v1.23.0-alpha.1

Downloads for v1.23.0-alpha.1

Source Code

filename sha512 hash
kubernetes.tar.gz f7c76f1e077b5d98019347b2c9b79eaa0c79d428542b9c15dab23886c276ca16314f200ca37af914c52264c0e1e5d0bde639d6adf37368d5e7b29d230df00d95
kubernetes-src.tar.gz f267f26eca20cd7018e68abeeed38aed5c10dbbae7c531c4e08e507196a4dd3f511eb8d41ee8b09495544337d8e1940a8ca04e94084f8dd172698a96564fb070

Client Binaries

filename sha512 hash
kubernetes-client-darwin-amd64.tar.gz deb110839c2c3cf94ca9b29df2f0b07b3fad6937d7bb6e9d2516d01345c8e324f6ab86fe1d34f1443f04c3d1fc328b53b3d756c295f4ed22f1994071fbc8c9cb
kubernetes-client-darwin-arm64.tar.gz 1473cb9fc4847b0daff6c9e3189ce55fadc22fb6190161e744e5438066a714cb467fdebfb35f6445a27f5010df94ee602fff492a2382e0f308fda111d53af1f4
kubernetes-client-linux-386.tar.gz ed5f5b0777ca51790d185764afc2c812f82ae27c35d897570fc86cabee90dc0a445d9d8c37c981bd3684ba9cd47dc0d75d0094578e79ef7b591d3c1b6564280f
kubernetes-client-linux-amd64.tar.gz 39f2a888e7a43c9e4a4018301894786f6babe23d79ab7a143e06444f69bc14aec2e158d355c5b48da4356e7bd72ec9b1268f8b12815c8b709395f36ad9a68a2f
kubernetes-client-linux-arm.tar.gz b6b8333d8adb4bc6a943bcd2c6cd1a0aeaf0b926d06aa03b759e3c723c81ccc91804debc64fedcd7d678eefdee9bdacc52b2891bd084a15fd5f7918a70e51a15
kubernetes-client-linux-arm64.tar.gz 3cb8217b9a5363cebad4989253e02c8a37259b61eafc2f08681508c11c5f68448cad43282257c3d90ad510cc9a62645b7f1adeb99fedf5e13c181495e3754ee4
kubernetes-client-linux-ppc64le.tar.gz e411700fb13b25deca6347983cdafe47199f0df00086ccd7b3e7d52a7b3bee7e96a85c2568dd52c956fd4ea8b4a6991859c57c9b73a13e06440b456c65b11687
kubernetes-client-linux-s390x.tar.gz 6c1395792a175de77436352d0893476363497b0f6a616f4415f91aed5e780d1f25b515021939a7563046237c7b651caba0d1fbf7c4c461677d1b9308b227e94c
kubernetes-client-windows-386.tar.gz f3aec7136c21d24a99145ce294a859078fcbf11bae132b8b4081555a6656c0d95ccbaca02a86dc257d557ecebc0673d0771b9cdd10593712a643e8cc0f61d681
kubernetes-client-windows-amd64.tar.gz b29697ba0a25f3d871ffbe5800dcb23ec9fd27c0122a284e17c21f1258f7dd9d341813aeb7826159c7999581a16db19fbb6eeeab48f5c89975df7595d19102c3

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz a5b3edca559b84cd9d22b43b23d0607951d434e185dcb313b831604d83dd306cfc017599994d3944ce77360116024eb59a302851325bb2c29c185a80db2e6eac
kubernetes-server-linux-arm.tar.gz 2334dbcff3ba22a50f252998eb63991b6c816659dbaa5f749370fc1b1f78f0af7739e50ab64c14a23c4e7dfa8917568e2a3b85bdffdb2cc691ee23ae8f5c8326
kubernetes-server-linux-arm64.tar.gz 58674443ce6e359a995dd7c4289bf730e616bcaf336837b77333a206d4e98693d9356a0a670ffbe0b274e2997a8b76a164153cf084f0ff5f91f40f00b5512684
kubernetes-server-linux-ppc64le.tar.gz f60ebdd04e2348b1ba51540cad93fa24cb133fd25db97150000bffaff8ccb41e1b6506bcde6b7d913aee7701478f975a97775430a82980105383fdb1cc13d260
kubernetes-server-linux-s390x.tar.gz ff008aa0ba1bf755f32c7251c6aceb12b6f9de00d2e2729302b51960e70e486bd82da62d21d70ad81c14e01910ab2afe0fd2509ebfdec050d36f88ee1f0330b2

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 352502f10fbc4579bd9556e3f73ca7513184371ea563d12a39d655d39bb14ccf0f485f4f2b54a77d984c91ff0de2acea7225f98532a1247da5b9ecc65081bc1a
kubernetes-node-linux-arm.tar.gz af9de95e2b9e4c1f39cb9757d4dca020f7d276b6702302a2d92e7a93e9986528615ce54531e62b96f6e8a0b9863cddbb264f42b1f59374948ac3499af60d9532
kubernetes-node-linux-arm64.tar.gz 45a286cb1d469b16d046af02047cf63a8407222e4a39fe696f5652e0587e0c9ffbdbab6505ce85e2726ba10db3189a7fbe70e316bc610caedc8cbb49fed28076
kubernetes-node-linux-ppc64le.tar.gz 7a540a3ff0295998a1679b0ccd50cb1825faf1d0afd6ed08138ab3767c83a2743aa43b122c8da89ee00161f57c0af8d76012e890f9fe6d77b4ee8aff4e32e50f
kubernetes-node-linux-s390x.tar.gz 3cd7656221ac2fa161abcf237878cff26c1d97cf77d9b784736c97a56841397ff859e43947d81a83f8fe4164701da41a1dad69b551c4e1fee49b3f8196878236
kubernetes-node-windows-amd64.tar.gz 21e63913024e88a48244a598cd400fbae6ce8f8910202f1b635812fbc9281b7c6097eb10a321dd18846484a198845bba58970d83b5119a367862cf8418d4d08c

Changelog since v1.22.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

  • Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

([#104389](kubernetes#104389), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node]
  • Kubeadm: remove the deprecated flag --experimental-patches for the init|join|upgrade commands. The flag --patches is no longer allowed in a mixture with the flag --config. Please use the kubeadm configuration for setting patches for a node using {Init|Join}Configuration.patches. (#104065, @pacoxu) [SIG Cluster Lifecycle]

Changes by Kind

Deprecation

  • Add apiserver_longrunning_requests metric to replace the soon to be deprecated apiserver_longrunning_gauge metric. (#103799, @jyz0309) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
  • Kubeadm: remove the --port flag from the manifest for the kube-controller-manager since the flag has been a NO-OP since 1.22 and insecure serving was removed for the component. (#104157, @knight42) [SIG Cluster Lifecycle]

API Change

  • CSIDriver.Spec.StorageCapacity can now be modified. (#101789, @pohly) [SIG Storage]
  • Kube-apiserver: The rbac.authorization.k8s.io/v1alpha1 API version is removed; use the rbac.authorization.k8s.io/v1 API, available since v1.8. The scheduling.k8s.io/v1alpha1 API version is removed; use the scheduling.k8s.io/v1 API, available since v1.14. (#104248, @liggitt) [SIG API Machinery, Auth, Network and Testing]
  • Kube-controller-manager supports '--concurrent-ephemeralvolume-syncs' flag to set the number of ephemeral volume controller workers. (#102981, @SataQiu) [SIG API Machinery and Apps]

Feature

  • Adding support for multiple --from-env-file flags (#101646, @lauchokyip) [SIG CLI]

  • All folks to build kubernetes with a custom kube-cross image (#104185, @dims) [SIG Release and Testing]

  • Allow node expansion of local volumes (#102886, @gnufied) [SIG Storage and Testing]

  • Client-go event library allows customizing spam filtering function. It is now possible to override SpamKeyFunc, which is used by event filtering to detect spam in the events. (#103918, @olagacek) [SIG API Machinery and Instrumentation]

  • Constants/variables from k8s.io for STABLE metrics is now supported (#103654, @coffeepac) [SIG Auth, Instrumentation, Node and Testing]

  • Display Labels when kubectl describe ingress (#103894, @kabab) [SIG CLI]

  • Expose a NewUnstructuredExtractor from apply configurations meta/v1 package that enables extracting objects into unstructured apply configurations (#103564, @kevindelgado) [SIG API Machinery, Cluster Lifecycle, Release and Testing]

  • Introduce a feature gate DisableKubeletCloudCredentialProviders which allows disabling the in-tree kubelet credential providers.

    The DisableKubeletCloudCredentialProviders FeatureGate is currently in Alpha, which means is currently disabled by default. Once the FeatureGate moves to beta, in-tree credential providers will be disabled by default, and users will need to migrate to using external credential providers. (#102507, @ostrain) [SIG Cloud Provider]

  • Introduces a new metric: admission_webhook_request_total with the following labels: name (string) - the webhook name, type (string) - the admission type, operation (string) - the requested verb, code (int) - the HTTP status code, rejected (bool) - whether the request was rejected, namespace (string) - the namespace of the requested resource. (#103162, @rmoriar1) [SIG API Machinery and Instrumentation]

  • Kube-up.sh installs csi-proxy v1.0.1-gke.0 (#104426, @mauriciopoppe) [SIG Cloud Provider, Storage and Windows]

  • Kubeadm: add support for dry running "kubeadm join". The new flag "kubeadm join --dry-run" is similar to the existing flag for "kubeadm init/upgrade" and allows you to see what changes would be applied. (#103027, @Haleygo) [SIG Cluster Lifecycle]

  • Kubernetes is now built with Golang 1.16.7 (#104199, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]

  • The ServiceAccountIssuerDiscovery feature gate is removed. It reached GA in Kubernetes 1.21. (#103685, @mengjiao-liu) [SIG API Machinery and Auth]

  • Updated Cluster Autosaler to version 1.22.0. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.22.0 (#104293, @x13n) [SIG Autoscaling and Cloud Provider]

  • Updates the following images to pick up CVE fixes:

    • debian to v1.9.0
    • debian-iptables to v1.6.6
    • setcap to v2.0.4 (#104142, @mengjiao-liu) [SIG API Machinery, Release and Testing]

Documentation

  • Update description of --audit-log-maxbackup to describe behavior when value = 0 (#103843, @Arkessler) [SIG API Machinery]

Bug or Regression

    1. Changes json representation for a conflicted taint to Key=Effect when a conflicted taint occurs in kubectl taint. (#104011, @manugupt1) [SIG CLI]
  • A new server run option 'shutdown-send-retry-after' has been introduced. If true the HTTP Server will continue listening until all non longrunning request(s) in flight have been drained, during this window all incoming requests will be rejected with a status code 429 and a 'Retry-After' response header. (#101257, @tkashem) [SIG API Machinery]
  • Adds Kubernetes Events to the Kubelet Graceful Shutdown feature (#101081, @rphillips) [SIG Node]
  • CA, certificate and key bundles for the generic-apiserver based servers will be reloaded immediately after the files are changed. (#104102, @tnqn) [SIG API Machinery and Testing]
  • Fix kube-apiserver metric reporting for the deprecated watch path of /api//watch/... (#104161, @wojtek-t) [SIG API Machinery and Instrumentation]
  • Fix: skip case sensitivity when checking Azure NSG rules (#104384, @feiskyer) [SIG Cloud Provider]
  • Fixed an issue which didn't append OS's environment variables with the one provided in Credential Provider Config file, which may lead to failed execution of external credential provider binary. See kubernetes#102750 (#103231, @n4j) [SIG Auth and Node]
  • Fixed architecture within manifest for non amd64 etcd images. (#104116, @saschagrunert) [SIG API Machinery]
  • Fixed bug where kubectl would emit duplicate warning messages for flag names that contain an underscore and recommend using a nonexistent flag in some cases (#103852, @brianpursley) [SIG CLI and Cluster Lifecycle]
  • Graceful node shutdown, allow the actual inhibit delay to be greater than the expected inhibit delay (#103137, @wzshiming) [SIG Node]
  • Kube-apiserver: Avoids unnecessary repeated calls to admission webhooks that reject an update or delete request. (#104182, @liggitt) [SIG API Machinery]
  • Kube-proxy: delete stale conntrack UDP entries for loadbalancer ingress IP. (#104009, @aojea) [SIG Network]
  • Kubeadm: When adding an etcd peer to an existing cluster, if an error is returned indicating the peer has already been added, this is accepted and a ListMembers call is used instead to return the existing cluster. This helps diminish the exponential backoff when the first AddMember call times out, while still retaining a similar performance when the peer had already been added from a previous call. (#104134, @ihgann) [SIG Cluster Lifecycle]
  • Pass additional flags to subpath mount to avoid flakes in certain conditions (#104253, @mauriciopoppe) [SIG Storage]
  • Update Go used to build migrate script in etcd image to v1.16.7 (#104301, @serathius) [SIG API Machinery and Release]

Other (Cleanup or Flake)

  • Deprecate apiserver_longrunning_gauge and apiserver_register_watchers in 1.23.0 (#103793, @yan-lgtm) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
  • Kube-apiserver: sets an upper-bound on the lifetime of idle keep-alive connections and time to read the headers of incoming requests (#103958, @liggitt) [SIG API Machinery and Node]
  • Kubeadm: external etcd endpoints passed in the ClusterConfiguration that have Unicode characters are no longer IDNA encoded (converted to Punycode). They are now just URL encoded as per Go's implementation of RFC-3986, have duplicate "/" removed from the URL paths, and passed like that directly to the kube-apiserver --etcd-servers flag. If you have etcd endpoints that have Unicode characters, it is advisable to encode them in advance with tooling that is fully IDNA compliant. If you don't do that, the Go standard library (used in k8s and etcd) would do it for you when making requests to the endpoints. (#103801, @gkarthiks) [SIG Cluster Lifecycle]
  • Kubeadm: update references to legacy artifacts locations, the 'ci-cross' prefix has been removed from the version match as it does not exist in the new 'gs://k8s-release-dev' bucket (#103813, @SataQiu) [SIG Cluster Lifecycle]
  • Migratecmd/kube-proxy/app logs to structured logging (#98913, @yxxhero) [SIG Network]
  • Surface warning when users don't set propagationPolicy for jobs while deleting (#104080, @ravisantoshgudimetla) [SIG Apps]
  • The AllowInsecureBackendProxy feature gate is removed. It reached GA in Kubernetes 1.21. (#103796, @mengjiao-liu) [SIG API Machinery]
  • The StartupProbe feature gate that is GA since v1.20 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#104168, @ialidzhikov) [SIG Node]
  • The apiserver exposes 4 new metrics that allow to track the status of the Service CIDRs allocations: - current number of available IPs per Service CIDR - current number of used IPs per Service CIDR - total number of allocation per Service CIDR - total number of allocation errors per ServiceCIDR (#104119, @aojea) [SIG Apps, Instrumentation and Network]
  • The flag --deployment-controller-sync-period has no effect now, deprecate it and will be removed in v1.24. (#103538, @Pingan2017) [SIG Apps]
  • Troubleshooting: informers log handlers that take more than 100 milliseconds to process an object if the DeltaFIFO queue starts to grow beyond 10 elements. (#103917, @aojea) [SIG API Machinery]
  • Update cri-tools dependency to v1.22.0 (#104430, @saschagrunert) [SIG Cloud Provider and Node]
  • gcr.io/kubernetes-e2e-test-images will no longer be used in E2E / CI testing, k8s.gcr.io/e2e-test-images will be used instead. (#103724, @claudiubelu) [SIG API Machinery and Testing]

Dependencies

Added

Changed

Removed