You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
Kube-apiserver: the unused '--master-service-namespace' flag is deprecated and will be removed in v1.27. (#112797, @SataQiu) [SIG API Machinery]
API Change
Add kubernetes_feature_enabled metric series to track whether each active feature gate is enabled. (#112690, @logicalhan) [SIG API Machinery, Architecture, Cluster Lifecycle, Instrumentation, Network, Node and Scheduling]
Introduce v1beta3 for Priority and Fairness with the following changes to the API spec:
rename 'assuredConcurrencyShares' (located under spec.limited') to 'nominalConcurrencyShares'
apply strategic merge patch annotations to 'Conditions' of flowschemas and prioritylevelconfigurations (#112306, @tkashem) [SIG API Machinery and Testing]
Legacy klog flags are no longer available. Only -v and -vmodule are still supported. (#112120, @pohly) [SIG Architecture, CLI, Instrumentation, Node and Testing]
The feature gates ServiceLoadBalancerClass and ServiceLBNodePortControl have been removed. These feature gates were enabled (and locked) since v1.24. (#112577, @andrewsykim) [SIG Apps]
Feature
A new --disable-compression flag has been added to kubectl (default = false). When true, it opts out of response compression for all requests to the apiserver. This can help improve list call latencies significantly when client-server network bandwidth is ample (>30MB/s) or if the server is CPU-constrained. (#112580, @shyamjvs) [SIG CLI and Testing]
A new pod_status_sync_duration_seconds histogram is reported at alpha metrics stability that estimates how long the Kubelet takes to write a pod status change once it is detected. (#107896, @smarterclayton) [SIG Apps, Architecture, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]
Added a new feature gate CELValidatingAdmission to enable expression validation for Admission Control. (#112792, @cici37) [SIG API Machinery]
Added validation for the --container-runtime-endpoint flag of kubelet to be non-empty. (#112542, @astraw99) [SIG Node]
Expose health check SLI metrics on "metrics/slis" for apiserver (#112741, @logicalhan) [SIG API Machinery, Architecture, Auth and Instrumentation]
Kubeadm: sub-phases are now able to support the dry-run mode, e.g. kubeadm reset phase cleanup-node --dry-run (#112945, @chendave) [SIG Cluster Lifecycle]
Kubeadm: support image repository format validation (#112732, @SataQiu) [SIG Cluster Lifecycle]
Kubernetes is now built with Go 1.19.2 (#112900, @xmudrii) [SIG Release and Testing]
Switch kubectl to use github.com/russross/blackfriday/v2 (#112731, @pacoxu) [SIG CLI]
registered_metric_total now reports the number of metrics broken down by stability level and deprecated version (#112907, @logicalhan) [SIG Architecture and Instrumentation]
Bug or Regression
Consider only plugin directory and not entire kubelet root when cleaning up mounts (#112607, @mattcary) [SIG Storage]
Fix that pods running on nodes tainted with NoExecute continue to run when the PodDisruptionConditions feature gate is enabled (#112518, @mimowo) [SIG Apps and Auth]
Fixes an issue in winkernel proxier that causes proxy rules to leak anytime service backends are modified. (#112837, @daschott) [SIG Network and Windows]
Kube-apiserver: redirects from backend API servers are no longer followed when checking availability with requests to /apis/$group/$version (#112772, @liggitt) [SIG API Machinery and Testing]
Kubeadm: fix a bug when performing validation on ClusterConfiguration networking fields (#112751, @SataQiu) [SIG Cluster Lifecycle]
Kubelet now cleans up the Node's cloud node IP annotation correctly if you
stop using --node-ip. (In particular, this fixes the problem where people who
were unnecessarily using --node-ip with an external cloud provider in 1.23,
and then running into problems with 1.24, could not fix the problem by just
removing the unnecessary --node-ip from the kubelet arguments, because
that wouldn't remove the annotation that caused the problems.) (#112184, @danwinship) [SIG Network and Node]
Kubelet: Fix log spam from kubelet_getters.go "Path does not exist" (#112650, @rphillips) [SIG Node]
Kubelet: when there are multi option lines in /etc/resolv.conf, merge all options into one line in a pod with the Default DNS policy. (#112414, @pacoxu) [SIG Network and Node]
The pod admission error message was improved for usability. (#112644, @vitorfhc) [SIG Node]
Other (Cleanup or Flake)
Adds a kubernetes_feature_enabled metric which will tell you if a feature is enabled. (#112652, @logicalhan) [SIG Architecture and Instrumentation]
Introduce ComponentSLIs alpha feature-gate for component SLIs metrics endpoint. (#112884, @logicalhan) [SIG API Machinery]
Lock ServerSideApply feature gate to true with the feature already being GA. (#112748, @wojtek-t) [SIG API Machinery, Apps, Instrumentation and Testing]
PodOverhead feature gate was removed as the feature is in GA since 1.24 (#112579, @SergeyKanzhelev) [SIG Node and Scheduling]
Reworded log message upon image garbage collection failure to be more clear. (#112631, @tzneal) [SIG Node]
The IndexedJob and SuspendJob feature gates that graduated to GA in 1.24 and were unconditionally enabled have been removed in v1.26 (#112589, @SataQiu) [SIG Apps]
The test/e2e/framework was refactored so that the core framework is smaller. Optional functionality like resource monitoring, log size monitoring, metrics gathering and debug information dumping must be imported by specific e2e test suites. Init packages are provided which can be imported to re-enable the functionality that traditionally was in the core framework. If you have code that no longer compiles because of this PR, you can use the script from a commit message to update that code. (#112043, @pohly) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling, Storage, Testing and Windows]
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
Add auth API to get self subject attributes (new selfsubjectreviews API is added).
The corresponding command for kubctl is provided - kubectl auth whoami. (#111333, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]
Clarified the CFS quota as 100ms in the code comments and set the minimum cpuCFSQuotaPeriod to 1ms to match Linux kernel expectations. (#112123, @paskal) [SIG API Machinery and Node]
Component-base: make the validation logic about LeaderElectionConfiguration consistent between component-base and client-go (#111758, @SataQiu) [SIG API Machinery and Scheduling]
Fixes spurious field is immutable errors validating updates to Event API objects via the events.k8s.io/v1 API (#112183, @liggitt) [SIG Apps]
Protobuf serialization of metav1.MicroTime timestamps (used in Lease and Event API objects) has been corrected to truncate to microsecond precision, to match the documented behavior and JSON/YAML serialization. Any existing persisted data is truncated to microsecond when read from etcd. (#111936, @haoruan) [SIG API Machinery]
Revert regression that prevented client-go latency metrics to be reported with a template URL to avoid label cardinality. (#111752, @aanm) [SIG API Machinery]
[kubelet] Change default cpuCFSQuotaPeriod value with enabled cpuCFSQuotaPeriod flag from 100ms to 100µs to match the Linux CFS and k8s defaults. cpuCFSQuotaPeriod of 100ms now requires customCPUCFSQuotaPeriod flag to be set to work. (#111520, @paskal) [SIG API Machinery and Node]
Feature
A new "DisableCompression" field (default = false) has been added to kubeconfig under cluster info. When set to true, clients using the kubeconfig opt out of response compression for all requests to the apiserver. This can help improve list call latencies significantly when client-server network bandwidth is ample (>30MB/s) or if the server is CPU-constrained. (#112309, @shyamjvs) [SIG API Machinery and Auth]
API Server tracing root span name for opentelemetry is changed from "KubernetesAPI" to "HTTP GET" (#112545, @dims) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing]
Add new Golang runtime-related metrics to Kubernetes components:
go_gc_cycles_automatic_gc_cycles_total
go_gc_cycles_forced_gc_cycles_total
go_gc_cycles_total_gc_cycles_total
go_gc_heap_allocs_by_size_bytes
go_gc_heap_allocs_bytes_total
go_gc_heap_allocs_objects_total
go_gc_heap_frees_by_size_bytes
go_gc_heap_frees_bytes_total
go_gc_heap_frees_objects_total
go_gc_heap_goal_bytes
go_gc_heap_objects_objects
go_gc_heap_tiny_allocs_objects_total
go_gc_pauses_seconds
go_memory_classes_heap_free_bytes
go_memory_classes_heap_objects_bytes
go_memory_classes_heap_released_bytes
go_memory_classes_heap_stacks_bytes
go_memory_classes_heap_unused_bytes
go_memory_classes_metadata_mcache_free_bytes
go_memory_classes_metadata_mcache_inuse_bytes
go_memory_classes_metadata_mspan_free_bytes
go_memory_classes_metadata_mspan_inuse_bytes
go_memory_classes_metadata_other_bytes
go_memory_classes_os_stacks_bytes
go_memory_classes_other_bytes
go_memory_classes_profiling_buckets_bytes
go_memory_classes_total_bytes
go_sched_goroutines_goroutines
go_sched_latencies_seconds (#111910, @tosi3k) [SIG API Machinery, Architecture, Auth, Cloud Provider and Instrumentation]
CSRDuration feature gate that graduated to GA in 1.24 and was unconditionally enabled has been removed in v1.26 (#112386, @Shubham82) [SIG Auth]
Client-go: SharedInformerFactory supports waiting for goroutines during shutdown (#112200, @pohly) [SIG API Machinery]
Kube-apiserver: gzip compression switched from level 4 to level 1 to improve large list call latencies in exchange for higher network bandwidth usage (10-50% higher). This increases the headroom before very large unpaged list calls exceed request timeout limits. (#112299, @shyamjvs) [SIG API Machinery]
Kubeadm: "show-join-command" has been added as a new separate phase at the end of "kubeadm init". You can skip printing the join information by using "kubeadm init --skip-phases=show-join-command". Executing only this phase on demand will throw an error because the phase needs dependencies such as bootstrap tokens to be pre-populated. (#111512, @SataQiu) [SIG Cluster Lifecycle]
Kubeadm: add the flag "--cleanup-tmp-dir" for "kubeadm reset". It will cleanup the contents of "/etc/kubernetes/tmp". The flag is off by default. (#112172, @chendave) [SIG Cluster Lifecycle]
Kubeadm: try to load CA cert from external CertificateAuthority file when CertificateAuthorityData is empty for existing kubeconfig (#111783, @SataQiu) [SIG Cluster Lifecycle]
Kubernetes is now built with Go 1.19.1 (#112287, @palnabarun) [SIG Release and Testing]
Scheduler now retries updating a pod's status on ServiceUnavailable and InternalError errors, in addition to net ConnectionRefused error. (#111809, @Huang-Wei) [SIG Scheduling]
The goroutines metric is newly added in the scheduler.
It replaces scheduler_goroutines metric and it counts the number of goroutine in more places than scheduler_goroutine does. (#112003, @sanposhiho) [SIG Instrumentation and Scheduling]
Documentation
Clarified the default CFS quota period as being 100µs and not 100ms. (#111554, @paskal) [SIG Node]
Bug or Regression
Adds back in unused flags on kubectl run command, which did not go through the required deprecation period before being removed. (#112243, @brianpursley) [SIG CLI]
Allow Label section in vsphere e2e cloudprovider configuration (#112427, @gnufied) [SIG Storage and Testing]
Apiserver /healthz/etcd endpoint rate limits the number of forwarded health check requests to the etcd backends, answering with the last known state if the rate limit is exceeded. The rate limit is based on 1/2 of the timeout configured, with no burst allowed. (#112046, @aojea) [SIG API Machinery]
Avoid propagating hosts' search . into containers' /etc/resolv.conf (#112157, @dghubble) [SIG Network and Node]
Callers using DelegatingAuthenticationOptions can use DisableAnonymous to disable Anonymous authentication. (#112181, @xueqzhan) [SIG API Machinery and Auth]
Change error message when resource is not supported by given patch type in kubectl patch (#112556, @ardaguclu) [SIG CLI]
Correct the calculating error in podTopologySpread plugin to avoid unexpected scheduling results. (#112507, @kerthcet) [SIG Scheduling]
Etcd: Update to v3.5.5 (#112489, @dims) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing]
Fix an ephemeral port exhaustion bug caused by improper connection management that occurred when a large number of objects were handled by kubectl while exec auth was in use. (#112017, @enj) [SIG API Machinery and Auth]
Fix list cost estimation in Priority and Fairness for list requests with metadata.name specified. (#112557, @marseel) [SIG API Machinery]
Fix race condition in GCE between containerized mounter setup in the kubelet and node startup. (#112195, @mattcary) [SIG Cloud Provider and Storage]
Fix relative cpu priority for pods where containers explicitly request zero cpu by giving the lowest priority instead of falling back to the cpu limit to avoid possible cpu starvation of other pods (#108832, @waynepeking348) [SIG Node]
Fixed bug in kubectl rollout history where only the latest revision was displayed when a specific revision was requested and an output format was specified (#111093, @brianpursley) [SIG CLI and Testing]
Fixed bug where dry run message was not printed when running kubectl label with --dry-run flag. (#111571, @brianpursley) [SIG CLI]
For raw block CSI volumes on Kubernetes, kubelet was incorrectly calling CSI NodeStageVolume for every single "map" (i.e. raw block "mount") operation for a volume already attached to the node. This PR ensures it is only called once per volume per node. (#112403, @akankshakumari393) [SIG Storage]
Improves kubectl display of invalid request errors returned by the API server (#112150, @liggitt) [SIG CLI]
Increase the maximum backoff delay of the endpointslice controller to match the expected sequence of delays when syncing Services. (#112353, @dgrisonnet) [SIG Apps and Network]
Kube-apiserver: redirect responses are no longer returned from backends by default. Set --aggregator-reject-forwarding-redirect=false to continue forwarding redirect responses. (#112193, @jindijamie) [SIG API Machinery and Testing]
Kube-apiserver: resolved a regression that treated 304 Not Modified responses from aggregated API servers as internal errors (#112526, @liggitt) [SIG API Machinery]
Kube-apiserver: x-kubernetes-list-type validation is now enforced when updating status of custom resources (#111866, @pacoxu) [SIG API Machinery]
Kube-proxy no longer falls back from ipvs mode to iptables mode if you ask it to do ipvs but the system is not correctly configured. Instead, it will just exit with an error. (#111806, @danwinship) [SIG Network]
Kube-scheduler: add taints filtering logic consistent with TaintToleration plugin for PodTopologySpread plugin (#112357, @SataQiu) [SIG Scheduling and Testing]
Kubeadm will cleanup the stale data on best effort basis. Stale data will be removed when each reset phase are executed, default etcd data directory will be cleanup when the remove-etcd-member phase are executed. (#110972, @chendave) [SIG Cluster Lifecycle]
Kubeadm: allow RSA and ECDSA format keys in preflight check (#112508, @SataQiu) [SIG Cluster Lifecycle]
Kubeadm: when a subcommand is needed but not provided for a kubeadm command, print a help screen instead of showing a short message. (#111277, @chymy) [SIG Cluster Lifecycle]
Log messages and metrics for the watch cache are now keyed by <resource>.<group> instead of go struct type. This means e.g. that *v1.Pod becomes pods. Additionally, resources that come from CustomResourceDefinitions are now displayed as the correct resource and group, instead of *unstructured.Unstructured. (#111807, @ncdc) [SIG API Machinery and Instrumentation]
Move LocalStorageCapacityIsolationFSQuotaMonitoring back to Alpha. (#112076, @rphillips) [SIG Node and Testing]
Pod failed in scheduling due to expected error will be updated with the reason of "SchedulerError"
rather than "Unschedulable" (#111999, @kerthcet) [SIG Scheduling and Testing]
Services of type LoadBalancer create fewer AWS security group rules in most cases (#112267, @sjenning) [SIG Cloud Provider]
The errors in k8s.io/apimachinery/pkg/api/meta gained support for the stdlibs errors.Is matching, including when wrapped (#111808, @alvaroaleman) [SIG API Machinery]
The metrics etcd_request_duration_seconds and etcd_bookmark_counts now differentiate by group resource instead of object type, allowing unique entries per CustomResourceDefinition, instead of grouping them all under *unstructured.Unstructured. (#112042, @ncdc) [SIG API Machinery]
Update the system-validators library to v1.8.0 (#112026, @pacoxu) [SIG Cluster Lifecycle]
Other (Cleanup or Flake)
E2e: tests can now register callbacks with ginkgo.BeforeEach/AfterEach/DeferCleanup directly after creating a framework instance and are guaranteed that their code is called after the framework is initialized and before it gets cleaned up. ginkgo.DeferCleanup replaces f.AddAfterEach and AddCleanupAction which got removed to simplify the framework. (#111998, @pohly) [SIG Storage and Testing]
GlusterFS in-tree storage driver which was deprecated at kubernetes 1.25 release has been removed entirely in 1.26. (#112015, @humblec) [SIG API Machinery, Cloud Provider, Instrumentation, Node, Scalability, Storage and Testing]
Kube scheduler Component Config release version v1beta3 is deprecated in v1.26 and will be removed in v1.29,
also v1beta2 will be removed in v1.28. (#112257, @kerthcet) [SIG Scheduling]
Kube-scheduler: the DefaultPodTopologySpread, NonPreemptingPriority, PodAffinityNamespaceSelector, PreferNominatedNode feature gates that graduated to GA in 1.24 and were unconditionally enabled have been removed in v1.26 (#112567, @SataQiu) [SIG Scheduling]
Kubeadm: remove the toleration for the "node-role.kubernetes.io/master" taint from the CoreDNS deployment of kubeadm. With the 1.25 release of kubeadm the taint "node-role.kubernetes.io/master" is no longer applied to control plane nodes and the toleration for it can be removed with the release of 1.26. You can also perform the same toleration removal from your own addon manifests. (#112008, @pacoxu) [SIG Cluster Lifecycle]
Kubeadm: remove the usage of the --container-runtime=remote flag for the kubelet during kubeadm init/join/upgrade. The flag value "remote" has been the only possible value since dockershim was removed from the kubelet. (#112000, @pacoxu) [SIG Cluster Lifecycle]
Scheduler dumper now exposes a summary to indicate the number of pending pods in each internal queue. (#111726, @Huang-Wei) [SIG Scheduling and Testing]
The IndexedJob and SuspendJob feature gates that graduated to GA in 1.24 and were unconditionally enabled have been removed in v1.26 (#112589, @SataQiu) [SIG Apps]
The in-tree cloud provider for OpenStack (and the cinder volume provider) has now been removed. Please use the external cloud provider and csi driver from https://github.com/kubernetes/cloud-provider-openstack instead. (#67782, @dims) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release, Scheduling, Storage and Testing]