Skip to content

Release 2021-02-05
Compare
Choose a tag to compare
@yanancai yanancai released this 09 Feb 07:12
· 2 commits to master since this release
2021-02-05
4393e92
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

This release applies for both HDInsight 3.6 and HDInsight 4.0. HDInsight release is made available to all regions over several days. The release date here indicates the first region release date. If you don't see below changes, wait for the release being live in your region in several days.

New features

Dav4-series support

HDInsight added Dav4-series support in this release. Learn more about Dav4-series here.

Kafka REST Proxy GA

Kafka REST Proxy enables you to interact with your Kafka cluster via a REST API over HTTPS. Kafka Rest Proxy is general available starting from this release. Learn more about Kafka REST Proxy here.

Moving to Azure virtual machine scale sets

HDInsight now uses Azure virtual machines to provision the cluster. The service is gradually migrating to Azure virtual machine scale sets. The entire process may take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

Deprecation

Disabled VM sizes

Starting form January 9 2021, HDInsight will block all customers creating clusters using standand_A8, standand_A9, standand_A10 and standand_A11 VM sizes. Existing clusters will run as is. Consider moving to HDInsight 4.0 to avoid potential system/support interruption.

Behavior changes

Default cluster VM size changes to Ev3-series

Default cluster VM sizes will be changed from D-series to Ev3-series. This change applies to head nodes and worker nodes. To avoid this change impacting your tested workflows, specify the VM sizes that you want to use in the ARM template.

Network interface resource not visible for clusters running on Azure virtual machine scale sets

HDInsight is gradually migrating to Azure virtual machine scale sets. Network interfaces for virtual machines are no longer visible to customers for clusters that use Azure virtual machine scale sets.

Breaking change for .NET for Apache Spark 1.0.0

HDInsight introduces the first major official release of .NET for Apache Spark in the next release. It provides DataFrame API completeness for Spark 2.4.x and Spark 3.0.x along with other features. There will be breaking changes for this major version, refer to this migration guide to understand steps needed to update your code and pipelines. Learn more here.

Upcoming changes

The following changes will happen in upcoming releases.

Default cluster version will be changed to 4.0

Starting February 2021, the default version of HDInsight cluster will be changed from 3.6 to 4.0. For more information about available versions, see available versions. Learn more about what is new in HDInsight 4.0.

OS version upgrade

HDInsight is upgrading OS version from Ubuntu 16.04 to 18.04. The upgrade will complete before April 2021.

HDInsight 3.6 end of support on June 30 2021

HDInsight 3.6 will be end of support. Starting form June 30 2021, customers can't create new HDInsight 3.6 clusters. Existing clusters will run as is without the support from Microsoft. Consider moving to HDInsight 4.0 to avoid potential system/support interruption.

Bug fixes

HDInsight continues to make cluster reliability and performance improvements.

Component version change

No component version change for this release. You can find the current component versions for HDInsight 4.0 and HDInsight 3.6 in this doc.

Assets 2
Loading