Skip to content

Release 2021-03-24
Compare
Choose a tag to compare
@yanancai yanancai released this 26 Mar 00:11
· 2 commits to master since this release
2021-03-24
4393e92
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

This release applies for both HDInsight 3.6 and HDInsight 4.0. HDInsight release is made available to all regions over several days. The release date here indicates the first region release date. If you don't see below changes, wait for the release being live in your region in several days.

New features

Spark 3.0 preview

HDInsight added Spark 3.0.0 support to HDInsight 4.0 as a Preview feature.

Kafka 2.4 preview

HDInsight added Kafka 2.4.1 support to HDInsight 4.0 as a Preview feature.

Moving to Azure virtual machine scale sets

HDInsight now uses Azure virtual machines to provision the cluster. The service is gradually migrating to Azure virtual machine scale sets. The entire process may take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

Eav4-series support

HDInsight added Eav4-series support in this release. Learn more about Dav4-series here. The series has been made available in below regions:

  • AUSTRALIA EAST
  • BRAZIL SOUTH
  • CENTRAL US
  • EAST ASIA
  • EAST US
  • JAPAN EAST
  • SOUTHEAST ASIA
  • UK SOUTH
  • WEST EUROPE
  • WEST US 2

Deprecation

No deprecation in this release.

Behavior changes

Default cluster version is changed to 4.0

The default version of HDInsight cluster is changed from 3.6 to 4.0. For more information about available versions, see available versions. Learn more about what is new in HDInsight 4.0.

Default cluster VM sizes are changed to Ev3-series

Default cluster VM sizes are changed from D-series to Ev3-series. This change applies to head nodes and worker nodes. To avoid this change impacting your tested workflows, specify the VM sizes that you want to use in the ARM template.

Network interface resource not visible for clusters running on Azure virtual machine scale sets

HDInsight is gradually migrating to Azure virtual machine scale sets. Network interfaces for virtual machines are no longer visible to customers for clusters that use Azure virtual machine scale sets.

Upcoming changes

The following changes will happen in upcoming releases.

OS version upgrade

HDInsight will be upgrading OS version from Ubuntu 16.04 to 18.04. The upgrade will complete before April 2021.

HDInsight 3.6 offers Basic support on July 2021

Starting July 2021, Microsoft will offer Basic support for certain HDInsight 3.6 cluster types. The Basic support plan will be available until 3 April 2022. You'll automatically be enrolled in Basic support starting July 2021. No action is required by you to opt in. See our documentation for which cluster types are included under Basic support.

We don't recommend building any new solutions on HDInsight 3.6, freeze changes on existing 3.6 environments.

We recommend that you migrate your clusters to HDInsight 4.0. Learn more about what's new in HDInsight 4.0.

Bug fixes

HDInsight continues to make cluster reliability and performance improvements.

Component version change

Added support for Spark 3.0.0 and Kafka 2.4.1 as Preview.
You can find the current component versions for HDInsight 4.0 and HDInsight 3.6 in this doc.

Assets 2
Loading