1.3.0

Features

• Allow deployment into a new or existing virtual network. When deploying into an existing virtual network, the resource group in which the virtual network resides must be in the same location as the target location for the deployment. Master, data and client nodes, as well as a kibana instance are all dynamically assigned IP addresses. The internal load balancer requires the user to specify an available IP address in the subnet, as this is used to connect to the cluster by both Kibana and the external load balancer (if one is selected).
• Set the default url in Sense to the IP address of the internal load balancer when plugins and kibana are installed
• Allow specifying a hostname prefix with which to prefix all nodes within the cluster. This is primarily to allow a deployment into an existing virtual network that contains an already deployed cluster; since master nodes are configured by hostname in the elasticsearch configuration, hostnames need to be different across different clusters.
• Send basic user information details to Elastic to understand who is using the template. These are required in the UI definition but are not in the deployment template. Details will only be sent when there is an email address.
• Conditionally show master node VM size configuration only when data nodes are not master eligible
• Conditionally show client node VM size configuration only when client node count is at least 1.
• Conditionally show kibana node VM size configuration only when install kibana is selected
• Validate admin username in the UI definition to prevent specifying a username that is invalid for a Ubuntu Administrator account.
• Update Microsoft.Storage components to use the 2016-01-01 api version
• Update Microsoft.Network components to use the 2016-03-30 api version

1.2.0

improvements

• data-node-* resources are now generated as part of npm run build and generate 32 and 40 disk variants as well
• Now we support also the vms that can have a maximum of 32 or 40 disks attached to them.
• dedicated empty variants of each role (client, kibana, etc) that is optional.
• Inputs and outputs of these need to be the same as their counterparts, this is now tested as part of npm run build
• add elasticsearch 2.3.3 support
• we inversed how osProfiles get created and attached to vms, this to hopefully catch quota errors in the validation phase of deployment (our tests are showing these in validation phase now)

testing

• npm test now tests the currently pushed template using the arm scenarios defined under build/arm-tests using the azure cli tooling.
• npm run deploy-all will do an actual deploy of all valid and opt in scenarios defined under build/arm-tests do several sanity checks on the deployed elasticsearch cluster and kibana.
• npm run release will do a build and deploy all and create a test zip with all the logs and results of the deploys that we can attach to the test results when we publish this solution to the microsoft publishers portal.
• npm run azure-cleanup will clean any resource groups starting with test- that might be lingering, the other targets should clean up after themselfs though.

limits

• The hard limit in the template of max 60 vms is now gone, this was their due to a binpackmap construct that created an even distribution over the storage accounts depending on the number of nodes chosen. Thanks to a nice trick by @trentmswanson we no longer need these map and we can deploy bigger clusters without having the increase the size of this baked in map.
• Even with the following limit lifted an availability set can only have 100 vms, to circumvent this the template will now create multiple av sets for every 100 vms. When choosing over 100vms the vms will be round robined over the av sets. This allows you to create >100 vm clusters.
• When selecting over 100 data nodes make sure to create <100 client nodes. The reason for this is that loadbalancers can not attach vms from multiple av-sets.
• Make sure to also choose dedicated master nodes if you are deploying clusters >10 nodes.The template will continue to set minimum_master_node appropiately but at some point dedicated master nodes is the way to go.
• If you are deploying from the market place ui there are still limits in place on the dropdown boxes.

1.1.0

Fixes and Features

• allowedSizes patched in ui and arm template through build
• number of nodes in UI is now controlled through allowedValues
• binpack map to distribute storage accounts evenly is now also generated based on number of allowed nodes
• can now pick the vm size for the kibana machine
• sshPublicKey is now a securestring
• all the public bits now get a DNS name (lb-, kb-, jb-*)
• The kibana url, public loadbalancer url and jumbox ssh info are now clearly output on the template.
• normalized some naming conventions of the resources e.g all data node resource start with data-node-
• we now always simply deploy to the resource of the resource group, deploying in a different location then the resource group location feels like an edgecase
• fixed default value for loadbalancer in UI
• use newer api version for storage accounts
• remove shield empty defaults
• Kibana size picker in UI used a label that mentioned client nodes
• recommended and default datanodes are now controlled through the build process
• Switched to recommending DS_v2 instances
• Add support for way more VM Sizes, the only ones we do not support now are:
  • GS, more compute heavy left out for now
  • D14, D5_v2, D14_v2, DS14, DS5_v2, DS14_v2. Since these allow 32 disks attached and our templates can only do 16 at the moment.
  • D15_v2, DS15_v2 can support 40 attached disks but our template only goes to 16

Provisioning reseliency

• vm-disk-utils is now loaded from our repository.
• Java installation failed quite often do to checksum failures post download, built in retries including a final nuclear clear cache and install again on the final 3rd retry.
• Sometimes mdadm failed to be installed completely, similary we now retry. Have not seen failures since.
• mdadm creation of the raid0 disk suffered from race condition and sometimes the attached disks were in use or "busy". Added a fix found here. Sent a PR to azure quickstart with these fixes too here Azure/azure-quickstart-templates#1978
• Quite often waagent crashes due to timeouts [1] and then eventhough the elasticsearch install scripts runs to completion noone is letting azure know causing the deployment to get stuck and eventually timeout (1h+). We now forcefully start this service (noop if already running) throughout and at the end of the deployment. This has helped tremendously.

Testing

Started with casperjs based testing of our UI (development version), it can login to azure and verify we did not break the UI from loading. In the future I would like to be able to write UI scenarios and it will fill the controls and assert the output on the console.

1.0.4

• Seperate loadbalancer to its own templates choosing internal only deploys internal loadbalancer external now deploys two loadbalancers (as a stop gap measure whilst kibana does not support seeding multiple urls/sniffing).
• Allow DS14 SKU (synching with elasticsearch quickstart)
• Elasticsearch and kibana updates to latest versions
• publicIp resource gets created even if internal loadbalancer is chosen #19
• Azure deployment - plugin:elasticsearch - Request Timeout after 1500ms #14
• Using External Load Balancer causes Kibana to be misconfigured #18

1.0.3

Put defaults back in place

1.0.2

• Limit max cluster size for now on the market place UI while the UI waits for stricter azure limit validations prior to deploying and we all dive into speeding up the bash installation script (#5)
• sync default values (#6)
• Market place's sizeSelector should only list allowed values, this is now done as part of our build (#7)

1.0.1

• Fix selecting Standard A0 and A1 broke the template #3 ty @crunchie84
• fixed the >9 client nodes bug #4 ty @trentmswanson
• Stricter mapping between elasticsearch and kibana version that gets deployed: #2

Initial Release

1.0.0

1.0.0