Image building strategy

[JimMadge]

Drawing on conversation in the comments of #1020

How should the process of building DSVM images be designed?

As this process is independent of other infrastructure, there is an opportunity to use new tools (Terraform/Ansible) to simply and accelerate the build process (which currently relies on quite a few bespoke bash/Python scripts).

Should we publish DSVM images on the Azure marketplace?

---

VM image development is going on here: https://github.com/alan-turing-institute/bureau

[JimMadge]

Copying some of my comments from the PR,

What are the advantages of that approach over using an Azure image gallery (is that what it is called?) like we currently do?

I think I prefer the idea that an organisation could customise their DSVM image with packages they use frequently.

Looking at the build times, it looks like an important difference between this and Mousehole is the extra Python versions and preloaded Python/R packages. On Mousehole I just have system Python3 (effectively leaving the users to deal with it) and don't install any packages. This leaves 'build' times around 30-45 minutes which feels acceptable as a part of deployment. What happens if we just approve but don't install the packages?

[jemrobinson]

Points in favour of publishing to the Marketplace:

• we can decouple the "Turing Data Science VM" from the Safe Haven (and allow others to use it)
• we can (hopefully) build the image in Azure DevOps or similar

Reasons for pre-installing packages:

• to provide a "batteries included" data science VM, similar to the Microsoft DSVM (which is based on Ubuntu 16.04).

[JimMadge]

The case against the Marketplace that I can see is,

• It will be easier for 'deployers' to make customise their own DSVM image with packages important to them/their organisation

Do we have any user feedback about the batteries included aspect? I feel we should be opinionated about reproducibility and encourage users to always user virtual environments and a requirements.txt or equivalent.

I also get the impression trying to include packages in the base image becomes a bit of nightmare when some AI/ML packages are very picky about the required versions of their dependencies.

I do like the idea of something like a Turing DSVM. If we were to go that way it should probably be a separate project which this (and others) could draw from.

[martintoreilly]

I do like the ideas of a Turing Data Science VM with a good set of packages for data science / AI work all working together. I definitely think this could be something we recommend for researchers at the Turing in general. We could potentially support a containerised version for more cloud-native deployment models and it could also be part of our ongoing discussions with IT around how to best mitigate security concerns while maintaining researcher productivity (thinking more about up to date OS/packages and configuration plus maybe some AV, rather than locking anything down, but maybe also an option to link into an Azure AD for MFA and group membership would be cool).

We actually used the MS DSVM initially for DSGs. We initially moved from it because we needed Ubuntu 18.04 to get the remote desktop working properly I think, but since then many of the DS/AI packages have become out of date.

[JimMadge]

Docs on publishing a VM offering to Azure Marketplace: https://docs.microsoft.com/en-us/azure/marketplace/marketplace-virtual-machines

Requires a commerical marketplace account

[jemrobinson]

• Here's a GitHub action that will build an image and add it to an Image Gallery.
• It looks like it should be possible to publish a Marketplace image from an Image Gallery image but I can't find the documentation explaining how

[JimMadge]

It would be good to use Ansible to automate the whole process https://docs.ansible.com/ansible/latest/scenario_guides/guide_azure.html

You don't need to deploy complicated infrastructure to build an image (just a single VM) so something like Terraform could be overkill. Ansible would be easier to run outside of actions and reduce dependancies.

Using the dynamic inventory should also keep the files/procedure neat.

Unfortunately this means everyone running this would need a service principal, as would the runner.

[JimMadge]

It might be prudent to have an az cli method for deploying/destroying the build vm to avoid the need for a service principal when experimenting.

[jemrobinson]

Also consider using:

• Azure VM Image Builder
• Packer on Azure

[JimMadge]

Options for publishing image ranked in order of preference

1. Azure marketplace
2. GitHub Releases/Packages (NB. this might be expensive)
3. vhd hosted elsewhere (public storage account, https, sha256)

In the case of Azure marketplace, there would be a simple "one click" deployment. For other hosting methods we could publish the VM image. Safe haven scripts can automate uploading the image to Azure so it can be used to deploy VMs.