Simplify setup of Docker-based machines

[jemrobinson]

💻 System information

• Data Safe Haven version: 2.0.0-beta

🚫 Problematic behaviour

We currently run full Ubuntu VMs to run applications in Docker containers. We should consider alternatives

🚂 Possible workarounds, remediations or solutions

CoreOS

There are examples of how to configure machines that only run Docker images using CoreOS on Azure. This would reduce the attack surface of these machines compared to an Ubuntu VM as they have less functionality.

Azure container instances

Azure container instances have been tested and found to work well in the v4prototype migration testing.

[JimMadge]

CoreOS is now end of life. It has been (more or less) replaced by Fedora CoreOS after being bought by Red Hat.

Unfortunately, there isn't a Fedora CoreOS image in the Azure market place yet. Although they do provide an image that could be uploaded to Azure.

[darenasc]

RancherOS could be an option as a distribution for containers.

[JimMadge]

@darenasc Have you had any experience with RancherOS? It's one I've seen as an option on cloud platforms but I've not heard much about it or anyone using it so I've always been (probably unfairly) suspicious of it.

[darenasc]

Sorry, I haven't used RancherOS (but I'd like to) I've only seen it on workshops and talks about it where it seems to be good for monitoring the containers as part of the SO. In Azure I have used its Container Instances (ACI) service to deploy ML models, and it is simple to deploy and monitor images and containers, there is also Azure Kubernetes Services (AKS) but I haven't use it for projects.

[JimMadge]

Exploring Fedora CoreOS a little more,

• It is possible and not too difficult to automate getting an image on Azure. However, unless there is a way to download directly to a storage account it requires using your local machine as a middle-man for the image. The process is,
  • Download image from Fedora
  • Put in blob storage
  • Build Azure image object from image in blob storage
• Configuration is simple, but adds one dependency (fcct)
  • Write a human-readable YAML configuration
  • Convert to a JSON format using fcct
• Creating a VM is straightforward, you pass the JSON configuration file as custom-data
• Operations on the CoreOS machine may be executed as systemd units.
• The preferred container manager is podman. This should be more secure than docker as it can be run as a non-privileged user, but this does mean there are some subtle difference to docker.

[JimMadge]

HackMD running on Fedora CoreOS in Azure

Configuration file

variant: fcos

version: 1.0.0

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa AAAA....

systemd:
  units:
    - name: postgres.service
      enabled: true
      contents: |
        [Unit]
        Description=Deploy a postgres container
        After=network-online.target
        Wants=network-online.target

        [Service]
        TimeoutStartSec=0
        ExecStartPre=-/bin/podman kill database
        ExecStartPre=-/bin/podman rm database
        ExecStartPre=/bin/podman pull postgres:11.6-alpine
        ExecStart=/bin/podman run --name database --pod new:hackmdpod -p 80:3000 -e POSTGRES_USER=hackmd -e POSTGRES_PASSWORD=change_password -e POSTGRES_DB=hackmd postgres:11.6-alpine

        [Install]
        WantedBy=multi-user.target
    - name: hackmd.service
      enabled: true
      contents: |
        [Unit]
        Description=Deploy a HackMD container
        After=network-online.target
        Wants=network-online.target
        Requires=postgres.service
        After=postgres.service

        [Service]
        TimeoutStartSec=0
        ExecStartPre=-/bin/podman kill hackmd
        ExecStartPre=-/bin/podman rm hackmd
        ExecStartPre=/bin/podman pull hackmdio/hackmd:2.1.0
        ExecStart=/bin/podman run --name hackmd --pod hackmdpod -e CMD_DB_URL=postgres://hackmd:change_password@localhost/hackmd -e CMD_USECDN=false hackmdio/hackmd:2.1.0

        [Install]
        WantedBy=multi-user.target

Notice that I use a 'pod' to allow the two containers to be connected via localhost. Being unable to use a docker (style) network is one of the restrictions of rootless containers.

[JimMadge]

Some podman notes,

• Can run containers as a normal user. If you, for example, mounted /etc/passwd into your container you would still not be able to edit it.
• Cannot use docker networks in rootless containers, but using pods seems like a reasonable alternative.
• There is currently no dns when using pods, unlike docker where you can refer to different containers by their name.
• Does not support docker-compose by design. However, you can use kubernetes project files podman play kube example.yaml and even export an existing pod to the kubernetes format.

[JimMadge]

A Rough plan for using Fedora CoreOS as a drop in replacement for the GitLab and HackMD servers,

1. Automate deploying Fedora CoreOS image to the image gallery.
2. Replace cloud-init files with CoreOS configuration files (taking advantage of the existing "expanding" in Setup_SRE_WebApp_Servers.ps1 to adjust configuration.
3. Add step to Setup_SRE_WebApp_Servers.ps1 to run fcct, converting YAML configurations to ignition files
4. Pass ignition files as custom data for VM creation to Deploy-ArmTemplate (replacing the existing HackMD_Cloud_Init and GitLab_Cloud_Init arguments)

[martintoreilly]

@JimMadge Could you post a quick summary of the benefits of making this switch? From @jemrobinson's note in the main description, it looks like the goal is improved security?

[martintoreilly]

If we do this, I'd propose scheduling for just before or after moving the HackMD / GitLab containers' file storage volume to a network file share (I think the GitLab volume is being moved as part of the code ingress review PR (#611). Also worth considering whether we want to consolidate these containerised services to a single VM to support scaling down to a cheaper base cost.

[JimMadge]

@martintoreilly Fedora CoreOS is designed for applications like this, from the documentation "Fedora CoreOS is an automatically updating, minimal, monolithic, container-focused operating system, designed for clusters but also operable standalone, optimized for Kubernetes but also great without it."

I think there are advantages in a number of dimensions,

• Security
  • The OS is minimal and focused around running containerised applications. It comes with just enough software to do this
  • Large parts of the installation are read only and immutable
  • Hardened with SELinux out of the box
  • Comes with, and prefers podman over docker. podman allows you to run containers as a normal user. If, for instance, you manage to mount data from the host inside the container, and you have access to root inside the container, you will still not have elevated privileges over the data mounted from the host.
• The configuration is simple, particularly as the OS comes with support for containers out of the box
• Automatic updates with support for rollbacks

I feel the main drawback at the moment is that as it is a fairly new project the documentation/examples are a bit sparse and there is no official image on Azure at this time.