Create new EFS volume shared between multiple EC2 instances

This page is documenting the steps taken to create and mount EFS on two or more EC2 instances. In our case we are creating a shared EFS between the grading and the student servers, and we are allowing the creation of user folders in the grading server when a user logs in.

Creating and mounting EFS

• Log in to AWS and go to services-EFS
• Create a new EFS and select the VPC and mount targets (if it will be used across multiple subnets/availability zones)
• Install EFS mount helper using sudo apt-get install nfs-common (ubuntu) sudo yum install -y nfs-utils
• Create a new EFS folder in the home directory using sudo mkdir efs
• Mount your file system using sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-f0e16a1d.efs.ca-central-1.amazonaws.com:/ efs
• Do the last 3 steps for both the grading and student servers.

Modifying the jupyterhub_config.py and zfs_homedir.sh files

/srv/jupyterhub/jupyterhub_config.py

import os
import subprocess
def create_fs_hook(spawner):
      username = spawner.user.name
      callysto_user = 'jupyter'
      course = 'dsci100'
      subprocess.check_call(['/srv/jupyterhub/zfs_homedir.sh', course, username, callysto_user])
c.Spawner.pre_spawn_hook = create_fs_hook
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.image = 'ubcstat/r-stat-201-grading:latest'
c.Spawner.default_url = '/lab'
#c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
c.DockerSpawner.remove_containers = True
c.JupyterHub.hub_ip = '172.17.0.1'
c.DockerSpawner.host_ip = '0.0.0.0'
c.DockerSpawner.volumes = {
        '/efs/home/dsci100/{username}': {
            'bind': '/home/jupyter',
            'mode': 'rw'
        }
}

zfs_home_dir.sh

#!/usr/bin/env bash
set -e
set -u
set -o pipefail
ZFSHOME="/efs/home"
ZFSOPTS="-o refquota=1G"
sys_user_exists() {
    local username="$1"
    ret=false
    getent passwd ${username} > /dev/null 2>&1 && ret=true
    if ! ${ret}; then
        echo "username \"${username}\" does not correspond to a system user!" >&2
        echo >&2
        exit 1
    fi
}
main() {
    local username
    local zfshome="/efs/home"
    coursename=${1}
    username=${2}
    owner=${3}
    sys_user_exists $owner
    local homedir="${zfshome}/${coursename}/${username}"
    if [[ -d "${homedir}" ]] ; then
        echo "Homedir already exists, not recreating"
    else
        echo "Creating Homedir: ${homedir}"
        mkdir ${homedir}
        chown -R ${owner}:${owner} ${homedir}
        chmod g+w ${homedir}
    fi
}
main "${@:-}"