Skip to content

Latest commit

 

History

History
131 lines (115 loc) · 4.82 KB

README.md

File metadata and controls

131 lines (115 loc) · 4.82 KB
Raw

NDSLabs GlusterFS Cluster Filesystem Support

Tooling and support for GlusterFS servers and clients. ##Supports:

  • container as client - mount in container
  • container as server - serve from container
  • host as client - mount in host via container
  • host as server - serve from host via container

Commands

  • usage - Consult this prior to use for instructions for you image - default command
docker run --rm -it ndslabs/gluster usage
  • server - Serves a Consult usage or see below
  • client - Consult usage or see below
  • Author/Maintainer: raila@illinois.edu --cap-add SYS_ADMIN --device /dev/fuse

Prerequisites

  • A cluster of system with mountable storage - GlusterFS filesystem nodes
  • A cluster of client systems that will mount GlusterFS volumes that the storage cluster serves
  • A basic understanding of GlusterFS

Conventions

  • LVM volume group per "pool" using one physical volume per volume group at provisioning
  • Multiple LV's can be cut from the VG for bricks in the pool - i.e. multiple independent cluster volumes from the same pool
  • LV's formatted as xfs -i size=512 - to support attributes in a single inode
  • LVM VG names match the cluster volume "name" identifying their "pool" on each server,
  • Brick local LV mounts on the servers are under /var/glfs//brick<#> - Supports known cluster kernel OS's
  • Each brick has a top-level brick directory containing the data, from Gluster conventions
    • WARNING: writing directly to a brick will corrupt the volume
  • Example node with 2 storage mount points and 2 pools:
Disk devices: /dev/vdb1 /dev/vdb1
Volume Groups:  HA scratch
PV assignments: vdb1->HA, vdb2->scratch
LVs and mounts:  /dev/mapper/HA-brick0:/var/glfs/HA/brick0  /dev/mapper/scratch-brick0:/var/glfs/scratch/brick0

Host storage setup - within the chroot in container

  • initialize PVs, create VG and LV per-pool, format the bricks, and mount
pvcreate /dev/vdb1
pvcreate /dev/vdb2
vgcreate HA /dev/vdb1
vgcreate scratch /dev/vdb2
lvcreate -l 100%VG HA -n brick0
lvcreate -l 100%VG scratch -n brick0
mkfs.xfs -i size=512 /dev/mapper/HA-brick0
mkfs.xfs -i size=512 /dev/mapper/scratch-brick0
echo "/dev/mapper/HA-brick0 /var/glfs/HA/brick0 xfs defaults 1 2" >> /etc/fstab 
echo "/dev/mapper/scratch-brick0 /var/glfs/scratch/brick0 xfs defaults 1 2" >> /etc/fstab 
mkdir /var/glfs/HA/brick0/brick
mkdir /var/glfs/scatch/brick0/brick

GFS setup

Start the gluster server container and exec into it

docker run --restart=always --name=glusterfs --net=host --privileged -v /:/hostroot -v /var/glfs:/var/glfs -d ndslabs/gluster
docker exec -it glusterfs bash
# chroot /hostroot

Initialize the cluster FS servers

gluster peer probe 172.16.1.161
gluster peer probe 172.16.1.162 
gluster peer probe 172.16.1.163 
  ...

Create the Gluster volumes per pool

  • In this example, dvol1 is a striped volume and rvol1 is a replicated volume
gluster volume create scratch transport tcp 172.16.1.162:/var/glfs/scratch/brick0 172.16.1.163:/var/glfs/scratch/brick0
gluster volume create rvol1 replica 2 transport tcp 172.16.1.162:/var/glfs/HA/brick0 172.16.1.163:/var/glfs/HA/brick0
gluster volume start HA
gluster volume start scratch
cluster volume list

Client mounts

mount -t glusterfs 172.16.1.163:/scratch /scratch

Enabling per-directory quotas

gluster vol quota HA enable
gluster volume quota HA limit-usage /qtest 5GB
gluster volume quota HA list
    root@gs1:/run/gluster/dvol1/qtest# gluster volume quota dvol1 list
    Path                   Hard-limit Soft-limit   Used  Available
    --------------------------------------------------------------------------------
    /qtest                                     5.0GB       80%       5.0GB  0Bytes
  • Test from client
root@gscli:/var/nds/dvol1/qtest# dd if=/dev/urandom of=random.img count=1024 bs=10M
dd: error writing 'random.img': Disk quota exceeded
dd: closing output file 'random.img': Disk quota exceeded

Mounting inside a container with the glusterfs client:

docker run --privileged -it ndslabs/gluster bash
mkdir -p /var/nds/dvol1
mkdir -p /dev/nds/dvol2
mount -t glusterfs 172.16.1.163:/scratch /scratch

Mounting in-host via container

docker run -v /:/hostroot --net=host --privileged -it ndslabs/gluster bash
mkdir -p /hostroot/scratch
mount -t glusterfs 172.16.1.163:/scratch /hostroot/scratch

Gluster client for Kubernetes cluster filesystem support

Temporary pending documentation update:

SC run: docker run --net=host --pid=host --privileged -v /dev:/dev -v /var:/var -v /run:/run -v /:/media/host -it ndslabs/gluster bash