docker-compose-air-gapper

If you are beginning your journey with Senzing, please start with Senzing Quick Start guides.

You are in the Senzing Garage where projects are "tinkered" on. Although this GitHub repository may help you understand an approach to using Senzing, it's not considered to be "production ready" and is not considered to be part of the Senzing product. Heck, it may not even be appropriate for your application of Senzing!

Synopsis

Create a TGZ bundle of docker images for air-gapped environments based on a docker-compose.yaml file.

Overview

Process:

1. In an internet-connected environment:
   1. "Normalize" a docker-compose.yaml file to instantiate variables.
   2. Scan through the normalized docker-compose.yaml file looking for image: tags.
   3. Create a shell script that does docker pull and docker save for the images. Example: save-images-example.sh
   4. Run shell script to create TGZ file containing docker images with a script to do docker load.
2. In an air-gapped environment:
   1. Uncompress the TGZ file.
   2. Run the script to perform docker load of the docker images.

Contents

1. Preamble
   1. Legend
2. Related artifacts
3. Expectations
4. In an internet-connected environment
   1. Internet-connected prerequisites
   2. Create save-images.sh
   3. Run save-images.sh
5. In an air-gapped environment
   1. Air-gapped prerequisites
   2. Load air-gapped docker repository
6. Develop
7. Advanced
   1. Download docker-compose-air-gapper.py
   2. Create save-images.sh using command-line
   3. Modified docker-compose.yaml file
8. Errors
9. References

Preamble

At Senzing, we strive to create GitHub documentation in a "don't make me think" style. For the most part, instructions are copy and paste. Whenever thinking is needed, it's marked with a "thinking" icon 🤔. Whenever customization is needed, it's marked with a "pencil" icon ✏️. If the instructions are not clear, please let us know by opening a new Documentation issue describing where we can improve. Now on with the show...

Legend

1. 🤔 - A "thinker" icon means that a little extra thinking may be required. Perhaps there are some choices to be made. Perhaps it's an optional step.
2. ✏️ - A "pencil" icon means that the instructions may need modification before performing.
3. ⚠️ - A "warning" icon means that something tricky is happening, so pay attention.

Related artifacts

1. DockerHub

Expectations

• Space: This repository and demonstration require 1-3 GB free disk space per image saved.
• Time: Budget 10 minutes per image saved, depending on CPU and network speeds.
• Background knowledge: This repository assumes a working knowledge of:
  • Docker
  • Docker-compose

In an internet-connected environment

Internet-connected prerequisites

1. Software requirements on the internet-connected (i.e. not the air-gapped) system:
   1. docker
   2. docker-compose

Create save-images.sh

1. ✏️ Identify the directory containing the docker-compose.yaml file. Example:

   export SENZING_DOCKER_COMPOSE_DIRECTORY=~/my-docker-compose

2. 🤔 Optional: Set any needed environment variables. For instance, to specify the latest docker image tags for docker-compose.yaml files in docker-compose-demo environment variables can be set in this manner. Example:

   curl -X GET \
     --output ${SENZING_DOCKER_COMPOSE_DIRECTORY}/docker-versions-latest.sh \
     https://raw.githubusercontent.com/Senzing/knowledge-base/main/lists/docker-versions-latest.sh
   
   source ${SENZING_DOCKER_COMPOSE_DIRECTORY}/docker-versions-latest.sh

3. Use docker-compose config to normalize the docker-compose.yaml file into a new docker-compose-normalized.yaml file.

   Example:

   cd ${SENZING_DOCKER_COMPOSE_DIRECTORY}
   
   docker-compose \
     --file docker-compose.yaml \
     config \
     > docker-compose-normalized.yaml

4. Get the latest version of senzing/docker-compose-air-gapper. Example:

   docker pull senzing/docker-compose-air-gapper:latest

5. Using a senzing/docker-compose-air-gapper docker container, create a save-images.sh file in the SENZING_DOCKER_COMPOSE_DIRECTORY directory. Example:

   docker run \
     --env SENZING_DOCKER_COMPOSE_FILE=/data/docker-compose-normalized.yaml \
     --env SENZING_OUTPUT_FILE=/data/save-images.sh \
     --env SENZING_SUBCOMMAND=create-save-images \
     --interactive \
     --rm \
     --tty \
     --volume ${SENZING_DOCKER_COMPOSE_DIRECTORY}:/data \
     senzing/docker-compose-air-gapper:latest

Run save-images.sh

1. Run save-images.sh Example:

   cd ${SENZING_DOCKER_COMPOSE_DIRECTORY}
   chmod +x save-images.sh
   
   ./save-images.sh

2. After save-images.sh has completed, there will be a new ~/docker-compose-air-gapper-0000000000.tgz file. This is the file to be transferred to the air-gapped system. Example output:

   Done.
       Output file: /home/senzing/docker-compose-air-gapper-0000000000.tgz
       Which is a compressed version of /home/senzing/docker-compose-air-gapper-0000000000

In an air-gapped environment

Air-gapped prerequisites

1. Software requirements on the air-gapped system:
   1. docker
   2. docker-compose
2. The docker-compose-air-gapper-0000000000.tgz needs to be transferred to the air-gapped system.

Load air-gapped docker repository

1. ✏️ Set Environment variables. Identify the location of the tgz file (SENZING_TGZ_FILE) and the location where the files should be extracted (SENZING_OUTPUT_DIRECTORY). Example:

   export SENZING_OUTPUT_DIRECTORY=/tmp
   export SENZING_TGZ_FILE=~/docker-compose-air-gapper-0000000000.tgz

2. Extract docker-compose-air-gapper-0000000000.tgz file into specified directory. Example:

   tar \
     --directory=${SENZING_OUTPUT_DIRECTORY} \
     --extract \
     --file=${SENZING_TGZ_FILE} \
     --verbose

3. Change directory to the location of the extracted files. Example:

   export SENZING_TGZ_FILENAME=${SENZING_TGZ_FILE##*/}
   export SENZING_INPUT_DIRECTORY=${SENZING_OUTPUT_DIRECTORY}/${SENZING_TGZ_FILENAME%%.tgz}
   
   cd ${SENZING_INPUT_DIRECTORY}

4. Run load-images.sh. This step loads the local docker repository with the images extracted from the docker-compose-air-gapper-0000000000.tgz file. Example:

   ./load-images.sh

Develop

The following instructions are used when modifying and building the Docker image.

Prerequisites for development

🤔 The following tasks need to be complete before proceeding. These are "one-time tasks" which may already have been completed.

1. The following software programs need to be installed:
   1. git
   2. make
   3. docker

Clone repository

For more information on environment variables, see Environment Variables.

1. Set these environment variable values:

   export GIT_ACCOUNT=senzing
   export GIT_REPOSITORY=docker-compose-air-gapper
   export GIT_ACCOUNT_DIR=~/${GIT_ACCOUNT}.git
   export GIT_REPOSITORY_DIR="${GIT_ACCOUNT_DIR}/${GIT_REPOSITORY}"

2. Using the environment variables values just set, follow steps in clone-repository to install the Git repository.

Build Docker image

1. Option #1: Using docker command and GitHub.

   sudo docker build \
     --tag senzing/docker-compose-air-gapper \
     https://github.com/senzing-garage/docker-compose-air-gapper.git#main

2. Option #2: Using docker command and local repository.

   cd ${GIT_REPOSITORY_DIR}
   sudo docker build --tag senzing/docker-compose-air-gapper .

3. Option #3: Using make command.

   cd ${GIT_REPOSITORY_DIR}
   sudo make docker-build

Advanced

Download docker-compose-air-gapper.py

1. Have python 3 installed.

   1. Using pip3, install Python requirements in requirements.txt.

2. Get a local copy of docker-compose-air-gapper.py. Example:

   1. ✏️ Specify where to download file. Example:

      export SENZING_DOWNLOAD_FILE=~/docker-compose-air-gapper.py

   2. Download file. Example:

      curl -X GET \
        --output ${SENZING_DOWNLOAD_FILE} \
        https://raw.githubusercontent.com/Senzing/docker-compose-air-gapper/main/docker-compose-air-gapper.py

   3. Make file executable. Example:

      chmod +x ${SENZING_DOWNLOAD_FILE}

Create save-images.sh using command-line

This step creates the save-images.sh shell script using a specified docker-compose.yaml file.

1. ✏️ Identify the file to hold a new shell script. This script will be used to download, save, and compress docker images into a single file. Example:

   export SENZING_SAVE_IMAGE_FILE=~/save-images.sh

2. Create save-images.sh by using docker-compose config to normalize docker-compose.yaml. The normalized output is used by docker-compose-air-gapper.py to create save-images.sh Example:

   cd ${SENZING_DOCKER_COMPOSE_DIRECTORY}
   
   docker-compose --file docker-compose.yaml config \
      | ${SENZING_DOWNLOAD_FILE} create-save-images \
      > ${SENZING_SAVE_IMAGE_FILE}

3. Make save-image.sh executable. Example:

   chmod +x ${SENZING_SAVE_IMAGE_FILE}

Modified docker-compose.yaml file

Since docker-compose-air-gapper.py only looks at a few properties in the docker-compose.yaml file, it's possible to make a skeleton docker-compose.yaml file for the purposes of creating a TGZ file for an air-gapped enviroment.

1. For example, a /tmp/docker-compose-skeleton.yaml file could look like this:

   services:
     x:
       image: senzing/senzing-console:1.0.3
     y:
       image: senzing/stream-loader:1.9.0
     z:
       image: senzing/senzing-api-server:2.7.4

2. Since the docker-compose-skeleton.yaml file is already "normalized" (i.e. there are no variables for substitution), there is no need to run docker-compose config.

3. Using a senzing/docker-compose-air-gapper docker container, create a save-images.sh file in the /tmp directory. Example:

   docker run \
     --env SENZING_DOCKER_COMPOSE_FILE=/data/docker-compose-skeleton.yaml \
     --env SENZING_OUTPUT_FILE=/data/save-images.sh \
     --env SENZING_SUBCOMMAND=create-save-images \
     --interactive \
     --rm \
     --tty \
     --volume /tmp:/data \
     senzing/docker-compose-air-gapper

4. Run save-images.sh. Example:

   chmod +x /tmp/save-images.sh
   /tmp/save-images.sh

Errors

1. See docs/errors.md.

References