Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

86c0t11yc - Extract Minio from monitoring package into an independent package #330

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+245 −124
Open

86c0t11yc - Extract Minio from monitoring package into an independent package #330

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5882931
86c0t11yc - Extract Minio from monitoring package into an independent…
drizzentic Oct 24, 2024
fb27418
cleanup
drizzentic Oct 25, 2024
38290f5
rename package
drizzentic Nov 4, 2024
7d3386b
Expose api port on dev mode
drizzentic Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ packages:
- database-postgres
- reprocess-mediator
- fhir-ig-importer
- datalake

profiles:
- name: cdr-dw
Expand All @@ -47,6 +48,7 @@ profiles:
- kafka-unbundler-consumer
- fhir-ig-importer
- reprocess-mediator
- datalake
envFiles:
- cdr-dw.env

Expand Down
82 changes: 82 additions & 0 deletions datalake/docker-compose.cluster.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
version: "3.9"

services:
minio-01:
deploy:
placement:
constraints:
- "node.labels.name==${MINIO_01_PLACEMENT}"

Comment on lines +4 to +9
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Critical: Incomplete configuration for minio-01 service

The minio-01 service configuration is incomplete and missing essential components that are present in other MinIO services:

  • Image specification
  • Entrypoint and command
  • Environment variables (MINIO_ROOT_USER, MINIO_ROOT_PASSWORD)
  • Healthcheck configuration
  • Volume mounts
  • Hostname
  • Replicas configuration

Add the missing configurations to match other MinIO services:

 minio-01:
+    image: ${MINIO_IMAGE}
+    entrypoint: sh
+    command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
+    environment:
+      MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
+      MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
+    healthcheck:
+      test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
+      interval: 30s
+      timeout: 20s
+      retries: 3
+    hostname: minio-01
+    volumes:
+      - minio-01-data1:/data1
+      - minio-01-data2:/data2
     deploy:
       placement:
         constraints:
           - "node.labels.name==${MINIO_01_PLACEMENT}"
+      replicas: 1
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
minio-01:
deploy:
placement:
constraints:
- "node.labels.name==${MINIO_01_PLACEMENT}"
minio-01:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
hostname: minio-01
volumes:
- minio-01-data1:/data1
- minio-01-data2:/data2
deploy:
placement:
constraints:
- "node.labels.name==${MINIO_01_PLACEMENT}"
replicas: 1

minio-02:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
hostname: minio-02
volumes:
- minio-02-data1:/data1
- minio-02-data2:/data2
deploy:
placement:
constraints:
- "node.labels.name==${MINIO_02_PLACEMENT}"
replicas: 1

Comment on lines +11 to +31
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Security and operational improvements needed for MinIO services

Several important configurations are missing or could be improved:

  1. Security:
    • Credentials are passed as environment variables without secrets management
    • Health check endpoint uses HTTP instead of HTTPS
  2. Operational:
    • No resource limits defined (CPU, memory)
    • No restart policy specified
    • Network configuration missing

Consider applying these improvements:

 minio-02:
     image: ${MINIO_IMAGE}
     entrypoint: sh
     command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
     environment:
-      MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
-      MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
+    secrets:
+      - minio_root_user
+      - minio_root_password
     healthcheck:
-      test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
+      test: [ "CMD", "curl", "-f", "-k", "https://localhost:9000/minio/health/live" ]
       interval: 30s
       timeout: 20s
       retries: 3
     hostname: minio-02
+    networks:
+      - datalake_network
     volumes:
       - minio-02-data1:/data1
       - minio-02-data2:/data2
     deploy:
+      restart_policy:
+        condition: on-failure
+        delay: 5s
+        max_attempts: 3
+      resources:
+        limits:
+          cpus: '1'
+          memory: 2G
+        reservations:
+          cpus: '0.5'
+          memory: 1G
       placement:
         constraints:
           - "node.labels.name==${MINIO_02_PLACEMENT}"
       replicas: 1

+secrets:
+  minio_root_user:
+    external: true
+  minio_root_password:
+    external: true
+
+networks:
+  datalake_network:
+    driver: overlay
+    attachable: true

Apply similar changes to minio-01, minio-03, and minio-04 services.

Also applies to: 33-53, 55-74

minio-03:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
hostname: minio-03
volumes:
- minio-03-data1:/data1
- minio-03-data2:/data2
deploy:
placement:
constraints:
- "node.labels.name==${MINIO_03_PLACEMENT}"
replicas: 1

minio-04:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
hostname: minio-04
volumes:
- minio-04-data1:/data1
- minio-04-data2:/data2
deploy:
placement:
constraints:
- "node.labels.name==${MINIO_04_PLACEMENT}"
replicas: 1

volumes:
minio-02-data1:
minio-02-data2:
minio-03-data1:
minio-03-data2:
minio-04-data1:
minio-04-data2:
Comment on lines +1 to +82
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codebase verification

MinIO extraction requires Loki configuration updates

The verification reveals important dependencies that need attention:

  1. Loki is configured to use MinIO as its storage backend, specifically pointing to minio-01:9000/loki
  2. The monitoring stack depends on the MinIO network for connectivity
  3. Node placement variables are properly defined in the package metadata

Action items:

  • Update Loki's storage configuration in monitoring/loki/loki-config.yml to use the new distributed MinIO setup
  • Ensure the MinIO network reference in monitoring/docker-compose.yml is updated to match the new datalake package network
🔗 Analysis chain

Verify architectural dependencies and requirements

As MinIO is being extracted from the monitoring package, please ensure:

  1. The monitoring package has alternative storage solutions or no longer requires object storage
  2. Node labels (MINIO_*_PLACEMENT) are properly configured in the deployment environment
  3. Service discovery and DNS resolution work correctly for the distributed MinIO setup

Let's verify the dependencies:

Let's gather more specific information about the MinIO references to understand the dependencies better:

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Check for MinIO dependencies in monitoring package
echo "Checking for MinIO references in monitoring package..."
rg -l "minio" monitoring/

# Check for environment variables in deployment configs
echo "Checking for MinIO placement variables..."
rg -l "MINIO_.*_PLACEMENT" .

Length of output: 605

Script:

#!/bin/bash
# Check MinIO references in monitoring config files
echo "=== monitoring/docker-compose.yml ==="
rg "minio" monitoring/docker-compose.yml -B 2 -A 2

echo -e "\n=== monitoring/loki/loki-config.yml ==="
rg "minio" monitoring/loki/loki-config.yml -B 2 -A 2

# Check package metadata for placement configuration
echo -e "\n=== datalake/package-metadata.json ==="
cat datalake/package-metadata.json

Length of output: 2864

8 changes: 8 additions & 0 deletions datalake/docker-compose.dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
version: '3.9'

services:
minio-01:
Comment on lines +3 to +4
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Service configuration is incomplete

The Minio service definition is missing essential configurations:

  • Image specification and version
  • Environment variables for access credentials
  • Volume mounts for data persistence
  • Health check configuration
  • Resource constraints

Here's a suggested configuration:

 services:
   minio-01:
+    image: minio/minio:RELEASE.2023-11-15T20-43-25Z
+    command: server --console-address ":9001" /data
+    environment:
+      - MINIO_ROOT_USER=minioadmin
+      - MINIO_ROOT_PASSWORD=minioadmin
+    volumes:
+      - minio_data:/data
+    healthcheck:
+      test: ["CMD", "mc", "ready", "local"]
+      interval: 30s
+      timeout: 20s
+      retries: 3
+    deploy:
+      resources:
+        limits:
+          memory: 2G
+        reservations:
+          memory: 1G
+
+ volumes:
+   minio_data:
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
services:
minio-01:
services:
minio-01:
image: minio/minio:RELEASE.2023-11-15T20-43-25Z
command: server --console-address ":9001" /data
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin
volumes:
- minio_data:/data
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 30s
timeout: 20s
retries: 3
deploy:
resources:
limits:
memory: 2G
reservations:
memory: 1G
volumes:
minio_data:

ports:
- target: 9001
published: 9005
mode: host
40 changes: 40 additions & 0 deletions datalake/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
version: '3.9'

services:
minio-01:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...${NUM_MINIO_SERVERS}}/data{1...2}'
Comment on lines +4 to +7
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Improve service initialization and configuration.

The current setup has several potential issues:

  1. Overriding the entrypoint with a shell command is not recommended. Consider using a custom init container or startup script.
  2. Hardcoded directory names (/data1/loki, /data2/loki) reduce flexibility.

Consider this alternative approach:

 minio-01:
   image: ${MINIO_IMAGE}
-  entrypoint: sh
-  command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...${NUM_MINIO_SERVERS}}/data{1...2}'
+  environment:
+    MINIO_DATA_DIRS: /data1/loki,/data2/loki
+  volumes:
+    - ./init-minio.sh:/docker-entrypoint.d/init-minio.sh
+  command: server --console-address ":9001" http://minio-0{1...${NUM_MINIO_SERVERS}}/data{1...2}

Create a new file init-minio.sh:

#!/bin/sh
set -e

# Create data directories from environment variable
IFS=',' read -ra DIRS <<< "$MINIO_DATA_DIRS"
for dir in "${DIRS[@]}"; do
    mkdir -p "$dir"
done

environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
MINIO_BROWSER_REDIRECT_URL: ${MINIO_BROWSER_REDIRECT_URL}
MINIO_SERVER_URL: http://localhost:9000
Comment on lines +9 to +12
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Review credential management and URL configuration.

  1. The service is using credentials from the monitoring package (MO_SECURITY_*). Since MinIO is being extracted into its own package, consider using dedicated environment variables.
  2. The MINIO_SERVER_URL is hardcoded to localhost:9000, which might not work in all deployment scenarios.

Consider these changes:

 environment:
-  MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
-  MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
+  MINIO_ROOT_USER: ${MINIO_ADMIN_USER}
+  MINIO_ROOT_PASSWORD: ${MINIO_ADMIN_PASSWORD}
   MINIO_BROWSER_REDIRECT_URL: ${MINIO_BROWSER_REDIRECT_URL}
-  MINIO_SERVER_URL: http://localhost:9000
+  MINIO_SERVER_URL: ${MINIO_SERVER_URL:-http://localhost:9000}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
MINIO_BROWSER_REDIRECT_URL: ${MINIO_BROWSER_REDIRECT_URL}
MINIO_SERVER_URL: http://localhost:9000
MINIO_ROOT_USER: ${MINIO_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MINIO_ADMIN_PASSWORD}
MINIO_BROWSER_REDIRECT_URL: ${MINIO_BROWSER_REDIRECT_URL}
MINIO_SERVER_URL: ${MINIO_SERVER_URL:-http://localhost:9000}

healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
hostname: minio-01
volumes:
- minio-01-data1:/data1
- minio-01-data2:/data2
deploy:
replicas: 1
labels:
- traefik.enable=true
- traefik.docker.network=reverse-proxy-traefik_public
- traefik.http.routers.minio.rule=${DOMAIN_NAME_HOST_TRAEFIK} && PathPrefix(`/minio`)
- traefik.http.services.minio.loadbalancer.server.port=9001
- traefik.http.middlewares.minio-stripprefix.stripprefix.prefixes=/minio
- traefik.http.routers.minio.middlewares=minio-stripprefix
networks:
public:
networks:
public:
name: minio_public
external: true

volumes:
minio-01-data1:
minio-01-data2:
21 changes: 21 additions & 0 deletions datalake/package-metadata.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"id": "datalake",
"name": "datalake",
"description": "Object storage for raw unstructured data, compatible with the s3 API",
"type": "infrastructure",
"version": "0.0.1",
"dependencies": [],
"environmentVariables": {
"NUM_MINIO_SERVERS": 1,
"MO_SECURITY_ADMIN_USER": "admin",
"MO_SECURITY_ADMIN_PASSWORD": "dev_password_only",
"MO_RETENTION_TIME": "15d",
"MINIO_BROWSER_REDIRECT_URL": "http://localhost:9001",
"DOMAIN_NAME_HOST_TRAEFIK": "localhost",
"MINIO_01_PLACEMENT": "minio-01",
"MINIO_02_PLACEMENT": "minio-02",
"MINIO_03_PLACEMENT": "minio-03",
"MINIO_04_PLACEMENT": "minio-04",
"MINIO_IMAGE": "minio/minio:RELEASE.2024-10-13T13-34-11Z.fips"
}
}
79 changes: 79 additions & 0 deletions datalake/swarm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
#!/bin/bash

declare ACTION=""
declare MODE=""
declare COMPOSE_FILE_PATH=""
declare UTILS_PATH=""
declare STACK="datalake"

function init_vars() {
ACTION=$1
MODE=$2

COMPOSE_FILE_PATH=$(
cd "$(dirname "${BASH_SOURCE[0]}")" || exit
pwd -P
)

UTILS_PATH="${COMPOSE_FILE_PATH}/../utils"

readonly ACTION
readonly MODE
readonly COMPOSE_FILE_PATH
readonly UTILS_PATH
readonly STACK
}
Comment on lines +9 to +25
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add parameter validation in init_vars function.

The function should validate the required parameters and their values before proceeding.

 function init_vars() {
+  # Validate required parameters
+  [[ $# -lt 2 ]] && log error "Missing required parameters. Usage: $0 <ACTION> <MODE>" && exit 1
+  
+  # Validate ACTION parameter
+  case $1 in
+    init|up|down|destroy) ;;
+    *) log error "Invalid ACTION. Must be one of: init, up, down, destroy" && exit 1 ;;
+  esac
+
+  # Validate MODE parameter
+  case $2 in
+    dev|prod) ;;
+    *) log error "Invalid MODE. Must be one of: dev, prod" && exit 1 ;;
+  esac
+
   ACTION=$1
   MODE=$2

Committable suggestion skipped: line range outside the PR's diff.


# shellcheck disable=SC1091
function import_sources() {
source "${UTILS_PATH}/docker-utils.sh"
source "${UTILS_PATH}/log.sh"
}
Comment on lines +27 to +31
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add validation for required utility files.

The function should verify the existence of required utility files before attempting to source them.

 # shellcheck disable=SC1091
 function import_sources() {
+  local required_utils=("docker-utils.sh" "log.sh")
+  for util in "${required_utils[@]}"; do
+    local util_path="${UTILS_PATH}/${util}"
+    [[ ! -f "${util_path}" ]] && echo "Error: Required utility file ${util_path} not found" && exit 1
+  done
+
   source "${UTILS_PATH}/docker-utils.sh"
   source "${UTILS_PATH}/log.sh"
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# shellcheck disable=SC1091
function import_sources() {
source "${UTILS_PATH}/docker-utils.sh"
source "${UTILS_PATH}/log.sh"
}
# shellcheck disable=SC1091
function import_sources() {
local required_utils=("docker-utils.sh" "log.sh")
for util in "${required_utils[@]}"; do
local util_path="${UTILS_PATH}/${util}"
[[ ! -f "${util_path}" ]] && echo "Error: Required utility file ${util_path} not found" && exit 1
done
source "${UTILS_PATH}/docker-utils.sh"
source "${UTILS_PATH}/log.sh"
}


function initialize_package() {
local package_dev_compose_filename=""
local minio_cluster_compose_filename=""
if [[ "${CLUSTERED_MODE}" == "true" ]]; then
minio_cluster_compose_filename="docker-compose.cluster.yml"
export NUM_MINIO_SERVERS=4
else
export NUM_MINIO_SERVERS=1
fi
if [[ "${MODE}" == "dev" ]]; then
log info "Running package in DEV mode"
package_dev_compose_filename="docker-compose.dev.yml"
else
log info "Running package in PROD mode"
fi

(
docker::deploy_service "$STACK" "${COMPOSE_FILE_PATH}" "docker-compose.yml" "$package_dev_compose_filename" "$minio_cluster_compose_filename"
) || {
log error "Failed to deploy package"
exit 1
}
}
Comment on lines +33 to +55
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add compose file validation and document environment variables.

The function should validate the existence of required compose files and document the environment variables it exports.

 function initialize_package() {
+  # NUM_MINIO_SERVERS: Number of MinIO servers to deploy
+  # Exports: NUM_MINIO_SERVERS=4 for clustered mode, NUM_MINIO_SERVERS=1 for non-clustered mode
+
   local package_dev_compose_filename=""
   local minio_cluster_compose_filename=""
+  local required_files=("docker-compose.yml")
+
   if [[ "${CLUSTERED_MODE}" == "true" ]]; then
     minio_cluster_compose_filename="docker-compose.cluster.yml"
+    required_files+=("${minio_cluster_compose_filename}")
     export NUM_MINIO_SERVERS=4
   else
     export NUM_MINIO_SERVERS=1
   fi
   if [[ "${MODE}" == "dev" ]]; then
     log info "Running package in DEV mode"
     package_dev_compose_filename="docker-compose.dev.yml"
+    required_files+=("${package_dev_compose_filename}")
   else
     log info "Running package in PROD mode"
   fi

+  # Validate required compose files
+  for file in "${required_files[@]}"; do
+    local file_path="${COMPOSE_FILE_PATH}/${file}"
+    [[ ! -f "${file_path}" ]] && log error "Required compose file ${file_path} not found" && exit 1
+  done
+
   (
     docker::deploy_service "$STACK" "${COMPOSE_FILE_PATH}" "docker-compose.yml" "$package_dev_compose_filename" "$minio_cluster_compose_filename"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
function initialize_package() {
local package_dev_compose_filename=""
local minio_cluster_compose_filename=""
if [[ "${CLUSTERED_MODE}" == "true" ]]; then
minio_cluster_compose_filename="docker-compose.cluster.yml"
export NUM_MINIO_SERVERS=4
else
export NUM_MINIO_SERVERS=1
fi
if [[ "${MODE}" == "dev" ]]; then
log info "Running package in DEV mode"
package_dev_compose_filename="docker-compose.dev.yml"
else
log info "Running package in PROD mode"
fi
(
docker::deploy_service "$STACK" "${COMPOSE_FILE_PATH}" "docker-compose.yml" "$package_dev_compose_filename" "$minio_cluster_compose_filename"
) || {
log error "Failed to deploy package"
exit 1
}
}
function initialize_package() {
# NUM_MINIO_SERVERS: Number of MinIO servers to deploy
# Exports: NUM_MINIO_SERVERS=4 for clustered mode, NUM_MINIO_SERVERS=1 for non-clustered mode
local package_dev_compose_filename=""
local minio_cluster_compose_filename=""
local required_files=("docker-compose.yml")
if [[ "${CLUSTERED_MODE}" == "true" ]]; then
minio_cluster_compose_filename="docker-compose.cluster.yml"
required_files+=("${minio_cluster_compose_filename}")
export NUM_MINIO_SERVERS=4
else
export NUM_MINIO_SERVERS=1
fi
if [[ "${MODE}" == "dev" ]]; then
log info "Running package in DEV mode"
package_dev_compose_filename="docker-compose.dev.yml"
required_files+=("${package_dev_compose_filename}")
else
log info "Running package in PROD mode"
fi
# Validate required compose files
for file in "${required_files[@]}"; do
local file_path="${COMPOSE_FILE_PATH}/${file}"
[[ ! -f "${file_path}" ]] && log error "Required compose file ${file_path} not found" && exit 1
done
(
docker::deploy_service "$STACK" "${COMPOSE_FILE_PATH}" "docker-compose.yml" "$package_dev_compose_filename" "$minio_cluster_compose_filename"
) || {
log error "Failed to deploy package"
exit 1
}
}


function destroy_package() {
docker::stack_destroy "$STACK"
}
Comment on lines +57 to +59
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add error handling and resource cleanup to destroy_package function.

The function should handle errors and ensure proper cleanup of resources.

 function destroy_package() {
-  docker::stack_destroy "$STACK"
+  log info "Destroying stack ${STACK}..."
+  if ! docker::stack_destroy "$STACK"; then
+    log error "Failed to destroy stack ${STACK}"
+    return 1
+  fi
+  
+  # Consider adding cleanup of persistent volumes/data if needed
+  log info "Stack ${STACK} destroyed successfully"
 }

Committable suggestion skipped: line range outside the PR's diff.


main() {
init_vars "$@"
import_sources

if [[ "${ACTION}" == "init" ]] || [[ "${ACTION}" == "up" ]]; then
log info "Running package in Single node mode"
initialize_package
elif [[ "${ACTION}" == "down" ]]; then
log info "Scaling down package"
docker::scale_services "$STACK" 0
elif [[ "${ACTION}" == "destroy" ]]; then
log info "Destroying package"
destroy_package
else
log error "Valid options are: init, up, down, or destroy"
fi
}

main "$@"
77 changes: 0 additions & 77 deletions monitoring/docker-compose.cluster.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,83 +32,6 @@ services:
public:
default:

minio-01:
deploy:
placement:
constraints:
- "node.labels.name==${MINIO_01_PLACEMENT}"

minio-02:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
hostname: minio-02
volumes:
- minio-02-data1:/data1
- minio-02-data2:/data2
deploy:
placement:
constraints:
- "node.labels.name==${MINIO_02_PLACEMENT}"
replicas: 1

minio-03:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
hostname: minio-03
volumes:
- minio-03-data1:/data1
- minio-03-data2:/data2
deploy:
placement:
constraints:
- "node.labels.name==${MINIO_03_PLACEMENT}"
replicas: 1

minio-04:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
hostname: minio-04
volumes:
- minio-04-data1:/data1
- minio-04-data2:/data2
deploy:
placement:
constraints:
- "node.labels.name==${MINIO_04_PLACEMENT}"
replicas: 1

volumes:
prometheus_data_backup:
minio-02-data1:
minio-02-data2:
minio-03-data1:
minio-03-data2:
minio-04-data1:
minio-04-data2:
6 changes: 0 additions & 6 deletions monitoring/docker-compose.dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,3 @@ services:
- target: 3100
published: 3100
mode: host

minio-01:
ports:
- target: 9001
published: 9005
mode: host
Loading
Loading