Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated deploy files #732

Open
wants to merge 9 commits into
base: develop
Choose a base branch
from
Open

Updated deploy files #732

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0a6c10b
updated deploy
raprasad Jan 23, 2023
12e74e5
working on #729
raprasad Jan 23, 2023
caa1af4
Merge remote-tracking branch 'origin' into 2023-deploy
raprasad Jan 25, 2023
3daeb4e
Merge remote-tracking branch 'origin' into 2023-deploy
raprasad Feb 3, 2023
d97645d
updated k8s;
raprasad Feb 3, 2023
eb624ec
demo k8s
raprasad Feb 3, 2023
d66fcf1
Update demo.dpcreator.org_08_2022_0719.yaml
raprasad Feb 3, 2023
45c7fc5
updated
raprasad Feb 3, 2023
69d2e6a
Merge remote-tracking branch 'origin' into 2023-deploy
raprasad Feb 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion deploy/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Deployment currently requires the building of two Docker images:
Set an environment variable called "today" of the form YYYY-MMDD. We will use this throughout the deploy:
```
# Example: (use today's date)
export today=2022-0717
export today=2023-0203
```

### (A) DPCreator app
Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s_maker/dpcreator_specs_01.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
#
dpcreator_app_container="ghcr.io/opendp/dpcreator/app", # app image (server + compiled Vue.js)
dpcreator_nginx_container="ghcr.io/opendp/dpcreator/nginx", # Nginx image
dpcreator_container_tag="2022-0719", # "latest", # Tag for Images
dpcreator_container_tag="2023-0203", # "latest", # Tag for Images
#
DEFAULT_FROM_EMAIL='info@opendp.org', # Related to the Sendgrid API call
#
Expand Down
344 changes: 344 additions & 0 deletions deploy/k8s_maker/rendered/demo.dpcreator.org_08_2023_0203.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,344 @@
---
# reference: https://github.com/kubernetes-operators-book/chapters/blob/master/ch05/database.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: dpcreator-db-data-configmap-demo
data:
#
# Database settings for postgres and Django
#
#
# DB_HOST Should be the same as the service name at the end of this file:
DB_HOST: "dpcreator-postgres-service-demo"
DB_PORT: "5432"
DB_ENGINE: "django.db.backends.postgresql_psycopg2"
#
# These two variables should have the same value (kludge):
POSTGRES_DB: "db_dpcreator"
DB_NAME: "db_dpcreator"
---
# ---------------------------
# DPCreator - Postgres Deployment
# ---------------------------
apiVersion: apps/v1
kind: Deployment
metadata:
name: dpcreator-database-demo
spec:
replicas: 1
selector:
matchLabels:
app: dpcreator-demo
tier: postgres
template:
metadata:
labels:
app: dpcreator-demo
tier: postgres
spec:
volumes:
# Persistent Disk space to store Postgres files
# Used by dpcreator-postgres container
#- name: postgres-persistent-volume
# azureDisk:
# kind: Managed
# diskName: dpcreator-postgres-02
# diskURI: /subscriptions/14d2f886-16d9-4a5b-87f0-7aa1c4608026/resourceGroups/MC_DPCreatorResourceGroup_DPCreatorCluster01_eastus/providers/Microsoft.Compute/disks/dpcreator-postgres-02
containers:
- name: dpcreator-postgres-demo
image: postgres:13
imagePullPolicy: Always
ports:
- name: postgres-port
containerPort: 5432
protocol: TCP
# Persistent Disk space to store Postgres files
#volumeMounts:
#- name: postgres-persistent-volume
# mountPath: /var/lib/postgresql/data
envFrom:
- configMapRef:
name: dpcreator-db-data-configmap-demo
env:
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: postgres-auth-secret
key: db_username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-auth-secret
key: db_password
---
# ---------------------------
# DPCreator - Postgres Service
# ---------------------------
apiVersion: v1
kind: Service
metadata:
name: dpcreator-postgres-service-demo
labels:
app: dpcreator-demo
tier: postgres
spec:
clusterIP: None
ports:
- port: 5432
selector:
app: dpcreator-demo
tier: postgres
---
# ----------------------------------------
# Configmap used for:
# dpcreator-app, dpcreator-celery
# ----------------------------------------
apiVersion: v1
kind: ConfigMap
metadata:
name: dpcreator-app-configmap-demo
data:
#
STATIC_ROOT: "/dpcreator_volume/static/dist"
STATIC_URL: "/static/dist/"
UPLOADED_FILE_STORAGE_ROOT: "/dpcreator_user_data"
RELEASE_FILE_STORAGE_ROOT: "/dpcreator_release_data"
#
# Should match the NGINX_MAX_UPLOAD_SIZE
# ref: https://docs.djangoproject.com/en/3.2/ref/settings/#std:setting-DATA_UPLOAD_MAX_MEMORY_SIZE
# 20971520 bytes = 20MB
DATA_UPLOAD_MAX_MEMORY_SIZE: "20971520"
#
# Set to default of 2.5MB
# ref: https://docs.djangoproject.com/en/3.2/ref/settings/#std:setting-FILE_UPLOAD_MAX_MEMORY_SIZE
FILE_UPLOAD_MAX_MEMORY_SIZE: "2621440"
#
# Require email verification for account creation
ACCOUNT_EMAIL_VERIFICATION: "mandatory"
#
DEFAULT_FROM_EMAIL: "info@opendp.org"
#
#
VUE_APP_GOOGLE_CLIENT_ID: "750757442540-4bg3aulcrm802i8pguo851lq8kikf5ge.apps.googleusercontent.com"
VUE_APP_ADOBE_PDF_CLIENT_ID: "44937032e26b4033a840626ed0cd8e79"
VUE_APP_WEBSOCKET_PREFIX: "wss://"
#
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dpcreator-app-demo
spec:
replicas: 1
selector:
matchLabels:
app: dpcreator-demo
tier: app
template:
metadata:
labels:
app: dpcreator-demo
tier: app
spec:
#restartPolicy: Always
volumes:
- name: dpcreator-volume-demo
emptyDir: {}
# azureDisk:
# kind: Managed
# diskName: storage-dpcreator-files
# diskURI:
containers:
# -------------------------------------------------
# (1) Nginx frontend: Separate requests to static files vs. Django app
# -------------------------------------------------
- name: dpcreator-nginx
image: ghcr.io/opendp/dpcreator/nginx:2023-0203
imagePullPolicy: Always
ports:
- containerPort: 80
name: http
protocol: TCP
volumeMounts:
# ----------------------------------
# shared between containers
# ----------------------------------
- name: dpcreator-volume-demo
mountPath: /dpcreator_volume
#subPath: 2ravens_org-apricot
#readOnly: true
#envFrom:
# - configMapRef:
# name: ravens-django-config-apricot
# -------------------------------------------------
# (2) Redis
# -------------------------------------------------
- name: redis
image: redis:6
imagePullPolicy: Always
ports:
- containerPort: 6379
# -------------------------------------------------
# (3) Core application: Django + bundled static files
# -------------------------------------------------
- name: dpcreator-app
image: ghcr.io/opendp/dpcreator/app:2023-0203
imagePullPolicy: Always
command: [ "azure_demo.dpcreator.org.sh" ]
#command: [ "/bin/sh" ]
# args: [ "-c", "./migrate.sh && python manage.py runserver 0.0.0.0:8000"]
ports:
- name: dpcreator
containerPort: 8000
protocol: TCP
volumeMounts:
# ----------------------------------
# shared between containers
# ----------------------------------
- name: dpcreator-volume-demo
mountPath: /dpcreator_volume
#subPath: 2ravens_org-apricot
#readOnly: false
envFrom:
- configMapRef:
name: dpcreator-db-data-configmap-demo
- configMapRef:
name: dpcreator-app-configmap-demo
env:
# Same storage roots on dpcreator-app and celery-worker
#
- name: UPLOADED_FILE_STORAGE_ROOT
value: /dpcreator_volume/private/user_uploaded_data
- name: RELEASE_FILE_STORAGE_ROOT
value: /dpcreator_volume/public/release_files
# ----------------------------------------------------
- name: DJANGO_SETTINGS_MODULE
value: opendp_project.settings.azure_test_01
- name: ALLOWED_HOSTS
value: "13.82.125.69,demo.dpcreator.org,127.0.0.1,0.0.0.0"
- name: DB_USER
valueFrom:
secretKeyRef:
name: postgres-auth-secret
key: db_username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-auth-secret
key: db_password
- name: SECRET_KEY
valueFrom:
secretKeyRef:
name: dpcreator-app-secrets
key: SECRET_KEY
- name: CRYPTOGRAPHY_KEY
valueFrom:
secretKeyRef:
name: dpcreator-app-secrets
key: CRYPTOGRAPHY_KEY
- name: SENDGRID_API_KEY
valueFrom:
secretKeyRef:
name: dpcreator-app-secrets
key: SENDGRID_API_KEY
- name: DJANGO_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: dpcreator-app-secrets
key: DJANGO_ADMIN_PASSWORD
- name: USER_DEPOSITOR_PASSWORD
valueFrom:
secretKeyRef:
name: dpcreator-app-secrets
key: USER_DEPOSITOR_PASSWORD
- name: USER_ANALYST_PASSWORD
valueFrom:
secretKeyRef:
name: dpcreator-app-secrets
key: USER_ANALYST_PASSWORD
# -------------------------------------------------
# (4) Celery Queue - same image/settings as except for "command"
# -------------------------------------------------
- name: celery-worker
image: ghcr.io/opendp/dpcreator/app:2023-0203
imagePullPolicy: Always
command: ['celery', '-A', 'opendp_project', 'worker', '-l', 'info', '-n', 'worker_dpcreator']
ports:
- name: dpcreator
containerPort: 8080
protocol: TCP
volumeMounts:
# ----------------------------------
# shared between containers
# ----------------------------------
- name: dpcreator-volume-demo
mountPath: /dpcreator_volume
#subPath: 2ravens_org-apricot
#readOnly: false
envFrom:
- configMapRef:
name: dpcreator-db-data-configmap-demo
- configMapRef:
name: dpcreator-app-configmap-demo
env:
# Same storage roots on dpcreator-app and celery-worker
#
- name: UPLOADED_FILE_STORAGE_ROOT
value: /dpcreator_volume/private/user_uploaded_data
- name: RELEASE_FILE_STORAGE_ROOT
value: /dpcreator_volume/public/release_files
- name: DJANGO_SETTINGS_MODULE
value: opendp_project.settings.azure_test_01
# ----------------------------------------------------
- name: ALLOWED_HOSTS
value: "13.82.125.69,demo.dpcreator.org,127.0.0.1,0.0.0.0"
- name: DB_USER
valueFrom:
secretKeyRef:
name: postgres-auth-secret
key: db_username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-auth-secret
key: db_password
- name: SECRET_KEY
valueFrom:
secretKeyRef:
name: dpcreator-app-secrets
key: SECRET_KEY
- name: CRYPTOGRAPHY_KEY
valueFrom:
secretKeyRef:
name: dpcreator-app-secrets
key: CRYPTOGRAPHY_KEY
- name: SENDGRID_API_KEY
valueFrom:
secretKeyRef:
name: dpcreator-app-secrets
key: SENDGRID_API_KEY
---
# ---------------------------
# DPCreator - Service
# ---------------------------
apiVersion: v1
kind: Service
metadata:
name: dpcreator-load-balancer-demo
labels:
app: dpcreator-demo
tier: load-balancer
spec:
type: LoadBalancer
# IP mapped to dpcreator.2ravens.org
loadBalancerIP: 13.82.125.69
selector:
app: dpcreator-demo
tier: app
ports:
- port: 80
# nginx port ->
targetPort: 80
# test - right to dpcreator-app port
#targetPort: 8000
Loading