Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: create local dev environment that mirrors kube #1891

Merged
merged 85 commits into from
Jan 11, 2024
Merged
Show file tree
Hide file tree
Changes from 72 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
51b1035
build: new local build environment
marvinroman Dec 2, 2022
7fc684d
build(deploy): used flock instead of manual lock, moved chown to later
marvinroman Dec 7, 2022
efe0fac
build: added some commented out deploy options for testing replicas
marvinroman Dec 7, 2022
986433b
feat: added proxy so that there can be multiple replicas for app layer
marvinroman Dec 7, 2022
e47bf11
feat: added mysql client to the app container for checking connection
marvinroman Dec 7, 2022
5a53c43
feat: temporarily remove ssl cert creation from local deploy entrypoint
marvinroman Dec 7, 2022
a37e992
feat: NGINX proxy container created and configs updated
marvinroman Dec 7, 2022
86fc446
Merge branch 'dev' into feat/create-local-dev-environment-that-mirror…
marvinroman Jan 25, 2023
0a63737
feat(command): created new table seeder backup command
marvinroman Jan 25, 2023
d0f2ccb
feat(config): added disk path for storing seeds
marvinroman Jan 25, 2023
1c305b2
feat(seeder): updated some seeders to pull from stored file first
marvinroman Jan 25, 2023
4e3ad59
feat(cron): updated cron commands with parameters
marvinroman Jan 25, 2023
e2f561f
feat(command): added parameter to command and seeder updates
marvinroman Jan 25, 2023
2b2dfa8
feat(command): updated command parameter
marvinroman Jan 25, 2023
df98ac2
feat(command): added ability to run all table backups at once
marvinroman Jan 25, 2023
328cebe
Merge branch '1483-create-seedersreplication-for-data-addededited-via…
marvinroman Jan 26, 2023
16de844
feat(command): added new commands --remove & --restore
marvinroman Jan 26, 2023
3458f66
refactor(command): removed the forced filament table seeding
marvinroman Jan 26, 2023
a6d5546
feat(config): added configuration for filament seeders
marvinroman Jan 26, 2023
3eeca54
feat(seeder): added InterpretationSeeder to the DevSeeder
marvinroman Jan 26, 2023
a8cb8a6
feat(seeder): don't run the filament seeder in testing or production
marvinroman Jan 26, 2023
8b88a68
feat(seeder): don't run on testing or production skip foreign key errors
marvinroman Jan 26, 2023
c57e641
Merge branch '1483-create-seedersreplication-for-data-addededited-via…
marvinroman Jan 26, 2023
81aa069
docs: added artisan command descriptions to the README
marvinroman Jan 27, 2023
8d40294
feat(command): --table option can have multiple values now
marvinroman Jan 27, 2023
1bec5f3
feat(seeder): added Interpretation Seeder to the Production Seeder
marvinroman Feb 8, 2023
8a531a0
Merge branch '1483-create-seedersreplication-for-data-addededited-via…
marvinroman Feb 8, 2023
2c99a97
feat(seeder): added in tagging for the environment to the seeder back…
marvinroman Feb 8, 2023
876fa3e
feat(seeder): added in tagging for the environment to the seeder back…
marvinroman Feb 8, 2023
d20d661
chore(config): removed resource tables and seeders from config until …
marvinroman Feb 8, 2023
22878cf
feat(seeder): added in tagging for the environment to the seeder back…
marvinroman Feb 8, 2023
cef0e58
chore(config): removed resource tables until we figure out handling o…
marvinroman Feb 8, 2023
bfd0929
feat(seeder): added --truncate option so that tables can be truncated…
marvinroman Feb 8, 2023
a826c98
Merge branch '1483-create-seedersreplication-for-data-addededited-via…
marvinroman Feb 9, 2023
710fd34
chore: fixed merge
marvinroman Feb 9, 2023
aeee435
Merge branch 'staging' into feat/create-local-dev-environment-that-mi…
marvinroman Feb 9, 2023
b9cf4b9
fix(seeder): resolved unmerged content
marvinroman Mar 20, 2023
cd5c2be
build: added user and group identifiers to .env.example
marvinroman Mar 20, 2023
ad74512
build: resolved docker compose issues
marvinroman Mar 20, 2023
bed0c69
docs: added documentation on how to setup an environment that mirrors…
marvinroman Mar 20, 2023
3c2a687
Merge branch 'staging' into feat/create-local-dev-environment-that-mi…
marvinroman Mar 20, 2023
3c253be
build(command): revised the deploy:global command to remove compiled …
marvinroman Mar 20, 2023
918960a
fix(deploy): make sure no additional cache files remain after deploy
marvinroman Mar 21, 2023
7fbebef
ci(deploy): local deploy allow checking of nginx status
marvinroman Mar 28, 2023
4a3212b
ci(deploy): changed healthcheck to use nginx_status uri
marvinroman Mar 28, 2023
fdb64db
chore: merged in staging branch
marvinroman Apr 26, 2023
6fb9096
ci(deploy): :construction_worker: testing limiting running by repository
marvinroman May 30, 2023
26ec264
ci(deploy): :construction_worker: added separate mirroring for each b…
marvinroman May 30, 2023
0dab938
ci(deploy): :construction_worker: testing deploy
marvinroman May 30, 2023
92a38cd
ci(deploy): :construction_worker: adding back repository restriction
marvinroman May 30, 2023
1e731f7
ci: :construction_worker: updated actions per PR notes
marvinroman May 31, 2023
76c0316
ci: :green_heart: swapped to using ssh webfactory action
marvinroman Jun 1, 2023
1c3f3dc
Merge branch '1721-automate-the-push-from-github-to-gitlab' into feat…
marvinroman Jun 11, 2023
8cb575d
build: local compose dockerfile path update
marvinroman Jun 12, 2023
225a7a7
fix(deploy): :bug: fixed npm install
marvinroman Aug 4, 2023
aee8bf0
chore: :twisted_rightwards_arrows: pulled latest staging updates
marvinroman Aug 21, 2023
32a0fd0
build(deploy): :construction_worker: updated entrypoint to better mat…
marvinroman Aug 21, 2023
2a61ae6
build(deploy): :adhesive_bandage: updated user id to come from .env
marvinroman Aug 21, 2023
20d5c57
docs: :pencil2: fixed typo in deployment instructions
marvinroman Aug 21, 2023
ca4bb7e
chore(deploy): reverted .env.example and created new one specific for…
marvinroman Aug 21, 2023
22e09ce
docs: :memo: fixed cyphersweet order of operation in deploying, add d…
marvinroman Oct 11, 2023
ac11a24
Merge branch 'dev' into feat/create-local-dev-environment-that-mirror…
marvinroman Oct 11, 2023
c7b773c
docs: :memo: added command to rebuild main container
marvinroman Oct 11, 2023
cb4b871
build(deploy): added local php development config and port overrides …
marvinroman Oct 12, 2023
63aa12c
build(deploy): :green_heart: fix entrypoint config that includes the …
marvinroman Oct 12, 2023
650e168
build: :building_construction: added in public directory to mapping i…
marvinroman Oct 12, 2023
5f9d08d
build(deploy): include resources directory into containers
marvinroman Oct 12, 2023
a3d2117
chore: merge from dev
marvinroman Oct 26, 2023
94b319e
Merge branch 'dev' into feat/create-local-dev-environment-that-mirror…
marvinroman Oct 26, 2023
af1c924
build: :fire: removed commented configuration
marvinroman Oct 26, 2023
4329f5e
chore: add example of MINIO variables needed to test s3 seeds sharing
marvinroman Oct 31, 2023
66f59a6
docs: :memo: fixed readme merge issue
marvinroman Oct 31, 2023
7dda5a8
build(deploy): changed local deploy to include mailpit instead of mai…
marvinroman Nov 1, 2023
aecd682
docs: :memo: updated instructions for local deployment
marvinroman Nov 1, 2023
5d689e5
docs: :memo: removed valet and added new troubleshooting instructions
marvinroman Nov 1, 2023
814e320
fix: fix so that npm commands can be run as www-data
marvinroman Nov 1, 2023
16175ac
build(deploy): :green_heart: consolidate php extensions, npm update
marvinroman Nov 1, 2023
0414e2a
docs: :memo: update laravel version, altered npm instructions
marvinroman Nov 1, 2023
57e5e5e
docs: :memo: added generating redis password to local deployment
marvinroman Nov 15, 2023
452db61
docs: :memo: fixed numbering on local deployment steps
marvinroman Nov 15, 2023
8609faf
chore: merge from dev
marvinroman Nov 15, 2023
373b7ef
chore: resolved merge conflicts
marvinroman Jan 11, 2024
fb20ed8
docs: :memo: added instructions to make sure it's setup properly use git
marvinroman Jan 11, 2024
369af6e
fix: update mailhost in example env to mailpit
jobara Jan 11, 2024
1bce650
style: run php formatting
jobara Jan 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions .env.local.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# accessibility-app variables
APP_NAME="The Accessibility Exchange"
APP_ENV=local
APP_KEY=
APP_URL=https://platform.test
APP_SERVICE=platform.test

DB_CONNECTION=mysql
DB_HOST=platform.mysql
DB_PORT=3306
DB_DATABASE=accessibilityexchange
DB_USERNAME=root
DB_PASSWORD=

FILES_PATH="/opt/data/storage"
CACHE_PATH="/opt/data/cache"

LOG_CHANNEL=stack
LOG_LEVEL=debug

BROADCAST_DRIVER=log
CACHE_DRIVER=file
QUEUE_CONNECTION=sync
SESSION_DRIVER=database
SESSION_LIFETIME=120

MEMCACHED_HOST=memcached

# REDIS_HOST=127.0.0.1
REDIS_HOST=platform.redis
REDIS_PASSWORD=
REDIS_PORT=6379


MAIL_MAILER=smtp
MAIL_HOST=platform.mailhog
MAIL_PORT=1025
MAIL_USERNAME=
MAIL_PASSWORD=
MAIL_ENCRYPTION=
MAIL_FROM_ADDRESS=noreply@example.com
MAIL_FROM_NAME="${APP_NAME}"

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=us-east-1
AWS_BUCKET=

SNAPSHOTS_DISK=snapshots
SNAPSHOTS_AWS_ACCESS_KEY_ID=
SNAPSHOTS_AWS_SECRET_ACCESS_KEY=
SNAPSHOTS_AWS_DEFAULT_REGION=us-east-1
SNAPSHOTS_AWS_BUCKET=

PUSHER_APP_ID=
PUSHER_APP_KEY=
PUSHER_APP_SECRET=
PUSHER_APP_CLUSTER=mt1

VITE_PUSHER_APP_KEY="${PUSHER_APP_KEY}"
VITE_PUSHER_APP_CLUSTER="${PUSHER_APP_CLUSTER}"

SAIL_XDEBUG_MODE=develop,debug,coverage

CIPHERSWEET_KEY=

FLARE_KEY=

WWWUSER=
WWWGROUP=

# if you want to test s3 connection from local setup and have proper credentials to an s3 bucket
# MINIO_DRIVER=s3
# MINIO_ENDPOINT=
# MINIO_REGION=
# MINIO_PROJECT_BUCKET=
# MINIO_ACCESS_KEY=
# MINIO_SECRET_KEY=
49 changes: 49 additions & 0 deletions .local-deploy/accessibility-app/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/bin/sh

set -e

# TODO permanent remove cache lines once testing on per/pod caching is tested
if [ -f /usr/local/etc/php/php.ini-development ]; then
rm -f /usr/local/etc/php/conf.d/php.ini
ln -s /usr/local/etc/php/php.ini-development /usr/local/etc/php/conf.d/php.ini
fi

# mirror developers user id so that they can edit live files in the docker
if [ -n "$USER_ID" ]; then
usermod -u $USER_ID www-data
chown -R www-data:root /app
fi

mkdir -p $FILES_PATH
# mkdir -p $CACHE_PATH removed per https://github.com/accessibility-exchange/platform/issues/1596

## fix permissions before syncing to existing storage and cache https://github.com/accessibility-exchange/platform/issues/1226
chown -R www-data:root /app/storage /app/bootstrap/cache $FILES_PATH $VIEW_COMPILED_PATH # $CACHE_PATH removed per https://github.com/accessibility-exchange/platform/issues/1596

## sync files from container storage to permanent storage then remove container storage
rsync -a /app/storage/ $FILES_PATH
rm -rf /app/storage

## sync files from container cache to permanent storage then remove container cache
## removed syncing to shared/permenant storage https://github.com/accessibility-exchange/platform/issues/1596
# rsync -a /app/bootstrap/cache/ $CACHE_PATH
# rm -rf /app/bootstrap/cache

## create symlinks from permanent storage & cache to application directory folders
ln -s $FILES_PATH /app/storage
## removed linked to shared/permenant storage https://github.com/accessibility-exchange/platform/issues/1596
# ln -s $CACHE_PATH /app/bootstrap/cache

# make sure to test mysql connection before running the php artisan commands
while ! mysqladmin ping -h$DB_HOST -uroot -p$DB_PASSWORD --silent; do
sleep 1
done

php artisan deploy:local # run before global so that storage is linked https://github.com/accessibility-exchange/platform/issues/1357

flock -n -E 0 /opt/data -c "php artisan deploy:global" # run exclusively on a single instance at once

## fix permissions after syncing to existing storage and cache https://github.com/accessibility-exchange/platform/issues/1236
chown -R www-data:root /app/bootstrap/cache $FILES_PATH # $CACHE_PATH removed per and added path to cache in the pod https://github.com/accessibility-exchange/platform/issues/1596

/usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
46 changes: 46 additions & 0 deletions .local-deploy/accessibility-app/etc/nginx/includes/laravel.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
sendfile off;
absolute_redirect off;

root /app/public;
index index.php index.html index.htm;

expires epoch;

location / {
try_files $uri $uri/ /index.php?$query_string;
}

location ~ /build/assets/.*\.(css|js)$ {
access_log off;
}

location ~ /_debugbar/assets/javascript/.*\.js$ {
access_log off;
}

location ~ /storage/fonts/.*\.woff2$ {
access_log off;
}

location = /favicon.ico {
access_log off;
log_not_found off;
}

location = /robots.txt {
access_log off;
log_not_found off;
}

error_page 404 /index.php;

location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
fastcgi_param HTTPS "on";
}

location ~ /\.(?!well-known).* {
deny all;
}
52 changes: 52 additions & 0 deletions .local-deploy/accessibility-app/etc/nginx/nginx.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
worker_processes auto;
error_log stderr warn;
pid /run/nginx.pid;

events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;

log_format main_timed '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'$request_time $upstream_response_time $pipe $upstream_cache_status';

access_log /dev/stdout main_timed;
error_log /dev/stderr notice;

keepalive_timeout 65;

client_body_temp_path /tmp/client_temp;
proxy_temp_path /tmp/proxy_temp_path;
fastcgi_temp_path /tmp/fastcgi_temp;
uwsgi_temp_path /tmp/uwsgi_temp;
scgi_temp_path /tmp/scgi_temp;

server {
listen [::]:8080;
listen 8080;
server_name _;

location /nginx_status {
stub_status on;
allow 127.0.0.1;
allow 10.0.0.0/8;
allow 172.0.0.0/8;
deny all;
}

include /etc/nginx/includes/laravel.conf;
}

gzip on;
gzip_proxied any;
gzip_types text/plain application/xml text/css text/js text/xml application/x-javascript text/javascript application/json application/xml+rss;
gzip_vary on;
gzip_disable "msie6";

include /etc/nginx/conf.d/*.conf;
}
5 changes: 5 additions & 0 deletions .local-deploy/accessibility-app/php.ini-development
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
[PHP]
post_max_size = 100M
upload_max_filesize = 100M
variables_order = EGPCS
memory_limit = 1G
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
[mysqld]
basedir=/opt/bitnami/mysql
bind-address=0.0.0.0
character-set-server=UTF8
collation-server=utf8_general_ci
datadir=/bitnami/mysql/data
default_authentication_plugin=mysql_native_password
explicit_defaults_for_timestamp
innodb_buffer_pool_size=1G
innodb_doublewrite=1
innodb_flush_log_at_trx_commit=1
innodb_flush_method=O_DIRECT
innodb_lock_wait_timeout=50
innodb_log_buffer_size=16M
innodb_max_dirty_pages_pct=80
innodb_thread_concurrency=0
log-error=/opt/bitnami/mysql/logs/mysqld.log
long_query_time=10.0
max_allowed_packet=16M
pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
plugin_dir=/opt/bitnami/mysql/lib/plugin
port=3306
skip-name-resolve
slow_query_log=0
slow_query_log_file=/opt/bitnami/mysql/logs/mysqld.log
socket=/opt/bitnami/mysql/tmp/mysql.sock
table_definition_cache=2048
table_open_cache=512
tmpdir=/opt/bitnami/mysql/tmp

[client]
default-character-set=UTF8
plugin_dir=/opt/bitnami/mysql/lib/plugin
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock

[manager]
pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
18 changes: 18 additions & 0 deletions .local-deploy/accessibility-proxy/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
FROM nginx:1.23-alpine

RUN apk update
RUN apk add --no-cache openssl

ARG SSL_SUBJECT

RUN openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

RUN openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-subj "$SSL_SUBJECT" \
-keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt

COPY etc /etc

RUN mkdir -p /etc/nginx/sites-enabled
RUN cp /etc/nginx/sites-available/default_ssl.conf /etc/nginx/sites-enabled/default_ssl.conf
RUN cp /etc/nginx/sites-available/default.conf /etc/nginx/sites-enabled/default.conf
40 changes: 40 additions & 0 deletions .local-deploy/accessibility-proxy/etc/nginx/nginx.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
worker_processes auto;
error_log stderr warn;
pid /run/nginx.pid;

events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;

log_format main_timed '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'$request_time $upstream_response_time $pipe $upstream_cache_status';

access_log /dev/stdout main_timed;
error_log /dev/stderr notice;

keepalive_timeout 65;

client_body_temp_path /tmp/client_temp;
proxy_temp_path /tmp/proxy_temp_path;
fastcgi_temp_path /tmp/fastcgi_temp;
uwsgi_temp_path /tmp/uwsgi_temp;
scgi_temp_path /tmp/scgi_temp;

gzip on;
gzip_proxied any;
gzip_types text/plain application/xml text/css text/js text/xml application/x-javascript text/javascript application/json application/xml+rss;
gzip_vary on;
gzip_disable "msie6";

upstream platform.test {
server platform.test:8080;
}

include /etc/nginx/sites-enabled/*.conf;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
server {
listen 80;
listen [::]:80;

location / {
proxy_pass http://platform.test;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-for $remote_addr;
proxy_connect_timeout 300;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
server {
listen 443 ssl;
listen [::]:443 ssl;

ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;

location / {
proxy_pass http://platform.test;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-for $remote_addr;
proxy_connect_timeout 300;
}
}
Loading