Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sandbox Environment #4383

Merged
merged 101 commits into from
Nov 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
7dfe697
Initial sandbox prototype
asteel-gsa Sep 6, 2024
9a144fc
Terraform runs, fails to stage
asteel-gsa Sep 9, 2024
de82d41
App and AV deploy.. but..
asteel-gsa Sep 10, 2024
930d87d
Modified setup to try and get a running instance
asteel-gsa Sep 11, 2024
c212c8b
Update gitignore
asteel-gsa Sep 11, 2024
02fa76c
Disable newrelic start
asteel-gsa Sep 11, 2024
91d7687
Add note about proxy
asteel-gsa Sep 11, 2024
343c4d7
Update Readme
asteel-gsa Sep 11, 2024
1654075
Use an old procfile to try and get system to boot
asteel-gsa Sep 11, 2024
53cd1ea
Add --preload to debug stack
asteel-gsa Sep 11, 2024
f270ec1
Test without collectstatic
asteel-gsa Sep 11, 2024
b9b6788
Modify profile again
asteel-gsa Sep 11, 2024
d17b9c7
Disable the entire .profile
asteel-gsa Sep 11, 2024
8ca2391
Try to just do something that will pass
asteel-gsa Sep 11, 2024
6600d6d
Just use an echo for now
asteel-gsa Sep 11, 2024
48dfc87
Preboot in sandbox
asteel-gsa Sep 12, 2024
5ee2cc8
Single Worker thread
asteel-gsa Sep 12, 2024
0dbbb68
executable for start command in terraform
asteel-gsa Sep 12, 2024
f3f5b90
Modify based on SO post
asteel-gsa Sep 12, 2024
8f9dae6
Fails gunicorn startup
asteel-gsa Sep 12, 2024
4b2b393
Reconfigure the app now that we can do strategy rolling
asteel-gsa Sep 17, 2024
9d6f0e5
update chmod for destroy
asteel-gsa Sep 17, 2024
4b34437
Reset everything to try and get a working deployment
asteel-gsa Sep 17, 2024
6342a67
Add fac-key-service
asteel-gsa Sep 17, 2024
da12557
disable api
asteel-gsa Sep 17, 2024
c2b80fe
Proxy configuration seems good
asteel-gsa Sep 18, 2024
5c4e951
Disable collectstatic
asteel-gsa Sep 18, 2024
fce290b
Reset proxy
asteel-gsa Sep 18, 2024
36867b2
Add direct route back
asteel-gsa Sep 18, 2024
d7e9ac4
Remove --preload, maybe fix whitenoise missing import?
asteel-gsa Sep 18, 2024
56ac663
Revert Procfile
asteel-gsa Sep 18, 2024
64d57c8
Add the correct routing and environment variables
asteel-gsa Sep 19, 2024
134118a
Make sure settings.py included sandbox space
asteel-gsa Sep 19, 2024
10bedfe
Remove depends_on
asteel-gsa Sep 19, 2024
eea61d0
Readd staticfiles in .profile
asteel-gsa Sep 19, 2024
68453bb
disable once more
asteel-gsa Sep 19, 2024
e747d64
Modify app
asteel-gsa Sep 19, 2024
3c74e6f
add collectstatic back
asteel-gsa Sep 19, 2024
b3bc858
disable once again
asteel-gsa Sep 19, 2024
99860d5
Get things back to stable
asteel-gsa Sep 19, 2024
b114fe7
Update System
asteel-gsa Sep 20, 2024
2d7925e
Modify bin/ops files to run
asteel-gsa Sep 23, 2024
68e8f56
Create a complete initial README.md
asteel-gsa Sep 23, 2024
396c0a5
Update gitignore
asteel-gsa Sep 23, 2024
9c304d5
Delete terraform lock file
asteel-gsa Sep 23, 2024
fe64805
Have a dedicated https-proxy for sandbox
asteel-gsa Sep 23, 2024
3a1e0ba
Add acl for client registration
asteel-gsa Sep 23, 2024
83ca01d
add acl support for routes
asteel-gsa Sep 23, 2024
aeba8a3
Specify only the branch
asteel-gsa Sep 23, 2024
ec8f60a
Infra config improvements
asteel-gsa Sep 23, 2024
ea62c23
Ensure we have compiled assets
asteel-gsa Sep 25, 2024
b6edeea
Bump the timeout to 15m
asteel-gsa Sep 25, 2024
3558da0
Disable smtp-proxy (for now)
asteel-gsa Sep 25, 2024
986abf6
Re-enable API boot sequence
asteel-gsa Sep 26, 2024
0087cc1
Disable collectstatic in .profile
asteel-gsa Sep 26, 2024
bc6cd6b
CRLF -> LF
asteel-gsa Sep 26, 2024
48b257c
Fix command
asteel-gsa Sep 26, 2024
dc9532b
add MV
asteel-gsa Sep 26, 2024
2efd035
Executable
asteel-gsa Sep 26, 2024
bd6205a
Copy paste error
asteel-gsa Sep 26, 2024
7212df1
Update Readme
asteel-gsa Oct 2, 2024
5644454
terraform fmt
asteel-gsa Oct 2, 2024
ac19e42
Add sandbox.tfvars helper
asteel-gsa Oct 2, 2024
1081aee
Add helper script
asteel-gsa Oct 2, 2024
8af3a04
Update helper scripts
asteel-gsa Oct 3, 2024
fa700b9
Indent fix
asteel-gsa Oct 3, 2024
011bcb0
Update Readme
asteel-gsa Oct 4, 2024
513fe90
Update readme
asteel-gsa Oct 4, 2024
0b73e4b
Update readme
asteel-gsa Oct 4, 2024
ccc2274
Update Readme
asteel-gsa Oct 4, 2024
437a464
Execution bit, README edits
jadudm Oct 4, 2024
6e8aa19
cleanup helper scripts
asteel-gsa Oct 8, 2024
dc6f867
Add backup db
asteel-gsa Oct 8, 2024
9b4e753
Enable partial config with s3 to store tfstate
asteel-gsa Oct 8, 2024
67d3590
Update Readme
asteel-gsa Oct 8, 2024
66e65aa
Update Readme
asteel-gsa Oct 11, 2024
a8e8b44
Fix cat
asteel-gsa Oct 15, 2024
6c8c7b6
Uncomment Destroy
asteel-gsa Oct 15, 2024
51669eb
Fix proxy
asteel-gsa Oct 15, 2024
360ffb5
Revert
asteel-gsa Oct 15, 2024
dcb6e7c
Update Readme
asteel-gsa Oct 17, 2024
e5e6931
Update Readme
asteel-gsa Oct 17, 2024
c05a6c3
Update Readme
asteel-gsa Oct 17, 2024
02fa11c
Update Readme
asteel-gsa Oct 17, 2024
684af23
Update Readme
asteel-gsa Oct 21, 2024
e050752
Update Readme
asteel-gsa Oct 21, 2024
a133616
Update Readme
asteel-gsa Oct 21, 2024
cc324e4
Test fixing migrations
asteel-gsa Oct 21, 2024
ca39751
revert
asteel-gsa Oct 21, 2024
a7e72de
Update Readme
asteel-gsa Oct 25, 2024
b2b6682
Undo smtp proxy change
asteel-gsa Oct 28, 2024
9d8b7f1
CRLF -> LF
asteel-gsa Oct 28, 2024
c7052e9
CRLF -> LF
asteel-gsa Oct 28, 2024
d4e8c16
CRLF -> LF
asteel-gsa Oct 28, 2024
d0e7a79
Cleanup .profile
asteel-gsa Oct 28, 2024
e3898ca
CRLF -> LF
asteel-gsa Oct 28, 2024
bee6a88
Regenerate Examples with Meta Module
asteel-gsa Nov 5, 2024
62b41e8
Upgrade Provider
asteel-gsa Nov 5, 2024
5a0df3e
Module Upgrades
asteel-gsa Nov 5, 2024
29e1183
terraform fmt
asteel-gsa Nov 5, 2024
9dc2697
Update provider
asteel-gsa Nov 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,14 @@ terraform/**/*.tfstate*
terraform/**/*.tfvars
terraform/shared/modules/egress-proxy/proxy.zip
terraform/shared/modules/egress-proxy/test/client.zip
terraform/shared/modules/app/app.zip
terraform/shared/modules/https-proxy/proxy.zip
terraform/shared/modules/https-proxy/.terraform.lock.hcl
terraform/shared/modules/stream-proxy/proxy.zip
terraform/shared/modules/https-proxy/.terraform.lock.hcl
terraform/shared/modules/sandbox-proxy/proxy.zip
terraform/shared/modules/sandbox-proxy/.terraform.lock.hcl


# XLSX ignores
.~*#
Expand Down
14 changes: 9 additions & 5 deletions backend/.profile
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#!/bin/bash

# Source everything; everything is now a function.
# Remember: bash has no idea if a function exists,
# Remember: bash has no idea if a function exists,
# so a typo in a function name will fail silently. Similarly,
# bash has horrible scoping, so use of `local` in functions is
# bash has horrible scoping, so use of `local` in functions is
# critical for cleanliness in the startup script.
source tools/util_startup.sh
# This will choose the correct environment
Expand All @@ -14,6 +14,7 @@ source tools/migrate_app_tables.sh
source tools/api_standup.sh
source tools/run_collectstatic.sh
source tools/seed_cog_baseline.sh
source tools/materialized_views.sh

#####
# SETUP THE CGOV ENVIRONMENT
Expand Down Expand Up @@ -43,15 +44,18 @@ if [[ "$CF_INSTANCE_INDEX" == 0 ]]; then
#####
# COLLECT STATIC
# Do Django things with static files.
run_collectstatic
gonogo "run_collectstatic"
# run_collectstatic
# gonogo "run_collectstatic"

#####
# SEED COG/OVER TABLES
# Setup tables for cog/over assignments
seed_cog_baseline
gonogo "seed_cog_baseline"

# materialized_views
# gonogo "materialized_views"
fi

# Make psql usable by scripts, for debugging, etc.
alias psql='/home/vcap/deps/0/apt/usr/lib/postgresql/*/bin/psql'
alias psql='/home/vcap/deps/0/apt/usr/lib/postgresql/*/bin/psql'
3 changes: 1 addition & 2 deletions backend/config/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,8 +227,7 @@

# Environment specific configurations
DEBUG = False

if ENVIRONMENT not in ["DEVELOPMENT", "PREVIEW", "STAGING", "PRODUCTION"]:
if ENVIRONMENT not in ["SANDBOX", "DEVELOPMENT", "PREVIEW", "STAGING", "PRODUCTION"]:
DATABASES = {
"default": env.dj_db_url(
"DATABASE_URL", default="postgres://postgres:password@0.0.0.0/backend"
Expand Down
9 changes: 9 additions & 0 deletions backend/tools/materialized_views.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
source tools/util_startup.sh

function materialized_views {
startup_log "RUN_MATERIALIZEDVIEWS" "BEGIN"
python manage.py materialized_views --create &&
local result=$?
startup_log "RUN_MATERIALIZEDVIEWS" "END"
return $result
}
2 changes: 1 addition & 1 deletion bin/ops/create_service_account.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,4 @@ cat << EOF

cf_user = $username
cf_password = $password
EOF
EOF
2 changes: 1 addition & 1 deletion bin/ops/destroy_service_account.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,4 @@ cf target -o $org -s $space
cf delete-service-key $service ${service}-key -f

# destroy service
cf delete-service $service -f
cf delete-service $service -f
70 changes: 70 additions & 0 deletions bin/ops/get_service_account.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
#!/usr/bin/env bash

org="gsa-tts-oros-fac"

usage="
$0: Create a Service User Account for a given space

Usage:
$0 -h
$0 -s <SPACE NAME> -u <USER NAME> [-r <ROLE NAME>] [-o <ORG NAME>]

Options:
-h: show help and exit
-s <SPACE NAME>: configure the space to act on. Required
-u <USER NAME>: set the service user name. Required
-r <ROLE NAME>: set the service user's role to either space-deployer or space-auditor. Default: space-deployer
-o <ORG NAME>: configure the organization to act on. Default: $org
"

set -e
set -o pipefail

space=""
service=""
role="space-deployer"

while getopts ":hs:u:r:o:" opt; do
case "$opt" in
s)
space=${OPTARG}
;;
u)
service=${OPTARG}
;;
r)
role=${OPTARG}
;;
o)
org=${OPTARG}
;;
h)
echo "$usage"
exit 0
;;
esac
done

if [[ $space = "" || $service = "" ]]; then
echo "$usage"
exit 1
fi

>&2 echo "Targeting org $org and space $space"
cf target -o $org -s $space > /dev/null 2>&1

# get service key
cf service-key $service ${service}-key > /dev/null 2>&1

# output service key to stdout in secrets.auto.tfvars format
creds=`cf service-key $service ${service}-key | tail -n 7`
username=`echo $creds | jq '.credentials.username'`
password=`echo $creds | jq '.credentials.password'`

cat <<EOM
# generated with $0 -s $space -u $service -r $role -o $org
# revoke with $(dirname $0)/destroy_service_account.sh -s $space -u $service -o $org

cf_user = $username
cf_password = $password
EOM
13 changes: 7 additions & 6 deletions terraform/.gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
*/backend.tfvars
*/.terraform
*/terraform.tfstate
*/terraform.tfstate.backup
*/deployer-creds.auto.tfvars
shared/config/backend.tfvars
*/backend.tfvars
*/.terraform
*/terraform.tfstate
*/terraform.tfstate.backup
*/deployer-creds.auto.tfvars
shared/config/backend.tfvars
*/*.tfplan
9 changes: 9 additions & 0 deletions terraform/dev/dev.tf-example
Original file line number Diff line number Diff line change
@@ -1,3 +1,12 @@
# The content of this file is managed by Terraform. If you modify it, it may
# be reverted the next time Terraform runs. If you want to make changes, do it
# in ../meta/bootstrap-env/templates.

# Add resources to this module describing what you want in the corresponding
# space in cloud.gov. You should probably just reference a shared module as in the
# example below to keep consistency across spaces, but you can also vary the
# content for each environment as needed.

module "dev" {
source = "../shared/modules/env"
cf_space_name = "dev"
Expand Down
5 changes: 5 additions & 0 deletions terraform/preview/preview.tf-example
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@
# be reverted the next time Terraform runs. If you want to make changes, do it
# in ../meta/bootstrap-env/templates.

# Add resources to this module describing what you want in the corresponding
# space in cloud.gov. You should probably just reference a shared module as in the
# example below to keep consistency across spaces, but you can also vary the
# content for each environment as needed.

module "preview" {
source = "../shared/modules/env"
cf_space_name = "preview"
Expand Down
9 changes: 9 additions & 0 deletions terraform/production/production.tf-example
Original file line number Diff line number Diff line change
@@ -1,3 +1,12 @@
# The content of this file is managed by Terraform. If you modify it, it may
# be reverted the next time Terraform runs. If you want to make changes, do it
# in ../meta/bootstrap-env/templates.

# Add resources to this module describing what you want in the corresponding
# space in cloud.gov. You should probably just reference a shared module as in the
# example below to keep consistency across spaces, but you can also vary the
# content for each environment as needed.

module "production" {
source = "../shared/modules/env"
cf_space_name = "production"
Expand Down
85 changes: 85 additions & 0 deletions terraform/sandbox/.terraform.lock.hcl

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading