From 6993f67ffba57e8e716c165ffa510d3aa5e70615 Mon Sep 17 00:00:00 2001 From: Mike Date: Tue, 6 Jun 2023 13:20:30 -0500 Subject: [PATCH 1/2] Add instructions for local dev with local VZ stack --- atd-etl/README.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/atd-etl/README.md b/atd-etl/README.md index cc861f2be..53967e7a6 100644 --- a/atd-etl/README.md +++ b/atd-etl/README.md @@ -14,9 +14,9 @@ In short, Docker allows us to have a fully built container with everything alrea #### 1. Download the environment variable files. -The ETL container depends on two specific files: `etl.production.env` and `etl.staging.env`. +The ETL container depends on three specific files: `etl.production.env`, `etl.staging.env`, and `etl.local.env`. -The files are located in 1Password and you can find the two files if you type `ETL` in the search bar. +The files are located in 1Password and you can find the two files if you type `ATD Splinter ETL` in the search bar. Once you download the files, put them a safe folder of your choice, example: @@ -34,6 +34,7 @@ $ chmod 600 ~/.ssh/atd-etl/*; $ ls -lha ~/.ssh/atd-etl/*; -rw------- 1 owner group 713B Nov 12 13:01 ~/.ssh/atd-etl/etl.production.env -rw------- 1 owner group 718B Nov 10 22:05 ~/.ssh/atd-etl/etl.staging.env + -rw------- 1 owner group 718B Nov 10 22:05 ~/.ssh/atd-etl/etl.local.env ``` If the file shows `-rw-------` (read-write access for you only) you did it correctly. @@ -49,10 +50,19 @@ Follow this example: ```bash $ runetl build +``` +Then, for local: +```bash +$ runetl ~/.ssh/atd-etl/etl.local.env app/process_test_run.py +``` +For staging: +```bash $ runetl ~/.ssh/atd-etl/etl.staging.env app/process_test_run.py +``` +For production: +```bash $ runetl ~/.ssh/atd-etl/etl.production.env app/process_test_run.py ``` - ## Development You should be able to make changes to the files (without need to re-build the container every time), just save your changes to the python file and then re-run the runetl command as needed. From 82b89447b61bdbd09748bbac162ba971640f124e Mon Sep 17 00:00:00 2001 From: Mike Date: Tue, 6 Jun 2023 16:44:40 -0500 Subject: [PATCH 2/2] Add detail about this only working locally for the Socrata ETL --- atd-etl/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/atd-etl/README.md b/atd-etl/README.md index 53967e7a6..c6f30446e 100644 --- a/atd-etl/README.md +++ b/atd-etl/README.md @@ -51,7 +51,7 @@ Follow this example: ```bash $ runetl build ``` -Then, for local: +Then, for local (only supports `process_socrata_export.py` for now): ```bash $ runetl ~/.ssh/atd-etl/etl.local.env app/process_test_run.py ```