Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Local Socrata ETL development #1252

Merged
merged 3 commits into from
Jul 18, 2023
Merged

Local Socrata ETL development #1252

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6993f67
Add instructions for local dev with local VZ stack
mddilley Jun 6, 2023
82b8944
Add detail about this only working locally for the Socrata ETL
mddilley Jun 6, 2023
32e4684
Merge branch 'master' into md_local_etl_dev
mddilley Jul 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions atd-etl/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ In short, Docker allows us to have a fully built container with everything alrea

#### 1. Download the environment variable files.

The ETL container depends on two specific files: `etl.production.env` and `etl.staging.env`.
The ETL container depends on three specific files: `etl.production.env`, `etl.staging.env`, and `etl.local.env`.

The files are located in 1Password and you can find the two files if you type `ETL` in the search bar.
The files are located in 1Password and you can find the two files if you type `ATD Splinter ETL` in the search bar.

Once you download the files, put them a safe folder of your choice, example:

Expand All @@ -34,6 +34,7 @@ $ chmod 600 ~/.ssh/atd-etl/*;
$ ls -lha ~/.ssh/atd-etl/*;
-rw------- 1 owner group 713B Nov 12 13:01 ~/.ssh/atd-etl/etl.production.env
-rw------- 1 owner group 718B Nov 10 22:05 ~/.ssh/atd-etl/etl.staging.env
-rw------- 1 owner group 718B Nov 10 22:05 ~/.ssh/atd-etl/etl.local.env
```

If the file shows `-rw-------` (read-write access for you only) you did it correctly.
Expand All @@ -49,10 +50,19 @@ Follow this example:

```bash
$ runetl build
```
Then, for local (only supports `process_socrata_export.py` for now):
```bash
$ runetl ~/.ssh/atd-etl/etl.local.env app/process_test_run.py
```
For staging:
```bash
$ runetl ~/.ssh/atd-etl/etl.staging.env app/process_test_run.py
```
For production:
```bash
$ runetl ~/.ssh/atd-etl/etl.production.env app/process_test_run.py
```

## Development

You should be able to make changes to the files (without need to re-build the container every time), just save your changes to the python file and then re-run the runetl command as needed.
Expand Down
Loading