Update the atd-vzd readme #1285
Conversation
| #FROM frankinaustin/postgis:14-3.3 | ||
| #FROM postgis/postgis:14-3.3 |
There was a problem hiding this comment.
We can keep this around if it is helpful to have a history of images here. I swept through the files this folder since I missed the readme in the migrations PR.
There was a problem hiding this comment.
Thanks for sweeping out this cruft! 🧹
atd-vzd/README.md
Outdated
| Currently the schema is being tracked in the following folders: | ||
| ## Backups | ||
|
|
||
| Currently there are two systems making backups, one in RDS and the other in S3 via CRON jobs for 60 days. The backups in S3 are table-based and contain both schema and data for each individual table. |
There was a problem hiding this comment.
I need to check in on this cron job described here. Not sure if we do that anymore.
There was a problem hiding this comment.
thats a good question, we have this issue open that i keep meaning to find time to discuss
cityofaustin/atd-data-tech#10276
There was a problem hiding this comment.
We had to turn down half cron'd backup, because it was depleting our EFS share of its burstable write capacity. While we were not using whole DB backup it was making, which included all the change log entries, we do regularly use the half that is still running -- that which makes the backup without the changelog included. The files it produces not only back up the DB in SQL form which could be very handy in a recovery situation, that same file can be dropped into the snapshots folder inside the VZD directory when replicating the database.
I think we do want to turn down this cron job, but only when we replace it with a airflow job that uses the limitless disk operations we have on our on-prem servers.
There was a problem hiding this comment.
Cool - thanks for the background, Frank! I was thinking that this was describing an outdated process. I'll update the language here to reflect what is currently going on.
mddilley
requested review from
frankhereford,
patrickm02L,
chiaberry,
johnclary,
tillyw and
roseeichelmann
|
|
||
| ## Pipeline | ||
|
|
||
| Currently there is no automated pipeline implemented yet; as of this iteration the modifications are made manually. | ||
| Changes to the schema and database are currently handled by manually applying migrations and metadata through the [Hasura CLI](https://hasura.io/docs/latest/hasura-cli/overview/). Run CLI commands from the local `atd-vzd` folder with a feature branch checked out. The CLI will often prompt you to choose which database to inspect (All or default). Default is the only database being tracked so either choice works. |
atd-vzd/README.md
Outdated
| Currently the schema is being tracked in the following folders: | ||
| ## Backups | ||
|
|
||
| Currently there are two systems making backups, one in RDS and the other in S3 via CRON jobs for 60 days. The backups in S3 are table-based and contain both schema and data for each individual table. |
There was a problem hiding this comment.
We had to turn down half cron'd backup, because it was depleting our EFS share of its burstable write capacity. While we were not using whole DB backup it was making, which included all the change log entries, we do regularly use the half that is still running -- that which makes the backup without the changelog included. The files it produces not only back up the DB in SQL form which could be very handy in a recovery situation, that same file can be dropped into the snapshots folder inside the VZD directory when replicating the database.
I think we do want to turn down this cron job, but only when we replace it with a airflow job that uses the limitless disk operations we have on our on-prem servers.
| #FROM frankinaustin/postgis:14-3.3 | ||
| #FROM postgis/postgis:14-3.3 |
There was a problem hiding this comment.
Thanks for sweeping out this cruft! 🧹
|
Thanks for reviewing this, y'all! Going to merge so that I can get #1286 up to date and testable. 🙏 Happy to make changes from any future eyes as well. |
Associated issues
This updates the atd-vzd readme to reflect our current Hasura migration and metadata update steps. I added couple of files to help us do the Hasura CLI stuff.
Testing
URL to test:
local
Steps to test:
Y'all can run through the steps if you want, but I think this one is mostly reading through the steps to see if I made sense and catching typos.
Ship list