Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrations for cr3 location triggers #1290

Merged
merged 31 commits into from
Sep 25, 2023

Conversation

roseeichelmann
Copy link
Contributor

@roseeichelmann roseeichelmann commented Sep 12, 2023

Associated issues

Closes cityofaustin/atd-data-tech#13055

Testing

URL to test:
Local

Steps to test:

  1. Use the readme in the /atd-vzd/ folder to get your local stack up and running with production data and the latest migrations and metadata applied
  2. Use the Hasura CLI instructions in that readme to apply the migrations and metadata in this branch. From /atd-vzd/:
hasura migrate apply --envfile .env.local
hasura metadata apply --envfile .env.local
  1. Follow the test steps 2-4 (the first step of those instructions is to manually make the changes that these migrations do for us) in Replace Lambda function location_from_latlon with stored generated column #1270
  2. You can also run ./vision-zero hasura-console to open your local Hasura console and make sure that the Event Trigger called location_from_latlon is no longer present in Hasura.
  3. If you want to test the down migration, run:

hasura migrate apply --down 1 --envfile .env.local


Ship list

  • Run migration steps in readme in /atd-vzd/ if needed
  • Code reviewed
  • Product manager approved

Copy link
Contributor

@mddilley mddilley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the test steps work perfectly! Just a couple small fixes and this will be 🚢 🚢 🚢 🙌

Comment on lines 5 to 6
- "!include public_find_location_for_noncr3_collision.yaml"
- "!include public_find_location_for_cr3_collision.yaml"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran into an inconsistent metadata warning when I applied the Hasura metadata locally. The first line added back here is a function that was dropped in #1286 and the second one is being dropped in this PR. These two lines can go since Hasura doesn't need to worry about tracking or not tracking them anymore.

Comment on lines 17 to 19
DROP FUNCTION find_cr3_mainlane_crash;
DROP FUNCTION find_location_for_cr3_collision;
DROP FUNCTION find_location_id_for_cr3_collision;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to make sure that we can migrate down and then back up again, these drop statements could use IF EXISTS since they aren't added back in the down migrations. I ran into this in my PR too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for catching this!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works great! I also tested nulling out latitude, longitude, latitude_primary, and longitude_primary because you caught that on mine, and I saw the location id clear out. 🚀

@roseeichelmann
Copy link
Contributor Author

@mddilley thanks for catching those things, just pushed up those updates!

@mddilley
Copy link
Contributor

@roseeichelmann awesome, just tested and there is a small bug with the syntax DROP IF EXISTS FUNCTION vs. DROP FUNCTION IF EXISTS. Ready to approve when that is patched up 🚀

@roseeichelmann
Copy link
Contributor Author

@mddilley ahh sorry i should have tested the migration again before pushing up lol but i just fixed that and tested the down/up and metadata, shoulddd be good to go now

Copy link
Contributor

@mddilley mddilley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So happy to see all your work pay off! 🙌 🥳 ✅

Copy link
Member

@frankhereford frankhereford left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So much delete!! I sure love seeing the events folder clearing out. 😅

This is superb, and it works great. So nice to see the fruits of the migration system already manifesting in such an organized and well presented change to the database. Thank you Rose!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rose, I'm curious what you think about if we need to keep these migration-like files in here with proper graphql-engine migration support in place. Like, I think now if I wanted to find what the latest and greatest version of this function is, I would probably get into the root of the repo and do:

find atd-vzd/migrations/default -name 'up.sql' -exec grep -l 'update_cr3_location' {} + | sort -r

The top line in the response is the most recent up.sql that mentions the string and going backwards is its history.

I totally think I may be missing something, and I'm super curious to what your reaction is.

CC/ @mddilley too - I'm curious if you have a take on this as well. Thanks to you both!!!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be misunderstanding what your comment is Frank, are you saying you don't see the need for this file because you could execute that command to find what the latest version is?

I do not work much in VZ, but I do find it easier that in moped we have the views in their own files to be able to easily see changes between migrations, and in moped the top of the file has what up.sql file is the latest aka
-- latest version 1694809058492_list_view_optimize

Copy link
Member

@frankhereford frankhereford Sep 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Chia, thanks for your comment. I'm sorry for not being very clear, but I think you've hit on it exactly. I'll try to add some more details, but first, I'll say that leaving the file in is totally fine if it's useful for devs. 👍

To your question about the top of the file -- what that command will do is tell you what files to look in, but not actually show you the contents of the file. You still need to open it up manually to read it.

The reason why I like to look in the migrations for the current state of the database is that they are what was actually applied to the database, so I feel like they have a very high likelihood of being accurate. We've struggled in the past with the VZ repo to keep all the "migration-like" files up to date when a change went in, mostly because changes were hand applied and it was easy for people to forget to update the views/triggers and what-have-you when they made the change, especially if it was an iterative process.

Now, that said, we've gotten a heck of a lot better at it, so if we continue to excel at that recording of the view and the files are helpful, then I'm all in for keeping them.

Thanks for asking!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, I understood that it would tell you what file to look in, but not the contents. In moped we manually add that line to the top of the git tracked file so devs can see what the latest migration is, and having the file is to help at a glance see changes in the PR from one migration to another.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah im cool with leaving this file out of the repo, I honestly was kinda just following Mikes lead and saw that he left his trigger function in his similar PR for the blueform crashes so figured I would do the same thing, but I agree that it's not necessary and maybe will lead to confusion now? What do you think @mddilley

Copy link
Contributor

@mddilley mddilley Sep 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey y'all - late to the party, but I think that we can leave it in for now and decide later so this can 🚢. I agree with Chia that the Moped views folder is great for seeing a diff during reviews, and we've succeeded at keeping files up to date over time in that project. I created cityofaustin/atd-data-tech#13950 to bring to a future refinement meeting.

Copy link
Member

@patrickm02L patrickm02L left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Send it 🥏

@roseeichelmann roseeichelmann merged commit 56bb92d into master Sep 25, 2023
9 checks passed
@roseeichelmann roseeichelmann deleted the 13055_location_id_trigger_migration branch September 25, 2023 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Replace Lambda function location_from_latlon
5 participants