VZE | Download CR3 "NoSuchKey The specified key does not exist" error #1272
Conversation
roseeichelmann
requested review from
frankhereford,
chiaberry,
johnclary,
tillyw,
mddilley and
patrickm02L
| crashes_with_cr3_flag_list = [] | ||
| for crash in crashes_with_cr3_flag: | ||
| crashes_with_cr3_flag_list.append(str(crash["crash_id"])) | ||
| crashes_to_update = list(set(crashes_with_cr3_flag_list) - set(crashes_with_cr3s)) |
There was a problem hiding this comment.
oooo nice that you used sets here 🚀
| "Writing names of cr3 pdfs we have in s3 to a new file in this directory called 's3_cr3_file_names.txt', this can take a few minutes..." | ||
| ) | ||
| aws = subprocess.Popen( | ||
| ["aws", "s3", "ls", "s3://atd-vision-zero-editor/production/cris-cr3-files/"], |
There was a problem hiding this comment.
cool to see an alternative to using boto3. Using that library would probably have produced more code too.
There was a problem hiding this comment.
lol i didnt even think to use that library/didnt know about it!
There was a problem hiding this comment.
Oh cool - totally valid either way! 😎
There was a problem hiding this comment.
Review
Approved. Thanks for running the script in Prod & Staging. Glad we have a new 🔧 for the 🧰
Testing
Testing the example crashes from Issue #13021 in Production worked
https://visionzero.austin.gov/editor/#/crashes/19662002
https://visionzero.austin.gov/editor/#/crashes/19659622
I don't think those crashes are in Staging. I put the crash ID in the URL and also did a search and couldn't find them.
I did not test locally
Associated issues
Closes cityofaustin/atd-data-tech#13021'
This is a script for our toolbox for fixing the above error in the VZE which is caused by crashes having the cr3_stored_flag set to Y when we dont actually have a CR3 for them in our AWS S3 bucket. This script updates that flag for those crashes to be N which allows the CR3 downloader to download the missing CR3s when it next runs. Follow up issue is to figure out why this desyncing happened in the first place.
I already ran the script for staging & prod which fixed the issue, this PR is just to review the code to put in our toolbox for potential future needs if this ever happens again.
Testing
URL to test:
You can test this script locally if youd like!
Steps to test:
Ship list