Local Socrata ETL development

[mddilley]

Associated issues

This is purely to unblock local development of the Socrata ETL. Thanks to @tillyw and @frankhereford for the discussion on this. I realized last night that I had started exploring this a while back but forgot. The biggest update here is that there is a new environment file in 1Password that hands the ETL details that point it to the local VZ DB so it can populate the test datasets. Since the data source is still production data, the test datasets look production-like but also makes schema change testing possible.

This should hold us over until we migrate this to Airflow v2 and refactor it a little.

Testing

URL to test:
n/a

Steps to test:
You can check that this worked by going to https://data.austintexas.gov/Transportation-and-Mobility/-UNDER-CONSTRUCTION-TEST-Crash-Report-Data/3aut-fhzp and filtering the dataset to micromobility_fl = Y.

or

1. Start up the local VZ stack and replicate today's backup
2. Add these two rows to the atd__mode_category_lkp table:
   • id: 10, atd_mode_category_mode_name: Micromobility device
   • id: 11, atd_mode_category_mode_name: E-scooter
3. Apply the database changes added in https://github.com/cityofaustin/atd-vz-data/pull/1233/files
   • get_crash_modes.sql
   • atd_txdot_units_mode_category_metadata_update.sql
   • atd_txdot_units_create_update.sql
   • migration_atd_txdot_units_2023_06_29--1509.sql
   • migration_atd_txdot_crashes_mode_category_metadata_2023_06_30--1624.sql
4. Run the ETL locally using the steps in this readme

$ runetl ~/.ssh/atd-etl/etl.local.env app/process_socrata_export.py

5. Check the Socrata dataset https://data.austintexas.gov/Transportation-and-Mobility/-UNDER-CONSTRUCTION-TEST-Crash-Report-Data/3aut-fhzp for the updated metadata

---

Ship list

• Code reviewed
• Product manager approved

[mddilley]

@tillyw The staging ETL overwrote the micromobility data in the test datasets over the weekend so I replaced the data again and turned off the Airflow DAG so it will stick around this time.

[chiaberry]

I see the Ys in Socrata, ✅

[patrickm02L]

That's a Y for me! 🛴

[roseeichelmann]

Looks good, I see Y's! Just to be sure, there should only be the option to filter by "Y", right?

[mddilley]

@roseeichelmann Yes, Socrata creates those filter options based on the content of the dataset. The "Y" values are proof that something that is not in the production dataset can be pushed to the test dataset to develop ETL/VZV updates, and the values could change in the future depending on where the rest of Tilly's work goes. 🚀 🙌

Going to go ahead and merge this - thanks y'all!