Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can we run opentransit-metrics/kubernetes/compute-new-cronjob.yaml as github action? #14

Open
sidetrackedmind opened this issue Mar 17, 2022 · 3 comments
Assignees
Labels
backend Issues related to the opentransit-metrics backend.

Comments

@sidetrackedmind
Copy link
Collaborator

Background

In order for the web app to show recent/current statistics, opentransit-metrics/backend/compute_new.py needs to run. This script pulls down the latest vehicle location data from S3 and runs compute_arrivals and compute_stats. The resultant data is put back in S3 for the web app to pick up and use to create data visualizations.

The compute_new.py used to run on Google Cloud (I think) using this yml:
https://github.com/codeforpdx/opentransit-metrics/blob/aba8f77506d93d91f68121569be58974ecdd130a/kubernetes/compute-new-cronjob.yaml

Issue / Ask

Is it possible to run this yml or run compute_new.py via github actions? From that doc, it looks like the first 2,000 minutes / month are free (or 3,000 if Code for PDX has a pro account):
image

If Github actions are not possible, what's the cheapest way to run that command daily? Can it run on the same heroku as the web app late at night?

@sidetrackedmind
Copy link
Collaborator Author

@sidetrackedmind
Copy link
Collaborator Author

Apologies for the rushed writeup @youngj . Let us know what you find out about the heroku scheduler 🙌

@sidetrackedmind sidetrackedmind added the backend Issues related to the opentransit-metrics backend. label Mar 18, 2022
@sidetrackedmind
Copy link
Collaborator Author

I created a github action in a forked version of this repo on my person github here - https://github.com/sidetrackedmind/opentransit-metrics-fork/blob/master/.github/workflows/run_compute.yml. I tested it and it seems to work. I'm going to let it run for a few days (weeks?) to get a sense for the timing. My main concern is going over the 20K minute free. I think we'll be fine 🤞 because the process takes ~10 mins. So 10 min / day * 31 days = 3,100 minutes.

I also just added another action to get schedule data weekly - https://github.com/sidetrackedmind/opentransit-metrics-fork/blob/master/.github/workflows/run_save_routes.yml.

If they run smoothly for a while and are not at risk of putting us over the free limit, I'll pull them into this repo so all the actions can run together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Issues related to the opentransit-metrics backend.
Projects
None yet
Development

No branches or pull requests

2 participants