Technology breakdown by-suburb over time

[lyricnz]

Like the current overall tallies/chart, but per-suburb.
https://forums.whirlpool.net.au/thread/3nklkn29?p=503#r10058

[LukePrior]

I think it's a good idea to modify the code such that generating such data is easily possible.

I'm not sure if it's worth storing all this data as it will quickly balloon the size of this repository but I definitely see the value behind it will have a think.

[lyricnz]

I already write the code, it's a fairly trivial change to the tech-breakdown code:

[lyricnz]

(yes, there's a bit more code around this to pull the new attribute, and write it to a different file, so the format/size of the current isn't changed; and updates to the GHA-run script to do the same; but that's the general idea)

[lyricnz]

Will have a PR shortly (just running the historical data)

[lyricnz]

It's about 1.2Mbyte per week

❯ jq '.["2023-08-01"]' results/breakdown-suburbs.json | wc
   67481  125613 1155430

Probably not "worth it". gzip gets about 5:1 on json like this.

[LukePrior]

Hmmm actually possibly we could generate and publish this file using the release function of GitHub solves the repo size issue.

[lyricnz]

The data (from 23/5 until now) is 20MB uncompressed. Took about 30 minutes to generate from git history. Updating it wouldn't add any time to the existing (weekly) process.

{
    "2023-05-23": {
        "ACT": {
            "Banks": {
                "FTTN": 1456,
                "FTTC": 428,
                "FTTP": 2
            }
        },
        "NSW": {
            "Liverpool": {
                "FTTP": 12490,
                "HFC": 1227,
                "FTTB": 666,
                "FTTN": 2,
                "FTTC": 60,
                "NULL": 33
            },
            "Wauchope": {
                "FTTN": 3138,
                "FTTP": 178,
                "WIRELESS": 35,
                "FTTC": 33,
                "SATELLITE": 3,
                "NULL": 15
            },

[lyricnz]

I don't know about that release function, how to run it etc.

[lyricnz]

One of our existing current geojson files is already 30M, I don't think 20M is a big deal?

❯ find results -name \*.geojson -size +10M | xargs ls -lh
-rw-r--r--@ 1 simonroberts  staff    13M 21 Sep 13:58 results/NSW/sydney.geojson
-rw-r--r--@ 1 simonroberts  staff    30M 27 Sep 12:38 results/VIC/melbourne.geojson
-rw-r--r--@ 1 simonroberts  staff    10M 27 Sep 12:38 results/VIC/southbank.geojson

[LukePrior]

You can see what I mean about releases here: https://github.com/LukePrior/open-energy-tracker/releases (note some logic unrelated to the daily release is failing that's why actions aren't fully running)

We could do a weekly release and then always grab the latest for the website.

https://docs.github.com/en/repositories/releasing-projects-on-github/linking-to-releases#linking-to-the-latest-release

[lyricnz]

(we still have a bunch of open issues to minimise, and/or compress the geojson files)

[LukePrior]

Yeah I think the file size should be fine either way however.

[lyricnz]

OK, pushed the json file into the PR. I'll leave the charting to you :)

[kiwisg124]

I would encourage this. We may be able to answer the question whether the orderable dates announced by NBN are realistic. Some suburbs have slow takeup for economic affordability reasons (closed subscriber wallets), but others due to delays for civil works ? I am basically interested in timelines and progress. Whether the 1.5BN will run out without all sites upgraded in the announced suburbs.

[LukePrior]

This has been merged in and will be generated going forwards, but it might be a while until I have time to get the website updated to show it.