v4.1.9 pre-release

[autoSteve]

Pre-release.

What's Changed

TL>DR:

• Granular dampening to dampen per half hour period by @autoSteve and @isorin
• Dampening applied at forecast fetch and not to forecast history @autoSteve and @isorin
• Retrieve un-dampened forecast values using service call by @autoSteve (thanks @Nilogax)
• Get presently set dampening factors using service call by @autoSteve (thanks @Nilogax)
• Migration of un-dampened forecast to un-dampened cache on startup by @autoSteve

Granular dampening

The only option to set dampening previously was at per-hour increments, and not the half hour forecast periods that Solcast provide to hobbyists. This can now not be a thing, because you can specify a total of 48 dampening values, corresponding to one for every half-hourly forecast period.

Plus, you can now dampen per Solcast site if desired, either hourly or at half-hourly resolution.

Implementation is by way of two methods. Either calling the set dampening service call, or by creating and maintaining a file of dampening factors. (More in the updated readme.)

Dampening at forecast fetch

Previously, dampening when seen was applied to all forecasts, but not saved in a cache for history. It was for display purposes only. Yes, the 730 days of history, today and the up-coming seven days were calculated and shown as dampened with the current values. Yuck. This is no more.

The history of dampened forecasts is now preserved, and only newly fetched forecasts have dampening adjusted for that refreshed data.

At 'day one' there will be no dampening applied to your history, but this will build over time. A nice segue into...

Retrieve un-dampened forecast values

Because history is now dampened as-at the dampening values set at the time, automations might actually need undampened data to calculate future dampening based on past performance. So the 'get forecasts' service call is extended to allow this by specifying undampened: true.

At upgrade, the past fourteen days of undampened history will be saved, allowing this modified call to be used immediately. Undampened forecasts get saved to a different cache than the dampened forecasts, and are kept at fourteen days of history. Additionally at upgrade, current 'today' and beyond forecasts will be dampened and saved to the dampened history cache, allowing a seamless display transition to this new mode of applying dampening.

Get currently set dampening values

To avoid an automation needing to keep track of dampening set, the current dampening values can be retrieved with a new service call.

Some bug fixes

There have also been some v4.1.8 corner-case bugs fixed that nobody noticed or reported.

Full Changelog: v4.1.8...v4.1.9

[autoSteve]

As has become tradition, notifying the usual early-adopting-and-reporting folk of v4.1.9 being set free. @dtswk, @EddieSpaghettie, @ffm777, @gcoan, @hwb45, @iainfogg, @isorin, @jeffb1974, @ladaan, @Markyt77, @Nilogax, @Noodleyman, @ProphetOfDoom, @rholligan, @sh00t2kill, @Unlik and @wimdebruyn.

Do ping if you would like to be off the list.

[BJReplay]

As is my tradition, I apply these changes over a VPN (Tailscale) while out (They Might Be Giants intermission) from my phone, and check that they worked by SSHing in to my WSL host from my phone.

All good.

[isorin]

I'm running 4.1.9 as well, with granular dampening loaded from the file, everything looks good, I've added a bit of extra debug logging (locally) and waiting for the next scheduled forecast update to check if the dampening is calculated correctly.

[rholligan]

Running 4.1.9, all looks good so far

[bert1111]

Somehow beta releases is active on my hacs and I installed version 4.1.9 because I got the update message. I see a few error messages in the logs. I set it up like in the example. One call using "force_update_forecasts" and I set auto update between sunrise and sunset with limit of 8. If you need more logs I'm happy to get them for you.

Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.solcast_solar.solcastapi
Bron: custom_components/solcast_solar/solcastapi.py:2358
integratie: Solcast PV Forecast (documentatie, problemen)
Eerst voorgekomen: 16:09:37 (1 gebeurtenissen)
Laatst gelogd: 16:09:37

API polling limit exhausted, not getting forecast for site x API used is 8/8`
`Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.solcast_solar.solcastapi
Bron: custom_components/solcast_solar/solcastapi.py:1986
integratie: Solcast PV Forecast (documentatie, problemen)
Eerst voorgekomen: 16:09:37 (1 gebeurtenissen)
Laatst gelogd: 16:09:37

Forecast update for site 4x failed so not getting remaining sites`
`Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.solcast_solar.solcastapi
Bron: custom_components/solcast_solar/solcastapi.py:2010
integratie: Solcast PV Forecast (documentatie, problemen)
Eerst voorgekomen: 16:09:37 (1 gebeurtenissen)
Laatst gelogd: 16:09:37

At least one site forecast failed to fetch, so forecast has not been built

[BJReplay]

Judging by the first error message, you had already used 8, 9, or even 10 calls out of your original limit of 10 calls when you set it up.

When you lowered your limit to 8, rather than attempt to make a call when you'd exceeded the now lowered limit, it reports that you've exceeded the limit (but the error message uses the same number for both the number used and the limit - so will report the lower number - even if you'd used 9 out of 10 previously, for a single site, it will report 8/8).

The second and third error messages follow from the first - if one forecast fails, then any others (even if there is only one - for a single site (array) forecast) are not fetched, so you'll see three error messages.

Chances are it will work correctly in 15 minutes from now (just after UTC midnight) when the usage count resets to zero).

If you see this reply before then, you can temporarily bump up your limit to 11 or 12 if you'd like) and this should allow the first fetches to work.

If you're not reading github discussion replies because it is the middle of the night, chances are it is working when you wake up.

[autoSteve]

Replicated.

As BJ explains, this can happen when transitioning to auto-updates if the existing automation uses a different period. It will only happen on the first day of transition, and also under certain configuration circumstances like reducing API limit from ten to eight (as an example), and probably also where midnight local is quite different to midnight UTC (as it is here in Australia ... +11 hours). The next day will not produce any "missed" forecasts.

Thanks for testing! All feedback is gratefully received, and I hope you enjoy the convenience of auto-update.

Here I reduced my API limit from ten to eight yesterday afternoon. This is what happened in the lead up to midnight UTC. (The 13:06:17 scheduled forecast will succeed uneventfully.)

2024-10-13 06:35:39.175 DEBUG (MainThread) [custom_components.solcast_solar.coordinator] Next forecast update scheduled for 2024-10-13 09:50:57
2024-10-13 09:50:00.311 DEBUG (MainThread) [custom_components.solcast_solar.coordinator] Forecast will update in 57 seconds
2024-10-13 09:50:57.312 DEBUG (MainThread) [custom_components.solcast_solar.coordinator] Checking for stale usage cache
2024-10-13 09:50:57.313 INFO (MainThread) [custom_components.solcast_solar.solcastapi] Getting forecast update for site b68d-c05a-c2b3-2cf9
2024-10-13 09:50:57.313 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Polling API for site b68d-c05a-c2b3-2cf9 lastday 2024-10-20 numhours 183
2024-10-13 09:50:57.315 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] API polling limit exhausted, not getting forecast for site b68d-c05a-c2b3-2cf9, API used is 8/8
2024-10-13 09:50:57.315 WARNING (MainThread) [custom_components.solcast_solar.solcastapi] Forecast update for site b68d-c05a-c2b3-2cf9 failed so not getting remaining sites
2024-10-13 09:50:57.315 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] At least one site forecast failed to fetch, so forecast has not been built
2024-10-13 09:50:57.335 DEBUG (MainThread) [custom_components.solcast_solar.coordinator] Next forecast update scheduled for 2024-10-13 13:06:17
2024-10-13 11:00:00.160 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Reset API usage
2024-10-13 11:00:00.161 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Writing API usage cache file: /config/solcast-usage.json```

[BJReplay]

Replicated.

I saw this when testing v4.1.8, but knew immediately what had happened, and burned some excess API calls to verify 🤣 and didn't think to mention it. I figured it's a corner case, but now that I think about it, it possibly isn't: if people update after sunset or before sunrise (e.g. when they've used all calls in a day that does align with a UTC day), especially at higher latitudes in the northern winter, it may be more common than not.

[autoSteve]

I've re-examined the code, @bert1111, and you have discovered a bug in the auto-update mechanism.

On both fetch and force fetch the API counter will be incremented. This should not be the case for a forced fetch, as the integration now relies on being able to set a reduced limit, and compare to an API use counter that must be in line with auto-update fetches.

Good pick up.

This will be fixed in the release version 4.2.0.

[autoSteve]

Bottom line:

There was a force fetch bug where it should not increment the API counter, and secondly there is a case when reducing the API limit could lead to a situation where the internal limit counter gets hit and fetch fails (which will sort itself at UTC midnight).

The second case is now documented in the readme. The force fetch bug is now fixed for 4.2.0 GA.

[dtswk]

Upgraded no issues, appears fine, no errors. Another boring release ;). I'm seriously impressed at how well these upgrades work.

[BJReplay]

Thanks. Your testing of this and the previous pre-release give us confidence that when we release to the larger community shortly that it should go well. It is appreciated.

[autoSteve]

I aim for "cool and useful boring", @dtswk as I am sure @BJReplay does, so this is music to my ears. Thanks for reporting nothing. It is as important as reporting something.

[autoSteve]

v4.2.0 is out.