v4.2.3

[autoSteve]

What's Changed

If you missed upgrading to v4.2.0 then you missed a lot of additions in past release notes, including bringing back auto-update. Check out the readme.

• Fix an issue that causes changing Solcast accounts to fail by @autoSteve
• Fix an issue with multi-API key where API usage reset was not handled correctly by @autoSteve
• Fix an issue with enabled detailed site breakdown for hourly attributes by @autoSteve
• Code clean-up and some refactoring by @autoSteve
• v4.2.1 had a major, but one-line issue. Release pulled.
• v4.2.2 had a major, but one-line issue. Release pulled.

Full Changelog: v4.2.0...v4.2.3

[autoSteve]

Sorry 'bout that folks. Let's do that again, and again... Please, no more.

Notifying the usual early-adopting-and-reporting folk about v4.2.3 being released. @dtswk, @EddieSpaghettie, @ffm777, @gcoan, @hwb45, @iainfogg, @isorin, @jeffb1974, @ladaan, @Markyt77, @Nilogax, @Noodleyman, @ProphetOfDoom, @rholligan, @sh00t2kill and @wimdebruyn.

As always, do ping if you would like to be off the list.

Notifying @Cycor and @Ben00793 who hit issues with v4.2.0.

[Unlik]

Hello @autoSteve, may I ask you to remove me from the list? Tx.

[BJReplay]

All good here.

I didn't get to try .22

Now to work out why my tariff automations are spamming my logs...

[dtswk]

Upgraded no errors waiting on an update.

Hang in there gentleman, sometimes things just go sideways. Very much appreciate the efforts.

Matt

[AdamGilchrist317]

All good!

[isorin]

Upgraded from 4.2.0 to 4.2.3, all good.

[dtswk]

15 updates so far and seems to be working fine.

[gcoan]

I'm still on 4.2.0 as HACS is on a go-slow in telling me to upgrade, but an observation of behaviour.

4 API's (out of 10) used so far I turned on auto updates between sunrise and sunset and set the API limit to 8 (as I still need a midnight-ish update).

Log produced:

2024-10-20 09:57:46.368 INFO (MainThread) [custom_components.solcast_solar.solcastapi] Usage loaded and cache updated with new limit
2024-10-20 09:57:46.368 INFO (MainThread) [custom_components.solcast_solar] Solcast integration version: v4.2.0
2024-10-20 09:57:46.893 INFO (MainThread) [custom_components.solcast_solar.coordinator] Auto-update will update forecasts 4 times between sunrise and sunset
2024-10-20 09:57:46.893 INFO (MainThread) [custom_components.solcast_solar.coordinator] Auto-scheduled forecast update for today at 10:09:17, 12:44:53, 15:20:29
2024-10-20 09:57:46.893 INFO (MainThread) [custom_components.solcast_solar.coordinator] Auto-scheduled forecast update for tomorrow at 07:35:27, 10:10:06, 12:44:45, 15:19:24

Neat.

The integration has scheduled 4x updates tomorrow and has adjusted the calls and times for today based on the API's already used. Good edge case proven.

[Ben00793]

Getting this today again. First 2 call happened flawlessly, the third call crapped out same as yesterday.
Something must be happening with solcast. Hopefully it will update agian later.

This error originated from a custom integration.

Logger: custom_components.solcast_solar.solcastapi
Source: custom_components/solcast_solar/solcastapi.py:2373
integration: Solcast PV Forecast (documentation, issues)
First occurred: October 19, 2024 at 12:32:31 PM (27 occurrences)
Last logged: 9:58:12 AM

API returned 'try later' (status 429), pausing 78 seconds before retry
API returned 'try later' (status 429), pausing 104 seconds before retry
API returned 'try later' (status 429), pausing 110 seconds before retry
API returned 'try later' (status 429), pausing 125 seconds before retry
API returned 'try later' (status 429), pausing 144 seconds before retry

[gcoan]

same story for me #187 (comment)

several of yesterday and today's scheduled updates are timing out after 10 retries

[Nilogax]

Same here. I've seen more 4290's (thats 10 x 429 😉) in the last week or so than since the auto retry was introduced.

Oh and 4.2.3 running OK so far for me.

[Ben00793]

I'm gonna be testing 4.2.3 with auto updates turned off, and using the old automation that I had with 4.1.3

alias: Solcast update sunlight
description: ""
mode: single
triggers:
  - value_template: >-
      {% set nr = as_datetime(state_attr('sun.sun','next_rising')) | as_local %}
      {% set ns = as_datetime(state_attr('sun.sun','next_setting')) | as_local
      %} {% set api_request_limit = 10 %} {% if nr > ns %}
        {% set nr = nr - timedelta(hours = 24) %} 
      {% endif %} {% set hours_difference = (ns - nr) %} {% set interval_hours =
      hours_difference / api_request_limit %} {% set ns = namespace(match =
      false) %} {% for i in range(api_request_limit) %}
        {% set start_time = nr + (i * interval_hours) %}
        {% if ((start_time - timedelta(seconds=30)) <= now()) and (now() <= (start_time + timedelta(seconds=30))) %}
          {% set ns.match = true %}
        {% endif %}
      {% endfor %} {{ ns.match }}
    trigger: template
conditions:
  - condition: sun
    before: sunset
    after: sunrise
actions:
  - delay:
      seconds: "{{ range(30, 360)|random|int }}"
  - data: {}
    action: solcast_solar.update_forecasts

[hwb45]

Several of mine had a couple of retries over the last three days but gets a forecast and the one at 09:30 ish always fails after 10 retries. It has to be a Solcast issue.Sent from my iPadOn 20 Oct 2024, at 10:32, Geoffrey Coan ***@***.***> wrote: same story for me #187 (comment) several of yesterday and today's scheduled updates are timing out after 10 retries —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[autoSteve]

Things are running like Forrest Gump here in GMT+11:00 land. No sign of gettin' tired. No sign of a 429 for days.

[autoSteve]

As a hobbyist, umm... I doubt they would hear all y'all @Ben00793. They protect their paying customers by rate limiting everyone who is not. Fair enough. Their service is about making a profit and providing a premium service for that profit. And they do not profit from us lot.

If I ran the world then there would be a "low use" paid subscription for pro hobbyists on offer. I'd gladly pay ten bucks a month to be rid of 429s given I make so few calls per day.

But I don't run their world. I was not part of that committee.

[autoSteve]

(Ten South-Pacific pesos per pro hob. user, or AUD120 per annum (which is about 75 Euro) x 2000 hobbyists who might be inclined to avoid 429s, or less than half our number on this integration, is equivalent to ~ a quarter of a million AUD, or 150k Euro per annum. Not chump change... For doing nothing but allowing traffic. A missed opportunity on their part, as it would pay for one senior FTE staff member. Again, if I was part of that committee I would jump at it.)

[BJReplay]

Australian protectionism. Solcast favouring you local boys

They're now owned by DNV - based in Norway.

I'd gladly pay ten bucks a month to be rid of 429s given I make so few calls per day.

I kinda agree. But...

I suspect that the cost of administering - collecting revenue and supporting and managing queries - and providing a nominal hobbyist account service - that hobbyists would be happy to pay - would be less than the revenue earned. So, they'd be worse off charging a cost that people would pay vs. not charging anything.

Do you think it's worth sending Solcast an email about this?

They're more or less aware.

[sh00t2kill]

Australian protectionism. Solcast favouring you local boys

There has to be some benefit to living in the middle of nowhere! :)

[BJReplay]

There has to be some benefit to living in the middle of nowhere! :)

It's not the quick flights to the Canary Islands :)

[autoSteve]

We need a separate dicussion about mind-bending 429 moments on this repo...

So the 429 / "go away we're busy" incidents are clearly nothing to do with v4.2.3 and auto-update.

I'm really glad we got to v4.2.3, as it is not possible to test every possible permultation of every installation out there. Go us! Things appear to be stable after a flurry of non-starter releases post-v4.2.0 (sorry). And the stability improvements in v4.2.3 are great I think, and owe thanks to those reporting issues so promptly. (They also grudgingly owe thanks to Solcast for incessant periods of 429s in the early Euro morning for me, which I utilised to improve exception handling...)

A heartfelt thank you from me for everyone who has stuck by this integration, and thanks to those who have reported issues. They will be fixed. There will be fails, yes, but hopefully there will be many more wins.

One thing that is a certainty is the dreaded 429 / go away... (Soz Europe it would seem.)

We try to cope with retries on schedule as best we can. If there's anything for the suggestions box on that front then I'm all ears. (I dunno, we fail, try it all half an hour later? An hour? Suggestions welcome.)

[sh00t2kill]

Just throwing it out there;

What about some sort of "proxy" API that we/you/someone maintains, which forwards requests on to solcast, and caches responses to reduce the drain on their infra.

Or even some form of onboarding flow, where users can setup their sites and keys, and the "proxy" service fetches from solcast on a schedule, and HA can hit it as much as it wants

Its addressing symptoms not issues, which i am generally not a fan of, and hosting may be an issue, compute costs money, but there may be merit to the general idea.

[BJReplay]

Yeah, Nah.

Not gonna help.

The issue isn't the caller (whether an individual running HA or a proxy) it's Solcast.

People handing over API keys to a proxy just shifts the problem further upstream, to a system that needs to maintain state, to hand back either responses, waiting state, or eventual failure back downstream.

It can't solve the problem, or fix the fact that the Solcast servers respond with the 429 at the time that the client wants the response.

Now, having done forecast caching in another life (for energy demand forecasting) - it can be done: when you identify sites that are otherwise identical, you can use the same forecast for those sites.

So, we could, in theory, identify each of the roughly 4000 sites that use the integration, and find out which ones share the same latitude, longitude, azimuth, tilt, installation date, efficiency factor, and collect a forecast for those, and then just scale by size.

That deals with the otherwise identical factor.

But that's not going to unload the Solcast servers by any significant amount.

[sh00t2kill]

True.
It makes the integration simpler, but either way that logic has to be handled somewhere, might as well handle it in HA at the integration layer.

Was just thinking out loud, especialy because i integrated HA to solcast using something similar originally, as using rest directly was flakey.

I was thinking about the identicalness too -- but the more I think about it, how many of those are actually identical, even if you ignore system size and a bunch of other factors its probably close enough to zero to not be worth it.

[autoSteve]

FYI, it turns out that many 429 status returns may likely be due to the integration not sending a user-agent in call headers.

A fix is in the works.

[sh00t2kill]

It might help if the user agent string has some uniqueness per client too!

Would be interesting to see if a 429 response followed by a different ua string gets a different response.

[BJReplay]

Lets limit this discussion to the release, please.

Correlation does not equal causation.

[Ben00793]

I emailed Solcast this morning. This is what they said.
Hope it helps.

[Hi Bence,

Glad to have you signed up and using our service.

The error code unfortunately isn't quite descriptive. In this case, I've had a look and it looks like rate limit would have applied for everyone.
So others who made the same API call as you at the same time would have got the error message as well. We recommend trying to randomise when you make the API calls to minimise the chance of this occurring.

If you have any further questions, please let me know.

Kind Regards,
Billy

Product Operation Specialist, Solcast
Email: billy@solcast.com.au
Web: solcast.com]

[autoSteve]

v4.2.4 is out. Closing.