oauth2: invalid_grant error when refreshing token

[tukane]

Hi,

i noticed, that the authorization is not working anymore.

The docker log says:
level=error msg="Error during refresh: Get "https://api.netatmo.net//api/getstationsdata?app_type=app_station\": oauth2: "invalid_grant""

Did Netatmo changed the authorization again?

Greetings
tukane

[xperimental]

I am not aware of any changes and, so far, my exporter seems to run fine.

I'll have a look the coming days, the only thing I can think of right now is whether you have selected the correct scope when creating the initial token.

[DEEJCUK]

I am seeing similar issues today, refreshing either by the built in GUI or by the api and copy token results in

netatmo-exporter  | level=info msg="Listen on :9210..."
netatmo-exporter  | level=error msg="Error during refresh: no token available"

[DEEJCUK]

ok a bit of a breakthrough for me, i went from latest docker image to master, pulled, and restarted, reauth'd, restarted container 2nd time and its started working for me

[xperimental]

@DEEJCUK Good that your issue is fixed. latest and master are "nearly identical" at the time of writing, though.

But you are giving me an idea there: Docker does not automatically update the local image when the repository has a newer one with the same tag (for example latest or master, which get "reused").

@tukane Can you check which version of the exporter you are running? Maybe you have the same issue as @DEEJCUK and have just pulled the exporter in a previous version that does not work anymore (if you're using Docker at all).

[DEEJCUK]

i noticed in the first comment there is a double slash in the URL after .net is this relevant ?

level=error msg="Error during refresh: Get "[https://api.netatmo.net//api/getstationsdata?app_type=app_station\](https://api.netatmo.net//api/getstationsdata?app_type=app_station%5C)": oauth2: "invalid_grant""

[xperimental]

No, I don't think the double slash has anything to do with this issue. While it is wrong, they are usually collapsed to a single slash on the server side.

I was able to reproduce the invalid_grant error message with a token that I generated a long time ago and never refreshed. A freshly generated token works fine, same for my long-running exporter which regularly refreshes the token.

@tukane Do you have any update on your situation?

[tukane]

Hi all,
I changed the docker image from "latest" to "master". Created a new token for the token-file and restarted the container, but the error message remains the same:

level=warning msg="Restored token has no expiry time! Token will be renewed immediately."
level=info msg="Loaded token from /var/lib/netatmo-exporter/netatmo-token.json."
level=info msg="Listen on :9210..."
level=debug msg="Refreshing data. Time since last refresh: 2562047h47m16.854775807s"
level=error msg="Error during refresh: Get \"https://api.netatmo.net//api/getstationsdata?app_type=app_station\": oauth2: \"invalid_grant\""

[lennartp92]

I'm having this issue, too. The container suddenly did not push data anymore. It still does not work after creating a new token and using the "master" instead of the "latest" image. 

[xperimental]

@lennartp92 Can you tell me what version of netatmo-exporter you are running? latest and master are both moving tags and, at this time of writing, both contain the 2.0.1 version released in October last year.

Docker (and other container runtimes) by default do not re-download an image if an image of the same name is already present on the host. This means that if you used latest or master in the past on the machine, but did not delete the previous version, it is still using an older release and not the current one.

In general, both latest and master are only provided for convenience, but you should rather use the explicitly versioned tags instead (or make sure to manually pull the images if needed).

My assumption on what happened for @DEEJCUK and @tukane where it started working when switching to master was that they never used "the tag that worked" before, so they got a fresh version.

See if you can replicate the issue with the latest version (ghcr.io/xperimental/netatmo-exporter:2.0.1) and a new token. If that still returns an error, then there's something broken that I need to fix.

Btw: While the exporter is running, there's also a /version handler available that will return "what it knows about itself". I need to change the exporter so that it also prints out this information during startup for easier debugging ...

[DEEJCUK]

my workflow went something like this :

look up the container with docker ps and stop the container, then you can delete it with docker rm <containername> and then delete the image using docker rmi <imagename> and then rerun docker pull <imagename> to pull the latest image
i also run docker system prune to remove all traces of containers, networks etc no longer connected

but please make sure you read about these commands before running to ensure they are suitable for your use case

[xperimental]

If the exporter is still running, first try to check the currently running version using the /version endpoint 🙂

[tukane]

My workflow was nearly the same, but I use portianer to manage my containers. I deleted the old container and the image with the latest/master tag and pulled explicit the version 2.0.1 from the repo, but the error message remains the same.

As I am using docker-compose, how can I add the /version handler?

[xperimental]

As I am using docker-compose, how can I add the /version handler?

The handler is available as an HTTP endpoint from the exporter. Similar to how Prometheus uses /metrics to get the metrics data you can use a HTTP client (Browser, curl, ...) to get the information from /version by pointing your client to the URL where the exporter is available, for example http://192.168.0.20/version if the exporter is running at that IP address.

edit: Just remembered that the exporter also has a very simple interface at / which shows the current token and also allows you to log in, if the token is not present. You could try that as another way of getting the token into your installation.

[xperimental]

@tukane Something completely different: I have read the documentation about the error in the developer docs (I know, should have done this earlier). It says this error should indicate a mismatch in the configuration between what is configured in the console and what the application is using.

Can you have a look in the NetAtmo developer console. Do you have something specified as "redirect URI" or "webhook URI"? If yes, can you try setting those fields to an empty value and try again?

[tukane]

Thanks for clarification. Here is the output of the /version page:

version	"2.0.1"
commit	"57952a2c2070406ea26f1c3b17930eba9c9c744a"

The redirect URI and web hook URI field are empty in the dev console from Netatmo.

BTW: When I try to open the web interface to enter the token there, I also get the Error getting token: oauth2: "invalid_grant" error message.

Just for reference, here is my docker-compose file.

version: "3.8"
services:
  netatmo:
    image: xperimental/netatmo-exporter:2.0.1
    container_name: netatmo
    ports:
      - 8040:8080
      - 9210:9210
    volumes:
      - /opt/netatmo:/var/lib/netatmo-exporter/
      - /opt/netatmo/netatmo-token.json:/var/lib/netatmo-exporter/netatmo-token.json
    environment:
      - NETATMO_CLIENT_ID= <my ID>
      - NETATMO_CLIENT_SECRET= <my secret>
      - NETATMO_EXPORTER_ADDR=:9210
      # - NETATMO_EXPORTER_TOKEN_FILE=/var/lib/netatmo-exporter/netatmo-token.json
      - NETATMO_EXPORTER_EXTERNAL_URL=http://192.168.0.140:9210
      - DEBUG_HANDLERS=
      - NETATMO_LOG_LEVEL=debug
      - NETATMO_REFRESH_INTERVAL=1m
      - NETATMO_AGE_STALE=60m
    restart: unless-stopped
    network_mode: bridge
networks: {}

[DEEJCUK]

here is mine, its a bit less than yours, but its working

 netatmo-exporter:
    container_name: netatmo-exporter
    image: ghcr.io/xperimental/netatmo-exporter:master
    restart: unless-stopped
    environment:
      - NETATMO_CLIENT_ID=${NETATMO_CLIENT_ID}
      - NETATMO_CLIENT_SECRET=${NETATMO_CLIENT_SECRET}
      - NETATMO_EXPORTER_EXTERNAL_URL=http://XXX.XXX.XXX.XXX:9210
    ports:
      - "9210:9210"

and my version is
{"version":"2.0.0-3-g57952a2","commit":"57952a2c2070406ea26f1c3b17930eba9c9c744a"}

[xperimental]

@tukane Sorry, forgot to reply earlier.

One thing I find odd about your configuration is having the two volume mounts "inside of each other". Just having the volume for the directory should suffice.

I can see that you have the "external URL" set. Have you tried the "interactive authentication" using the exporter's web interface already? (The "login" link, not the textbox)

Also, you can see that you are both using the same code (commit 57952a2c...), even though one is using the image built as "2.0.1" (tagged) and the other as "master" (before the tag existed).

[tukane]

@xperimental: Thanks for your annotations, I've deleted the mount for the json file.

I'm not able to reach the web interface. The container is running on a server with the IP 192.168.0.13. So I've tried to open the URL http://192.168.0.13:9210, but I'm getting also the same error message Error getting token: oauth2: "invalid_grant" when I try to access it.

The version 2.0.1 and the latest version are using the same code (commit "57952a2c2070406ea26f1c3b17930eba9c9c744a")

[mblaschke]

@xperimental
The root cause of this problem is that the new token is only saved on shutdown but never when a new one if fetched from the API.

If the exporter runs for weeks and then is killed (eg. power outage, OOM kill,...) the exporter still has the initial token from the start and lost the updated one.

This happened to me multiple times now. Please regularly update the token file, eg. every hour.

[xperimental]

@mblaschke I might have misread the thread so far, but I was under the impression that the failure described here is immediate and not once the exporter has been killed after a longer runtime. The issue you're describing is #54.

[mblaschke]

@xperimental
not sure 🤔

i was running multiple times into this issue (even today) after i had to restart the physical box and was running into the "oauth2: invalid_grant error when refreshing token" issue.
The token was expired and so the exporter was failing.

Log output:

level=info msg="Listen on :9210..."
level=debug msg="Refreshing data. Time since last refresh: 2562047h47m16.854775807s"
level=error msg="Error during refresh: Get \"https://api.netatmo.net//api/getstationsdata?app_type=app_station\": oauth2: \"invalid_grant\""

workaround: restart the container every day or recreate the token every time the server is rebootet

[xperimental]

@mblaschke I've already accepted that saving the token after a refresh is probably a good idea for people that have issues with spontaneous power outages or other circumstances which cause the exporter to not be able to save on shutdown. This should not be the normal case "after a restart" though. For me the exporter has survived a lot of restarts already. Is there some special circumstance for you that makes the exporter unable to save the token on shutdown?

[mblaschke]

not sure if docker daemons "live restore" causes the issues but it's definitely not storing the file when the machine reboots.
strange.

[xperimental]

@mblaschke I have been unaware of that feature and have not used it so far. The documentation is, unfortunately, a bit short on details. There is a section about "Live restore upon restart", but reading that and a few articles about the feature from other sources, I get the feeling this is more about a (docker-)service restart not a machine restart.

[xperimental]

@tukane I can reproduce the invalid_grant error with an expired token that also has an expired refresh token. Do you also have this issue after a restart of the exporter as @mblaschke described?

--edit: I have just added a small fix to the exporter, so that it ignores expired tokens on startup. This should make it easier to recover from this situation instead of just producing more errors.

[tukane]

@xperimental I don't know what I'm doing wrong but it's not working after a restart either.
I've created a new access and refresh token on the Netatmo website and entered this tokens into the netatmo-token.json file in my docker volume.
This is how the file looks like:

{
     "access_token": "xxxxxxxx|yyyyyyy",
     "refresh_token": "xxxxxxxx|yyyyyyy"
}

I've stopped the container, deleted the image and started the container once again.

Here is the log output:

level=debug msg="Got signal: terminated"
level=error msg="Error persisting token: error retrieving token: oauth2: \"invalid_grant\""
level=warning msg="Restored token has no expiry time! Token will be renewed immediately."
level=info msg="Loaded token from /var/lib/netatmo-exporter/netatmo-token.json."
level=info msg="Listen on :9210..."
level=debug msg="Refreshing data. Time since last refresh: 2562047h47m16.854775807s"
level=error msg="Error during refresh: Get \"https://api.netatmo.net//api/getstationsdata?app_type=app_station\": oauth2: \"invalid_grant\""
level=debug msg="Refreshing data. Time since last refresh: 1m59.999039506s"

[ebfg]

My plugincontainer is having the same issue that @tukane's is.

[xperimental]

@tukane I'm guessing here, but if you're modifying the token file while the container is running and then stop the exporter, then it will overwrite your changes with its own status again. You have to modify the file while the exporter is not running.

The main branch also now contains a change that should make the exporter ignore tokens which are expired on startup which will make recovery from this error easier. It's not part of a release yet though, because I wanted to wait for another feature first. You can get that version by using the main tag on the Docker image though.

@ebfg Also, I don't know what you mean with "plugin".

[ebfg]

Meant container.

[tukane]

@tukane I'm guessing here, but if you're modifying the token file while the container is running and then stop the exporter, then it will overwrite your changes with its own status again. You have to modify the file while the exporter is not running.

The main branch also now contains a change that should make the exporter ignore tokens which are expired on startup which will make recovery from this error easier. It's not part of a release yet though, because I wanted to wait for another feature first. You can get that version by using the main tag on the Docker image though.

@ebfg Also, I don't know what you mean with "plugin".

I stopped the container, created a new token on the dev page of Netatmo, updated the token-file and started the container again. No success. I don't understand what's wrong here :-(

netatmo  | level=info msg="Loaded token from /var/lib/netatmo-exporter/netatmo-token.json."
netatmo  | level=info msg="Listen on :9210..."
netatmo  | level=debug msg="Refreshing data. Time since last refresh: 2562047h47m16.854775807s"
netatmo  | level=error msg="Error during refresh: Get \"https://api.netatmo.net//api/getstationsdata?app_type=app_station\": oauth2: \"invalid_grant\"" 

The main tag is not working for me.
Error response from daemon: manifest for xperimental/netatmo-exporter:main not found: manifest unknown: manifest unknown

[xperimental]

@tukane I'm sorry, I got this mixed up with another project. I meant to say master tag, not main. The current image would be this:

ghcr.io/xperimental/netatmo-exporter@sha256:5bf34072169a2294f9205677d1b4fb9d95854cd4e9e6a234db2557f087a61787

[tukane]

@xperimental No problem. With the master tag I was able to pull the new image.
But the error remains the same.

[mblaschke]

In mean time i've converted my system to a k8s (actually k3s) system. After a cluster reboot (only one node) due to a power loss the netatmo-exporter is failing again with "invalid_grant" error.
It was actually running for several weeks but the token was never saved to the persistent volume because it only save it when there is a clean shutdown (SIGTERM) but fails for all other reasons.

Deleting the Pod seems to trigger the save, i'm thinking about an automatic Pod delete to ensure the token is saved at least once a day as workaround.

I still suggest to save the token eg. every 5 minutes inside the exporter.

[xperimental]

@mblaschke It might interest you then, that I finally found time this weekend to get that change in (see #54). I'm also using k3s for my local installation... 🙂

[xperimental]

@tukane You might want to retry with 2.1.0 and see if the fixes introduced in the meantime have also fixed your issue.

[tukane]

@xperimental Unfortunately, I don't see any difference with this version.

I'm also wondering why I get the same error when I open the URL http://192.168.0.13:9210 in my browser. The - NETATMO_EXPORTER_EXTERNAL_URLparameter is set. From your documentation I suspected to get a form where I can enter my token.

[xperimental]

@tukane Somewhere there's a flaw in your workflow. Are you sure that a) the exporter you are starting is using the new image and b) the exporter you are accessing on that host:port is the exporter you started with the new image?

I don't think we can get anywhere without you providing steps for me to reproduce the issue. Because I don't know what you did so far, I would suggest you remove everything belonging to the exporter (container, images, volumes, networks) and start fresh. docker compose down --remove-orphans -v is probably a good starting point, but I would check manually if everything is removed.

Then start again with a simple configuration containing only the client-id, client-secret and a volume for the token file. Look at the exporter's main page and work towards a working configuration from there.

[tukane]

@xperimental I'm one step ahead. I deleted everything (container, image, folders, volumes, yaml-files) and started from scratch.

With this docker-compose file I was able to get the web-interface working:

version: "3.8"
services:
  netatmo:
    image: xperimental/netatmo-exporter:2.1.0
    container_name: netatmo
    volumes:
      - /opt/netatmo:/var/lib/netatmo-exporter/
    ports:
      - 9210:9210
    environment:
      - NETATMO_CLIENT_ID=[myClientID]
      - NETATMO_CLIENT_SECRET=[myClientSecret]
      - NETATMO_LOG_LEVEL=debug
      - NETATMO_EXPORTER_EXTERNAL_URL=http://192.168.0.13:9210
    network_mode: bridge

So I restarted the container, logged into the Dev portal of Netatmo, created a new token (scope: read_station), copied the refresh_token to the web-interface and clicked "update token". This leads to the same error as before.
But when I clicked on the first which redirects me to the Netatmo DEV portal, it worked!!

The web-interface shows:

Token is valid until 2024-10-21 18:23:33.688986973 +0000 UTC m=+10834.095749806 (2h52m19.936183692s)

Metrics are available [here](http://192.168.0.13:9210/metrics). 

But in the logs I see this error message:
level=error msg="Error during refresh: bad HTTP return code: 403

I also miss a lot of metrics. This are the only ones starting with Netatmo:

I'm sorry, I think the problem is me or my setup, but I couldn't find out what's wrong here. So I'm very thankful if you can help me out, but I can also understand when you can no longer spend time in troubleshooting my problem.

[DEEJCUK]

I see this quite often, try docker restart (container-name) and look at the metrics page to see if they update there

[xperimental]

@tukane Thanks for going through it again. I currently have two ideas:

• Do you have more than one application defined in the NetAtmo dev portal? If yes, make sure to use the correct one (the one matching the client-ID you used in the exporter) when using the manual flow.
• Maybe there's something interfering with the manual flow when the external URL is set. You can try the manual flow without setting NETATMO_EXPORTER_EXTERNAL_URL. It's only needed for the redirect, not for the manual flow with copy&pasting the token from the website.

Can you attach a full log of the exporter (ideally from "empty state") instead of just a single line? Just having a single line without context makes debugging what happened nearly impossible. If it's large you might need to attach a file with the log messages.

[tukane]

What an odyssey!! :-(

I've created a new second application (deleting the old one is not possible) in the dev portal from Netatmo.
When I use this app, everything is working. When I switch back to the old app, I get the error message Error getting token: oauth2: "invalid_grant" after entering the refresh_token on the web GUI.
I don't know what's wrong with the first app, but I'm happy that the exporter is working again.

Thanks a lot for your support and your time @xperimental @DEEJCUK

[giuliomagnifico]

I'm having the same problem with the latest update. I tried creating a new app and it works for a few hours, then stops. I'm not using Docker, I compiled it. Any suggestions?

( The old executable worked fine for years)

[xperimental]

@giuliomagnifico I'm not sure, if you have the same problem, because for the others the updates seem to have fixed it. It also looks like there might have been a misconfiguration issue.

What I recommend you do is the same as I have written before: Start fresh, note down what you did. If you then have an issue, provide that information to me. I am not able to help you when you just provide a vague description and a screenshot that contains no information other than "does not work".