401 Unauthorized when getting public room list

[govynnus]

Description

I recently moved my Synapse install to a different VPS by installing afresh and moving the database, media directory, homeserver.yaml and homeserver.signing.key. Of possible relevance is that the old VPS was running Ubuntu 18.04 whereas the new one is on 16.04 (so I can run BigBlueButton). I also changed from Apache to Nginx, and from using an SRV record to listening directly on 8448 for federation.

Everything works as expected apart from 2 things:

• Viewing and joining rooms on matrix.org
• Adding some servers, for example postmarketos.org

I think these 2 problems are actually the same underneath.

It's worth noting that:

• On the old install the only server I federated with was matrix.org and it worked
• I can view and join rooms on some servers, for example mozilla.org and privacytools.io

Viewing and joining rooms on matrix.org

• Use my homeserver calcuode.com
• On Riot go to Explore
• Change to matrix.org using the drop-down
• Error occurs

The error Riot shows is:

Riot failed to get the public room list. No key for calcuode.com with ids in ['ed25519:a_FHGZ'] (min_validity 1593532866008)

And in Synapse's homeserver.log:

2020-06-12 18:50:58,236 - synapse.storage.SQL - 223 - DEBUG - POST-2668 - [SQL] {get_destination_retry_timings-34eb} SELECT destination, failure_ts, retry_last_ts, retry_interval FROM destinations WHERE destination = ?
2020-06-12 18:50:58,236 - synapse.storage.SQL - 228 - DEBUG - POST-2668 - [SQL values] {get_destination_retry_timings-34eb} ['matrix.org']
2020-06-12 18:50:58,239 - synapse.storage.SQL - 242 - DEBUG - POST-2668 - [SQL time] {get_destination_retry_timings-34eb} 0.002315 sec
2020-06-12 18:50:58,247 - synapse.http.matrixfederationclient - 409 - INFO - POST-2668 - {GET-O-132} [matrix.org] Sending request: GET matrix://matrix.org/_matrix/federation/v1/publicRooms?include_all_networks=false&limit=20; timeout 60.000000s
2020-06-12 18:50:58,258 - synapse.http.federation.matrix_federation_agent - 242 - INFO - POST-2668 - Connecting to matrix-federation.matrix.org:443
2020-06-12 18:50:58,448 - synapse.http.matrixfederationclient - 461 - INFO - POST-2668 - {GET-O-132} [matrix.org] Got response headers: 401 Unauthorized
2020-06-12 18:50:58,451 - synapse.http.matrixfederationclient - 537 - WARNING - POST-2668 - {GET-O-132} [matrix.org] Request failed: GET matrix://matrix.org/_matrix/federation/v1/publicRooms?include_all_networks=false&limit=20: HttpResponseException("401: b'Unauthorized'",)
2020-06-12 18:50:58,453 - synapse.http.server - 84 - INFO - POST-2668 - <XForwardedForRequest at 0x7f97ce031da0 method='POST' uri='/_matrix/client/r0/publicRooms?server=matrix.org' clientproto='HTTP/1.0' site=8008> SynapseError: 401 - No key for calcuode.com with ids in ['ed25519:a_FHGZ'] (min_validity 1591980658386)
2020-06-12 18:50:58,456 - synapse.access.http.8008 - 311 - INFO - POST-2668 - 82.14.158.103 - 8008 - {@callum:calcuode.com} Processed request: 0.227sec/0.001sec (0.041sec, 0.001sec) (0.002sec/0.004sec/1) 122B 401 "POST /_matrix/client/r0/publicRooms?server=matrix.org HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0" [0 dbevts]

When trying to add a new server

• Use my homeserver calcuode.com
• On Riot go to Explore
• On the drop-down select Add a new server
• Enter a server name (e.g. postmarketos.org)
• Error occurs

The error Riot shows (in a pop-up that sometimes doesn't pop up) is:

Can't find this server or its room list

Synapse's homeserver.log:

2020-06-30 18:39:11,537 - synapse.access.http.8008 - 311 - INFO - OPTIONS-1945 - 82.14.158.103 - 8008 - {None} Processed request: 0.001sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 2B 200 "OPTIONS /_matrix/client/r0/publicRooms?server=postmarketos.org HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.6.4 Chrome/83.0.4103.119 Electron/9.0.5 Safari/537.36" [0 dbevts]
2020-06-30 18:39:11,582 - synapse.storage.SQL - 223 - DEBUG - POST-1946 - [SQL] {get_destination_retry_timings-1070} SELECT destination, failure_ts, retry_last_ts, retry_interval FROM destinations WHERE destination = ?
2020-06-30 18:39:11,583 - synapse.storage.SQL - 228 - DEBUG - POST-1946 - [SQL values] {get_destination_retry_timings-1070} ['postmarketos.org']
2020-06-30 18:39:11,585 - synapse.storage.SQL - 242 - DEBUG - POST-1946 - [SQL time] {get_destination_retry_timings-1070} 0.001775 sec
2020-06-30 18:39:11,587 - synapse.http.matrixfederationclient - 409 - INFO - POST-1946 - {GET-O-110} [postmarketos.org] Sending request: GET matrix://postmarketos.org/_matrix/federation/v1/publicRooms?include_all_networks=false&limit=1; timeout 60.000000s
2020-06-30 18:39:11,589 - synapse.http.federation.matrix_federation_agent - 242 - INFO - POST-1946 - Connecting to synapse.postmarketos.org:8448
2020-06-30 18:39:15,680 - synapse.http.matrixfederationclient - 461 - INFO - POST-1946 - {GET-O-110} [postmarketos.org] Got response headers: 401 Unauthorized
2020-06-30 18:39:15,681 - synapse.http.matrixfederationclient - 537 - WARNING - POST-1946 - {GET-O-110} [postmarketos.org] Request failed: GET matrix://postmarketos.org/_matrix/federation/v1/publicRooms?include_all_networks=false&limit=1: HttpResponseException("401: b'Unauthorized'",)
2020-06-30 18:39:15,685 - synapse.http.server - 84 - INFO - POST-1946 - <XForwardedForRequest at 0x7f534403c278 method='POST' uri='/_matrix/client/r0/publicRooms?server=postmarketos.org' clientproto='HTTP/1.0' site=8008> SynapseError: 401 - No key for calcuode.com with ids in ['ed25519:a_FHGZ'] (min_validity 1593535151757)
2020-06-30 18:39:15,688 - synapse.access.http.8008 - 311 - INFO - POST-1946 - 82.14.158.103 - 8008 - {@callum:calcuode.com} Processed request: 4.106sec/0.002sec (0.004sec, 0.000sec) (0.001sec/0.004sec/1) 122B 401 "POST /_matrix/client/r0/publicRooms?server=postmarketos.org HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.6.4 Chrome/83.0.4103.119 Electron/9.0.5 Safari/537.36" [0 dbevts]

I also get the same error when trying to add matrix.org manually.

Version information

• Homeserver: calcuode.com
• Version: 1.15.1
• Python version: 3.5.2
• Install method: from https://packages.matrix.org/debian/ using apt
• Platform: Ubuntu 16.04
• Machine: Virtual Private Server

If you need any more info just let me know, thanks for taking a look.

[richvdh]

a note to whoever picks this up: I'd like us to investigate why matrix.org is reporting

No key for calcuode.com with ids in ['ed25519:a_FHGZ']

[richvdh]

the reason this is happening is that google's public DNS is giving us a SERVFAIL when we try to look up the SRV record:

dig -t SRV _matrix._tcp.calcuode.com 

; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> -t SRV _matrix._tcp.calcuode.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 20829
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 33feb80a37d17adbf8da74af5eff7ff8a18a7bbd7f5fc0d8 (good)
;; QUESTION SECTION:
;_matrix._tcp.calcuode.com.	IN	SRV

So this is kinda the same as #6589.

The problem is that I don't really know why it is returning SERVFAIL for this request. I've raised an issue about it here: https://issuetracker.google.com/issues/160447762. I'll be interested to see if we get an answer.

[richvdh]

ahh I think it's a DNSSEC fail:

$ delv _matrix._tcp.calcuode.com SRV
;; no valid RRSIG resolving '_matrix._tcp.calcuode.com/DS/IN': 2a01:7e00::5#53
;; no valid RRSIG resolving '_matrix._tcp.calcuode.com/DS/IN': 2a01:7e00::6#53
;; no valid RRSIG resolving '_matrix._tcp.calcuode.com/DS/IN': 109.74.192.20#53
;; no valid DS resolving '_matrix._tcp.calcuode.com/SRV/IN': 2a01:7e00::5#53
;; validating _matrix._tcp.calcuode.com/SRV: bad cache hit (_matrix._tcp.calcuode.com/DS)
;; broken trust chain resolving '_matrix._tcp.calcuode.com/SRV/IN': 2a01:7e00::6#53
;; resolution failed: broken trust chain

[govynnus]

Thanks for looking into it - not what I expected.
So this sounds like it could be a problem on my end with DNSSEC?

Or is it just different DNS servers handling stuff differently/incorrectly?
Results from 8.8.8.8 give a SERVFAIL (as you put above):

; <<>> DiG 9.10.3-P4-Ubuntu <<>> -t SRV _matrix._tcp.calcuode.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 34917
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_matrix._tcp.calcuode.com.	IN	SRV

But 1.1.1.1 gives NXDOMAIN:

; <<>> DiG 9.10.3-P4-Ubuntu <<>> -t SRV _matrix._tcp.calcuode.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 12852
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;_matrix._tcp.calcuode.com.	IN	SRV

If the error is DNS server dependent it kind of make sense that it federating works fine with some servers because they are likely to use different DNS. Though that doesn't explain why the federation tester is OK unless it and synapse handle SERVFAIL and NXDOMAIN differently, which would be confusing.

Also it seems a bit weird to me that SRV lookups are being done in the first place considering the first line of delegate.md:

By default, other homeservers will expect to be able to reach yours via your server_name, on port 8448.

I must admit I'm a bit out of my depth with some of this DNS stuff, so apologies if I've said anything that doesn't make sense.

[richvdh]

I'm no dnssec expert myself, but the answer at https://issuetracker.google.com/issues/160447762 confirms that this is a dnssec problem. Essentially your server is not returning the correct records to confirm the non-existence of _matrix._tcp.calcuode.com. If someone else hosts your DNS, I suggest talking to them, but if you run it yourself I guess you need to do some research!

Though that doesn't explain why the federation tester is OK unless it and synapse handle SERVFAIL and NXDOMAIN differently, which would be confusing.

Unfortunately that is the case: this is matrix-org/matrix-federation-tester#92.

Also it seems a bit weird to me that SRV lookups are being done in the first place considering the first line of delegate.md

Well, delegate.md simplifies the case slightly. We have to check whether you have a SRV record so that we can follow it if it exists. Normally we expect that to return a NXDOMAIN ("no such record"), in which case, we fall back to the default. But because your server returns an invalid response, we're a bit stuck.

One workaround which might work for you is to use a .well-known file with an explicit port number. This will mean the SRV lookup gets skipped. Again, see delegate.md. I'd really recommend you get your DNSSEC sorted though.

This is confirmed as a configuration issue on your side, so I'm going to go ahead and close this. Good luck getting it sorted though!

[govynnus]

Alright, thanks a lot

[chThie]

Just in case someone with the same problem as me is stumbling over this issue as well:
I had the same "401 - Unauthorized" error and was not able to fetch rooms from matrix.com, while the Federation Tester did not report any error.
The DNS entry of the domain I used had just been updated to a new server/ip an hour ago and it seems to have been some kind of DNS cache problem.
Eating lunch and retrying after an hour solved the problem in my case.