Trying to fetch a public room list results in a 401.

[VVelox]

Description

Trying to pull a list of matrix.org purlib rooms results in the error 'Riot failed to get the public room list. Internal server error'. Checking my servers logs I am seeing the item below being logged.

2019-12-01 18:47:50,003 - synapse.access.https.8448 - 233 - INFO - OPTIONS-882- 192.168.15.2 - 8448 - Received request: OPTIONS /_matrix/client/r0/publicRooms?server=matrix.org
2019-12-01 18:47:50,007 - synapse.access.https.8448 - 302 - INFO - OPTIONS-882- 192.168.15.2 - 8448 - {None} Processed request: 0.002sec/0.002sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 2B 200 "OPTIONS /_matrix/client/r0/publicRooms?server=matrix.org HTTP/1.1" "Mozilla/5.0 (X11; FreeBSD amd64; rv:70.0) Gecko/20100101 Firefox/70.0" [0 dbevts]
2019-12-01 18:47:50,024 - synapse.access.https.8448 - 233 - INFO - POST-883- 192.168.15.2 - 8448 - Received request: POST /_matrix/client/r0/publicRooms?server=matrix.org
2019-12-01 18:47:50,028 - synapse.util.caches.response_cache - 148 - INFO - POST-883- [remote_room_list]: no cached result for [('matrix.org', 20, None, True, None)], calculating new one
2019-12-01 18:47:50,031 - synapse.http.matrixfederationclient - 408 - INFO - POST-883- {GET-O-14} [matrix.org] Sending request: GET matrix://matrix.org/_matrix/federation/v1/publicRooms?include_all_networks=true&limit=20; timeout 60.000000s
2019-12-01 18:47:50,054 - synapse.http.federation.matrix_federation_agent - 242 - INFO - POST-883- Connecting to matrix-federation.matrix.org.cdn.cloudflare.net:8443
2019-12-01 18:47:50,816 - synapse.http.matrixfederationclient - 440 - INFO - POST-883- {GET-O-14} [matrix.org] Got response headers: 401 Unauthorized
2019-12-01 18:47:50,819 - synapse.http.matrixfederationclient - 522 - WARNING - POST-883- {GET-O-14} [matrix.org] Request failed: GET matrix://matrix.org/_matrix/federation/v1/publicRooms?include_all_networks=true&limit=20: HttpResponseException("401: b'Unauthorized'",)
2019-12-01 18:47:50,823 - synapse.http.server - 109 - ERROR - POST-883- Failed handle request via 'PublicRoomListRestServlet': <SynapseRequest at 0x80c0bc470 method='POST' uri='/_matrix/client/r0/publicRooms?server=matrix.org' clientproto='HTTP/1.1' site=8448>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/synapse/http/server.py", line 77, in wrapped_request_handler
    await h(self, request)
  File "/usr/local/lib/python3.6/site-packages/synapse/http/server.py", line 326, in _async_render
    callback_return = await callback_return
  File "/usr/local/lib/python3.6/site-packages/synapse/rest/client/v1/room.py", line 408, in on_POST
    third_party_instance_id=third_party_instance_id,
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/synapse/handlers/room_list.py", line 391, in get_remote_public_room_list
    third_party_instance_id=third_party_instance_id,
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/synapse/federation/transport/client.py", line 359, in get_public_rooms
    destination=remote_server, path=path, args=args, ignore_backoff=True
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/synapse/http/matrixfederationclient.py", line 776, in get_json
    timeout=timeout,
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/synapse/http/matrixfederationclient.py", line 246, in _send_request_with_optional_trailing_slash
    response = yield self._send_request(request, **send_request_args)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python3.6/site-packages/synapse/http/matrixfederationclient.py", line 477, in _send_request
    raise e
synapse.api.errors.HttpResponseException: 401: b'Unauthorized'
2019-12-01 18:47:50,831 - synapse.access.https.8448 - 302 - INFO - POST-883- 192.168.15.2 - 8448 - {@vvelox:vvelox.net} Processed request: 0.805sec/0.002sec (0.035sec, 0.000sec) (0.000sec/0.000sec/0) 55B 500 "POST /_matrix/client/r0/publicRooms?server=matrix.org HTTP/1.1" "Mozilla/5.0 (X11; FreeBSD amd64; rv:70.0) Gecko/20100101 Firefox/70.0" [0 dbevts]

### Steps to reproduce

Attempt to pull up the public room list on matrix.org.

### Version information

- **Homeserver**:

server: 1.6.1, LDAP & PostgreSQL backend
python: 3.6.9

- **Install method**:

FreeBSD package build via poudriere.

- **Platform**:

uname -a
FreeBSD vulpes.vvelox.net 12.1-STABLE FreeBSD 12.1-STABLE r354803 vixen42 amd64

[aaronraimist]

You don't have valid certificates so your federation is not going to work. https://federationtester.matrix.org/#vvelox.net

[VVelox]

@aaronraimist Actually that is wrong. If you point a browser at https://matrix.vvelox.net:8448/ you will see it in does have a valid cert.

The tool is also failing to do a DNS lookup as well as 'dig -t SRV _matrix._tcp.vvelox.net' shows.

_matrix._tcp.vvelox.net. 360    IN      SRV     10 0 8448 matrix.vvelox.net.

[aaronraimist]

Are you sure that certificate includes the full certificate chain? Browsers may accept it but that doesn't mean Matrix will. If you are using Certbot for example the correct file is called fullchain.pem rather than cert.pem.

It isn't looking up the SRV record because you have a .well-known/matrix/server file. You want one or the other, you don't need both.

[VVelox]

@aaronraimist Ahh! Derp!

Okay! Yeah, that is definitely a bit confusing. Okay, working now. :)

That said just though I mention the SRV thing as it said there was none. Thanks! :)

[mozzieongit]

I would add some information on certificates and homeserver delegation, if anyone also gets 401 Unauthorized and uses SRV.
I used a different Server than the webserver my domain-part pointed to. The matrix server didn't have valid certificate for the main domain, as my server was located on a subdomain.
The problem is, that DNS is too easy to spoof and therefore not a sufficient proof of ownership for that domain. So one must either have a valid cert for the main domain part and the subdomains and can use SRV or use a .well-known entry on the web-server that serves the main domain.
Thanks to @richvdh for that information