SSL verification fails for certificates unknown to certifi

[timobrembeck]

Python requests uses certifi to check the SSL certificates.
However, this package only ships a limited set of root certificates, which means that for a substantial portion of other valid certificates (e.g. https://stadtwerke-hall.de/ for certifi 2022.12.7), the link checker shows an error:

SSL Error: certificate verify failed: unable to get local issuer certificate

$ python
Python 3.10.9 (main, Dec 19 2022, 17:35:49) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("https://www.kleinmachnow.de/")
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

In general, this should be addressed in the library (psf/requests#2966). However, I don't think this will be fixed anytime soon, so I suggest one of the following workarounds:

1. Try to build a larger root certificate database at runtime by downloading e.g. the Mozilla CA certificate store and use:
   response = requests.get(url, verify ='/path/to/ca_bundle')
2. Treat such links as valid and change the message to e.g. 200 OK, SSL certificate could not be verified or similar

[claudep]

This most probably explains why linkcheck used to not verify certificates before 080c5aa.

For me, it should be either 2. or revert 080c5aa.

It could be interesting to explore the possibility of using Python's own HTTPSConnection as recent versions seem to use the system-installed CAs.