Can this be used with requests?

[danechitoaie]

Can this be used with requests? As replacement for the built in SSL lib (currently I'm referring to a Sublime Text 3 plugin context).

[wbond]

I don’t know how requests is structured, unfortunately. I think the question may be better suited to ask the requests maintainers. Specifically, if the transport layer is modular. oscrypto.tls in its current implementation controls the raw socket connection and provides methods to read and write data.

I do have the intention of using oscrypto with Sublime Text in the future, however my use case for oscrypto.tls would be FTPS or a very basic HTTP 1.1 implementation. Requests is big enough and complicated enough that I’d prefer not to become dependent on it. Additionally, I currently still support Python 2.6 for ST2, which many Python packages are starting to phase out.

The other downside is that oscrypto does not support Windows XP (about 1% of Package Control users) or OS X 10.6. OS X 10.6 is only supported by ST2, and is probably a very small percentage of users.

[danechitoaie]

I guess I'll have to dig in the requests source and see what can be done. Thanks.

[wbond]

It looks like requests uses urllib3, which has a contrib module for using the cryptography package as an alternative to the ssl module. That may be a basis upon which to add oscrypto support. http://urllib3.readthedocs.org/en/latest/contrib.html

Certainly let me know what you find out. I don't really have time to do development related to this, but I'd be happy to answer questions and try to remove any blockers.

[danechitoaie]

Yes, if I make any progress I'll let you know.

[Lukasa]

Howdy! @wbond mentioned this was a thing in IRC, and probably didn't realise that I'm the current urllib3 lead maintainer, so didn't think to ask me what would be needed.

The relevant example is this contrib module. It shims PyOpenSSL into an interface that looks a lot like the standard library's ssl module. urllib3 currently codes to that interface, so that's the interface you need to shim into.

The specific requirements are:

• You must provide an SSLContext object that takes one argument (the equivalent of Python's SSLv23_METHOD). You can probably get away with ignoring this argument for libraries that don't have anything meaningfully similar to it.
  • This SSLContext object must have a wrap_socket method that acts as a TLS socket factory. This needs to match the API the PyOpenSSL context provides. Note that you can assume that server_side is always False, do_handshake_on_connect is always True, and suppress_ragged_eofs is always True, as we do not ever set those differently.
  • You must tolerate both blocking and non-blocking sockets. You must accept timeouts in all I/O parameters.
  • Supporting loading certificate locations is optional: if you only want to support the system certs, without adding custom trust roots, you can choose to do so by causing load_verify_locations to throw an exception.
  • Supporting client certificates is optional: if you don't want to support them, you can cause load_cert_chain to throw an exception.
  • set_ciphers is mandatory. Annoyingly, this API uses an OpenSSL cipher string, so you'll probably need to include a translation. Happily, urllib3 uses a very restricted OpenSSL cipher string in regular use, though our users can and do customise them.
  • set_default_verify_paths must be implemented, but needn't do anything.
  • verify_mode must be implemented, and must accept ssl.CERT_NONE and ssl.CERT_REQUIRED. Supporting ssl.CERT_OPTIONAL is, ironically, optional.
  • options must be implemented, and must accept OpenSSL option flags. Again, urllib3 mostly uses a restricted subset here, so you can probably avoid supporting many OpenSSL option flags.
• You also need a socket wrapper.
  • This needs to support the weird file-object wrapping that Python's SSL sockets do. That means you'll need to duplicate _decref_socketios, _reuse, _drop, and makefile from the PyOpenSSL wrapper.
  • You need to support recv, recv_into, settimeout, sendall, shutdown, and close (though we never use shutdown).
  • You need to support getpeercert, which must be able to support both providing the binary form of the peer certificate and the wacky Python-data-structure parsed form. See what PyOpenSSL does here for guidance. You will need to be able to provide at least the common name of the subject and the subject alternative name field in a form that the Python match_hostname function can handle.
  • For future-proofing, you should also implement send and setblocking. Both of these are likely to be used in urllib3 v2, which is being actively developed.
• Finally, you need two monkeypatching functions, one that injects your code into urllib3 and one that removes it. These should look like the PyOpenSSL contrib module's one.

Assuming you do all of these things, everything should just work. If you start working on this in a concrete way, please let me know: I'm happy to do code review and testing, and if it gets into a good shape I'd also be happy to merge a contrib module into urllib3. That would make it possible for Requests to use it.

[wbond]

@Lukasa Thanks for the extensive info! From reading over it, I believe that everything exists right now in order to be able to implement what you described. Hopefully when I have some free time for hacking I can take a pass at this.

[notatallshaw]

Out of curiosity did anyone ever give this a try?

[wbond]

Requests I believe uses urllib3. Urllib3 now has a MacOS-specific backend that is derived from the code in oscrypto, even though it doesn’t use it directly.

I don’t believe it has a Windows (SecureChannel) backend, though. I may be wrong about that.

[notatallshaw]

I don’t believe it has a Windows (SecureChannel) backend, though. I may be wrong about that.

I don't believe so either because we (the company I work for) hacked together our own SSL context that uses OpenSSL's CAPI Engine to be able to do Windows Cert authentication. Unfortunately this breaks with TLS 1.2+ due to CAPI being deprecated in favor of CNG.

I was wondering if this could be an alternative. I'll try and spend a little time seeing if I can reproduce the same sort of hacking we did but with oscrypto.

[notatallshaw]

I have a proof of concept working of using oscrypto to wrap the socket in an sslcontext and then providing that context to requests and it working well on Windows.

I want to clean up the code and debug it a little bit before sharing.

In particular for some sites I am getting the TLSError: SECURITY_STATUS error 0x80090327: The parameter is incorrect.

Is there any debug mode I can set in oscrypto for it to give me more info than that?

[wbond]

Heh, welcome to the world of debugging Windows APIs! In my experience you’ll need to look up the preceding API call and try to deduce what isn’t working properly.

There are a number of tests in oscrypto for the TLS layer. If you can provide some basic steps to reproduce, I can see what I can find. Even knowing a public site the error occurs on with what version of Windows and Python would be helpful.

[notatallshaw]

If you can provide some basic steps to reproduce, I can see what I can find. Even knowing a public site the error occurs on with what version of Windows and Python would be helpful.

Oh dear, this will probably never be reproducible publicly. These sites are using SSO with smart-card certificates and an internal CA. I'm on Windows 10 and using Python 3.9 to test, I think it's on TLS 1.2, and the error happens during handshake. I'll keep debugging on my end and hope for the best.

Anyway, here is my proof of concept code to get requests to use oscrypto. If anyone uses this be aware I am not well versed in SSL/TLS logic, I've just shoved together a few APIs and hoped it worked, mostly based on @Lukasa 's excellent comment.

import requests
import ipaddress
import socket as socket_
import requests.adapters
from ssl import SSLContext
from oscrypto._errors import pretty_message
from oscrypto._types import type_name, str_cls
from oscrypto.tls import TLSSocket, TLSSession

SSL_WRITE_BLOCKSIZE = 16385

def is_ip(ip):
    try:
        ipaddress.ip_address(ip)
    except ValueError:
        return False
    else:
        return True

class OsCryptoWrappedSocket:
    """API-compatibility wrapper"""

    def __init__(self, sock, session, timeout=10):
        if sock:
            ip, port = sock.getpeername()
            self.oscrypto_socket = TLSSocket(ip, port, timeout, session)
        else:
            self.oscrypto_socket = TLSSocket(None, None, timeout, session)
        self.suppress_ragged_eofs = True
        self._io_refs = 0
        self._closed = False

    def fileno(self) -> int:
        return self.oscrypto_socket._socket.fileno()

    def _decref_socketios(self) -> None:
        if self._io_refs > 0:
            self._io_refs -= 1
        if self._closed:
            self.close()

    def recv(self, bufsize, flags=None) -> bytes:
        return self.oscrypto_socket.read(bufsize)

    def recv_into(self, buffer, nbytes=None, flags=None) -> int:
        buffer_len = len(buffer)
        if nbytes:
            max_length = min(buffer_len, nbytes)
        else:
            max_length = buffer_len
        
        read_bytes = self.oscrypto_socket.read(max_length)
        if not read_bytes:
            return 0

        buffer[:len(read_bytes)] = read_bytes
        return len(read_bytes)

    def settimeout(self, timeout: float) -> None:
        return self.oscrypto_socket._socket.settimeout(timeout)

    def _send_until_done(self, data: bytes) -> int:
        self.oscrypto_socket.write(data)
        return len(data)

    def sendall(self, data: bytes) -> None:
        total_sent = 0
        while total_sent < len(data):
            sent = self._send_until_done(
                data[total_sent : total_sent + SSL_WRITE_BLOCKSIZE]
            )
            total_sent += sent

    def shutdown(self) -> None:
        self.oscrypto_socket.shutdown()

    def close(self) -> None:
        if self._io_refs < 1:
            try:
                self._closed = True
                return self.oscrypto_socket.close()
            except Exception:
                return
        else:
            self._io_refs -= 1

    def getpeercert(self, binary_form: bool = False):
        if not self.oscrypto_socket.certificate:
            return self.oscrypto_socket.certificate

        if binary_form:
            return self.oscrypto_socket.certificate.dump()

        return {
            "subject": ((("commonName", self.oscrypto_socket.certificate.subject.native["common_name"]),),),
            "subjectAltName": (("IP Address" if is_ip(v) else "DNS", v)
                               for v in self.oscrypto_socket.certificate.subject_alt_name_value.native),
        }

    def version(self):
        return self.oscrypto_socket.protocol

    @classmethod
    def wrap(cls, socket, hostname, session=None):
        if not isinstance(socket, socket_.socket):
            raise TypeError(pretty_message(
                '''
                socket must be an instance of socket.socket, not %s
                ''',
                type_name(socket)
            ))

        if not isinstance(hostname, str_cls):
            raise TypeError(pretty_message(
                '''
                hostname must be a unicode string, not %s
                ''',
                type_name(hostname)
            ))

        if session is not None and not isinstance(session, TLSSession):
            raise TypeError(pretty_message(
                '''
                session must be an instance of oscrypto.tls.TLSSession, not %s
                ''',
                type_name(session)
            ))

        new_socket = cls(None, session=session)
        new_socket.oscrypto_socket._socket = socket
        new_socket.oscrypto_socket._hostname = hostname
        new_socket.oscrypto_socket._handshake()

        return new_socket

OsCryptoWrappedSocket.makefile = socket_.socket.makefile 


class OSCryptoSSLContext(SSLContext):
    def wrap_socket(self,
                    sock,
                    server_side=False,
                    do_handshake_on_connect=True,
                    suppress_ragged_eofs=True,
                    server_hostname=None,
                    session=None):
        return OsCryptoWrappedSocket.wrap(sock, server_hostname, session)


class HTTPAdapterOSCrpyto(requests.adapters.HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=..., **pool_kwargs):
        return super().init_poolmanager(
            connections, maxsize, block=block, ssl_context=OSCryptoSSLContext(),
            **pool_kwargs
            )


with requests.Session() as session:
    session.mount("https://", HTTPAdapterOSCrpyto())
    response = session.get("https://www.bbc.co.uk/")
    print(response.status_code)

If I figure out the handshake issue I am getting I will update this sample code.

[notatallshaw]

Oh dear, this will probably never be reproducible publicly. These sites are using SSO with smart-card certificates and an internal CA. I'm on Windows 10 and using Python 3.9 to test, I think it's on TLS 1.2, and the error happens during handshake. I'll keep debugging on my end and hope for the best.

A colleague has tracked it down to failing when this function is called: https://docs.microsoft.com/en-us/windows/win32/api/sspi/nf-sspi-initializesecuritycontextw

So some progress, except it has over 10 parameters and no real way to debug them yet, aha. If you have any pointers they'd be appreciated, otherwise we'll very slowly debug when we have time.

[notatallshaw]

So for anyone reading this I'm fairly sure my code in #10 (comment) does get requests to use oscrypto, but probably because of #4 it doesn't work for my particular use case.