OpenLDAP: unable to get local issuer certificate

[pappapo]

If anyone have a recipe to have this plugin and nagios working with ldap please let me know.
We have check_ssl_cert working just fine with http, smtp etc., but unable to get it to work with OpenLDAP, it seems OpenLDAP works different from all the other protocols. Also, other SSL check plugins does not complain when checking same ldap hosts.

Thank you in advance,
Per

[olcTLSCACertificatePath]
[olcTLSCACertificateFile]
[olcTLSCertificateFile]
[olcTLSCertificateKeyFile]

are all properly defined and all clients can access the directory with TLS.

The command will return

"Cannot verify certificate: unable to get local issuer certificate, unable to verify the first certificate"

openssl s_client -showcerts -verify 5 -connect db.nethead.se:389 
verify depth is 2
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 317 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

[matteocorti]

With the plug-in you just have to specify --protocol ldap, with OpenSSL you need to specify -starttls ldap.

[pappapo]

Yes, but that does not change
Verify return code: 21 (unable to verify the first certificate)
the command is,
check_ssl_cert -H db5.nethead.se -w 21 -c 14 -p 389 --protocol ldap --rootcert-dir /etc/ssl/certs
adding the -A switch will make the error go away of course but that sort of negates the whole idea.

Anyway, thanks a lot for a great plugin!

[matteocorti]

Strange. Do you also have the debugging output? As far as I can see the host ist not publicly available so I cannot test ...

[pappapo]

Note that with all other protocols and files we have no errors, only ldap

SSL_CERT CRITICAL db5.nethead.se:ldap: Cannot verify certificate: unable to get local issuer certificate
unable to verify the first certificate|days_chain_elem1=26;21;14;; 
Sat Jul 29 13:10:20 CEST 2023
Command line arguments: -H db5.nethead.se -w 21 -c 14 -p 389 --protocol ldap --rootcert-dir /etc/ssl/certs --debug-file 1
  TMPDIR = /tmp
Adding the domain if missing
HOST = db5.nethead.se
SNI                 = 
HOST_NAME           = db5.nethead.se
HOST_ADDR           = db5.nethead.se
NAMES_TO_BE_CHECKED = __HOST__
Checking if db5.nethead.se is an IP address
db5.nethead.se is not an IP address
HOST_IS_IP.         = 0
Checking if db5.nethead.se is an IP address
db5.nethead.se is not an IP address
Adding db5.nethead.se to NAMES_TO_BE_CHECKED
NAMES_TO_BE_CHECKED = db5.nethead.se
curl binary needed. SSL Labs = , OCSP = 1, CURL = , IGNORE_CONNECTION_STATE=, FILE_URI=
curl binary not specified
curl available: /usr/local/bin/curl
curl 8.1.2 (amd64-portbld-freebsd12.4) libcurl/8.1.2 OpenSSL/1.1.1t zlib/1.2.13 libssh2/1.11.0 nghttp2/1.55.1
Release-Date: 2023-05-30
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL threadsafe TLS-SRP UnixSockets
-c specified: 14
-w specified: 21
Executing comparison '1814400 <= 1209600'
  bc result = 0
  returning 1
ROOT_CA = -CApath /etc/ssl/certs
mktemp available: /usr/bin/mktemp
file version: file-5.43
magic file from /usr/share/misc/magic
nmap binary not needed. No disallowed protocols
perl available: /usr/local/bin/perl
date available: /bin/date
checking date version
date computation type: BSD
'/usr/bin/openssl s_client' supports '-servername': using -servername db5.nethead.se
Proxy settings (before):
  http_proxy  = 
  https_proxy = 
  HTTP_PROXY  = 
  HTTPS_PROXY = 
Proxy settings (after):
  http_proxy  = 
  https_proxy = 
  HTTP_PROXY  = 
  HTTPS_PROXY = 
  s_client    =  
  curl        =  
'/usr/bin/openssl s_client' supports '-name': using nagios.nethead.se
'/usr/bin/openssl s_client' supports '-xmpphost': using -xmpphost db5.nethead.se
HOST_HEADER = db5.nethead.se
Sanity checks: OK
temporary file /tmp/q9a93X created
temporary file /tmp/5AUOZe created
temporary file /tmp/J5jucU created
temporary file /tmp/FNor3j created
temporary file /tmp/heTnmi created
temporary file /tmp/2wImPD created
temporary file /tmp/vMky8q created
Temporary files created
db5.nethead.se is not an IP address
fetch_certificate: PROTOCOL = ldap
exec_with_timeout echo | /usr/bin/openssl s_client    -starttls ldap -showcerts -connect db5.nethead.se:389 -servername db5.nethead.se   -verify 6 -CApath /etc/ssl/certs       2> /tmp/5AUOZe 1> /tmp/q9a93X  
  TIMEOUT_REASON = fetching certificate
executing with timeout (120s): echo | /usr/bin/openssl s_client    -starttls ldap -showcerts -connect db5.nethead.se:389 -servername db5.nethead.se   -verify 6 -CApath /etc/ssl/certs       2> /tmp/5AUOZe 1> /tmp/q9a93X
  start time = 1690629020
/usr/bin/timeout 120 /bin/sh -c "echo | /usr/bin/openssl s_client    -starttls ldap -showcerts -connect db5.nethead.se:389 -servername db5.nethead.se   -verify 6 -CApath /etc/ssl/certs       2> /tmp/5AUOZe 1> /tmp/q9a93X"
  end time = 1690629020
  new timeout = 120
Return value of the command = 0
Negotiated protocol: 
openssl_version 3.0.0
Checking if OpenSSL version is at least 3.0.0 ( '3' '0' '0' ':0' )
openssl version: OpenSSL 1.1.1t-freebsd  7 Feb 2023
Current version 1.1.1t ( '1' '1' '1' 't:116' )
checking TLS renegotiation
exec_with_timeout printf 'R\n' | /usr/bin/openssl s_client  -crlf -connect db5.nethead.se:389 -servername db5.nethead.se   -starttls ldap 2>&1 | grep -F -q err  
  TIMEOUT_REASON = checking TLS renegotiation
executing with timeout (120s): printf 'R\n' | /usr/bin/openssl s_client  -crlf -connect db5.nethead.se:389 -servername db5.nethead.se   -starttls ldap 2>&1 | grep -F -q err
  start time = 1690629020
/usr/bin/timeout 120 /bin/sh -c "printf 'R\n' | /usr/bin/openssl s_client  -crlf -connect db5.nethead.se:389 -servername db5.nethead.se   -starttls ldap 2>&1 | grep -F -q err"
  end time = 1690629020
  new timeout = 120
extracting cert attribute enddate
extracting cert attribute startdate
extracting cert attribute cn
extracting cert attribute subject
SUBJECT = subject=CN = db5.nethead.se
extracting cert attribute serial
SERIAL = 04EDABF4D15394BEED8322B1D14ACBB4A729
extracting cert attribute version
X509_VERSION = 3 (0x2)
extracting cert attribute fingerprint
FINGERPRINT = CB:59:35:1D:4A:D0:35:D0:7D:C2:AC:24:CA:05:61:8A:F6:80:CB:A8
Checking if x509 supports the -ext option
extracting cert attribute keyUsage
Certificate purpose is not defined as critical
extracting cert attribute oscp_uri_single
extracting cert attribute oscp_uri
OCSP_URI = http://r3.o.lencr.org
Extracting issuers
  Number of certificates in the chain: 1
Checking certificate chain
    extracting issuer for element 1
extracting cert attribute issuer
ELEMENT_ISSUER=Let's Encrypt
ELEMENT_ISSUER=R3
ISSUERS=Let's Encrypt
ISSUERS=R3
Certificate chain check finished
ISSUERS = 
Let's Encrypt
R3
extracting cert attribute issuer_uri_single
extracting cert attribute issuer_uri
extracting cert attribute sig_algo
extracting cert attribute subjectAlternativeName
subjectAlternativeName = db5.nethead.se
Check the common name and alternative names
CN                       = db5.nethead.se
ALTNAMES                 = 1
NAMES_TO_BE_CHECKED      = db5.nethead.se
  checking 'db5.nethead.se'
    common name
      checking if (3) db5.nethead.se matches ^db5.nethead.se$
        db5.nethead.se matches ^db5.nethead.se$
    alternative names
      check altname: db5.nethead.se
      db5.nethead.se matches ^db5.nethead.se$
 CN check finished
Checking expiration date
Number of certificates in CA chain: 1
------------------------------------------------------------------------------
-- Checking element 1
extracting cert attribute cn
Checking expiration date of element 1 (db5.nethead.se)
extracting cert attribute enddate
Validity date on cert element 1 (db5.nethead.se) is Aug 25 01:23:13 2023 GMT
Date computations: BSD
Computing number of hours until 'Aug 25 01:23:13 2023 GMT' with BSD
Computing '(1692926593-1690629021)/3600'
Hours until Aug 25 01:23:13 2023 GMT: 638
Computing '638/24'
Computing '638 * 3600'
Adding line to prometheus days output: cert_days_chain_elem{cn="db5.nethead.se", element="1"} 26
  valid for 26 days
executing: /usr/bin/openssl x509 -noout -checkend 0 on cert element 1 (db5.nethead.se)
executing: /usr/bin/openssl x509 -noout -checkend 1209600 on cert element 1 (db5.nethead.se)
executing: /usr/bin/openssl x509 -noout -checkend 1814400 on cert element 1
Adding line to prometheus validity output: cert_valid_chain_elem{cn="db5.nethead.se", element="1"} 0
------------------------------------------------------------------------------
Checking OCSP status of element 1
temporary file /tmp/opt49W created
Storing the chain element in /tmp/opt49W
Checking revocation via OCSP
extracting cert attribute issuer_hash
Issuer hash: 8d33f237
extracting cert attribute issuer_uri
Chain element issuer URIs: http://r3.i.lencr.org/
checking issuer URIs: http://r3.i.lencr.org/
OCSP: fetching issuer certificate http://r3.i.lencr.org/ to /tmp/2wImPD
exec_with_timeout /usr/local/bin/curl    --silent --user-agent 'check_ssl_cert/2.37.0' --location \"http://r3.i.lencr.org/\" > /tmp/2wImPD  
  TIMEOUT_REASON = OCSP: fetching issuer http://r3.i.lencr.org/
executing with timeout (120s): /usr/local/bin/curl    --silent --user-agent 'check_ssl_cert/2.37.0' --location \"http://r3.i.lencr.org/\" > /tmp/2wImPD
  start time = 1690629021
/usr/bin/timeout 120 /bin/sh -c "/usr/local/bin/curl    --silent --user-agent 'check_ssl_cert/2.37.0' --location \"http://r3.i.lencr.org/\" > /tmp/2wImPD"
  end time = 1690629021
  new timeout = 120
OCSP: issuer certificate type (1): Certificate, Version=3
OCSP: issuer certificate type (2): Certificate, Version=3
OCSP: converting issuer certificate from DER to PEM
OCSP: issuer certificate type (3): PEM certificate
extracting cert attribute oscp_uri
OCSP: URIs = http://r3.o.lencr.org
OCSP: URI = http://r3.o.lencr.org
OCSP: host = r3.o.lencr.org
openssl ocsp supports the -header option
/usr/bin/openssl ocsp -header requires 'key=value'
executing /usr/bin/openssl ocsp -timeout "120" -no_nonce -issuer /tmp/2wImPD -cert /tmp/opt49W  -url http://r3.o.lencr.org  -header HOST=r3.o.lencr.org
OCSP: response = Response verify OK
OCSP: response = /tmp/opt49W: good
OCSP: response =        This Update: Jul 28 18:00:00 2023 GMT
OCSP: response =        Next Update: Aug  4 17:59:58 2023 GMT
Timeout before OCSP check: 120
Timeout after OCSP check:  120
------------------------------------------------------------------------------
extracting cert attribute email
EMAIL = 
Checking if the certificate was self signed
Error: verify depth is 6
Error: depth=0 CN = db5.nethead.se
Error: verify error:num=20:unable to get local issuer certificate
Error: verify return:1
Error: depth=0 CN = db5.nethead.se
Error: verify error:num=21:unable to verify the first certificate
Error: verify return:1
Error: depth=0 CN = db5.nethead.se
Error: verify return:1
Error: DONE
CRITICAL >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
prepend_critical_message: new message    = Cannot verify certificate: unable to get local issuer certificate
unable to verify the first certificate
prepend_critical_message: CRITICAL_MSG   = 
prepend_critical_message: ALL_MSG 1      = 
prepend_critical_message: MSG 2          = SSL_CERT CRITICAL db5.nethead.se:ldap: Cannot verify certificate: unable to get local issuer certificate
unable to verify the first certificate
prepend_critical_message: CRITICAL_MSG 2 = SSL_CERT CRITICAL db5.nethead.se:ldap: Cannot verify certificate: unable to get local issuer certificate
unable to verify the first certificate
prepend_critical_message: ALL_MSG 2      = \n    SSL_CERT CRITICAL db5.nethead.se:ldap: Cannot verify certificate: unable to get local issuer certificate
unable to verify the first certificate
CRITICAL <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
openssl_version 1.1.0
Checking if OpenSSL version is at least 1.1.0 ( '1' '1' '0' ':0' )
openssl version: 1.1.1t
Current version 1.1.1t ( '1' '1' '1' 't:116' )
Checking Signed Certificate Timestamps (SCTs)
extracting cert attribute sct
cleaning up temporary files

/tmp/q9a93X
/tmp/5AUOZe
/tmp/J5jucU
/tmp/FNor3j
/tmp/heTnmi
/tmp/2wImPD
/tmp/vMky8q
/tmp/opt49W
exiting with CRITICAL
ALL_MSG = \n    SSL_CERT CRITICAL db5.nethead.se:ldap: Cannot verify certificate: unable to get local issuer certificate
unable to verify the first certificate
number of errors =        2

[matteocorti]

Thanks. It's strange as OpenSSL does not complain. I'll check when I'll have some time with my laptop (I'll be traveling until end of next week).
Seems a bug in the cert validation.

[matteocorti]

I see the problem.

/usr/bin/openssl s_client  -crlf -connect db5.nethead.se:389 -servername db5.nethead.se   -starttls ldap 

Is called without the root CA directory.

But should, since ${ROOT_CA} is there:

        irc | ldap)
            exec_with_timeout "echo | ${OPENSSL} s_client ${SECURITY_LEVEL} ${INETPROTO} ${CLIENT} ${CLIENTPASS} -starttls ${PROTOCOL} -showcerts -connect ${HOST_ADDR}:${PORT} ${SERVERNAME} ${SCLIENT_PROXY} ${SCLIENT_PROXY_ARGUMENT} -verify 6 ${ROOT_CA} ${SSL_VERSION} ${SSL_VERSION_DISABLED} ${SSL_AU} ${STATUS} ${DANE} ${RENEGOTIATION} 2> ${ERROR} 1> ${CERT}"
            RET=$?
            ;;

[matteocorti]

Can you try with -r /etc/ssl/certs instead of --rootcert-dir /etc/ssl/certs

[pappapo]

Already tested several variations of -r and --rootcert-dir/usr/local/etc/ssl and --rootcert-file /usr/local/etc/ssl/cert.pem

Could it be an FreeBSD openssl/ldap specific issue?

[matteocorti]

Seems a bug but I cannot reproduce it with the current version.

[pappapo]

Some more info,
the server log file indicates that bind with TLS succeeds and tcpdump shows certs being sent although one cannot see which ones of course, but there are several.
Will try the FreeBSD community now as this may be OS specific.
What OS is you testing ground?

[matteocorti]

See the related issue: the root cert directory is not correctly handled, but I cannot reproduce the problem with the current version.

[matteocorti]

Hi It’s a bug. The CA root directory you specify is not used. You are using version 2.39. Please upgrade to the latest (2.71). With the new version, it should work.

[pappapo]

Already test several versions including latest, same error. Also tested with two versions of FreeBSD. Because the openssl command
openssl s_client -showcerts -verify 5 -connect db5.nethead.se:389
fails but
openssl s_client -showcerts -verify 5 -connect www5.nethead.se:443
works
would that not show it is a local bug and not with your code?

[matteocorti]

OK if it works without the root cert directory for HTTP is strange that it does not with LDAP. Since the error points in that direction, can you also try

openssl s_client -showcerts -verify 5 -connect db5.nethead.se:389 -CApath /etc/ssl/certs

Just to exclude the problem with the missing parameter in the plugin.

[pappapo]

Same result as above.

[matteocorti]

Strange. Are you sure that the certificate is delivered correctly (e.g. with the intermediate certificate)? Is there a way I could test?

[lukastribus]

So is 389 starttls based or not? Seems like you guys keep changing between starttls and non starttls commands.

[pappapo]

[olcTLSCACertificateFile] /usr/local/etc/ssl/lets-encrypt-r3.pem
solved it, thank you very much for the hint.
[olcTLSCACertificatePath] /etc/ssl/certs
is not enough for slapd apparently. Why it works for the other protocols without pointing to the intermediate is beyond me but perhaps you know. Thanks again /Per

[pappapo]

We are 389 starttls based since forever and this is the first time we saw a problem.

[matteocorti]

Strange indeed. It could be interesting to ask on an OpenSSL related forum.