Implement SSL Verification

[dadoonet]

We now support both ssl_verification (default to true) and ca_certificate (default to null).

In order to ingest documents to Elasticsearch over HTTPS based connection, you obviously need to set the URL to https://your-server-address. If your server is using a certificate that has been signed by a Certificate Authority, then you're good to go. For example, that's the case if you are running Elasticsearch from cloud.elastic.co.

But if you are using a self signed certificate, which is the case in development mode, you need to either ignore the ssl check (not recommended) or provide the certificate to the Elasticsearch client.

To bypass the SSL Certificate verification, you can use the ssl_verification option:

name: "test"
elasticsearch:
  api_key: "VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw=="
  ssl_verification: false

If you are running Elasticsearch from a Docker container, you can copy the self-signed certificate generated in /usr/share/elasticsearch/config/certs/http_ca.crt to your local machine:

docker cp CONTAINER_NAME:/usr/share/elasticsearch/config/certs/http_ca.crt /path/to/certificate

And then, you can specify this file in the elasticsearch.ca_certificate option:

name: "test"
elasticsearch:
  api_key: "VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw=="
  ca_certificate: /path/to/certificate/http_ca.crt

You can also import your certificate into <JAVA_HOME>\lib\security\cacerts.

For example, if you have a root CA chain certificate or Elasticsearch server certificate in DER format (it's a binary format using a .cer extension), you need to:

1. Logon to server (or client machine) where FSCrawler is running
2. Run:

keytool -import -alias <alias name> -keystore "<JAVA_HOME>\lib\security\cacerts" -file <Path of Elasticsearch Server certificate or Root certificate>

It will prompt you for the password. Enter the certificate password like changeit.

3. Make changes to FSCrawler _settings.json file to connect to your Elasticsearch server over HTTPS:

name: "test"
elasticsearch:
  api_key: "VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw=="
  nodes:
  - url: "https://localhost:9243"

If you can not find keytool, it probably means that you did not add your JAVA_HOME/bin directory to your path.

Closes #1538.

[sonarcloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud