feat(helm): add live logs deployment (#827)

[jlemesh]

Closes #824

OpenSearch

Optionally deploy OpenSearch alongside REANA (enabled: false by default).

In dev environment, deploy one instance with security features off (i. e. no TLS, no authn/authz)

In prod, also only one OpenSearch node is configured. To add more nodes, each of these nodes needs to have its own PersistentVolumeClaim and PersistentVolume as each node writes to a directory with the same name, but different contents - this will not work with current reana-shared-persistent-volume or reana-infrastructure-persistent-volume. It is possible to use volumeClaimTemplates with the StorageClass of choice (configured in OpenSearch Helm chart persistence.storageClass).

Volumes configuration tested with nfs-ganesha-server-and-external-provisioner/nfs-server-provisioner, for one and multiple nodes.

OpenSearch allows deploying nodes of different types, e. g. master and data. This configuration would not allow to do it, we need to add the second opensearch dependency, e. g. opensearch-master, and use the same Helm chart with different configuration. I think it is an overkill at this stage, as this is needed for bigger clusters.

TLS certificates for OpenSearch nodes are generated automatically by a Helm function, similar to how it is done for Ingress, but without automatic rotation on each helm upgrade. It is possible for a user to supply his own certificates, by putting them in a secret and mounting to an OpenSearch pod (in opensearch.secretMounts). Admin TLS certificates provide superadmin permissions to whoever uses them, and are stored in the same secret - there is no specific admin user.

Two users are configured - reana and fluentbit. Their passwords need to be prepared by first spinning up OpenSearch instance in dev environment, connecting to a pod and running hash.sh script (described in docs). After getting the hashes for the passwords, they should be supplied to Helm with --set opensearch.customSecurityConfig.internalUsers.reana.hash='$So$Me$pASsWOrD.HasH' --set opensearch.customSecurityConfig.internalUsers.fluentbit.hash='$So$Me$pASsWOrD.HasH'.

Also enabled SSL cert reload, which allows updating SSL certs without restarting OpenSearch instance. Link to docs provided in the comments for those who will want to use it.

FluentBit

Optionally deploy FluentBit alongside REANA (enabled: false by default). Collects job and workflow logs with tail input plugin and pushes logs to OpenSearch. Allows configuring custom TLS certificates for OpenSearch connection.

reana-dev

Update cluster-deploy command and add --live-logs flag, which enables opensearch and fluent-bit dependencies.

The command breaks Helm configuration if it contains long (multiline) strings. PyYAML library first reads the YAML config file, converts multiline strings to one long string delimited with " and then wraps it while dumping, i. e.:

<...>
key: "value\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\n"\
"value\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\n"\
"value\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\n"
<...>

If the string is used with Helm template values (tpl function), it adds whitespaces in places where \ was used, which results in malformed configuration for OpenSearch and FluentBit.

dump(width=100000) will add \ only if the string is longer than 100000 characters.

How to test

Dev setup

reana-dev cluster-deploy --admin-email john.doe@example.org --admin-password mypwd1 --mode debug --live-logs
reana-dev run-example -c r-d-helloworld -w serial --submit-only
curl -XGET "https://localhost:9200/fluentbit-job_log/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "kubernetes.labels.job-name.keyword": "<reana run job name>"
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}' | jq

Prod setup

Password hash is for ReanaOS1= password.

helm install reana reana/helm/reana --wait --debug \
  --set opensearch.customSecurityConfig.internalUsers.reana.hash='$2y$12$e4RWvWxRW0QO8QtnDd.70eV/fJDJ2F7uUo2ox1P07bIpRArYDSUWK' \
  --set opensearch.customSecurityConfig.internalUsers.fluentbit.hash='$2y$12$e4RWvWxRW0QO8QtnDd.70eV/fJDJ2F7uUo2ox1P07bIpRArYDSUWK' \
  --set fluent-bit.outputConfig.httpPasswd='ReanaOS1=' \
  --set opensearch.enabled=true \
  --set fluent-bit.enabled=true
reana-dev run-example -c r-d-helloworld -w serial --submit-only
curl -u "reana:ReanaOS1=" --insecure -XGET "https://localhost:9200/fluentbit-job_log/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "kubernetes.labels.job-name.keyword": "<reana job name>"
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}' | jq

[codecov]

Codecov Report

Attention: Patch coverage is 75.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 31.19%. Comparing base (834b3eb) to head (4faf1f0).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
reana/reana_dev/cluster.py	75.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #827       +/-   ##
===========================================
+ Coverage   19.22%   31.19%   +11.96%     
===========================================
  Files          26       26               
  Lines        2512     2491       -21     
===========================================
+ Hits          483      777      +294     
+ Misses       2029     1714      -315     

Files with missing lines	Coverage Δ
reana/reana_dev/cluster.py	50.28% <75.00%> (+50.28%)	⬆️

... and 9 files with indirect coverage changes