Cannot install Prometheus with remoteWrite enabled

[Hy3n4]

App Name

prometheus

Operating System

TrueNAS-SCALE-23.10.0.1

App Version

13.0.17

Application Events

Error: [EFAULT] Failed to install chart release: Error: INSTALLATION FAILED: Prometheus.monitoring.coreos.com "prometheus-prometheus" is invalid: [spec.remoteWrite[0].basicAuth.password: Invalid value: "string": spec.remoteWrite[0].basicAuth.password in body must be of type object: "string", spec.remoteWrite[0].basicAuth.username: Invalid value: "string": spec.remoteWrite[0].basicAuth.username in body must be of type object: "string"]

Application Logs

[2023/11/09 08:58:21] (ERROR) middlewared.job.run():440 - Job <bound method returns.<locals>.returns_internal.<locals>.nf of <middlewared.plugins.chart_releases_linux.chart_release.ChartReleaseService object at 0x7f097256e670>> failed: CallError('Failed to install chart release: Error: INSTALLATION FAILED: Prometheus.monitoring.coreos.com "prometheus-prometheus" is invalid: [spec.remoteWrite[0].basicAuth.password: Invalid value: "string": spec.remoteWrite[0].basicAuth.password in body must be of type object: "string", spec.remoteWrite[0].basicAuth.username: Invalid value: "string": spec.remoteWrite[0].basicAuth.username in body must be of type object: "string"]\n')

Application Configuration

Not sure how to provide configuration since the chart cannot be even installed.

Describe the bug

Somehow helm thinks that username and password are not type string even if the error message says something different.
Tried to be creative in the username and password field and fill in the base64 encoded string, just numbers, just letter. Even empty password (questions.yaml says, that it's optional).

To Reproduce

1. Open TruneNAS UI
2. Go to Apps
3. Open Available Applications
4. Scroll down or search for Prometheus (The TrueCharts one - currently in version 2.47.2_13.0.12)
5. Click Install
6. Fill all needed information
7. In the Remote Write section click Add
8. Fill the URL, Username and Password fields
9. Scroll down and click on Save
10. Wait for the error message

Expected Behavior

Install chart without any problems.

Screenshots

Additional Context

I want to store metrics in a remote Mimir instance.

I've read and agree with the following

• I've checked all open and closed issues and my issue is not there.

[PrivatePuffin]

Please update to a supported version of TrueNAS SCALE before opening a bugreport...
Feel free to request reopen when updated.

[Hy3n4]

Sorry, I'm not sure what you mean by the upgrade, because I am running the latest version TrueNAS-SCALE-22.12.4.2 🤷‍♂️ If it's not supported version can you redirect me to a list of supported TrueNAS SCALE versions?

[PrivatePuffin]

Sorry, I'm not sure what you mean by the upgrade, because I am running the latest version TrueNAS-SCALE-22.12.4.2 🤷‍♂️ If it's not supported version can you redirect me to a list of supported TrueNAS SCALE versions?

You've just edited your PR, previously you've stated to be using a non-supported older version in your PR.
The list of supported versions is on the website.

[PrivatePuffin]

Even so, we do not accept bugreports for TrueNAS SCALE 22.12.4.2 either, please update to a version for which we accept bugreport, in accordance to the supported version list on the website

[Hy3n4]

So, I just upgraded to TrueNAS-SCALE-23.10.0.1 and the error is still persisting and is still pretty much the same.

I've also updated the version in the original BR.

Actual error trace is below:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 426, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 464, in __run_body
    rv = await self.method(*([self] + args))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 194, in nf
    rv = await func(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 487, in do_create
    await self.middleware.call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1398, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1352, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/helm.py", line 49, in helm_action
    raise CallError(f'Failed to {tn_action} App: {errmsg}')
middlewared.service_exception.CallError: [EFAULT] Failed to install App: Error: INSTALLATION FAILED: 1 error occurred:
	* Prometheus.monitoring.coreos.com "prometheus-prometheus" is invalid: [spec.remoteWrite[0].basicAuth.username: Invalid value: "string": spec.remoteWrite[0].basicAuth.username in body must be of type object: "string", spec.remoteWrite[0].basicAuth.password: Invalid value: "string": spec.remoteWrite[0].basicAuth.password in body must be of type object: "string"]

[Hy3n4]

Hi,
I've been digging a bit and it seems that prometheus api spec is expecting SecretKeySelector instead of pure paintext.

IMHO there has to be a secret file created as a part of the installation process and it should be referenced in the _prometheus.tpl template instead of passing the pure values from values.yaml.

[PrivatePuffin]

To be clear: Prometheus needs to be completely refactored from scratch.
There is no use commenting on this further. so i'll lock this.

[PrivatePuffin]

ALso: removeWrite isn't even a feature we ever aimed to support at this stage.

[alfi0812]

Closing as SCALE isnt a supported platform any longer. Feel free to open a new Issue if the issue persists on one of the currently supported platforms.