[BUG] Admin password getting lost after pod restart

[git-hyagi]

Describe the bug
The Pulp admin password from admin-password secret is overwriting the password defined through Pulp API.

To Reproduce
Steps to reproduce the behavior:
Modify the admin password through Pulp API.
Delete Pulp pods (didn't test yet, but maybe just reprovisioning pulp-api pods would be enough).
Try to use the modified password and it will not work, but the "old" stored in the secret will.

Expected behavior
The password defined in the admin-password secret should be used only as a default password. As soon as a user changed it, the secret should not be checked anymore.

Additional context
In a quick chat with @dkliban++ we identified a possible fix:

Before the operator set the password, we need to:
* if `password_is_set: false`
  * run a query in the database to check if the password is still null:
  > select password from auth_user where username='admin';

  * if the password field (from the last query) is null, set it with the default value from secret
  * set a status field `password_is_set: true` in Pulp CR (this will be used to control if the password has already been set)
* if `password_is_set: true` do nothing 

Notes
We will probably need to also fix this in ansible version.
Ref https://issues.redhat.com/browse/AAP-8344

[git-hyagi]

We'll also need to update the doc:
https://docs.pulpproject.org/pulp_operator/configuring/secrets/#pulp-admin-password

[dkliban]

Thank you @git-hyagi for pointing out that this is an issue in the image itself[0].

[0] https://github.com/pulp/pulp-oci-images/blob/latest/images/assets/pulp-api#L19-L27

[dkliban]

It sounds like we need to write a file 'password-has-been-set' somewhere that is mounted into the container. Then we should check for the existence of this file and not reset the password when it is present.

[mikedep333]

Consider (but you're not required to implement) the example of the Nexus container.

It sets the password to a random default, and let's you view the password.

If you set the password, it gets written to a file, and it prevents the random default from being set.

[mdellweg]

In the s6 images we only set the admin password (from $PULP_DEFAULT_ADMIN_PASSWORD) if there is no admin account yet. Can we just do the same here?
How much backwards compatibility do we need to maintain?

[git-hyagi]

For the golang version of pulp-operator, I believe this is not needed anymore (but we still need to decide how to proceed with Galaxy/AH/Ansible version).
If we consider that the operator should be the "source of truth" for Pulp deployment, then I think it is fine to let the admin password in the Secret. One example where this can be helpful is for clusters with vault/eso integration. Modifying the password in vault would update it in Pulp without having to manually execute pulpcore-manager reset-admin-password or make a Pulp API request.

To avoid having to reprovision an API pod to re-run the pulpcore-manager reset-admin-password, we added a watcher in the admin-pwd-secret and, whenever it is modified, the operator will create a k8s job to update the password: https://docs.pulpproject.org/pulp_operator/configuring/reset_admin_pwd/