Add support for priorityClassName (fixes #1034)

[olim7t]

What this PR does:

Which issue(s) this PR fixes:
Fixes #1034

Checklist

• Changes manually tested
• Automated Tests added/updated
• Documentation added/updated
• CHANGELOG.md updated (not required for documentation PRs)
• CLA Signed: DataStax CLA

[sonarcloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[burmanm]

suggestion: This was not part of the original ticket, so these are not mandatory changes, but if you feel like these would worth doing doing later (but not part of this PR), we could create a ticket for it:

That is, we don't actually check if the PriorityClass exists. Right now that could cause the Pod to have a priorityClassName that points to nothing and this probably prevents the pod from being scheduled correctly. I think this check could be implemented in cass-operator also, I have no preference to that right now.

Second is actually creating PriorityClasses in k8ssandra-operator, but I would have to think about this feature more. Most users should probably set something like preemptionPolicy to Never to prevent new Cassandra pods from killing existing pods. Right now, that would be up to the user to understand what they're doing (like the entire feature, which I think we could consider "experienced users only").

If we use this feature in downstream products, then tracking PodStatus' nominatedNodeName might make sense also. And our PodDisruptionBudget policy is probably not very compatible with this feature either. The scheduling is quite complex on some of these and the documentation is a bit hand waving in some cases "we might or might not do this and maybe something else".