Add Support for merging multiple different NotificationPolicies

[consooo]

Description

Currently, there is no support for merging NotificationPolicies. Issue #55 proposed a workaround that involves specifying notification settings directly in Grafana alerts. However, this approach only applies to Grafana alerts and does not offer the flexibility or centralized control that NotificationPolicies could provide.

This limitation presents the following challenges:

• Limited Alert Engine Compatibility: The current workaround applies only to Grafana alerts. Users cannot manage notifications from other alert engines such as Cortex, or Mimir under a unified policy. In our case for instance, we collect all cluster alerts based on labels. For each new cluster, the NotificationPolicy needs to be adjusted. Otherwise we would need to replicate all alert rules to grafana rules.
• fixed Notification Endpoint: Since notification settings are embedded directly in the Grafana alert, the endpoint is fixed and cannot be dynamically adjusted based on different alert sources or conditions. This limits flexibility and prevents users from routing alerts dynamically or tailoring responses based on the alert engine or alert characteristics.

Proposed Solution

The proposed solution from b-m-f in #55 (comment) would suit us fine: a new CRD, like NotificationPolicyPatch, allows to specify a desired addition to the NotificationPolicy.policy field. The provider would then update the NotificationPolicy accordingly.

[MasterExecuson]

We are having the same issue.

We control multiple environments in the different cluster poiting to the same Grafana instance. Everything else works fine, except the NotificationPolicy.

It's pretty much impossible with only the CRDs to make that work. This could be similar to the OnCall Integration and OnCall Route approach, where we have the main one with the default and add a route for each condition.

[Duologic]

Have you looked into creating a pipeline function in Crossplane Compositions?

That might be a good stepping stone towards solving this in Crossplane land.

[MasterExecuson]

Hey!

Yes I have, we use it to sync OnCall Schedules and OnCall Integrations, but as I mentioned, with the ContactPoint and Alerts CRDs on different cluster, there is very little I can do to actually sync all the ContactPoints into the same NotificationPolicy. This is even harder due to how the NotificationPolicy handles non-existent ContactPoints, pretty much refusing to reconsile until one of them is fixed.

[Duologic]

I don't think this should be solved as part of the Crossplane provider, this should probably become a feature request towards the alerting product so that this 'patch' feature can be supported at the API level instead.

[Duologic]

Closing this awaiting upstream to take this into consideration.