Modelmesh Prometheus monitoring does not work in RHODS 2.4

[lugi0]

Describe the bug

The Prometheus instance of modelmesh monitoring does not work in RHODS 2.4 - there are no targets present and no queries can be run against it

To Reproduce
Steps to reproduce the behavior:

1. Go to Networking -> Routes in NS redhat-ods-monitoring
2. Click on Prometheus Route
3. Check targets in Status -> Targets or try to run a query
4. No results / no targets are present

Expected behavior

Targets can be found and queries can be run

Screenshots

Environment (please complete the following information):

RHODS 2.4

Additional context

Metrics can be run in Openshift's Metrics page, they are present and working correctly. I could not find the expected targets anywhere else though.

[zdtsw]

what i can see from cluster:
https://github.com/red-hat-data-services/odh-manifests/blob/rhods-2.4/modelmesh-monitoring/base/prometheus/rhods-prometheus.yaml#L36 is trying to use servicemonitor "modelmesh-serving" which uses service "modelmesh-metrics-monitor" to select pod with label "modelmesh-service=modelmesh-serving" but there is no pod with this lable in application namespace.