fix(cluster): remove raleted metrics label when cluster is deleted

[CharlesQQ]

What type of PR is this?
/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:
Fixes #5843

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

[karmada-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign garrybest for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 12.50000% with 14 lines in your changes missing coverage. Please review.

Project coverage is 46.28%. Comparing base (88c3e44) to head (bd68ef8).
Report is 38 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/metrics/cluster.go	0.00%	11 Missing ⚠️
pkg/controllers/cluster/cluster_controller.go	40.00%	2 Missing and 1 partial ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5866      +/-   ##
==========================================
+ Coverage   46.20%   46.28%   +0.07%     
==========================================
  Files         663      663              
  Lines       54580    54777     +197     
==========================================
+ Hits        25218    25352     +134     
- Misses      27739    27796      +57     
- Partials     1623     1629       +6     

Flag	Coverage Δ
unittests	46.28% <12.50%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[RainbowMango]

Have you tested it? Can you share the metrics before and after cluster is deleted?

[RainbowMango]

What if the removeCluster raises an error? Do we still clean up the metrics?

[CharlesQQ]

What if the removeCluster raises an error? Do we still clean up the metrics?

shouldn't clean up the metrics, I'll fix it!

[CharlesQQ]

Have you tested it? Can you share the metrics before and after cluster is deleted?

yes, I join a cluster named host

Before metrics: following metrics not removed after host deleted from karmada

cluster_ready_node_number{cluster_name="host"} 0
cluster_ready_state{cluster_name="host"} 1
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="0.005"} 0
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="0.01"} 0
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="0.025"} 0
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="0.05"} 1
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="0.1"} 3
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="0.25"} 3
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="0.5"} 4
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="1"} 4
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="2.5"} 4
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="5"} 4
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="10"} 4
cluster_sync_status_duration_seconds_bucket{cluster_name="host",le="+Inf"} 4
cluster_sync_status_duration_seconds_sum{cluster_name="host"} 0.667916613
cluster_sync_status_duration_seconds_count{cluster_name="host"} 4

After: metrics removed successfully.

[XiShanYongYe-Chang]

Thanks~
It tests with expection. And when I rejoin the cluster, the related cluster metrics occur as while.

[XiShanYongYe-Chang]

/lgtm
Can you help add the release note?