koordlet: add resctrl qos collector #2005

Rouzip · 2024-04-15T13:03:53Z

Ⅰ. Describe what this PR does

Add resctrl qos collector.

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it

After enable resctrl flag in config:

curl http://localhost:9316/metrics|grep resctrl

Ⅳ. Special notes for reviews

V. Checklist

I have written necessary docs and comments
I have added necessary unit tests and integration tests
All checks passed in make test

codecov · 2024-04-15T13:14:16Z

Codecov Report

Attention: Patch coverage is 57.64192% with 97 lines in your changes missing coverage. Please review.

Project coverage is 67.14%. Comparing base (83267af) to head (1dca501).

Files with missing lines	Patch %	Lines
...icsadvisor/collectors/resctrl/resctrl_collector.go	19.40%	54 Missing ⚠️
pkg/koordlet/resourceexecutor/resctrl.go	69.44%	12 Missing and 10 partials ⚠️
pkg/koordlet/metrics/resctrl.go	0.00%	9 Missing ⚠️
pkg/koordlet/metriccache/metric_types.go	0.00%	7 Missing ⚠️
pkg/koordlet/util/system/resctrl.go	90.56%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2005      +/-   ##
==========================================
- Coverage   67.19%   67.14%   -0.06%     
==========================================
  Files         451      454       +3     
  Lines       43468    43686     +218     
==========================================
+ Hits        29208    29331     +123     
- Misses      11714    11798      +84     
- Partials     2546     2557      +11

Flag	Coverage Δ
unittests	`67.14% <57.64%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Rouzip <1226015390@qq.com>

Rouzip · 2024-05-09T12:32:31Z

Sorry for the late pr, any comments is welcome.

Signed-off-by: Rouzip <1226015390@qq.com>

pkg/koordlet/metrics/resctrl.go

pkg/koordlet/metricsadvisor/collectors/resctrl/resctrl_collector.go

pkg/koordlet/util/system/resctrl.go

pkg/koordlet/resourceexecutor/resctrl.go

koordinator-bot · 2024-05-14T04:11:08Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign fillzpp, zwzhang0107 after the PR has been reviewed.
You can assign the PR to them by writing /assign @fillzpp @zwzhang0107 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

pkg/features/OWNERS
pkg/koordlet/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/koordlet/metricsadvisor/framework/options.go

pkg/koordlet/resourceexecutor/resctrl.go

pkg/koordlet/util/system/resctrl.go

Rouzip · 2024-06-25T03:11:31Z

hello, PTAL 😊 @saintube

pkg/koordlet/resourceexecutor/resctrl.go

pkg/koordlet/util/system/resctrl.go

Signed-off-by: Rouzip <1226015390@qq.com>

pkg/koordlet/util/system/resctrl.go

saintube · 2024-07-12T03:01:00Z

@Rouzip Thanks for your great contributions. Since it is a large patch, we need to make some tests before it is merged.

Rouzip · 2024-07-15T06:49:11Z

@Rouzip Thanks for your great contributions. Since it is a large patch, we need to make some tests before it is merged.

😊 Anything I can do?

saintube · 2024-07-16T02:01:10Z

@Rouzip Thanks for your great contributions. Since it is a large patch, we need to make some tests before it is merged.

😊 Anything I can do?

We will verify this patch on some test environments later. It would also be appreciated if you could add more UTs to increase the patch coverage to no less than the target of 70%.

Signed-off-by: Rouzip <1226015390@qq.com>

Signed-off-by: Frame <saintube@foxmail.com>

saintube

/lgtm
PTAL /cc @zwzhang0107 @hormes

zwzhang0107 · 2024-10-08T06:19:10Z

pkg/koordlet/metricsadvisor/collectors/resctrl/resctrl_collector.go

+	"github.com/koordinator-sh/koordinator/pkg/koordlet/metriccache"
+	"github.com/koordinator-sh/koordinator/pkg/koordlet/metrics"
+	"github.com/koordinator-sh/koordinator/pkg/koordlet/metricsadvisor/framework"
+	"github.com/koordinator-sh/koordinator/pkg/koordlet/qosmanager/plugins/resctrl"


it's better not not import qos plugin
these const can be moved to utils

zwzhang0107 · 2024-10-08T06:26:27Z

pkg/koordlet/resourceexecutor/resctrl.go

+	return &ResctrlAMDReader{}
+}
+
+func (rr *ResctrlBaseReader) ReadResctrlL3Stat(parent string) (map[CacheId]uint64, error) {


more comments about ReadResctrlL3Stat/ReadResctrlMBStat can make the reader easy understand

zwzhang0107 · 2024-10-08T06:28:34Z

pkg/koordlet/metricsadvisor/framework/config.go

@@ -48,7 +50,9 @@ func NewDefaultConfig() *Config {
 		PSICollectorInterval:             10 * time.Second,
 		CPICollectorTimeWindow:           10 * time.Second,
 		ColdPageCollectorInterval:        5 * time.Second,
+		ResctrlCollectorInterval:         1 * time.Second,


now 1 second seems to short and not necessary now, how about 10 seconds by default?

zwzhang0107 · 2024-10-08T06:33:23Z

pkg/koordlet/metrics/resctrl.go

+	QosResctrl = prometheus.NewGaugeVec(prometheus.GaugeOpts{
+		Subsystem: KoordletSubsystem,
+		Name:      "qos_resctrl",
+		Help:      "qos resctrl collected by koordlet",


detailed help msg will be helpful since resctrl is really an advanced metrics.

zwzhang0107 · 2024-10-08T06:36:41Z

pkg/koordlet/metrics/resctrl.go

+var (
+	QosResctrl = prometheus.NewGaugeVec(prometheus.GaugeOpts{
+		Subsystem: KoordletSubsystem,
+		Name:      "qos_resctrl",


"qos" is already included in labels, maybe define as two metric such as "llc_occupancy" and "mbm_occupancy". so that we don't need to set "MetricPropertyResctrlMbType="") during record metrics.

so does the MetricPropertiesFunc

zwzhang0107 · 2024-10-08T06:37:43Z

pkg/koordlet/metriccache/metric_types.go

@@ -102,6 +106,12 @@ const (

 	MetricPropertyCPIResource MetricProperty = "cpi_resource"

+	MetricPropertyNodeQos MetricProperty = "node_qos"


MetricPropertyQos MetricProperty = "qos"

koordinator-bot bot requested review from jasonliu747 and zwzhang0107 April 15, 2024 13:04

koordinator-bot bot added the size/XL label Apr 15, 2024

Rouzip force-pushed the collect branch from 2fc65f2 to e36fbcf Compare April 15, 2024 13:50

feat: add resctrl qos collector

ce54ce2

Signed-off-by: Rouzip <1226015390@qq.com>

Rouzip force-pushed the collect branch from e36fbcf to ce54ce2 Compare April 16, 2024 08:38

Rouzip force-pushed the collect branch from cba748a to b64cd0b Compare May 9, 2024 11:43

koordinator-bot bot added size/XXL and removed size/XL labels May 9, 2024

Rouzip changed the title ~~koordlet: add resctrl qos collector(WIP)~~ koordlet: add resctrl qos collector May 9, 2024

Rouzip force-pushed the collect branch from b64cd0b to ffa2cd9 Compare May 9, 2024 12:22

feat: add unit test

2904004

Signed-off-by: Rouzip <1226015390@qq.com>

Rouzip force-pushed the collect branch from ffa2cd9 to 2904004 Compare May 9, 2024 12:41

saintube reviewed May 13, 2024

View reviewed changes

saintube reviewed May 23, 2024

View reviewed changes

pkg/koordlet/metricsadvisor/framework/options.go Outdated Show resolved Hide resolved

pkg/koordlet/resourceexecutor/resctrl.go Show resolved Hide resolved

pkg/koordlet/util/system/resctrl.go Show resolved Hide resolved

Rouzip force-pushed the collect branch 2 times, most recently from 2d1efd6 to c28c849 Compare May 29, 2024 03:29

saintube reviewed Jun 25, 2024

View reviewed changes

pkg/koordlet/resourceexecutor/resctrl.go Show resolved Hide resolved

saintube reviewed Jun 25, 2024

View reviewed changes

pkg/koordlet/util/system/resctrl.go Outdated Show resolved Hide resolved

fix comments

a56f235

Signed-off-by: Rouzip <1226015390@qq.com>

Rouzip force-pushed the collect branch from c28c849 to a56f235 Compare June 29, 2024 07:55

saintube reviewed Jul 12, 2024

View reviewed changes

pkg/koordlet/util/system/resctrl.go Show resolved Hide resolved

Rouzip force-pushed the collect branch from 13e06eb to 1b4da0f Compare July 16, 2024 14:04

fix comments

b5e9a96

Signed-off-by: Rouzip <1226015390@qq.com>

Rouzip force-pushed the collect branch from 1b4da0f to b5e9a96 Compare July 16, 2024 14:09

saintube added 2 commits September 30, 2024 09:44

Merge branch 'main' into collect

1e368c2

Signed-off-by: Frame <saintube@foxmail.com>

Fix resctrl.go

1dca501

Signed-off-by: Frame <saintube@foxmail.com>

saintube approved these changes Sep 30, 2024

View reviewed changes

saintube added the lgtm label Sep 30, 2024

zwzhang0107 reviewed Oct 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

koordlet: add resctrl qos collector #2005

koordlet: add resctrl qos collector #2005

Rouzip commented Apr 15, 2024 •

edited

Loading

codecov bot commented Apr 15, 2024 •

edited

Loading

Rouzip commented May 9, 2024

koordinator-bot bot commented May 14, 2024

Rouzip commented Jun 25, 2024

saintube commented Jul 12, 2024

Rouzip commented Jul 15, 2024

saintube commented Jul 16, 2024 •

edited

Loading

saintube left a comment

zwzhang0107 Oct 8, 2024

zwzhang0107 Oct 8, 2024

zwzhang0107 Oct 8, 2024

zwzhang0107 Oct 8, 2024

zwzhang0107 Oct 8, 2024

zwzhang0107 Oct 8, 2024

		@@ -102,6 +106,12 @@ const (

		MetricPropertyCPIResource MetricProperty = "cpi_resource"

		MetricPropertyNodeQos MetricProperty = "node_qos"

koordlet: add resctrl qos collector #2005

Are you sure you want to change the base?

koordlet: add resctrl qos collector #2005

Conversation

Rouzip commented Apr 15, 2024 • edited Loading

Ⅰ. Describe what this PR does

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it

Ⅳ. Special notes for reviews

V. Checklist

codecov bot commented Apr 15, 2024 • edited Loading

Codecov Report

Rouzip commented May 9, 2024

koordinator-bot bot commented May 14, 2024

Rouzip commented Jun 25, 2024

saintube commented Jul 12, 2024

Rouzip commented Jul 15, 2024

saintube commented Jul 16, 2024 • edited Loading

saintube left a comment

Choose a reason for hiding this comment

zwzhang0107 Oct 8, 2024

Choose a reason for hiding this comment

zwzhang0107 Oct 8, 2024

Choose a reason for hiding this comment

zwzhang0107 Oct 8, 2024

Choose a reason for hiding this comment

zwzhang0107 Oct 8, 2024

Choose a reason for hiding this comment

zwzhang0107 Oct 8, 2024

Choose a reason for hiding this comment

zwzhang0107 Oct 8, 2024

Choose a reason for hiding this comment

Rouzip commented Apr 15, 2024 •

edited

Loading

codecov bot commented Apr 15, 2024 •

edited

Loading

saintube commented Jul 16, 2024 •

edited

Loading