-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Metrics Support in tritonfrontend
#7703
Conversation
…' into kprashanth-tritonfrontend-metrics
@@ -57,14 +57,18 @@ Note: `model_path` may need to be edited depending on your setup. | |||
|
|||
2. Now, to start up the respective services with `tritonfrontend` | |||
```python | |||
from tritonfrontend import KServeHttp, KServeGrpc | |||
from tritonfrontend import KServeHttp, KServeGrpc, Metrics |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't love that the Metrics
object is a web server, so it makes me wonder if we should rename these down the line, ex: KServeHttpService
, MetricsService
, etc
But I don't have a strong opinion on an alternative right now so I think it's fine, just mentioning for later. We will probably be restructing some packaging and naming in the near-mid future.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only minor comments - nice work!
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
What does the PR do?
Adding support for
Metrics
intritonfrontend
. This involves two components:HTTPMetricsServer
Metrics
classWith this PR, similar to
KServeHttp
andKServeGrpc
, the metrics service can used with:Additional Changes made in this PR:
request.post(...)
based on this commentTest plan:
Added 3 test function to L0_python_api:
test_metrics_default_port()
: Tests whether the metrics service can start as expectedtest_metrics_custom_port()
: Tests whether arguments defined intritonfrontend.Metrics.Options
are passed successfully toHTTPMetrics
test_metrics_update()
: Tests whethernv_inference_count
value goes from 0 to 1 if inference request is performed.