telemetry: Adding metrics for amazonq unit test generation #915
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,20 @@ | ||
| { | ||
| "types": [ | ||
| { | ||
| "name": "acceptedCharactersCount", | ||
| "type": "int", | ||
| "description": "The number of accepted characters" | ||
| }, | ||
| { | ||
| "name": "acceptedCount", | ||
| "type": "int", | ||
| "description": "The number of accepted cases" | ||
| }, | ||
| { | ||
| "name": "acceptedLinesCount", | ||
| "type": "int", | ||
| "description": "The number of accepted lines of code" | ||
| }, | ||
| { | ||
| "name": "action", | ||
| "type": "string", | ||
|
|
@@ -178,11 +193,21 @@ | |
| "type": "int", | ||
| "description": "The amount of time required for the build to complete (in seconds)." | ||
| }, | ||
| { | ||
| "name": "buildPayloadBytes", | ||
| "type": "int", | ||
| "description": "The uncompressed payload size in bytes of the source files in customer project context" | ||
| }, | ||
| { | ||
| "name": "buildSystemVersion", | ||
| "type": "string", | ||
| "description": "The build system version on the user's machine" | ||
| }, | ||
| { | ||
| "name": "buildZipFileBytes", | ||
| "type": "int", | ||
| "description": "The compressed payload size of source files in bytes of customer project context sent" | ||
| }, | ||
| { | ||
| "name": "causedBy", | ||
| "type": "string", | ||
|
|
@@ -1218,6 +1243,11 @@ | |
| "type": "string", | ||
| "description": "The name of the EventBridge Schema used in the operation" | ||
| }, | ||
| { | ||
| "name": "executedCount", | ||
| "type": "int", | ||
| "description": "The number of executed operations" | ||
| }, | ||
| { | ||
| "name": "experimentId", | ||
| "type": "string", | ||
|
|
@@ -1284,6 +1314,21 @@ | |
| "type": "string", | ||
| "description": "Application framework being used" | ||
| }, | ||
| { | ||
| "name": "generatedCharactersCount", | ||
| "type": "int", | ||
| "description": "Number of characters of code generated" | ||
| }, | ||
| { | ||
| "name": "generatedCount", | ||
|
There was a problem hiding this comment. this name is unhelpful. why not There was a problem hiding this comment. Please refer this PR There was a problem hiding this comment. Each field is "namespaced" in a metric, so there is no need to name each field in a globally-unique way, and it's actually counterproductive (because it results in 10x or 100x the number of field names, which reduces discoverability, re-use, and leads to less-helpful dashboards and queries). So for example, this field would live on a metric named There was a problem hiding this comment. You probably don't need this field, just use the |
||
| "type": "int", | ||
| "description": "The number of generated cases" | ||
| }, | ||
| { | ||
| "name": "generatedLinesCount", | ||
| "type": "int", | ||
| "description": "The number of generated lines of code" | ||
| }, | ||
| { | ||
| "name": "generateFailure", | ||
| "type": "string", | ||
|
|
@@ -1405,6 +1450,21 @@ | |
| "type": "boolean", | ||
| "description": "Whether or not the operation was a retry" | ||
| }, | ||
| { | ||
| "name": "isSupportedLanguage", | ||
| "type": "boolean", | ||
| "description": "Indicate if the language is supported" | ||
| }, | ||
| { | ||
| "name": "jobGroup", | ||
| "type": "string", | ||
| "description": "Job group name used in the operation" | ||
| }, | ||
| { | ||
| "name": "jobId", | ||
| "type": "string", | ||
| "description": "Job id used in the operation" | ||
| }, | ||
| { | ||
| "name": "lambdaArchitecture", | ||
| "type": "string", | ||
|
|
@@ -1696,6 +1756,11 @@ | |
| "type": "string", | ||
| "description": "Date/time that an SSO client registration expires." | ||
| }, | ||
| { | ||
| "name": "step", | ||
| "type": "string", | ||
| "description": "Indicates the stage at which a user interface click action was performed." | ||
| }, | ||
| { | ||
| "name": "successCount", | ||
| "type": "int", | ||
|
|
@@ -1758,6 +1823,11 @@ | |
| "type": "string", | ||
| "description": "User selection from a predefined menu (not user-provided input). See also `action`." | ||
| }, | ||
| { | ||
| "name": "userEnteredPromptMessage", | ||
| "type": "boolean", | ||
| "description": "True if user enter prompt message as input else false" | ||
| }, | ||
| { | ||
| "name": "userId", | ||
| "type": "string", | ||
|
|
@@ -2240,6 +2310,175 @@ | |
| } | ||
| ] | ||
| }, | ||
| { | ||
| "name": "amazonq_unitTestGeneration", | ||
| "description": "Client side metrics of AmazonQ Unit Test Generation", | ||
| "metadata": [ | ||
| { | ||
| "type": "acceptedCharactersCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "acceptedCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "acceptedLinesCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "credentialStartUrl", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "cwsprChatProgrammingLanguage" | ||
|
There was a problem hiding this comment. why is this a chat programming language? There was a problem hiding this comment. We planned to reuse existing fields as much as possible. Please refer this PR |
||
| }, | ||
| { | ||
| "type": "executedCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "failedCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "generatedCharactersCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "generatedCount", | ||
|
There was a problem hiding this comment. This is on a metric named each field does not need its own namespace; it's already namespaced in a metric. There was a problem hiding this comment. Follow this suggestion. There was a problem hiding this comment. Sure, but |
||
| "required": false | ||
| }, | ||
| { | ||
| "type": "generatedLinesCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "isSupportedLanguage" | ||
| }, | ||
| { | ||
| "type": "jobGroup", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "successCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "userEnteredPromptMessage" | ||
|
There was a problem hiding this comment. you cannot save this as telemetry. This is customer data. There was a problem hiding this comment. As declared this is boolean, so there is no harm for customer data. |
||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "name": "amazonq_utg_buildLoop", | ||
| "description": "Client side invocation of the AmazonQ Unit Test Generation build loop", | ||
| "metadata": [ | ||
| { | ||
| "type": "credentialStartUrl", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "cwsprChatProgrammingLanguage" | ||
| }, | ||
| { | ||
| "type": "isSupportedLanguage" | ||
| }, | ||
| { | ||
| "type": "jobGroup", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "jobId", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "perfClientLatency", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "result", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "source", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "userEnteredPromptMessage" | ||
|
|
||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "name": "amazonq_utg_generateTests", | ||
| "description": "Client side invocation of the AmazonQ Unit Test Generation", | ||
|
Comment on lines
+2412
to
+2413
There was a problem hiding this comment. how is this different than
There was a problem hiding this comment. This will be the E2E metric after Build execution loop There was a problem hiding this comment. That isn't indicated in the description, and still doesn't explain how it's different There was a problem hiding this comment. This is not being added as part of RIV. |
||
| "metadata": [ | ||
| { | ||
| "type": "acceptedCharactersCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "acceptedCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "acceptedLinesCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "artifactsUploadDuration", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "buildPayloadBytes", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "buildZipFileBytes", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "credentialStartUrl", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "cwsprChatProgrammingLanguage" | ||
| }, | ||
| { | ||
| "type": "generatedCharactersCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "generatedCount", | ||
|
There was a problem hiding this comment. This is on a metric named each field does not need its own namespace; it's already namespaced in a metric. |
||
| "required": false | ||
| }, | ||
| { | ||
| "type": "generatedLinesCount", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "isSupportedLanguage" | ||
| }, | ||
| { | ||
| "type": "jobGroup", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "jobId", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "perfClientLatency", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "source", | ||
| "required": false | ||
| }, | ||
| { | ||
| "type": "userEnteredPromptMessage" | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "name": "amazonq_viewChatPanel", | ||
| "description": "Captures if Q chat panel is successfully viewed or not", | ||
|
|
@@ -7035,6 +7274,10 @@ | |
| "metadata": [ | ||
| { | ||
| "type": "elementId" | ||
| }, | ||
| { | ||
| "type": "step", | ||
| "required": false | ||
| } | ||
| ] | ||
| }, | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what about number of files?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We does not have a field for
Number of filesand I do not think it adds a value as file can be of any size and number does not define our latency or test generation experience.Correct me If I am missing something.