EM ListTrialStatus API - scenarios

[chandrams]

EM listTrialStatus API - https://github.com/kruize/autotune/blob/b89b57f7764245f0b018a8d43ea20b2890aa5e8b/design/API.md

GET /listTrialStatus?runId=<RUN_ID>&completeStatus=true&summary=true

runId - Required
completeStatus - Optional, default is true
summary - Optional

• Test with valid / invalid runids

• ListTrial Status to be checked for querying each of the below scenarios

  • If Status is COMPLETED, then query with summary=true
  • If Status is IN PROGRESS, then query with completeStatus=false?

• Do we need both completeStatus and Summary? What's the difference?

• Shouldn't it just be status & metrics?

• Test with valid runid and summary=true, validate if a summary of the metrics alone is displayed?

• Test with valid runid and summary=false, validate if metrics for all warmup/measurement cycles are displayed or there is no summary?

• Test with valid runid and completeStatus=true, what is the output?

• Test with valid runid and completeStatus=false, what is the output?

• Output is too verbose

• What about Iterations?

• Validate metrics

            {
              "score": "",
              "Error": "",
              "mean": "",
              "mode": "",
              "95.0": "",
              "99.0": "",
              "99.9": "",
              "99.99": "",
              "99.999": "",
              "99.9999": "",
              "100.0": "",
              "spike": ""
            },
  

  • error is zero, rest are non-zero
  • Percentiles - Should not be in descending order (can be ascending or same)
  • Unit tests to validate metrics - score, with mock data

• Validate config

  • If values are same as in input json
  • Env parameters are same as in the input json
  • List of valid env parameters for a layer (probably unit tests at analyzer)

• Stop the load abruptly in between the experiment

• Query the EM multiple times and see if the output JSON values varies(check for consistency of result)

• Delete the pod deployed in between the experiments

• Delete the deployment in between the experiments