Skip to content

Commit

Permalink
Add integration test
Browse files Browse the repository at this point in the history
  • Loading branch information
@kathy-t
kathy-t committed Nov 10, 2023
1 parent 572319a commit dfe495b
Show file tree
Hide file tree
Showing 3 changed files with 77 additions and 23 deletions.
12 changes: 6 additions & 6 deletions THIRD-PARTY-LICENSES.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -124,10 +124,10 @@ Lists of 402 third-party dependencies.
(The Apache Software License, Version 2.0) docker-java-core (com.github.docker-java:docker-java-core:3.3.0 - https://github.com/docker-java/docker-java)
(The Apache Software License, Version 2.0) docker-java-transport (com.github.docker-java:docker-java-transport:3.3.0 - https://github.com/docker-java/docker-java)
(The Apache Software License, Version 2.0) docker-java-transport-httpclient5 (com.github.docker-java:docker-java-transport-httpclient5:3.3.0 - https://github.com/docker-java/docker-java)
(Apache Software License, Version 2.0) dockstore-common (io.dockstore:dockstore-common:1.15.0-20231109.155709-113 - no url defined)
(Apache Software License, Version 2.0) dockstore-integration-testing (io.dockstore:dockstore-integration-testing:1.15.0-20231109.155709-113 - no url defined)
(Apache Software License, Version 2.0) dockstore-language-plugin-parent (io.dockstore:dockstore-language-plugin-parent:1.15.0-20231109.155709-113 - no url defined)
(Apache Software License, Version 2.0) dockstore-webservice (io.dockstore:dockstore-webservice:1.15.0-20231109.155709-113 - no url defined)
(Apache Software License, Version 2.0) dockstore-common (io.dockstore:dockstore-common:1.15.0-20231110.184257-114 - no url defined)
(Apache Software License, Version 2.0) dockstore-integration-testing (io.dockstore:dockstore-integration-testing:1.15.0-20231110.184257-114 - no url defined)
(Apache Software License, Version 2.0) dockstore-language-plugin-parent (io.dockstore:dockstore-language-plugin-parent:1.15.0-20231110.184257-114 - no url defined)
(Apache Software License, Version 2.0) dockstore-webservice (io.dockstore:dockstore-webservice:1.15.0-20231110.184257-114 - no url defined)
(Apache License 2.0) Dropwizard (io.dropwizard:dropwizard-core:4.0.2 - http://www.dropwizard.io/4.0.2/dropwizard-bom/dropwizard-dependencies/dropwizard-parent/dropwizard-core)
(Apache License 2.0) Dropwizard Asset Bundle (io.dropwizard:dropwizard-assets:4.0.2 - http://www.dropwizard.io/4.0.2/dropwizard-bom/dropwizard-dependencies/dropwizard-parent/dropwizard-assets)
(Apache License 2.0) Dropwizard Authentication (io.dropwizard:dropwizard-auth:4.0.2 - http://www.dropwizard.io/4.0.2/dropwizard-bom/dropwizard-dependencies/dropwizard-parent/dropwizard-auth)
Expand Down Expand Up @@ -343,7 +343,7 @@ Lists of 402 third-party dependencies.
(Apache License, Version 2.0) Objenesis (org.objenesis:objenesis:3.2 - http://objenesis.org/objenesis)
(The Apache Software License, Version 2.0) okhttp (com.squareup.okhttp3:okhttp:4.10.0 - https://square.github.io/okhttp/)
(The Apache Software License, Version 2.0) okio (com.squareup.okio:okio-jvm:3.0.0 - https://github.com/square/okio/)
(Apache Software License, Version 2.0) openapi-java-client (io.dockstore:openapi-java-client:1.15.0-20231109.155709-113 - no url defined)
(Apache Software License, Version 2.0) openapi-java-client (io.dockstore:openapi-java-client:1.15.0-20231110.184257-114 - no url defined)
(The Apache License, Version 2.0) OpenCensus (io.opencensus:opencensus-api:0.31.0 - https://github.com/census-instrumentation/opencensus-java)
(Apache 2) opencsv (com.opencsv:opencsv:5.7.1 - http://opencsv.sf.net)
(MIT License) ORCID - Model (org.orcid:orcid-model-jakarta:3.3.0 - https://github.com/ORCID/orcid-model)
Expand Down Expand Up @@ -381,7 +381,7 @@ Lists of 402 third-party dependencies.
(Apache License 2.0) swagger-core-jakarta (io.swagger.core.v3:swagger-core-jakarta:2.2.15 - https://github.com/swagger-api/swagger-core/modules/swagger-core-jakarta)
(Apache License 2.0) swagger-integration-jakarta (io.swagger.core.v3:swagger-integration-jakarta:2.2.15 - https://github.com/swagger-api/swagger-core/modules/swagger-integration-jakarta)
(Apache Software License, Version 2.0) swagger-java-bitbucket-client (io.dockstore:swagger-java-bitbucket-client:2.0.3 - no url defined)
(Apache Software License, Version 2.0) swagger-java-client (io.dockstore:swagger-java-client:1.15.0-20231109.155709-113 - no url defined)
(Apache Software License, Version 2.0) swagger-java-client (io.dockstore:swagger-java-client:1.15.0-20231110.184257-114 - no url defined)
(Apache Software License, Version 2.0) swagger-java-discourse-client (io.dockstore:swagger-java-discourse-client:2.0.1 - no url defined)
(Apache Software License, Version 2.0) swagger-java-quay-client (io.dockstore:swagger-java-quay-client:2.0.2 - no url defined)
(Apache Software License, Version 2.0) swagger-java-sam-client (io.dockstore:swagger-java-sam-client:2.0.2 - no url defined)
Expand Down
4 changes: 4 additions & 0 deletions ...icsaggregator/src/main/java/io/dockstore/metricsaggregator/MetricsAggregatorS3Client.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
import io.dockstore.openapi.client.model.ExecutionsRequestBody;
import io.dockstore.openapi.client.model.Metrics;
import io.dockstore.openapi.client.model.RunExecution;
import io.dockstore.openapi.client.model.TaskExecutions;
import io.dockstore.openapi.client.model.ValidationExecution;
import java.io.IOException;
import java.net.URISyntaxException;
Expand Down Expand Up @@ -132,6 +133,7 @@ public void aggregateMetrics(ExtendedGa4GhApi extendedGa4GhApi) {
private ExecutionsRequestBody getExecutions(String toolId, String versionName, String platform) throws IOException, JsonSyntaxException {
List<MetricsData> metricsDataList = metricsDataS3Client.getMetricsData(toolId, versionName, Partner.valueOf(platform));
List<RunExecution> runExecutionsFromAllSubmissions = new ArrayList<>();
List<TaskExecutions> taskExecutionsFromAllSubmissions = new ArrayList<>();
List<ValidationExecution> validationExecutionsFromAllSubmissions = new ArrayList<>();
List<Metrics> aggregatedExecutionsFromAllSubmissions = new ArrayList<>();

Expand All @@ -140,12 +142,14 @@ private ExecutionsRequestBody getExecutions(String toolId, String versionName, S
metricsData.platform(), metricsData.fileName());
ExecutionsRequestBody executionsFromOneSubmission = GSON.fromJson(fileContent, ExecutionsRequestBody.class);
runExecutionsFromAllSubmissions.addAll(executionsFromOneSubmission.getRunExecutions());
taskExecutionsFromAllSubmissions.addAll(executionsFromOneSubmission.getTaskExecutions());
validationExecutionsFromAllSubmissions.addAll(executionsFromOneSubmission.getValidationExecutions());
aggregatedExecutionsFromAllSubmissions.addAll(executionsFromOneSubmission.getAggregatedExecutions());
}

return new ExecutionsRequestBody()
.runExecutions(runExecutionsFromAllSubmissions)
.taskExecutions(taskExecutionsFromAllSubmissions)
.validationExecutions(validationExecutionsFromAllSubmissions)
.aggregatedExecutions(aggregatedExecutionsFromAllSubmissions);
}
Expand Down
84 changes: 67 additions & 17 deletions ...or/src/test/java/io/dockstore/metricsaggregator/client/cli/MetricsAggregatorClientIT.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
import io.dockstore.openapi.client.model.ExecutionsRequestBody;
import io.dockstore.openapi.client.model.Metrics;
import io.dockstore.openapi.client.model.RunExecution;
import io.dockstore.openapi.client.model.TaskExecutions;
import io.dockstore.openapi.client.model.ValidationExecution;
import io.dockstore.openapi.client.model.ValidatorInfo;
import io.dockstore.openapi.client.model.ValidatorVersionInfo;
Expand Down Expand Up @@ -138,17 +139,17 @@ void testAggregateMetrics() {
String id = "#workflow/" + workflow.getFullWorkflowPath();
String versionId = version.getName();

// A successful run execution that ran for 5 minutes, requires 2 CPUs and 2 GBs of memory
List<RunExecution> runExecutions = List.of(createRunExecution(SUCCESSFUL, "PT5M", 2, 2.0, new Cost().value(2.00), "us-central1"));
// A successful workflow run execution that ran for 5 minutes, requires 2 CPUs and 2 GBs of memory
List<RunExecution> workflowExecutions = List.of(createRunExecution(SUCCESSFUL, "PT5M", 2, 2.0, new Cost().value(2.00), "us-central1"));
// A successful miniwdl validation
final String validatorToolVersion1 = "1.0";
ValidationExecution validationExecution1 = createValidationExecution(MINIWDL, validatorToolVersion1, true);
final String validatorToolVersion2 = "2.0";
ValidationExecution validationExecution2 = createValidationExecution(WOMTOOL, validatorToolVersion2, false);

// Submit metrics for two platforms
extendedGa4GhApi.executionMetricsPost(new ExecutionsRequestBody().runExecutions(runExecutions).validationExecutions(List.of(validationExecution1)), platform1, id, versionId, "");
extendedGa4GhApi.executionMetricsPost(new ExecutionsRequestBody().runExecutions(runExecutions).validationExecutions(List.of(validationExecution2)), platform2, id, versionId, "");
extendedGa4GhApi.executionMetricsPost(new ExecutionsRequestBody().runExecutions(workflowExecutions).validationExecutions(List.of(validationExecution1)), platform1, id, versionId, "");
extendedGa4GhApi.executionMetricsPost(new ExecutionsRequestBody().runExecutions(workflowExecutions).validationExecutions(List.of(validationExecution2)), platform2, id, versionId, "");
int expectedNumberOfPlatforms = 3; // 2 for platform1 and platform2, and 1 for ALL platforms
// Aggregate metrics
MetricsAggregatorClient.main(new String[] {"aggregate-metrics", "--config", CONFIG_FILE_PATH});
Expand All @@ -167,11 +168,11 @@ void testAggregateMetrics() {
ValidatorInfo validationInfo;

// A failed run execution that ran for 1 second, requires 2 CPUs and 4.5 GBs of memory
runExecutions = List.of(createRunExecution(FAILED_RUNTIME_INVALID, "PT1S", 4, 4.5, new Cost().value(2.00), "us-central1"));
workflowExecutions = List.of(createRunExecution(FAILED_RUNTIME_INVALID, "PT1S", 4, 4.5, new Cost().value(2.00), "us-central1"));
// A failed miniwdl validation for the same validator version
List<ValidationExecution> validationExecutions = List.of(createValidationExecution(MINIWDL, "1.0", false));
ExecutionsRequestBody executionsRequestBody = new ExecutionsRequestBody().runExecutions(runExecutions).validationExecutions(validationExecutions);
// Submit metrics for the same workflow version for platform 2
ExecutionsRequestBody executionsRequestBody = new ExecutionsRequestBody().runExecutions(workflowExecutions).validationExecutions(validationExecutions);
// Submit metrics for the same workflow version for platform 1
extendedGa4GhApi.executionMetricsPost(executionsRequestBody, platform1, id, versionId, "");
// Aggregate metrics
MetricsAggregatorClient.main(new String[] {"aggregate-metrics", "--config", CONFIG_FILE_PATH});
Expand Down Expand Up @@ -225,6 +226,52 @@ void testAggregateMetrics() {
assertEquals(50d, validationInfo.getPassingRate());
assertEquals(2, validationInfo.getNumberOfRuns());

// Submit two TaskExecutions, each one representing the task metrics for a single workflow execution
// A successful task execution that ran for 11 seconds, requires 6 CPUs and 5.5 GBs of memory. Signifies that this workflow execution only executed one task
TaskExecutions taskExecutions = new TaskExecutions().taskExecutions(List.of(createRunExecution(SUCCESSFUL, "PT11S", 6, 5.5, new Cost().value(2.00), "us-central1")));
executionsRequestBody = new ExecutionsRequestBody().taskExecutions(List.of(taskExecutions));
// Submit metrics for the same workflow version for platform 1
extendedGa4GhApi.executionMetricsPost(executionsRequestBody, platform1, id, versionId, "");
// Aggregate metrics
MetricsAggregatorClient.main(new String[] {"aggregate-metrics", "--config", CONFIG_FILE_PATH});
// Get workflow version to verify aggregated metrics
workflow = workflowsApi.getPublishedWorkflow(32L, "metrics");
version = workflow.getWorkflowVersions().stream().filter(v -> "master".equals(v.getName())).findFirst().orElse(null);
assertNotNull(version);
assertEquals(expectedNumberOfPlatforms, version.getMetricsByPlatform().size());
platform1Metrics = version.getMetricsByPlatform().get(platform1);

// This version now has three submissions of execution metrics data. Verify that the aggregated metrics are correct
assertEquals(2, platform1Metrics.getExecutionStatusCount().getNumberOfSuccessfulExecutions());
assertEquals(1, platform1Metrics.getExecutionStatusCount().getNumberOfFailedExecutions());
assertEquals(2, platform1Metrics.getExecutionStatusCount().getCount().get(SUCCESSFUL.name()));
assertEquals(1, platform1Metrics.getExecutionStatusCount().getCount().get(FAILED_RUNTIME_INVALID.name()));
assertFalse(platform1Metrics.getExecutionStatusCount().getCount().containsKey(FAILED_SEMANTIC_INVALID.name()));

assertEquals(3, platform1Metrics.getCpu().getNumberOfDataPointsForAverage());
assertEquals(2, platform1Metrics.getCpu().getMinimum());
assertEquals(6, platform1Metrics.getCpu().getMaximum());
assertEquals(4, platform1Metrics.getCpu().getAverage());
assertNull(platform1Metrics.getCpu().getUnit());

assertEquals(3, platform1Metrics.getMemory().getNumberOfDataPointsForAverage());
assertEquals(2, platform1Metrics.getMemory().getMinimum());
assertEquals(5.5, platform1Metrics.getMemory().getMaximum());
assertEquals(4, platform1Metrics.getMemory().getAverage());
assertNotNull(platform1Metrics.getMemory().getUnit());

assertEquals(3, platform1Metrics.getCost().getNumberOfDataPointsForAverage());
assertEquals(2, platform1Metrics.getCost().getMinimum());
assertEquals(2, platform1Metrics.getCost().getMaximum());
assertEquals(2, platform1Metrics.getCost().getAverage());
assertNotNull(platform1Metrics.getCost().getUnit());

assertEquals(3, platform1Metrics.getExecutionTime().getNumberOfDataPointsForAverage());
assertEquals(1, platform1Metrics.getExecutionTime().getMinimum());
assertEquals(300, platform1Metrics.getExecutionTime().getMaximum());
assertEquals(104, platform1Metrics.getExecutionTime().getAverage());
assertNotNull(platform1Metrics.getExecutionTime().getUnit());

testOverallAggregatedMetrics(version, validatorToolVersion1, validatorToolVersion2, platform1Metrics);
}

Expand Down Expand Up @@ -290,28 +337,31 @@ private static void testOverallAggregatedMetrics(WorkflowVersion version, String
// Verify that the metrics aggregated across ALL platforms are correct
Metrics overallMetrics = version.getMetricsByPlatform().get(Partner.ALL.name());
assertNotNull(overallMetrics);
assertEquals(2, overallMetrics.getExecutionStatusCount().getNumberOfSuccessfulExecutions());
assertEquals(3, overallMetrics.getExecutionStatusCount().getNumberOfSuccessfulExecutions());
assertEquals(1, overallMetrics.getExecutionStatusCount().getNumberOfFailedExecutions());
assertEquals(2, overallMetrics.getExecutionStatusCount().getCount().get(SUCCESSFUL.name()));
assertEquals(3, overallMetrics.getExecutionStatusCount().getCount().get(SUCCESSFUL.name()));
assertEquals(1, platform1Metrics.getExecutionStatusCount().getCount().get(FAILED_RUNTIME_INVALID.name()));
assertFalse(overallMetrics.getExecutionStatusCount().getCount().containsKey(FAILED_SEMANTIC_INVALID.name()));

assertEquals(3, overallMetrics.getCpu().getNumberOfDataPointsForAverage());
// The CPU values submitted were 2, 2, 4, 6
assertEquals(4, overallMetrics.getCpu().getNumberOfDataPointsForAverage());
assertEquals(2, overallMetrics.getCpu().getMinimum());
assertEquals(4, overallMetrics.getCpu().getMaximum());
assertEquals(2.6666666666666665, overallMetrics.getCpu().getAverage());
assertEquals(6, overallMetrics.getCpu().getMaximum());
assertEquals(3.5, overallMetrics.getCpu().getAverage());
assertNull(overallMetrics.getCpu().getUnit());

assertEquals(3, overallMetrics.getMemory().getNumberOfDataPointsForAverage());
// The memory values submitted were 2, 2, 4.5, 5.5
assertEquals(4, overallMetrics.getMemory().getNumberOfDataPointsForAverage());
assertEquals(2, overallMetrics.getMemory().getMinimum());
assertEquals(4.5, overallMetrics.getMemory().getMaximum());
assertEquals(2.833333333333333, overallMetrics.getMemory().getAverage());
assertEquals(5.5, overallMetrics.getMemory().getMaximum());
assertEquals(3.5, overallMetrics.getMemory().getAverage());
assertNotNull(overallMetrics.getMemory().getUnit());

assertEquals(3, overallMetrics.getExecutionTime().getNumberOfDataPointsForAverage());
// The execution times submitted were PT5M, PT5M, PT1S, PT11S
assertEquals(4, overallMetrics.getExecutionTime().getNumberOfDataPointsForAverage());
assertEquals(1, overallMetrics.getExecutionTime().getMinimum());
assertEquals(300, overallMetrics.getExecutionTime().getMaximum());
assertEquals(200.33333333333331, overallMetrics.getExecutionTime().getAverage());
assertEquals(153, overallMetrics.getExecutionTime().getAverage());
assertNotNull(overallMetrics.getExecutionTime().getUnit());

assertEquals(2, overallMetrics.getValidationStatus().getValidatorTools().size());
Expand Down

0 comments on commit dfe495b

Please sign in to comment.