Remove backwards compatible fields (#569)

docs/storage-schema.md

@@ -5,7 +5,10 @@ The data is split across 2 tables ([`summary`](#summary) and
 [`details`](#details)) to better store common items, reduce duplication and
 allow for more targeted queries. Below is a breakdown of the various tables and
 there columns as well as the corresponding JSON path within the report format
-the data is sourced from.
+the data is sourced from. The current `measure_name` for both tables is
+`report_v2`. Old format records will be marked with `report_2_bc` or
+`*_test_run` and can require special handling. See notes under various
+**Removed** sections for details.
 
 > [!NOTE]
   The storage schema is always based on the [latest report schema] and uses the
@@ -22,8 +25,6 @@ as some rolled up counts of various test statuses.
 
 * `duration_total` (`BIGINT`): Stored as **milliseconds** from report JSON
   `report.summary.duration.total`.
-* `total_duration` (`BIGINT`) **[deprecated]**: Stored as **milliseconds**,
-  sourced from report JSON `report.summary.duration.total`.
 * `status` (`VARCHAR`): Will either be `passed` or `failed`, sourced rom report
   JSON `report.summary.status`.
 * `count_passed` (`BIGINT`): Sourced from report JSON
@@ -39,6 +40,13 @@ as some rolled up counts of various test statuses.
   Anything marked with **[deprecated]** should be moved away from. Sending of
   this column will be removed in the near future.
 
+##### Removed
+
+* `total_duration` (`BIGINT`): Starting with records that have `measure_name` of
+  `report_v2` this field will no longer be present. Please use `duration_total`
+  instead. If wanting to get data from records prior to `report_v2` please use
+  `COALESCE(total_duration, duration_total)` to get a reasonable value.
+
 #### Dimensions
 
 > [!NOTE]
@@ -63,6 +71,10 @@ as some rolled up counts of various test statuses.
 * `lms_instance_url` (`NULLABLE`): Sourced from report JSON
   `report.summary.lms.instanceUrl`.
 
+> [!WARNING]
+  Anything marked with **[deprecated]** should be moved away from. Sending of
+  this column will be removed in the near future.
+
 ### `details`
 
 This table contains information about each individual test that was run. A
@@ -73,12 +85,8 @@ combination of the data is desired.
 
 * `duration_final` (`BIGINT`): Stored as **milliseconds**, sourced from report
   JSON `report.details[].duration.final`.
-* `duration` (`BIGINT`) **[deprecated]**: Stored as **milliseconds**, sourced
-  from report JSON `report.details[].duration.final`.
 * `duration_total` (`BIGINT`): Stored as **milliseconds**, sourced from report
   `JSON report.details[].duration.total`.
-* `total_duration` (`BIGINT`) **[deprecated]**: Stored as **milliseconds**,
-  sourced from report JSON `report.details[].duration.total`.
 * `retries` (`BIGINT`): Sourced from report JSON `report.details[].retries`.
 * `status` (`VARCHAR`): Will be one of `passed`, `skipped` or `failed`, sourced
   from report JSON `report.details[].status`.
@@ -87,15 +95,24 @@ combination of the data is desired.
   Anything marked with **[deprecated]** should be moved away from. Sending of
   this column will be removed in the near future.
 
+##### Removed
+
+* `duration` (`BIGINT`): Starting with records that have `measure_name` of
+  `report_v2` this field will no longer be present. Please use `duration_final`
+  instead. If wanting to get data from records prior to `report_v2` please use
+  `COALESCE(duration, duration_final)` to get a reasonable value.
+* `total_duration` (`BIGINT`): Starting with records that have `measure_name` of
+  `report_v2` this field will no longer be present. Please use `duration_total`
+  instead. If wanting to get data from records prior to `report_v2` please use
+  `COALESCE(total_duration, duration_total)` to get a reasonable value.
+
 #### Dimensions
 
 > [!NOTE]
   All dimensions are stored as `VARCHAR` as it is the only format available.
 
 * `report_id`: Sourced from report JSON `report.id`.
 * `name`: Sourced from report JSON `report.details[].name`.
-* `location` **[deprecated]**: Sourced from report JSON
-  `report.details[].location.file`.
 * `location_file`: Sourced from report JSON `report.details[].location.file`.
 * `location_line` (`NULLABLE`): Sourced from report JSON
   `report.details[].location.line`.
@@ -115,6 +132,13 @@ combination of the data is desired.
   Anything marked with **[deprecated]** should be moved away from. Sending of
   this column will be removed in the near future.
 
+##### Removed
+
+* `location`: Starting with records that have `measure_name` of
+  `report_v2` this field will no longer be present. Please use `location_file`
+  instead. If wanting to get data from records prior to `report_v2` please use
+  `COALESCE(location, location_file)` to get a reasonable value.
+
 <!-- links -->
 [AWS Timestream]: https://aws.amazon.com/timestream
 [latest report schema]: https://github.com/Brightspace/test-reporting-node/tree/main/schemas/report

src/report.js

@@ -69,12 +69,9 @@ const makeSummaryWriteRequest = (report) => {
 			Version: 1,
 			Time: (Date.parse(started)).toString(),
 			TimeUnit: MILLISECONDS,
-			MeasureName: `report_${version}_bc`,
+			MeasureName: `report_v${version}`,
 			MeasureValueType: MULTI,
 			MeasureValues: [
-				// kept for backwards compat. Once all dashboards are updated will be removed
-				{ Name: 'total_duration', Value: total.toString(), Type: BIGINT },
-				/////////////////////////////////////////////////////////////////////////////
 				{ Name: 'duration_total', Value: total.toString(), Type: BIGINT },
 				{ Name: 'status', Value: status, Type: VARCHAR },
 				{ Name: 'count_passed', Value: passed.toString(), Type: BIGINT },
@@ -107,9 +104,6 @@ const makeDetailRecord = (detail) => {
 	const { file, line, column } = location;
 	const dimensions = [
 		{ Name: 'name', Value: name },
-		// kept for backwards compat. Once all dashboards are updated will be removed
-		{ Name: 'location', Value: file },
-		/////////////////////////////////////////////////////////////////////////////
 		{ Name: 'location_file', Value: file }
 	];
 
@@ -145,10 +139,6 @@ const makeDetailRecord = (detail) => {
 		Time: (Date.parse(started)).toString(),
 		TimeUnit: MILLISECONDS,
 		MeasureValues: [
-			// kept for backwards compat. Once all dashboards are updated will be removed
-			{ Name: 'duration', Value: final.toString(), Type: BIGINT },
-			{ Name: 'total_duration', Value: total.toString(), Type: BIGINT },
-			/////////////////////////////////////////////////////////////////////////////
 			{ Name: 'duration_final', Value: final.toString(), Type: BIGINT },
 			{ Name: 'duration_total', Value: total.toString(), Type: BIGINT },
 			{ Name: 'retries', Value: retries.toString(), Type: BIGINT },
@@ -174,7 +164,7 @@ const makeDetailWriteRequests = (report) => {
 				Records: detailRecordBatch,
 				CommonAttributes: {
 					Version: 1,
-					MeasureName: `report_${version}_bc`,
+					MeasureName: `report_v${version}`,
 					MeasureValueType: MULTI,
 					Dimensions: [
 						{ Name: 'report_id', Value: id, Type: VARCHAR }