Additional Steps for BQ_Source #1472

Praveena2607 · 2024-12-11T07:18:20Z

Please verify the Scenarios.

google-cla · 2024-12-11T07:18:25Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

masivesky

Please update the PR description to reflect details and context about the changes. For eg: #1444 (comment)

masivesky

The BUTs seems to be failing, please take a look and fix the errors.

…ugin: Added and modified.

Praveena2607 · 2024-12-12T06:04:51Z

Description:

Updated the BigQuery plugin to include additional steps for better handling.
Modified property locators in Cdfplugin property locators.java to support new BigQuery functionality.
Updated the errorMessage.properties and pluginParameters.properties file to ensure proper error handling and parameter management for the new configuration.

Modified Files:

Cdfplugin property locators.java
errorMessage.properties
pluginParameters.properties

File Modified:

src/e2e-test/java/io/cdap/plugin/utils/CdfPluginPropertyLocator.java

Unit Tests:

No changes made to unit tests.
Verified that all new test cases pass successfully.

masivesky · 2024-12-12T09:48:08Z

Build with unit tests is still failing, there are a bunch of checkstyle errors, for eg:

/tmp/google-cloud/google-cloud/src/e2e-test/java/io/cdap/plugin/utils/E2ETestConstants.java:20:68: '=' is not preceded with whitespace. [WhitespaceAround]

Please check and fix the same

…fully.

itsankit-google · 2024-12-16T06:02:18Z

pom.xml

@@ -86,7 +86,7 @@
    <google.cloud.storage.version>2.3.0</google.cloud.storage.version>
    <google.cloud.datastore.version>1.105.1</google.cloud.datastore.version>
    <google.protobuf.java.version>3.19.4</google.protobuf.java.version>
-    <google.tink.version>1.3.0-rc3</google.tink.version>
+    <google.tink.version>1.5.0</google.tink.version>


why this change?

I encountered dependency errors while running mvn clean install after resolving all the checkstyle issues. The build was failing due to the outdated version of google.tink.version. Updating it to 1.5.0 resolved the issue, and no other changes were made.

please do not make dependency changes in the e2e tests PR.

Refactor CdfPluginPropertyLocator alignment and update build dependencies.
-Fixed indentation alignment for partitionFrom, partitionTo, filter and Output Schema.
-Updated pom.xml as it is.

itsankit-google · 2024-12-16T06:03:12Z

src/e2e-test/java/io/cdap/plugin/utils/CdfPluginPropertyLocator.java

+            .put("projectId", CdfPluginPropertyLocator.PROJECT_ID)
+            .put("datasetProjectId", CdfPluginPropertyLocator.DATASET_PROJECT_ID)
+            .put("dataset", CdfPluginPropertyLocator.DATASET)
+            .put("table", CdfPluginPropertyLocator.TABLE)
+            .put("format", CdfPluginPropertyLocator.FORMAT)
+            .put("path", CdfPluginPropertyLocator.PATH)
+            .put("sampleSize", CdfPluginPropertyLocator.SAMPLE_SIZE)
+            .put("delimiter", CdfPluginPropertyLocator.DELIMITER)
+            .put("skipHeader", CdfPluginPropertyLocator.SKIP_HEADER)
+            .put("pathSuffix", CdfPluginPropertyLocator.SUFFIX)
+            .put("encryptionKeyName", CdfPluginPropertyLocator.CMEK_KEY)
+            .put("serviceAccountType", CdfPluginPropertyLocator.SERVICE_ACCOUNT_TYPE)
+            .put("serviceAccountFilePath", CdfPluginPropertyLocator.SERVICE_ACCOUNT_PATH)
+            .put("serviceAccountJSON", CdfPluginPropertyLocator.SERVICE_ACCOUNT_JSON)
+            .put("truncateTable", CdfPluginPropertyLocator.TRUNCATE_TABLE)
+            .put("updateTableSchema", CdfPluginPropertyLocator.UPDATE_TABLE_SCHEMA)
+            .put("topic", CdfPluginPropertyLocator.PUBSUB_TOPIC)
+            .put("maximumBatchCount", CdfPluginPropertyLocator.PUBSUB_MAXIMUM_BATCH_COUNT)
+            .put("maximumBatchSize", CdfPluginPropertyLocator.PUBSUB_MAXIMUM_BATCH_SIZE)
+            .put("publishDelayThreshold", CdfPluginPropertyLocator.PUBSUB_PUBLISH_DELAY_THRESHOLD)
+            .put("retryTimeout", CdfPluginPropertyLocator.PUBSUB_RETRY_TIMEOUT)
+            .put("errorThreshold", CdfPluginPropertyLocator.PUBSUB_ERROR_THRESHOLD)
+            .put("outputSchema", CdfPluginPropertyLocator.OUTPUT_SCHEMA_MACRO_INPUT)
+            .put("objectsToDelete", CdfPluginPropertyLocator.GCS_DELETE_OBJECTS_TO_DELETE)
+            .put("objectsToCreate", CdfPluginPropertyLocator.GCS_CREATE_OBJECTS_TO_CREATE)
+            .put("createFailIfObjectExists", CdfPluginPropertyLocator.GCS_CREATE_FAIL_IF_OBJECT_EXISTS)
+            .put("gcsMoveSourcePath", CdfPluginPropertyLocator.GCS_MOVE_SOURCE_PATH)
+            .put("gcsMoveDestinationPath", CdfPluginPropertyLocator.GCS_MOVE_DESTINATION_PATH)
+            .put("partitionFrom", CdfPluginPropertyLocator.PARTITION_START_DATE)
+            .put("partitionTo", CdfPluginPropertyLocator.PARTITION_END_DATE)
+            .put("filter", CdfPluginPropertyLocator.FILTER)
+            .put("Output Schema-macro-input", CdfPluginPropertyLocator.OUTPUT_SCHEMA)
+            .build();


why this change?

I was unable to add my steps properly in this field because the alignments were incorrect. To fix this, I copied and pasted the same content and added my steps without making any other changes.

please fix the code style in your editor.

Refactor CdfPluginPropertyLocator alignment and update build dependencies.
-Fixed indentation alignment for partitionFrom, partitionTo, filter and Output Schema.
-Updated pom.xml as it is.

itsmekumari · 2024-12-16T08:08:00Z

src/e2e-test/features/bigquery/source/BigQuerySourceError.feature

+    Then Replace input plugin property: "table" with value: "bqSourceTable"
+    Then Click on the Get Schema button
+    Then Enter BigQuery source properties partitionFrom and partitionTo
+    Then Validate BigQuery source incorrect property error for Partition Start date "<property>" value "<value>"


@Praveena2607 I don't think the internal review comments have been addressed. Please address all the internal review comments provided earlier in this PR as well. Ref Praveena2607#1

As discussed over call, Please resolve all the discussed points and mentioned in internal PR. To use the latest framework steps in all scenarios, for entering the values in property, for validating inline error messages, for selecting output schema macro.

…cies. -Fixed indentation alignment for partitionFrom, partitionTo, filter and Output Schema. -Updated pom.xml for as it is.

itsmekumari · 2024-12-19T05:18:15Z

src/e2e-test/features/bigquery/source/BigQueryToGCS_WithMacro.feature

+    Then Validate the cmek key "cmekGCS" of target GCS bucket if cmek is enabled
+
+  @CMEK @BQ_SOURCE_TEST @GCS_SINK_TEST
+  Scenario:Validate successful records transfer from BigQuery to GCS with macro arguments for filter and outputschema


Output schema macro step is not present in scenario but in title it is written. Make sure if not using in scenario remove from title, and add in the scenario where step is used.

itsmekumari · 2024-12-19T05:18:48Z

src/e2e-test/features/bigquery/source/BigQueryToGCS_WithMacro.feature

+  Scenario:Validate successful records transfer from BigQuery to GCS with macro arguments for filter and outputschema
+    Given Open Datafusion Project to configure pipeline
+    When Source is BigQuery
+    When Sink is GCS


Use the latest steps from framework.

itsmekumari · 2024-12-19T05:20:02Z

src/e2e-test/features/bigquery/source/BigQueryToGCS_WithMacro.feature

+    Then Enter BigQuery property "serviceAccountJSON" as macro argument "serviceAccount"
+    Then Enter BigQuery property "dataset" as macro argument "bqDataset"
+    Then Enter BigQuery property "table" as macro argument "bqSourceTable"
+    Then Enter BigQuery source property output schema "outputSchema" as macro argument "bqOutputSchema"


Use the existing step discussed earlier, for selecting macro action for output schema.

itsmekumari · 2024-12-19T05:24:06Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BigQueryBase.java

@@ -260,4 +264,124 @@ public void validateRecordsTransferredToTargetTableIsEqualToNumberOfRecordsFromS
    BeforeActions.scenario.write("Number of records transferred from source table to target table:" + count);
    Assert.assertEquals(count, countRecordsTarget);
  }
+
+  @Then("Enter BigQuery source properties partitionFrom and partitionTo")


No need to add the step definition. Please make use of existing steps. This was discussed may times in our calls.

itsmekumari · 2024-12-19T05:24:27Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BigQueryBase.java

+    CdfBigQueryPropertiesActions.enterPartitionEndDate(new SimpleDateFormat("dd-MM-yyyy")
+            .format(DateUtils.addDays(new Date(), 1)));
+  }
+  @Then("Validate BigQuery source incorrect property error for Partition Start date {string} value {string}")


No need to add the step definition. Please make use of existing steps.

itsmekumari · 2024-12-19T05:25:02Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BigQueryBase.java

+    Assert.assertEquals(expectedColor, actualColor);
+  }
+
+  @Then("Validate BigQuery source incorrect property error for Partition End date {string} value {string}")


Here as well.

itsmekumari · 2024-12-19T05:25:22Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BigQueryBase.java

+    Assert.assertEquals(expectedColor, actualColor);
+  }
+
+  @Then("Validate BigQuery source incorrect property error for reference name{string} value {string}")


Here as well.

itsmekumari · 2024-12-19T05:26:49Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BigQueryBase.java

+
+  }
+
+  @Then("Enter BigQuery source properties filter")


All the step definitions added here in base file are all not needed. Use the existing steps, and execute the tests.

itsankit-google · 2024-12-19T06:30:44Z

pom.xml

@@ -86,7 +86,7 @@
    <google.cloud.storage.version>2.3.0</google.cloud.storage.version>
    <google.cloud.datastore.version>1.105.1</google.cloud.datastore.version>
    <google.protobuf.java.version>3.19.4</google.protobuf.java.version>
-    <google.tink.version>1.3.0-rc3</google.tink.version>
+    <google.tink.version>1.3.0</google.tink.version>


this is still not reverted to older version

itsankit-google

please revert back the checkstyle changes in existing steps.

masivesky · 2024-12-25T16:07:34Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BigQueryBase.java

+import io.cdap.e2e.pages.actions.CdfPluginPropertiesActions;
 import io.cdap.e2e.pages.actions.CdfStudioActions;
+import io.cdap.e2e.pages.locators.CdfBigQueryPropertiesLocators;
 import io.cdap.e2e.pages.locators.CdfStudioLocators;
 import io.cdap.e2e.utils.BigQueryClient;
 import io.cdap.e2e.utils.ConstantsUtil;
 import io.cdap.e2e.utils.ElementHelper;
 import io.cdap.e2e.utils.PluginPropertyUtils;
 import io.cdap.e2e.utils.SeleniumHelper;
+import io.cdap.e2e.utils.WaitHelper;
 import io.cdap.plugin.common.stepsdesign.TestSetupHooks;
+import io.cdap.plugin.utils.CdfPluginPropertyLocator;
 import io.cdap.plugin.utils.E2EHelper;
 import io.cdap.plugin.utils.E2ETestConstants;
 import io.cucumber.java.en.Then;
 import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.time.DateUtils;
 import org.junit.Assert;
 import stepsdesign.BeforeActions;

 import java.io.IOException;
 import java.text.SimpleDateFormat;
 import java.util.Date;
 import java.util.Optional;
+import java.util.Properties;


Any reason we are adding additional imports here? Don't see them being used in the file anywhere

Additional Steps for BQ.

2f6ad0f

itsankit-google added the build Trigger unit test build label Dec 11, 2024

itsankit-google requested a review from masivesky December 11, 2024 07:25

masivesky reviewed Dec 11, 2024

View reviewed changes

Updated CdfPluginPropertyLocator and associated fiels for BigQuery Pl…

60fb70f

…ugin: Added and modified.

Fixed Checkstyle issues and ensured all scenarios are running success…

7e71bc2

…fully.

itsankit-google reviewed Dec 16, 2024

View reviewed changes

itsmekumari reviewed Dec 16, 2024

View reviewed changes

Refactor CdfPluginPropertyLocator alignment and update build dependen…

0870576

…cies. -Fixed indentation alignment for partitionFrom, partitionTo, filter and Output Schema. -Updated pom.xml for as it is.

itsmekumari reviewed Dec 19, 2024

View reviewed changes

All the changes are made.Please check.

06ac469

itsankit-google reviewed Dec 19, 2024

View reviewed changes

itsankit-google requested changes Dec 19, 2024

View reviewed changes

masivesky reviewed Dec 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Additional Steps for BQ_Source #1472

Additional Steps for BQ_Source #1472

Praveena2607 commented Dec 11, 2024 •

edited

Loading

google-cla bot commented Dec 11, 2024

masivesky left a comment

masivesky left a comment

Praveena2607 commented Dec 12, 2024

masivesky commented Dec 12, 2024

itsankit-google Dec 16, 2024

Praveena2607 Dec 16, 2024

itsankit-google Dec 16, 2024

Praveena2607 Dec 16, 2024

itsankit-google Dec 16, 2024

Praveena2607 Dec 16, 2024

itsankit-google Dec 16, 2024

Praveena2607 Dec 16, 2024

itsmekumari Dec 16, 2024 •

edited

Loading

itsmekumari Dec 19, 2024

itsmekumari Dec 19, 2024

itsmekumari Dec 19, 2024

itsmekumari Dec 19, 2024 •

edited

Loading

itsmekumari Dec 19, 2024

itsmekumari Dec 19, 2024

itsmekumari Dec 19, 2024

itsmekumari Dec 19, 2024

itsmekumari Dec 19, 2024

itsankit-google Dec 19, 2024

itsankit-google left a comment

masivesky Dec 25, 2024

Additional Steps for BQ_Source #1472

Are you sure you want to change the base?

Additional Steps for BQ_Source #1472

Conversation

Praveena2607 commented Dec 11, 2024 • edited Loading

google-cla bot commented Dec 11, 2024

masivesky left a comment

Choose a reason for hiding this comment

masivesky left a comment

Choose a reason for hiding this comment

Praveena2607 commented Dec 12, 2024

masivesky commented Dec 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsmekumari Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsmekumari Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsankit-google left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Praveena2607 commented Dec 11, 2024 •

edited

Loading

itsmekumari Dec 16, 2024 •

edited

Loading

itsmekumari Dec 19, 2024 •

edited

Loading