GCS Source additional scenarios.

[bijay27bit]

@itsankit-google Please review the GCS source additional scenarios.

Thanks

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[itsankit-google]

why can't we combine scenarios To verify successful records transfer from GCS source to BigQuery sink with macro fields enabled at source and To verify successful data transfer from GCS to BigQuery with macro Path Field?

[bijay27bit]

why can't we combine scenarios To verify successful records transfer from GCS source to BigQuery sink with macro fields enabled at source and To verify successful data transfer from GCS to BigQuery with macro Path Field?

Thanks Ankit,

The test data used for the 'path field' was not supported. Following an internal discussion with the team, a separate scenario was created to ensure coverage.

[itsankit-google]

why can't we combine scenarios To verify successful records transfer from GCS source to BigQuery sink with macro fields enabled at source and To verify successful data transfer from GCS to BigQuery with macro Path Field?

Thanks Ankit,

The test data used for the 'path field' was not supported. Following an internal discussion with the team, a separate scenario was created to ensure coverage.

What do you mean by testdata not supported? @itsmekumari @AnkitCLI can you provide more clarification?

[itsankit-google]

PTAL at test failures.

[itsmekumari]

why can't we combine scenarios To verify successful records transfer from GCS source to BigQuery sink with macro fields enabled at source and To verify successful data transfer from GCS to BigQuery with macro Path Field?

Thanks Ankit,
The test data used for the 'path field' was not supported. Following an internal discussion with the team, a separate scenario was created to ensure coverage.

What do you mean by testdata not supported? @itsmekumari @AnkitCLI can you provide more clarification?

I think, reference was taken from the existing test case for path field scenario. @bijay27bit you can add any path field column name and add the same in output schema which is set as macro in same scenario.

[bijay27bit]

why can't we combine scenarios To verify successful records transfer from GCS source to BigQuery sink with macro fields enabled at source and To verify successful data transfer from GCS to BigQuery with macro Path Field?

Thanks Ankit,
The test data used for the 'path field' was not supported. Following an internal discussion with the team, a separate scenario was created to ensure coverage.

What do you mean by testdata not supported? @itsmekumari @AnkitCLI can you provide more clarification?

I think, reference was taken from the existing test case for path field scenario. @bijay27bit you can add any path field column name and add the same in output schema which is set as macro in same scenario.

Merged both the scenario and it is running at local branch now. Looking below test failures as per Ankit's comment. Thanks

[itsankit-google]

PTAL at test failures.

Also the count of records transferred is zero in failing tests, not sure if it is expected.

[bijay27bit]

PTAL at test failures.

Also the count of records transferred is zero in failing tests, not sure if it is expected.

Hi @itsankit-google I have made the changes and above failed test scenarios are not created by me. Please review the latest code. Thanks

[itsankit-google]

No of records transferred is 0 in this test, is it expected?