Skip to content

Commit

Permalink
Added sample session and dump file
Browse files Browse the repository at this point in the history
Support check constraint frontend (#22)

* chore(deps): update dependency jasmine-core to ~5.5.0 (GoogleCloudPlatform#953)

Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>

* removed duplicate function

* Fixed UT

* feat: make spanner config mandatory for schema migration (GoogleCloudPlatform#970)

* changes

* change

* change

* chore(deps): update gcr.io/cloud-spanner-emulator/emulator docker tag to v1.5.28 (GoogleCloudPlatform#974)

* chore(deps): update dependency @types/node to v20.17.10 (GoogleCloudPlatform#973)

Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>

* Add Support for Check Constraint - Backend code changes (GoogleCloudPlatform#945)

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

* Updating version of msprod (GoogleCloudPlatform#969)

* fix(deps): update module golang.org/x/net to v0.33.0 [security] (GoogleCloudPlatform#967)

Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>

* feat: APIs for Backend Changes for Default Values (GoogleCloudPlatform#965)

* backend apis

* linting

* comment changes

* comment changes

* feat: default value for mysql source  (GoogleCloudPlatform#963)

* source dv

* test fix

* change

* comment changes

* test fix

* change

* fix github v

* change

* Check constraint backend (#9)

Backend Support for Check Constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* removed duplicate function

* Fixed UT

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com>

* Check constraint backend (#9)

Backend Support for Check Constraint

* removed duplicate code after rebase

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

* feat: APIs for Backend Changes for Default Values (GoogleCloudPlatform#965)

* backend apis

* linting

* comment changes

* comment changes

* feat: default value for mysql source  (GoogleCloudPlatform#963)

* source dv

* test fix

* change

* comment changes

* test fix

* change

* fix github v

* change

* Check constraint backend (#9)

Backend Support for Check Constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* removed duplicate function

* Fixed UT

* Check constraint backend (#9)

Backend Support for Check Constraint

* removed duplicate code after rebase

* removed duplicate constants

* Support check constraint backend (GoogleCloudPlatform#962)

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>

* Feat/fe check contraints (#10)

* Add the UI changes  for supporting check constraints

---------

Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>
Co-authored-by: taherkl <taher.lakdawala@ollion.com>

* updated the regrex for constraint name

* address the comment

* addressed the comment
1. change checkIfCcColumn to check in Spschema

* address the comment
1.Modify the checkIfCcColumn to handle pk and check constraint

* address the comment
1. when  interleave tab is shown 4 or 3

* fixed regular expression for check constraints name

* remove the node-sql-parser package

* handle check constraint without name.

* resolve the duplicate issue

* Check constraint backend (#9)

Backend Support for Check Constraint

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* fix: added regular exprression to match the exact column

* Added test case for the column rename for check constraint

* PR and UT fixes

* added the verification api integration and struct changes

* added the verification at connect button

* added the integration of expression api

* code fix for expression api implementation

* code fix for expression verification on initial call

* added partial unit test of verify check constraint expression

* refactored expression verfication api mock

* Test added for verification api

* fix the UT issue

* Support check constraint backend (GoogleCloudPlatform#962)

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>

* Fixed unit test and code refactor

* fixed unit tests

* added source in conv object

* front-end ut fixed

* removed node sql parser import

* removed node dependencies

* rename handler

* added the generation of constraint name

* added dump flow

* added dump flow

* Feat/fe check contraints (#10)

* Add the UI changes  for supporting check constraints

---------

Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>
Co-authored-by: taherkl <taher.lakdawala@ollion.com>

* address the comment

* remove the node-sql-parser package

* Check constraint backend (#9)

Backend Support for Check Constraint

* fixed PR comments

* fix: added regular exprression to match the exact column

* Added test case for the column rename for check constraint

* nit: doesCheckConstraintNameExist

* moved regex globally

* modify the query to fetch the check constraint

* added the verification api integration and struct changes

* added partial unit test of verify check constraint expression

* refactored expression verfication api mock

* Test added for verification api

* Support check constraint backend (GoogleCloudPlatform#962)

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>

* Fixed unit test and code refactor

* removed node dependencies

* added dump flow

* Feat/fe check contraints (#10)

* Add the UI changes  for supporting check constraints

---------

Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>
Co-authored-by: taherkl <taher.lakdawala@ollion.com>

* address the comment

* remove the node-sql-parser package

* Check constraint backend (#9)

Backend Support for Check Constraint

* added the verification api integration and struct changes

* added partial unit test of verify check constraint expression

* refactored expression verfication api mock

* Test added for verification api

* Support check constraint backend (GoogleCloudPlatform#962)

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>

* removed node dependencies

* Feat/fe check contraints (#10)

* Add the UI changes  for supporting check constraints

---------

Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>
Co-authored-by: taherkl <taher.lakdawala@ollion.com>

* address the comment

* remove the node-sql-parser package

* Check constraint backend (#9)

Backend Support for Check Constraint

* added the verification api integration and struct changes

* added partial unit test of verify check constraint expression

* refactored expression verfication api mock

* Test added for verification api

* Support check constraint backend (GoogleCloudPlatform#962)

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>

* removed node dependencies

* added UT

* address the comment
1. handle the unexpected error
2. resuse the code

* updated the table exist query for version8.0.16

* Feat/fe check contraints (#10)

* Add the UI changes  for supporting check constraints

---------

Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>
Co-authored-by: taherkl <taher.lakdawala@ollion.com>

* address the comment

* address the comment
1.Modify the checkIfCcColumn to handle pk and check constraint

* rebase commit

* fixed regular expression for check constraints name

* remove the node-sql-parser package

* resolve the duplicate issue

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fix: added regular exprression to match the exact column

* Added test case for the column rename for check constraint

* reverted white spaces

* reverted white spaces

* added the verification api integration and struct changes

* added the integration of expression api

* code fix for expression api implementation

* code fix for expression verification on initial call

* added partial unit test of verify check constraint expression

* refactored expression verfication api mock

* Test added for verification api

* Support check constraint backend (GoogleCloudPlatform#962)

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>

* fixed unit tests

* front-end ut fixed

* removed node sql parser import

* removed node dependencies

* rename handler

* added the generation of constraint name

* added dump flow

* Feat/fe check contraints (#10)

* Add the UI changes  for supporting check constraints

---------

Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>
Co-authored-by: taherkl <taher.lakdawala@ollion.com>

* address the comment

* remove the node-sql-parser package

* Check constraint backend (#9)

Backend Support for Check Constraint

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* fix: added regular exprression to match the exact column

* Added test case for the column rename for check constraint

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* moved regex globally

* modify the query to fetch the check constraint

* added the verification api integration and struct changes

* added partial unit test of verify check constraint expression

* refactored expression verfication api mock

* Test added for verification api

* Support check constraint backend (GoogleCloudPlatform#962)

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>

* Fixed unit test and code refactor

* removed node dependencies

* added dump flow

* Feat/fe check contraints (#10)

* Add the UI changes  for supporting check constraints

---------

Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>
Co-authored-by: taherkl <taher.lakdawala@ollion.com>

* address the comment

* address the comment
1.Modify the checkIfCcColumn to handle pk and check constraint

* fixed regular expression for check constraints name

* remove the node-sql-parser package

* resolve the duplicate issue

* Check constraint backend (#9)

Backend Support for Check Constraint

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* moved regex globally

* modify the query to fetch the check constraint

* added the verification api integration and struct changes

* added partial unit test of verify check constraint expression

* refactored expression verfication api mock

* Test added for verification api

* Support check constraint backend (GoogleCloudPlatform#962)

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>

* fixed unit tests

* front-end ut fixed

* remove unwanted code

* removed node sql parser import

* removed node dependencies

* rename handler

* added the generation of constraint name

* added dump flow

* Feat/fe check contraints (#10)

* Add the UI changes  for supporting check constraints

---------

Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>
Co-authored-by: taherkl <taher.lakdawala@ollion.com>

* address the comment

* address the comment
1.Modify the checkIfCcColumn to handle pk and check constraint

* fixed regular expression for check constraints name

* remove the node-sql-parser package

* resolve the duplicate issue

* Check constraint backend (#9)

Backend Support for Check Constraint

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* moved regex globally

* Fix UT

* modify the query to fetch the check constraint

* added the verification api integration and struct changes

* added partial unit test of verify check constraint expression

* refactored expression verfication api mock

* Test added for verification api

* Support check constraint backend (GoogleCloudPlatform#962)

* Check constraint backend (#9)

Backend Support for Check Constraint

* update api

* fix PR comment

* remove api call to while validating constraints

* Fixed db collation regex to remove collation name from the results

* renamed function name to formatCheckConstraints and added check if constraint name is empty

* fixed PR comments

* added test case for the empty check constraint name

* fix: added regular exprression to match the exact column

* fix: added regular expression to replace table name

* Added test case for the column rename for check constraint

* 1. Refactored GetConstraint function
2. Fixed inforschema unit tests

* added comment at handling case for check constraints

* reverted white spaces

* reverted white spaces

* nit: doesCheckConstraintNameExist

* added comments for doesCheckConstraintNameExist

* PR and UT fixes

* fix UT

* UT fix

* Removed isCheckConstraintsTablePresent function

* moved regex globally

* Fix UT

* fixed UT

* fixed handling of the constraints

* removed unused function

* added unit tests for incompatable name

* Combined unit tests

* added test case for the renaming column having substring of other column

* added the query changes which return distinct value

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>

* fixed unit tests

* front-end ut fixed

* remove unwanted code

* removed node sql parser import

* removed node dependencies

* rename handler

* added the generation of constraint name

* added dump flow

* added UT

* address the comment
1. handle the unexpected error
2. resuse the code

* added mock for RefreshSpannerClient function

* addressed the comment and feedback

---------

Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com>
Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com>
Co-authored-by: Vivek Yadav <vivek.yadav@ollion.com>
Co-authored-by: Vivek Yadav <105432992+VivekY1098@users.noreply.github.com>
  • Loading branch information
8 people committed Dec 27, 2024
1 parent a5067cd commit 891bc56
Show file tree
Hide file tree
Showing 51 changed files with 2,539 additions and 329 deletions.
23 changes: 15 additions & 8 deletions .github/workflows/integration-tests-against-emulator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ jobs:

services:
spanner_emulator:
image: gcr.io/cloud-spanner-emulator/emulator:1.5.27
image: gcr.io/cloud-spanner-emulator/emulator:1.5.28
ports:
- 9010:9010
- 9020:9020
Expand Down Expand Up @@ -114,21 +114,28 @@ jobs:
- run: mysql -v -P 3306 --protocol=tcp -u root -proot < test_data/mysql_foreignkeyaction_dump.test.out

# init sql server with test_data
# since we use ubuntu-latest container, we should ensure that the path matches the latest from https://packages.microsoft.com/config/ubuntu/
# while its possible to infer the latest from the path in the run script, it will make the run section more complex and hard to maintian.
- name: Install sqlcmd required for loading .sql files
run: |
curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list | sudo tee /etc/apt/sources.list.d/msprod.list
sudo apt-get update
sudo apt-get install mssql-tools unixodbc-dev
echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile
- run: sqlcmd -?
- run: sqlcmd -U sa -P ${MSSQL_SA_PASSWORD} -i test_data/sqlserver.test.out
curl https://packages.microsoft.com/config/ubuntu/24.04/prod.list | sudo tee /etc/apt/sources.list.d/msprod.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18
sudo apt-get install mssql-tools18 unixodbc-dev
set -x
ls /opt/mssql-tools18/bin/
set +x
echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> ~/.bashrc
- run: /opt/mssql-tools18/bin/sqlcmd -C -?
- run: /opt/mssql-tools18/bin/sqlcmd -U sa -P ${MSSQL_SA_PASSWORD} -i test_data/sqlserver.test.out -C

# sqlplus set up init oracle db.
- name: Install sqlplus required for loading .sql files
run: |
sudo apt-get update
sudo apt-get install -y libaio1 rpm2cpio cpio
sudo apt-get install -y libaio1t64 rpm2cpio cpio
sudo ln -s /usr/lib/x86_64-linux-gnu/libaio.so.1t64 /usr/lib/libaio.so.1
curl -O https://download.oracle.com/otn_software/linux/instantclient/2340000/oracle-instantclient-basic-23.4.0.24.05-1.el9.x86_64.rpm
curl -O https://download.oracle.com/otn_software/linux/instantclient/2340000/oracle-instantclient-sqlplus-23.4.0.24.05-1.el9.x86_64.rpm
rpm2cpio oracle-instantclient-basic-23.4.0.24.05-1.el9.x86_64.rpm | sudo cpio -idmv
Expand Down
4 changes: 3 additions & 1 deletion common/constants/constants.go
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,9 @@ const (
// VerifyExpresions API
CHECK_EXPRESSION = "CHECK"
DEFAUT_EXPRESSION = "DEFAULT"

DEFAULT_GENERATED = "DEFAULT_GENERATED"
TEMP_DB = "smt-staging-db"

// Regex for matching database collation
DB_COLLATION_REGEX = `(_[a-zA-Z0-9]+\\|\\)`
)
3 changes: 3 additions & 0 deletions conversion/conversion_from_source.go
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,9 @@ func (sads *DataFromSourceImpl) dataFromCSV(ctx context.Context, sourceProfile p
return nil, fmt.Errorf("dbName is mandatory in target-profile for csv source")
}
conv.SpDialect = targetProfile.Conn.Sp.Dialect
conv.SpProjectId = targetProfile.Conn.Sp.Project
conv.SpInstanceId = targetProfile.Conn.Sp.Instance
conv.Source = sourceProfile.Driver
dialect, err := targetProfile.FetchTargetDialect(ctx)
if err != nil {
return nil, fmt.Errorf("could not fetch dialect: %v", err)
Expand Down
108 changes: 105 additions & 3 deletions expressions_api/expression_verify.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"fmt"
"sync"

spannerclient "github.com/GoogleCloudPlatform/spanner-migration-tool/accessors/clients/spanner/client"
spanneraccessor "github.com/GoogleCloudPlatform/spanner-migration-tool/accessors/spanner"
"github.com/GoogleCloudPlatform/spanner-migration-tool/common/constants"
"github.com/GoogleCloudPlatform/spanner-migration-tool/common/task"
Expand All @@ -18,22 +19,50 @@ const THREAD_POOL = 500
type ExpressionVerificationAccessor interface {
//Batch API which parallelizes expression verification calls
VerifyExpressions(ctx context.Context, verifyExpressionsInput internal.VerifyExpressionsInput) internal.VerifyExpressionsOutput
RefreshSpannerClient(ctx context.Context, project string, instance string) error
}

type ExpressionVerificationAccessorImpl struct {
SpannerAccessor *spanneraccessor.SpannerAccessorImpl
}

func NewExpressionVerificationAccessorImpl(ctx context.Context, project string, instance string) (*ExpressionVerificationAccessorImpl, error) {
spannerAccessor, err := spanneraccessor.NewSpannerAccessorClientImplWithSpannerClient(ctx, fmt.Sprintf("projects/%s/instances/%s/databases/%s", project, instance, "smt-staging-db"))
if err != nil {
return nil, err
var spannerAccessor *spanneraccessor.SpannerAccessorImpl
var err error
if project != "" && instance != "" {
spannerAccessor, err = spanneraccessor.NewSpannerAccessorClientImplWithSpannerClient(ctx, fmt.Sprintf("projects/%s/instances/%s/databases/%s", project, instance, constants.TEMP_DB))
if err != nil {
return nil, err
}
} else {
spannerAccessor, err = spanneraccessor.NewSpannerAccessorClientImpl(ctx)
if err != nil {
return nil, err
}
}
return &ExpressionVerificationAccessorImpl{
SpannerAccessor: spannerAccessor,
}, nil
}

// APIs to verify and process Spanner DLL features such as Default Values, Check Constraints
type DDLVerifier interface {
VerifySpannerDDL(conv *internal.Conv, expressionDetails []internal.ExpressionDetail) (internal.VerifyExpressionsOutput, error)
GetSourceExpressionDetails(conv *internal.Conv, tableIds []string) []internal.ExpressionDetail
GetSpannerExpressionDetails(conv *internal.Conv, tableIds []string) []internal.ExpressionDetail
RefreshSpannerClient(ctx context.Context, project string, instance string) error
}
type DDLVerifierImpl struct {
Expressions ExpressionVerificationAccessor
}

func NewDDLVerifierImpl(ctx context.Context, project string, instance string) (*DDLVerifierImpl, error) {
expVerifier, err := NewExpressionVerificationAccessorImpl(ctx, project, instance)
return &DDLVerifierImpl{
Expressions: expVerifier,
}, err
}

func (ev *ExpressionVerificationAccessorImpl) VerifyExpressions(ctx context.Context, verifyExpressionsInput internal.VerifyExpressionsInput) internal.VerifyExpressionsOutput {
err := ev.validateRequest(verifyExpressionsInput)
if err != nil {
Expand Down Expand Up @@ -79,6 +108,15 @@ func (ev *ExpressionVerificationAccessorImpl) VerifyExpressions(ctx context.Cont
return verifyExpressionsOutput
}

func (ev *ExpressionVerificationAccessorImpl) RefreshSpannerClient(ctx context.Context, project string, instance string) error {
spannerClient, err := spannerclient.NewSpannerClientImpl(ctx, fmt.Sprintf("projects/%s/instances/%s/databases/%s", project, instance, constants.TEMP_DB))
if err != nil {
return err
}
ev.SpannerAccessor.SpannerClient = spannerClient
return nil
}

func (ev *ExpressionVerificationAccessorImpl) verifyExpressionInternal(expressionDetail internal.ExpressionDetail, mutex *sync.Mutex) task.TaskResult[internal.ExpressionVerificationOutput] {
var sqlStatement string
switch expressionDetail.Type {
Expand Down Expand Up @@ -132,3 +170,67 @@ func (ev *ExpressionVerificationAccessorImpl) removeExpressions(inputConv *inter
}
return convCopy, nil
}

func (ddlv *DDLVerifierImpl) VerifySpannerDDL(conv *internal.Conv, expressionDetails []internal.ExpressionDetail) (internal.VerifyExpressionsOutput, error) {
ctx := context.Background()
verifyExpressionsInput := internal.VerifyExpressionsInput{
Conv: conv,
Source: conv.Source,
ExpressionDetailList: expressionDetails,
}
verificationResults := ddlv.Expressions.VerifyExpressions(ctx, verifyExpressionsInput)

return verificationResults, verificationResults.Err
}

func (ddlv *DDLVerifierImpl) GetSourceExpressionDetails(conv *internal.Conv, tableIds []string) []internal.ExpressionDetail {
expressionDetails := []internal.ExpressionDetail{}
// Collect default values for verification
for _, tableId := range tableIds {
srcTable := conv.SrcSchema[tableId]
for _, srcColId := range srcTable.ColIds {
srcCol := srcTable.ColDefs[srcColId]
if srcCol.DefaultValue.IsPresent {
defaultValueExp := internal.ExpressionDetail{
ReferenceElement: internal.ReferenceElement{
Name: conv.SpSchema[tableId].ColDefs[srcColId].T.Name,
},
ExpressionId: srcCol.DefaultValue.Value.ExpressionId,
Expression: srcCol.DefaultValue.Value.Statement,
Type: "DEFAULT",
Metadata: map[string]string{"TableId": tableId, "ColId": srcColId},
}
expressionDetails = append(expressionDetails, defaultValueExp)
}
}
}
return expressionDetails
}

func (ddlv *DDLVerifierImpl) GetSpannerExpressionDetails(conv *internal.Conv, tableIds []string) []internal.ExpressionDetail {
expressionDetails := []internal.ExpressionDetail{}
// Collect default values for verification
for _, tableId := range tableIds {
spTable := conv.SpSchema[tableId]
for _, spColId := range spTable.ColIds {
spCol := spTable.ColDefs[spColId]
if spCol.DefaultValue.IsPresent {
defaultValueExp := internal.ExpressionDetail{
ReferenceElement: internal.ReferenceElement{
Name: conv.SpSchema[tableId].ColDefs[spColId].T.Name,
},
ExpressionId: spCol.DefaultValue.Value.ExpressionId,
Expression: spCol.DefaultValue.Value.Statement,
Type: "DEFAULT",
Metadata: map[string]string{"TableId": tableId, "ColId": spColId},
}
expressionDetails = append(expressionDetails, defaultValueExp)
}
}
}
return expressionDetails
}

func (ddlv *DDLVerifierImpl) RefreshSpannerClient(ctx context.Context, project string, instance string) error {
return ddlv.Expressions.RefreshSpannerClient(ctx, project, instance)
}
Loading

0 comments on commit 891bc56

Please sign in to comment.