Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DAT-18792: added cluster by support for snapshot and diff table related change types #199

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

SvampX
Copy link
Contributor

@SvampX SvampX commented Oct 15, 2024

Tested following liquibase commands:

  • diff
  • diff-changelog
  • snapshot
  • generate-changelog

Changelog generates with present CLUSTER BY part with appropriate columns. Tested case with 1 and 2 clustering columns.
Generated changeset example:

    <changeSet author="msav1 (generated)" id="1728984464968-4">
        <databricks:createTable clusterColumns="test_id,test_present_new" tableName="test_table_clustered_new">
            <column name="test_id" type="INT"/>
            <column name="test_present_new" type="INT"/>
            <databricks:extendedTableProperties tableLocation="s3://databricks-test-harness-metastore/c93bad90-fc3e-4083-a910-1cad48d283f2/tables/3b4b3312-cd20-41d1-9d74-451be370869f" tblProperties="delta.checkpoint.writeStatsAsJson=false,delta.checkpoint.writeStatsAsStruct=true,delta.checkpointPolicy=v2,delta.columnMapping.maxColumnId=3,delta.columnMapping.mode=name,delta.enableDeletionVectors=true,delta.enableRowTracking=true,delta.feature.allowColumnDefaults=supported,delta.feature.clustering=supported,delta.feature.columnMapping=supported,delta.feature.deletionVectors=supported,delta.feature.domainMetadata=supported,delta.feature.rowTracking=supported,delta.feature.v2Checkpoint=supported,delta.minReaderVersion=3,delta.minWriterVersion=7,delta.rowTracking.materializedRowCommitVersionColumnName=_row-commit-version-col-f039a2c1-7756-4597-9259-825d27ddce58,delta.rowTracking.materializedRowIdColumnName=_row-id-col-4af75f6c-2d74-4fcf-bc2b-8a2ddce45ef4"/>
        </databricks:createTable>
    </changeSet>

@SvampX
Copy link
Contributor Author

SvampX commented Oct 15, 2024

As per described in extracting and cleaning methods possible issues we might try different way to read and store TBLPROPERTIES. Partially it was done on the view side here https://github.com/liquibase/liquibase-databricks/blob/main/src/main/java/liquibase/ext/databricks/snapshot/jvm/ViewSnapshotGeneratorDatabricks.java#L81-L89
I suggest storing in the https://github.com/liquibase/liquibase-databricks/blob/main/src/main/java/liquibase/ext/databricks/change/createTable/ExtendedTableProperties.java tblProperties parameter as a Map, not a raw String. This would make easier and fail proof cleaning up of system table properties and parsing needed stored there data.

Base automatically changed from DAT-18787 to main October 16, 2024 11:39
Copy link

sonarcloud bot commented Oct 16, 2024

@suryaaki2
Copy link

Are we writing tests for this PR?

@filipelautert
Copy link
Contributor

Are we writing tests for this PR?
@suryaaki2 so far for Databricks we are only adding new test cases for test harness, as in createClusteredTableNew.* files added in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants