ADAP-940: Add change monitoring for partitioning clause #962
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…940' into feature/materialized-views/ADAP-940
mikealfare
changed the base branch from
main
to
feature/materialized-views/ADAP-394
…ialized-views/ADAP-940
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-bigquery contributing guide. |
mikealfare
merged commit 2581b62
into
feature/materialized-views/ADAP-394
9 of 10 checks passed
mikealfare
added a commit
that referenced
this pull request
Oct 11, 2023
* init push of ADAP-394 * update lots of framework and update to main * updating based on feedback * add changelog * remove in preivew option * fill out changeset and config change classes for specific options * change partition_by and cluster_by to FrozenSet, initial attempt at describe.sql * create utility.py to add bool_setting method, update parse_model_node to use new method * update describe.sql query * update describe sql to be able to create list of cluster by field names * initital attempt at modifying get_alter_materialized_view_as_sql * update to main and add space * initial build out of mini classes for bigquery cluster, partition, auto_refresh in materialized_view dict. still need to fix some mypy issues * remove local package (dbt-bigquery) on `make dev-uninstall` * update changelog entry to encompass all features in this branch * remove alteration to setup/teardown for materialized view materialization * fix spelling error, prepend underscore on base class module to mark as private to its package * update call to relation to include quote and include policies, update case to match convention * update create statement to include partition, cluster, and options clauses * update partition config to align with existing dbt-bigquery table config * update cluster config to align with existing dbt-bigquery table config * update auto refresh config to align with other configs * revert parse results to accept an agate Row * update how defaults are handled * add description option to materialized view since it is handled for tables * add description option to materialized view since it is handled for tables * fix method call chain in parse_relation_results on cluster, partition, and auto_refresh * move PartitionConfig into relation_configs to be used by materialized views, update references * move PartitionConfig into relation_configs to be used by materialized views, update references; add get_materialized_view_options in alignment with get_table_options; fix wild import order; add factory method for materialized views to be used in the jinja template; update expiration timestamp attribute; * update create materialized view to use the relation config * condition on existence of properties before templating them * allow for "drop if exists" functionality via the google sdk * remove unnecessary trailing semicolon * implement replace based on create * implement clustering, partitioning, and auto refresh for materialized views * remove include_policy from BigQueryRelation, it's causing unit tests to fail and is not used at the moment * partition type cannot be queried for materialized views, adjust the describe query and equality check to account for that * add describe_relation for materialized views * break out common utilities into a mixin for materialized view tests * change refresh_interval_minutes from an int to a float to match the bigquery docs * make partition optional on relation results since it cannot be queried yet * initial draft of materialized view change tests * build changeset for materialized view * implement change monitoring for autorefresh and clustering on materialized views, add describe_relation method on BigQueryAdapter for utility * committing to park changes and wrap up other 1.7 items * update describe to use the sdk instead of sql to pick up partition information * basic tests pass * existing change monitoring tests pass * partition change monitoring tests pass * ADAP-940: Add change monitoring for partitioning clause (#962) * committing to park changes and wrap up other 1.7 items * update describe to use the sdk instead of sql to pick up partition information * basic tests pass * existing change monitoring tests pass * partition change monitoring tests pass --------- Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com> * implement PR review feedback * delete empty file * add MV tests for cluster and partition alone, update combined tests to perform all checks --------- Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com> Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com> Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #
docs dbt-labs/docs.getdbt.com/#
Problem
Solution
Checklist