Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable setting dbt_valid_to snapshot column to new setting dbt_valid_to_current #1113

Merged
merged 13 commits into from
Oct 15, 2024
Merged

Enable setting dbt_valid_to snapshot column to new setting dbt_valid_to_current #1113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a0064fc
Use dbt_valid_to_current config
gshank Sep 25, 2024
1ef4fbc
Merge branch 'main' into snapshot_dbt_valid_to_current
gshank Sep 25, 2024
a5a25ab
Check both dbt_valid_to_current and null
gshank Sep 27, 2024
ff3fc50
Changie
gshank Sep 27, 2024
3ba89b2
Merge branch 'main' into snapshot_dbt_valid_to_current
gshank Sep 30, 2024
0bcd63c
Try skipping databricks_http_cluster profile for test
gshank Oct 11, 2024
dafda99
Skip right profile
gshank Oct 11, 2024
dce16ff
Skip another test
gshank Oct 11, 2024
80e38cc
Another
gshank Oct 11, 2024
46a5729
Merge branch 'main' into snapshot_dbt_valid_to_current
gshank Oct 15, 2024
896ee81
Revert "Another"
gshank Oct 15, 2024
b1ff98b
Revert "Skip another test"
gshank Oct 15, 2024
cb8dad9
Revert "Skip right profile"
gshank Oct 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20240927-133927.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: Enable setting current value of dbt_valid_to
time: 2024-09-27T13:39:27.268886-04:00
custom:
Author: gshank
Issue: "1112"
7 changes: 6 additions & 1 deletion dbt/include/spark/macros/materializations/snapshot.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,12 @@
{% endif %}
on DBT_INTERNAL_SOURCE.{{ columns.dbt_scd_id }} = DBT_INTERNAL_DEST.{{ columns.dbt_scd_id }}
when matched
and DBT_INTERNAL_DEST.{{ columns.dbt_valid_to }} is null
{% if config.get("dbt_valid_to_current") %}
and ( DBT_INTERNAL_DEST.{{ columns.dbt_valid_to }} = {{ config.get('dbt_valid_to_current') }} or
DBT_INTERNAL_DEST.{{ columns.dbt_valid_to }} is null )
{% else %}
and DBT_INTERNAL_DEST.{{ columns.dbt_valid_to }} is null
{% endif %}
and DBT_INTERNAL_SOURCE.dbt_change_type in ('update', 'delete')
then update
set {{ columns.dbt_valid_to }} = DBT_INTERNAL_SOURCE.{{ columns.dbt_valid_to }}
Expand Down
6 changes: 5 additions & 1 deletion tests/functional/adapter/test_python_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,11 @@ def model(dbt, spark):


@pytest.mark.skip_profile(
"apache_spark", "spark_session", "databricks_sql_endpoint", "spark_http_odbc"
"apache_spark",
"spark_session",
"databricks_sql_endpoint",
"spark_http_odbc",
"databricks_http_cluster",
)
class TestChangingSchemaSpark:
"""
Expand Down
Loading