1.0.0-b1

🥳 Beta 1 of dbt utils 1.0.0 🥳

🔜 we're aiming to release the final version in late October (post-Coalesce, alongside dbt Core 1.3) 📆

Scrappy upstart no longer, dbt utils 1.0 is a major milestone.

• Following on from the discussions in #487, this release tightly focuses the project on user-facing tests and convenience macros. Experimental materializations and cross-database macros have moved to the experiments repo and dbt Core respectively.
• In 1.0, we are finally implementing a flurry of long-promised deprecations and knotty bugs to create a cleaner, tidier, stabler API that people can build against well into the future. We have chosen to collate all of the breaking changes into one hit; after this, functionality in dbt utils 1.0.0 will work in all subsequent 1.x.y (if you don't know about semantic versioning, this is a good time to read up on it!)

New Features

• Add not_empty_string test by @epapineau in #634
• Many tests can now be validated against subgroups of a table, thanks to @emilyriederer in #633

Fixes

• Make union_relations include/exclude case insensitive by @jeremyyeo in #587

Deprecations and breaking changes

Long-forewarned deprecations:

• Remove table argument from unpivot() by @miles170 in #671
• Remove varargs in surrogate_key and safe_add, in favour of a single fields list argument by @miles170 in #674
• Remove group_by and relation_alias from deduplicate macro, make order by mandatory by @miles170 in #673
• Totally remove the identifier macro (use adapter.quote() instead) by @miles170 in #672
• Remove not_null_where and unique_where by @joellabes in #678

Recently-squared circles (not implemented yet; this is a heads-up)

• 🚨 Treat blank strings differently to nulls in surrogate_key (#488). 🚨
  • This one could cause trouble. We are going to default to correct behaviour going forward, but if you have a surrogate key which depends on the old incorrect behaviour, you will be able to opt back into it.
• Fully delete all the cross-database macros that have moved to dbt Core.

New Contributors

• @jeremyyeo made their first contribution in #587
• @miles170 made their first contribution in #671
• @emilyriederer made their first contribution in #633

Full Changelog: 0.9.2...1.0.0-b1

0.9.2

What's Changed

• Remove unnecessary generated new lines in star by @courentin in #651
• fix: Actually suppress union_relations source_column_name when passing none by @kmclaugh in #661
• Make mutually_exclusive_ranges' test deterministic by adding upper_bound_column to order by clause by @sfc-gh-ancoleman in #660
• update union_relations to use core string literal macro by @dave-connors-3 in #665
• Add where clause example to get_column_values documentation by @arsenkhy in #623

New Contributors

• @courentin made their first contribution in #651
• @kmclaugh made their first contribution in #661
• @sfc-gh-ancoleman made their first contribution in #660
• @dave-connors-3 made their first contribution in #665
• @arsenkhy made their first contribution in #623

Full Changelog: 0.9.1...0.9.2

0.9.1

What's Changed

• Remove cross-db dbt_utils references by @clausherther in #650

0.9.0

What's Changed

Changed functionality

• 🚨 (Almost all) cross-db macros are now implemented in dbt Core instead of dbt-utils. A backwards-compatibility layer remains for now and will be removed in dbt utils 1.0 later this year. Completed by @dbeatty10 and @jtcohen6 in #597, #586 and #615
  • See #487 for further discussion on the backstory
  • If you are a package maintainer with a dependency on these macros, prepare for their removal by switching to {{ dbt.some_macro() }}. Refer to #package-ecosystem in the Community Slack for further assistance
• Feature: Add option to remove the source_column_name on the union_relations macro by @christineberger in #624

Fixes

• Use adapter.quote() instead of hardcoded BQ quoting for get_table_types_sql by @alla-bongard in #636

Documentation

• standardize yml indentation under the 'models:' line on the README by @leoebfolsom in #613
• Use MADR 3.0.0 for formatting decision records by @dbeatty10 in #614
• Docs cleanup by @dbeatty10 in #620
• Add not_accepted_values to README ToC by @david-beallor in #646

New Contributors

• @leoebfolsom made their first contribution in #613
• @christineberger made their first contribution in #624
• @alla-bongard made their first contribution in #636
• @david-beallor made their first contribution in #646

Full Changelog: 0.8.6...0.9.0

0.8.6

dbt-utils v0.8.6

New features

• New macros array_append and array_construct (#595)

Fixes

• Use * in star macro if no columns (for SQLFluff) (#605, #561)
• Only raise error within union_relations for build/run sub-commands (#606, #607)

Quality of life

• Add slugify to list of Jinja Helpers (#602)

Under the hood

• Fix make test for running integration tests locally (#344, #564, #591)

Contributors:

• @swanjson (#561)
• @dataders (#561)
• @epapineau (#583)
• @graciegoheen (#595)
• @jeremyyeo (#606)

0.8.5

dbt-utils v0.8.5

🚨 deduplicate (#542, #548)

The call signature of deduplicate has changed. The previous call signature is marked as deprecated and will be removed in the next minor version.

• The group_by argument is now deprecated and replaced by partition_by.
• The order_by argument is now required.
• The relation_alias argument has been removed as the macro now supports relation as a string directly. If you were using relation_alias to point to a CTE previously then you can now pass the alias directly to relation.

Before:

{% macro deduplicate(relation, group_by, order_by=none, relation_alias=none) -%}
...
{% endmacro %}

After:

{% macro deduplicate(relation, partition_by, order_by) -%}
...
{% endmacro %}

New features

• Add an optional where clause parameter to get_column_values() to filter values returned (#511, #583)
• Add where parameter to union_relations macro (#554)
• Add Postgres specific implementation of deduplicate() (#548)
• Add Snowflake specific implementation of deduplicate() (#543, #548)

Fixes

• Enable a negative part_number for split_part() (#557, #559)

Quality of life

• Documentation about listagg macro (#544, #560)
• Fix links to macro section in table of contents (#555)
• Use the ADR (Architectural Design Record) pattern for documenting significant decisions (#573)
• Contributing guide (#574)
• Add better documentation for deduplicate() (#542, #548)

Under the hood

• Fail integration tests appropriately (#540, #545)
• Upgrade CircleCI postgres convenience image (#584, #585)
• Run test for deduplicate (#579, #580)
• Reduce warnings when executing integration tests (#558, #581)
• Framework for functional testing using pytest (#588)

Contributors:

• @graciegoheen (#560)
• @judahrand (#548)
• @clausherther (#555)
• @LewisDavies (#554)
• @epapineau (#583)
• @b-per (#559)

0.8.4

Fixes get_relations_by_pattern in BigQuery (#536, #537)

0.8.3

dbt-utils v0.8.3

New features

• A macro for deduplicating data, deduplicate() (#335, #512)
• A cross-database implementation of listagg() (#530)
• A new macro to get the columns in a relation as a list, get_filtered_columns_in_relation(). This is similar to the star() macro, but creates a Jinja list instead of a comma-separated string. (#516)

Fixes

• get_column_values() once more raises an error when the model doesn't exist and there is no default provided (#531, #533)
• get_column_values() raises an error when used with an ephemeral model, instead of getting stuck in a compilation loop (#358, #518)
• BigQuery materialized views work correctly with get_relations_by_pattern() (#525)

Quality of life

• Updated references to 'schema test' in project file structure and documentation (#485, #521)
• date_trunc() and datediff() default macros now have whitespace control to assist with linting and readability #529
• star() no longer raises an error during SQLFluff linting (#506, #532)

Contributors:

• @judahrand (#512)
• @b-moynihan (#521)
• @sunriselong (#529)
• @jpmmcneill (#533)
• @KamranAMalik (#532)
• @graciegoheen (#530)
• @luisleon90 (#525)
• @epapineau (#518)
• @patkearns10 (#516)

0.8.2

Fixes

• Fixed an error with union_relations() when no include/exclude parameters were provided

0.8.1

dbt-utils v0.8.1

New features

• A cross-database implementation of any_value() (#497, #501)
• A cross-database implementation of bool_or() (#504)

Under the hood

• add dbt_packages/ to .gitignore #463
• Remove block comments to make date_spine() macro compatible with the Athena connector (#462)

Fixes

• type_timestamp macro now explicitly casts postgres and redshift warehouse timestamp data types as timestamp without time zone, to be consistent with Snowflake behaviour (timestamp_ntz).
• union_relations macro will now raise an exception if the use of include or exclude results in no columns (#473, #266).
• get_relations_by_pattern() works with foreign data wrappers on Postgres again. (#357, #476)
• star() will only alias columns if a prefix/suffix is provided, to allow the unmodified output to still be used in group by clauses etc. #468
• The sequential_values test is now compatible with quoted columns #479
• pivot() escapes values containing apostrophes #503

Contributors:

• grahamwetzler (#473)
• Aesthet (#476)
• Kamitenshi (#462)
• nickperrott (#468)
• jelstongreen (#468)
• armandduijn (#479)
• mdutoo (#503)