Skip to content

dbt-spark 1.3.0

Compare
Choose a tag to compare
@leahwicz leahwicz released this 12 Oct 13:22
· 17 commits to 1.3.latest since this release
94b9e7e

dbt-spark 1.3.0 - October 12, 2022

Features

  • Support python model through notebook, currently supported materializations are table and incremental (#417, #377)
  • Add changie to dbt-spark (#416, #418)
  • merge exclude columns for spark models (#5260, #390)
  • Array macros (#453, #454)
  • implement testing for type_boolean in spark (#470, #471)
  • Support job cluster in notebook submission method, remove requirement for user for python model submission (#444, #467)
  • Migrate dbt-utils current_timestamp macros into core + adapters (#483, #480)

Fixes

  • Pin pyodbc to version 4.0.32 to prevent overwriting libodbc.so and libltdl.so on Linux (#397, #398)
  • python incremental model tmp table using correct schema (#441, #445)
  • change to get_columns_in_relation to fix cache inconsistencies to fix cache issues in incremental models causing failure on on_schema_change (#447, #451)

Under the Hood

  • Support core incremental refactor (#4402, #394)
  • specify supported_languages for materialization that support python models (#437, #440)
  • Submit python model with Command API by default. Adjusted run name (#424, #442)
  • Better interface for python submission (#452, #452)
  • ignore mypy typing issues (#461, #462)
  • Enable Pandas and Pandas-on-Spark DataFrames for dbt python models (#468, #469)
  • Convert df to pyspark DataFrame if it is koalas before writing (#473, #474)

Dependency

  • Bump pyodbc from 4.0.32 to 4.0.34 (#417, #459)
  • Bump black from 22.3.0 to 22.8.0 (#417, #458)
  • Update click requirement from ~=8.0.4 to ~=8.1.3 (#417, #457)
  • Bump mypy from 0.950 to 0.971 (#417, #456)
  • Bump thrift-sasl from 0.4.1 to 0.4.3 (#417, #455)

Contributors