adding config option for table and incremental materializaions

[gregology]

resolves #
docs dbt-labs/docs.getdbt.com/#

Problem

BigQuery generates inefficient execution plans when doing comparisons on nested queries. For example;

{{
  config(
    materialized='table'
  )
}}

SELECT id
FROM `shopify-stg-dw`.`greg_sandbox`.`another_table`
WHERE created_at < (SELECT MAX(created_at) FROM `shopify-stg-dw`.`greg_sandbox`.`my_table`)

BigQuery also generates inefficient execution plans using a CTE approach. For example;

{{
  config(
    materialized='table'
  )
}}

WITH max_created_at AS (
  SELECT MAX(created_at) AS max_date
  FROM `shopify-stg-dw`.`greg_sandbox`.`my_table`
)
SELECT id
FROM `shopify-stg-dw`.`greg_sandbox`.`another_table`
WHERE created_at < (SELECT max_date FROM max_created_at)

BigQuery variables are unable to be used as they require declaration at the top of a script. The existing pre_hook options are not suitable as they run in a separate script.

So the current best practices to solve this in dbt is to use a run_query macro to set a dbt variable from a SQL query. This results in two separate SQL scripts being generated. For example;

{{
  config(
    materialized='table'
  )
}}

{%- set latest_my_table_data = run_query("SELECT MAX(created_at) FROM `shopify-stg-dw`.`greg_sandbox`.`my_table`")[0][0] -%}

SELECT id
FROM `shopify-stg-dw`.`greg_sandbox`.`another_table`
WHERE created_at < TIMESTAMP("{{ latest_my_table_data }}")

This approach executes these two separate scripts;

/* {"app": "dbt", "dbt_version": "1.7.13", "profile_name": "data_warehouse", "target_name": "dev", "node_id": "model.data_warehouse.bq_variables_example"} */

SELECT MAX(created_at) FROM `shopify-stg-dw`.`greg_sandbox`.`my_table`

/* {"app": "dbt", "dbt_version": "1.7.13", "profile_name": "data_warehouse", "target_name": "dev", "node_id": "model.data_warehouse.bq_variables_example"} */

CREATE OR REPLACE TABLE `shopify-stg-dw`.`greg_sandbox`.`bq_variables_example` OPTIONS( 
  expiration_timestamp=TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 168 hour)
) AS (
  SELECT id
  FROM `shopify-stg-dw`.`greg_sandbox`.`another_table`
  WHERE created_at < TIMESTAMP("2024-07-31 18:45:08.176533 UTC")
);

The current solution has a few drawbacks;

• It requires an additional separate SQL script to be run for each defined variable.
• There are limitations on the size of the dbt variable because of BigQuery's script limits.
• The type of variable is not necessarily maintained, for instance in the example above the query output is a TIMESTAMP which is converted to a string in the dbt variable and then CAST back to a TIMESTAMP in the subsequent query.
• All of these issues are exabated when a model requires multiple such variables.

Solution

This macro and materialization update enables running SQL code at the beginning of a BigQuery SQL script. Which allows dbt models to DECLARE and SET BigQuery variables before the CREATE OR REPLACE TABLE statement.

{{
  config(
    materialized='table',
    script_headers=[
      "DECLARE latest_my_table_data TIMESTAMP;"
      "SET latest_my_table_data = (SELECT MAX(created_at) FROM `data-warehouse`.`greg_sandbox`.`my_table`);"
    ]
  )
}}

SELECT id
FROM `shopify-stg-dw`.`greg_sandbox`.`another_table`
WHERE created_at < latest_my_table_data

Using this approach, what would have required two separate scripts to produce an efficient execution plan in BigQuery can now be achieved in a single script.

/* {"app": "dbt", "dbt_version": "1.7.13", "profile_name": "data_warehouse", "target_name": "dev", "node_id": "model.data_warehouse.bq_variables_example"} */

DECLARE latest_my_table_data TIMESTAMP;
SET latest_my_table_data = (SELECT MAX(created_at) FROM `data-warehouse`.`greg_sandbox`.`my_table`);

CREATE OR REPLACE TABLE `data-warehouse`.`sandbox`.`bq_variables_example`
OPTIONS(
  expiration_timestamp=TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 168 hour)
) AS (
  SELECT id
  FROM `shopify-stg-dw`.`greg_sandbox`.`another_table`
  WHERE created_at < latest_my_table_data
);

Alternative approaches considered;

Having a bq_varables option that accepted a list of dicts with information about the variable type and defaults. I decided against this as there may be other use cases for running SQL within a single script at the beginning of a model.

Not making these changes and only using dbt variables instead of BigQuery variables. I decided against this as setting dbt variables from a SQL query currently requires executing the run_query() macro which generates a separate SQL script. As mentioned above, this can result in a single model creating dozens of separate SQL scripts to run.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX