-
Notifications
You must be signed in to change notification settings - Fork 153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ADAP-945] [Bug] submission_method
from dbt profile not being applied to dbt Python models
#967
Comments
Thanks for reporting this @gbmarc1 It sounds like this didn't work for you: def model(dbt, session):
my_sql_model_df = dbt.source("safe_content_moderation", "safe_content_moderation")
final_df = my_sql_model_df
return final_df But this did work: def model(dbt, session):
dbt.config(
submission_method="cluster",
dataproc_cluster_name="ml-adhoc-dataproc-us-central1",
)
my_sql_model_df = dbt.source("safe_content_moderation", "safe_content_moderation")
final_df = my_sql_model_df
return final_df To help troubleshootDid you happen to try either of these as well? This could help nail down where the missing piece(s) might be. Configuring def model(dbt, session):
dbt.config(
submission_method="cluster",
)
my_sql_model_df = dbt.source("safe_content_moderation", "safe_content_moderation")
final_df = my_sql_model_df
return final_df Or configuring def model(dbt, session):
dbt.config(
dataproc_cluster_name="ml-adhoc-dataproc-us-central1",
)
my_sql_model_df = dbt.source("safe_content_moderation", "safe_content_moderation")
final_df = my_sql_model_df
return final_df |
Hello, Seems the profile's submission_method get ignored.
|
submission_method
from dbt profile not being applied to dbt Python models
Thanks @gbmarc1 -- that gives us the info we need 👍 Acceptance criteriaAs noted in the original issue, dbt should use the
ml:
target: dev
outputs:
dev: &dev_config
type: bigquery
dataset: "{{ env_var('USER') }}"
project: shopify-ml-adhoc
priority: interactive
method: oauth
location: US
job_execution_timeout_seconds: 600
job_retries: 1
threads: 2
submission_method: cluster
dataproc_region: us-central1
gcs_bucket: ml-adhoc-dataproc-jobs
dataproc_cluster_name: ml-adhoc-dataproc-us-central1
def model(dbt, session):
dbt.config(
dataproc_cluster_name="ml-adhoc-dataproc-us-central1",
)
my_sql_model_df = dbt.source("safe_content_moderation", "safe_content_moderation")
final_df = my_sql_model_df
return final_df Relevant code |
Is this a new bug in dbt-bigquery?
Current Behavior
I have the following profile. I want a job to be created in the provided cluster name but it always end up as a serverless batch.
This is the model. If I uncomment the dbt.config it works properly. But I want this config in the profile not in the model itself.
Expected Behavior
The profile config is respected and the job is executed in the cluster.
Steps To Reproduce
dbt run
Relevant log output
Environment
Additional Context
No response
The text was updated successfully, but these errors were encountered: