[CT-462] [Feature] .yaml and .yml extensions are interchangeable for profiles.yml, etc.

[DharamLietz-BO]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

If you provide the --profiles-dir flag and a valid directory with a file named profiles.yaml, dbt will fail to initialize.

Expected Behavior

dbt should look for both .yml and .yaml extensions

Steps To Reproduce

1. Create a new project
2. Name your profiles file profiles.yaml
3. Run dbt --profiles-dir /directory/with/profiles/yaml/file/in/it debug

Relevant log output

$ dbt --profiles-dir "/Users/dharam.lietz/bo/airflow-datawarehouse/.dbt" debug
18:34:45  Running with dbt=1.1.0-b1
dbt version: 1.1.0-b1
python version: 3.9.9
python path: /Users/dharam.lietz/.pyenv/versions/3.9.9/envs/airflow-dwh-3.9.9/bin/python3.9
os info: macOS-12.3-x86_64-i386-64bit
Using profiles.yml file at /Users/dharam.lietz/bo/airflow-datawarehouse/.dbt/profiles.yml
Using dbt_project.yml file at /Users/dharam.lietz/bo/airflow-datawarehouse/dbt/ecs_gl/dbt_project.yml

Configuration:
  profiles.yml file [ERROR not found]
  dbt_project.yml file [OK found and valid]

Required dependencies:
 - git [OK found]

1 check failed:
dbt looked for a profiles.yml file in /Users/dharam.lietz/bo/airflow-datawarehouse/.dbt/profiles.yml, but did
not find one. For more information on configuring your profile, consult the
documentation:

https://docs.getdbt.com/docs/configure-your-profile

Environment

- OS: macOS-12.3-x86_64-i386-64bit
- Python: 3.9.9
- dbt: 1.1.0-b1

What database are you using dbt with?

No response

Additional Context

No response

[gshank]

I think if we were going to do this, it would have to be for all config type yaml files, i.e. dbt_project.yml, profiles.yml, selectors.yml, packages.yml. We've lifted the restriction against .yaml files in project schema files. It would, oddly, be a bit harder to do for these other files because that code is sprinkled around the codebase.

[DharamLietz-BO]

I think it's the "file doesn't exist" error that threw me for a loop. If it had just said "use .yml, the .yaml extension is not supported", it would've been an easy fix.

Are there command line flags that reference dbt_project, selectors and packages config files that also accept a directory but only look for files with the .yml extension?

I imagined this being a very small change to update the code that looks for the profiles.yml in the directory provided to also look for the .yaml extension.

[DharamLietz-BO]

Something like this: main...DharamLietz-BO:patch-1

[jtcohen6]

@DharamLietz-BO This isn't something we're going to prioritize ourselves, but we'd welcome a PR from a community contributor who feels motivated :)

Here's what we think a good solution would look like:

1. New reusable method, read_yaml_file, in clients.system, with handling for both .yml + .yaml extensions
2. In all the places where we hard-code a yaml file name today ("profiles.yml", "dbt_project.yml", "selectors.yml", "packages.yml"), replace hard-coded file path with the new method clients.system.read_yaml_file
3. Add a test case for this (profiles.yaml, dbt_project.yaml, etc). We think this would just require the test case to reimplement the "project" fixtures used by pytest: profiles_yml, dbt_project_yml, packages_yml, etc.

Is that something you or another community member might be interested in working on?

[DharamLietz-BO]

Sure, I'll put together sometime this week probably. Thanks for providing a suggested approach 💯 that all makes sense

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[dbeatty10]

Issue #8738 prompted me to review this issue, and we're re-designating this as a handy feature request rather than a bug.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[morgan-dgk]

I'm interested in getting this done. I can see clients.system has been extracted to the dbt_common library. Should the suggested changes to clients.system be made there?

Had a poke around dbt_core and it seems like there is some variation in how reading project files happens. E.g. Some project file paths are defined as constants whereas profiles.yml is passed directly as a string to os.path.join in profiles.yml

edit: maybe yaml_helper would be a better place to implement this?