[Bug] issue with PyYAML when installing dbt-snowflake

[adanin0]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I am trying to install dbt-snowflake with homebrew on Mac OS, and the install always stops when installing PyYAML. I have tried with versions 1.3.2, 1.4.3 and 1.5.2 and they all give the same outcome.

Expected Behavior

install dbt-snowflake

Steps To Reproduce

1. Mac OX 13.4.1 Ventura
2. brew install dbt-snowflake
3. see error below

Relevant log output

==> /usr/local/Cellar/dbt-snowflake@1.3.2/1.3.2/libexec/bin/pip install -v --no-deps --no-binary :all: --use-feature=no-binary-enable-wheel-cache --ignore-installed /private/tmp/dbt-snowflakeA1.3.2--PyYAML-20230719-4512-e6x4zw/PyYAML-6.0
Last 15 lines from /Users/*********/Library/Logs/Homebrew/dbt-snowflake@1.3.2/48.pip:
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/local/Cellar/dbt-snowflake@1.3.2/1.3.2/libexec/bin/python3.9 /usr/local/Cellar/dbt-snowflake@1.3.2/1.3.2/libexec/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /private/tmp/tmpr4k5xszi
  cwd: /private/tmp/dbt-snowflakeA1.3.2--PyYAML-20230719-4512-e6x4zw/PyYAML-6.0
  Getting requirements to build wheel: finished with status 'error'
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip available: 22.3.1 -> 23.2
[notice] To update, run: python3.9 -m pip install --upgrade pip

Environment

- OS: Mac OS 13.4.1 Ventura
- Python:Python 3.9.6
- dbt: not installed

What Homebrew formula are you trying to install?

snowflake

Additional Context

No response

[ksbohon]

+1 for dbt-bigquery

[dhruv-marwa]

pyYAML 6.0.0 can't build because Cython 3.0 recently had an update. The solution that should work is to update the formula to pyYAML 6.0.1. Below is the correct resource section - I will attempt a pull request but in the meantime I am sharing a temporary solution.

This is applies to the dbt-snowflake and dbt-redshift formula and possibly some others.

resource "PyYAML" do
    url "https://files.pythonhosted.org/packages/cd/e5/af35f7ea75cf72f2cd079c95ee16797de7cd71f29ea7c68ae5ce7be1eda0/PyYAML-6.0.1.tar.gz"
    sha256 "bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43"
  end 

[carlkibler]

I worked with Dhruv on the above fix and want to give a bit more details while we put together a PR. The dbt homebrew formulas on your machine are probably at /opt/homebrew/Library/Taps/dbt-labs/homebrew-dbt/Formula/. Within that you can edit the affected formulas like dbt-redshift.rb, dbt-snowflake.rb, and dbt-bigquery.rb.

Replace the resource PyYAML section with his snippet above and you should be able to brew install dbt-redshift, for example. When a real fix comes out you can revert to default and let homebrew update the files normally.

Hope this helps. The PyYAML 6.0.0 is causing a lot of problems in the Python world right now since so many packages rely on it.

If someone beats us to a PR - great! Neither of us are familiar with the versioning for this repo's formulas and it will take us a bit to find time to do it right.

[dataders]

so sorry to hear you're experiencing this issue everyone. It is incredible to see the interconnectedness of the Python package ecosystem via all the linked issues that can be seen in yaml/pyyaml#724 (most importantly cython/cython#4568).

That said, what I feel exacerbates this issue is brew itself which insists on rebuilding all packages from source and ignoring pre-built wheels. The command brew is invoking (below) is the default behavior.

Library/Homebrew/language/python.rb#L302-L311

pip install -v --no-deps --no-binary :all: \
    --use-feature=no-binary-enable-wheel-cache \
    --ignore-installed /private/tmp/dbt-snowflakeA1.3.2--PyYAML-20230719-4512-e6x4zw/PyYAML-6.0

@dhruv-marwa @carlkibler we'd very much appreciate a pull request if you all are willing the current proposal seems to me to be a monkeypatch, and not a permanent solution. I'd be hard-pressed to make a temporary change given that everyone is unblocked by invoking pip install directly in the interim. Happy to be wrong here, just let me know!

other miscellaneous reasons why brew has been challenging

Here's a PR list for all the times we've had to manually intervene to fix brew formulas. Perhaps this will become a discussion on dbt-core at some point, but for now let me document some of internally-enumerated challenges.

• we often see issues with brew that either have nothing to do with us or have to do with changes to brew itself
• we don't see nearly the same level of adoption of brew install as we do with pip install
• brew is Mac OS only, so these efforts don't serve Windows or Linux users
• we don't have any resident brew experts which makes solving problems more difficult
• automating formula updates as part of releases is challenging so it slows down our overall release process

a brighter future

Internally at dbt Labs, we've been developing an alternative abstraction to brew that is more deterministic and reliable than brew: dbt-core-bundles. It is still currently in beta, but we plan to offer a stronger "General Availability" guarantee within the next few months. There is a future state envisioned for brew install dbt, though all I can say on that front is "stay tuned!".

[jschulz2]

Thanks @carlkibler @dhruv-marwa and @dataders ! Have you heard anywhere else if anyone has able to get this fix to solve the issue? I updated the dbt-snowflake.rb and dbt-snowflake@1.5.2.rb and am still running into the same error.

[klayhb]

We have a team member who can't install dbt with brew due to this issue. We tried the workaround (pip) but then ran into ANOTHER issue: snowflakedb/snowflake-connector-python#1626

Can anyone please advise...

[dataders]

@klayhb assumably, you're on Snowflake, right? can you try this?

pip install dbt-snowflake~=1.5.0 snowflake-connector-python --only-binary :all:

[colin-rogers-dbt]

This should be fixed now

[johnAirRobe]

When I run brew install dbt-snowflake I get no available formula with the name "dbt-snowflake".

[colin-rogers-dbt]

@johnAirRobe thanks for surfacing, looks like this is a new issue, see: #318

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.