undefined symbol: cblas_drot

[mrgordon]

I believe we are hitting some dependency issues after upgrading to the newest scipy/numpy

Here is the installation of the newest packages: https://gist.github.com/beedub/43d10721887ab5668a0b

Then we get a runtime error for sklearn/utils/arrayfuncs.so: undefined symbol: cblas_drot: https://gist.github.com/beedub/bacf2f52e2a3fe1544e7

So I'm guessing this is a problem with the buildpacks installation of BLAS?

[branliu0]

Can you clarify which versions of scipy/numpy you were using before, what versions you're using now, and also the respective versions of sklearn?

I haven't investigated the installation of sklearn extensively -- so I wouldn't be surprised if there are issues.

I don't think there is some issue with the installation of BLAS; no one has mentioned it as an issue up to this point.

[beedub]

I think these were the most recent upgrades, but I'll confirm tmr that older ones aren't the issue / were working
scipy: 0.14.0 => 0.15.1
numpy 1.8.1 => 1.9.2

[mrgordon]

Yes I never observed the error on the earlier versions that @beedub mentions but we'll reverify as well

[mrgordon]

After further testing, we found that it's possible to install the newer versions if we first install the old versions and then upgrade

In other words, the earlier versions appear to leave artifacts on Heroku that enable the later versions to successfully build

[branliu0]

Hmm, that's interesting. When I have a chance I'll add precompiled version of sklearn to this project too.

[mrgordon]

FYI hit this issue again when trying to make a minor code change. I'm guessing somehow I ended up with a dyno that didn't have the pre-existing artifacts and now I am having issues getting "undefined symbol: cblas_drot" to go away again

[mrgordon]

Here is, I think, where it can't find BLAS:

remote: -----> Fetching set buildpack https://github.com/CrowdFlower/heroku-buildpack-scipy#testing... done
remote: -----> Python app detected
remote: -----> Noticed cffi. Bootstrapping libffi.
remote: -----> Detected numpy/scipy in requirements.txt.
remote: -----> Using cached binaries.
remote: -----> Existing numpy (1.9.2) package detected.
remote: -----> Existing scipy (0.15.1) package detected.
remote: -----> Installing dependencies with pip
remote:        Collecting scikit-learn==0.17.1 (from -r requirements.txt (line 33))
remote:          Downloading scikit-learn-0.17.1.tar.gz (7.9MB)
remote:            Partial import of sklearn during the build process.
remote:        Installing collected packages: scikit-learn
remote:          Found existing installation: scikit-learn 0.16.1
remote:            Uninstalling scikit-learn-0.16.1:
remote:              Successfully uninstalled scikit-learn-0.16.1
remote:          Running setup.py install for scikit-learn
remote:            Partial import of sklearn during the build process.
remote:            blas_opt_info:
remote:            blas_mkl_info:
remote:              libraries mkl,vml,guide not found in ['/app/.heroku/python/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
remote:              NOT AVAILABLE
remote:            openblas_info:
remote:              libraries openblas not found in ['/app/.heroku/python/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
remote:              NOT AVAILABLE
remote:            atlas_3_10_blas_threads_info:
remote:            Setting PTATLAS=ATLAS
remote:            Replacing _lib_names[0]=='tatlas' with 'atlas'
remote:            Setting PTATLAS=ATLAS
remote:            Replacing _lib_names[0]=='atlas' with 'atlas'

Note that the only change between your buildpack and the one I am using is that it sets the directory to projects/deep because its within a larger repo. I tried creating a fresh Heroku project with the code at the root of the project to no avail.

[branliu0]

It appears this is an issue with installing sklearn... Unfortunately this buildpack does not officially support sklearn and there are known issues with installing sklearn on this buildpack

[FreshDonut]

Seeing the same issue. Thought it was a pkl loading issue, but simply defining something like clf = LogisticRegression() crashes the app (Although locally it works correctly). Hmm...