error: linker command failed with exit code 1 #2033

Runtrons · 2023-09-29T21:06:13Z

Describe the bug
While attempting to install Wenet Python bindings on macOS, I encountered a linker error indicating "Undefined symbols for architecture x86_64."

Error:

wenet::PostProcessor::InitITNResource(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&) in libpost_processor.a(post_processor.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [api/libwenet_api.dylib] Error 1
make[2]: *** [api/CMakeFiles/wenet_api.dir/all] Error 2
make[1]: *** [CMakeFiles/_wenet.dir/rule] Error 2
make: *** [_wenet] Error 2

To Reproduce
(I had experienced an error before this that got resolved, that is why the steps look a little different.)
Steps to reproduce the behavior:

Clone the Wenet repository via git clone https://github.com/wenet-e2e/wenet.git
Navigate to the runtime/libtorch directory. # To fix bug that was resolved here Setup.py fatal error: 'processor/wetext_processor.h' file not found #2032
Run mkdir build && cd build && cmake -DGRAPH_TOOLS=ON .. && cmake --build . # To fix bug # To fix bug that was resolved here Setup.py fatal error: 'processor/wetext_processor.h' file not found #2032
Navigate to wenet/runtime/binding/python # To fix bug # To fix bug that was resolved here Setup.py fatal error: 'processor/wetext_processor.h' file not found #2032
Run python setup.py install

linker error related to "Undefined symbols for architecture x86_64."

Expected behavior
I expected the Python bindings for Wenet to install successfully without any linker errors.

OS: macOS
Python 3.8
The error appears to be a linker issue related to missing or misconfigured libraries. I was following the Wenet Python binding installation steps as outlined in the GitHub repository.

Runtrons · 2023-10-06T22:14:53Z

Still haven't been able to figure this out, can anyone help me?

xingchensong · 2023-10-10T14:42:58Z

sorry for late response. we are on vacation last week~. could u try #2042

xingchensong · 2023-10-10T14:53:04Z

I can build from source successfully using that fix

Runtrons · 2023-10-10T23:28:31Z

Thanks @xingchensong for your response! I tried it and it worked! Now when I try to run a test script found in the documentation.

import sys
import wave
import wenetruntime as wenet

test_wav = sys.argv[1]

with wave.open(test_wav, 'rb') as fin:
    assert fin.getnchannels() == 1
    wav = fin.readframes(fin.getnframes())

decoder = wenet.Decoder(lang='chs', streaming=True)
# We suppose the wav is 16k, 16bits, and decode every 0.5 seconds
interval = int(0.5 * 16000) * 2
for i in range(0, len(wav), interval):
    last = False if i + interval < len(wav) else True
    chunk_wav = wav[i: min(i + interval, len(wav))]
    ans = decoder.decode(chunk_wav, last)
    print(ans)

When I attempt to run the above script from the documentation, I'm encountering an ImportError related to the _wenet module. The error suggests there might be an issue with the expected symbols between the libwenet_api.dylib and libtorch_cpu.dylib.

Here's the error for reference:

/Users/ronangrant/anaconda3/envs/wenet_fix_3.8/bin/python /Users/ronangrant/Safe/wenet_fix_3.8/wenet_fix.py 
Traceback (most recent call last):
  File "/Users/ronangrant/Safe/wenet_fix_3.8/wenet_fix.py", line 3, in <module>
    import wenetruntime as wenet
  File "/Users/ronangrant/anaconda3/envs/wenet_fix_3.8/lib/python3.8/site-packages/wenetruntime-1.14.0-py3.8-macosx-10.9-x86_64.egg/wenetruntime/__init__.py", line 1, in <module>
    from .decoder import Decoder  # noqa
  File "/Users/ronangrant/anaconda3/envs/wenet_fix_3.8/lib/python3.8/site-packages/wenetruntime-1.14.0-py3.8-macosx-10.9-x86_64.egg/wenetruntime/decoder.py", line 23, in <module>
    import _wenet
ImportError: dlopen(/Users/ronangrant/anaconda3/envs/wenet_fix_3.8/lib/python3.8/site-packages/wenetruntime-1.14.0-py3.8-macosx-10.9-x86_64.egg/_wenet.cpython-38-darwin.so, 0x0002): Symbol not found: __ZN2at4_ops10select_int4callERKNS_6TensorExx
  Referenced from: <2E55E5A7-2892-3CC8-A309-41F225CE39F3> /Users/ronangrant/Safe/wenet_fix_3.8/wenet/runtime/binding/python/build/temp.macosx-10.9-x86_64-cpython-38/api/libwenet_api.dylib
  Expected in:     <F8D5F274-4B4B-3A83-8226-5DF5A38267D6> /Users/ronangrant/anaconda3/envs/wenet_fix_3.8/lib/python3.8/site-packages/torch-2.1.0-py3.8-macosx-10.9-x86_64.egg/torch/lib/libtorch_cpu.dylib

I've tried this on both Python 3.8 and 3.10, and I consistently get the same error. Based on the error message, it appears to be related to a discrepancy in the expected symbols between the provided library and the Torch library.

Can you advise on the specific versions of the dependencies (especially PyTorch) you've used to successfully build from source? Additionally, any other configurations or environment setups that might be crucial would be greatly appreciated.

Thank you immensely for your continued support and guidance. I can't tell you how much I appreciate your help

Here is a list of my dependencies by running pip list:

decorator 5.1.1
gmpy2 2.1.2
joblib 1.3.2
lazy_loader 0.3
librosa 0.10.1
mpmath 1.3.0
msgpack 1.0.7
numba 0.58.0
numpy 1.24.4
pip 23.2.1
pooch 1.7.0
scikit-learn 1.3.1
setuptools 68.2.2
soundfile 0.12.1
soxr 0.3.7
sympy 1.12
torch 2.1.0
typing_extensions 4.8.0
wenetruntime 1.14.0
wheel 0.41.2

Also, for future reference should I open a new error or continue the thread here? Thanks so much.

xingchensong · 2023-10-11T01:14:32Z

I think the current problem is relevant to the torch version, not itn (OR post_processor), @robin1001 any ideas?

Runtrons · 2023-10-13T14:41:16Z

@xingchensong really what I am looking for is being able to stream. I do not need to use python but would like to. Is there any other method you think wouldn't give me this torch error? Should I just try different torch versions until it works? Thanks for your time an help!

xingchensong · 2023-10-14T02:29:00Z

@Mddct could u do us a favor😁

xingchensong · 2023-10-16T02:25:22Z

could u try to add import torch to your script

Runtrons · 2023-10-16T03:08:41Z

Thanks for your help! I ran:

import sys
import wave
import wenetruntime as wenet
import torch

test_wav = sys.argv[1]

with wave.open(test_wav, 'rb') as fin:
    assert fin.getnchannels() == 1
    wav = fin.readframes(fin.getnframes())

decoder = wenet.Decoder(lang='en', streaming=True)
# We suppose the wav is 16k, 16bits, and decode every 0.5 seconds
interval = int(0.5 * 16000) * 2
for i in range(0, len(wav), interval):
    last = False if i + interval < len(wav) else True
    chunk_wav = wav[i: min(i + interval, len(wav))]
    ans = decoder.decode(chunk_wav, last)
    print(ans)

And still got this error:


/Users/ronangrant/anaconda3/envs/wenet_fix_3.8/bin/python /Users/ronangrant/Safe/wenet_fix_3.8/wenet_fix.py 
Traceback (most recent call last):
  File "/Users/ronangrant/Safe/wenet_fix_3.8/wenet_fix.py", line 3, in <module>
    import wenetruntime as wenet
  File "/Users/ronangrant/anaconda3/envs/wenet_fix_3.8/lib/python3.8/site-packages/wenetruntime-1.14.0-py3.8-macosx-10.9-x86_64.egg/wenetruntime/__init__.py", line 1, in <module>
    from .decoder import Decoder  # noqa
  File "/Users/ronangrant/anaconda3/envs/wenet_fix_3.8/lib/python3.8/site-packages/wenetruntime-1.14.0-py3.8-macosx-10.9-x86_64.egg/wenetruntime/decoder.py", line 23, in <module>
    import _wenet
ImportError: dlopen(/Users/ronangrant/anaconda3/envs/wenet_fix_3.8/lib/python3.8/site-packages/wenetruntime-1.14.0-py3.8-macosx-10.9-x86_64.egg/_wenet.cpython-38-darwin.so, 0x0002): Symbol not found: __ZN2at4_ops10select_int4callERKNS_6TensorExx
  Referenced from: <2E55E5A7-2892-3CC8-A309-41F225CE39F3> /Users/ronangrant/Safe/wenet_fix_3.8/wenet/runtime/binding/python/build/temp.macosx-10.9-x86_64-cpython-38/api/libwenet_api.dylib
  Expected in:     <F8D5F274-4B4B-3A83-8226-5DF5A38267D6> /Users/ronangrant/anaconda3/envs/wenet_fix_3.8/lib/python3.8/site-packages/torch-2.1.0-py3.8-macosx-10.9-x86_64.egg/torch/lib/libtorch_cpu.dylib

Process finished with exit code 1

xingchensong · 2023-10-16T06:19:14Z

How about downgrade torch to 1.13.0

Runtrons · 2023-10-16T22:47:11Z

Thanks @xingchensong ! That worked! I can now stream!

I edited the code to time the seconds it takes in between chunks and it is pretty slow. I have verified that the wav file is 16k, 16 bits.

On CPU, is this normal? I was expecting it to be near the speed from the trimtail paper. Is there anything I can do to speed it up? For what I am doing every millisecond matters.

Time taken for this chunk: 1.3440 seconds { "nbest" : [{ "sentence" : "hello" }], "type" : "partial_result" } Time taken for this chunk: 5.5059 seconds { "nbest" : [{ "sentence" : "hello this" }], "type" : "partial_result" } Time taken for this chunk: 0.0009 seconds { "nbest" : [{ "sentence" : "hello this" }], "type" : "partial_result" } Time taken for this chunk: 7.8264 seconds { "nbest" : [{ "sentence" : "hello this is a test hopefully" }], "type" : "partial_result" } Time taken for this chunk: 0.0009 seconds { "nbest" : [{ "sentence" : "hello this is a test hopefully" }], "type" : "partial_result" } Time taken for this chunk: 5.8348 seconds { "nbest" : [{ "sentence" : "hello this is a test hopefully this works i" }], "type" : "partial_result" }

xingchensong · 2023-10-17T00:53:46Z

speed is highly corelate with hardware

Runtrons mentioned this issue Oct 9, 2023

Error when running python package #2040

Closed

xingchensong closed this as completed Oct 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error: linker command failed with exit code 1 #2033

error: linker command failed with exit code 1 #2033

Runtrons commented Sep 29, 2023 •

edited

Loading

Runtrons commented Oct 6, 2023

xingchensong commented Oct 10, 2023

xingchensong commented Oct 10, 2023

Runtrons commented Oct 10, 2023

xingchensong commented Oct 11, 2023

Runtrons commented Oct 13, 2023

xingchensong commented Oct 14, 2023

xingchensong commented Oct 16, 2023

Runtrons commented Oct 16, 2023

xingchensong commented Oct 16, 2023

Runtrons commented Oct 16, 2023

xingchensong commented Oct 17, 2023

error: linker command failed with exit code 1 #2033

error: linker command failed with exit code 1 #2033

Comments

Runtrons commented Sep 29, 2023 • edited Loading

Runtrons commented Oct 6, 2023

xingchensong commented Oct 10, 2023

xingchensong commented Oct 10, 2023

Runtrons commented Oct 10, 2023

xingchensong commented Oct 11, 2023

Runtrons commented Oct 13, 2023

xingchensong commented Oct 14, 2023

xingchensong commented Oct 16, 2023

Runtrons commented Oct 16, 2023

xingchensong commented Oct 16, 2023

Runtrons commented Oct 16, 2023

xingchensong commented Oct 17, 2023

Runtrons commented Sep 29, 2023 •

edited

Loading