bitarray dependency is unfortunate

[nmigen-issue-migration]

Issue by emilazy
Monday Sep 30, 2019 at 00:22 GMT
Originally opened as m-labs/nmigen#242

---

It's currently broken in PyPy and I feel like I've managed to poke it into memory unsafe behaviour from the CPython REPL before. Only pysim uses it so I imagine it could pretty easily be replaced with a combination of Python's bigints and lists.

[nmigen-issue-migration]

Comment by emilazy
Monday Sep 30, 2019 at 00:29 GMT

---

Maybe Glasgow's glasgow.support.bits should be upstreamed and used instead?

[nmigen-issue-migration]

Comment by whitequark
Monday Sep 30, 2019 at 08:56 GMT

---

I'm not going to touch pysim until 0.1, and there's a complete rewrite planned for 0.2 that will remove the dependency on bitarray.

Note that Glasgow depends on both anyway. glasgow.support.bits is immutable.

[nmigen-issue-migration]

Comment by emilazy
Monday Sep 30, 2019 at 12:16 GMT

---

Glasgow only uses bitarray in a few applets, so I figured it could probably be migrated away.

[nmigen-issue-migration]

Comment by whitequark
Monday Sep 30, 2019 at 12:25 GMT

---

I already tried to do that during migration to bits. I eventually decided to keep bitarray for mutable cases, though it might be possible to use a list of chunks anyway, assuming someone benchmarks it and figures out a good chunk size.

[nmigen-issue-migration]

Comment by emilazy
Monday Sep 30, 2019 at 12:26 GMT

---

Fair enough; but in any case, using Glasgow on PyPy is a lot more niche than using nmigen on PyPy, and it has a lot more dependencies regardless.

[nmigen-issue-migration]

Comment by xobs
Tuesday Oct 01, 2019 at 03:27 GMT

---

I thought I'd take a look at nmigen (migrating from migen), and I just ran into this issue.

❯ python .\haddecks.py --document-only
lxbuildenv: v2019.8.19.1 (run .\haddecks.py --lx-help for help)                                                                                                                                                            Traceback (most recent call last):
  File ".\haddecks.py", line 17, in <module>
    from litex.build.lattice import LatticePlatform
  File "D:\Code\had\haddecks\deps\litex\litex\__init__.py", line 4, in <module>
    from litex.soc.interconnect import packet
  File "D:\Code\had\haddecks\deps\litex\litex\soc\interconnect\packet.py", line 4, in <module>
    from nmigen.compat import *
  File "D:\Code\had\haddecks\deps\nmigen\nmigen\compat\__init__.py", line 8, in <module>
    from .sim import *
  File "D:\Code\had\haddecks\deps\nmigen\nmigen\compat\sim\__init__.py", line 5, in <module>
    from ...back.pysim import *
  File "D:\Code\had\haddecks\deps\nmigen\nmigen\back\pysim.py", line 5, in <module>
    from bitarray import bitarray
ModuleNotFoundError: No module named 'bitarray'
❯

I'm not actually using the simulator, so this dependency is definitely unfortunate.

[nmigen-issue-migration]

Comment by whitequark
Tuesday Oct 01, 2019 at 17:20 GMT

---

@xobs Have you actually installed nMigen's dependencies? Because there is no goal to reduce the amount of dependencies to zero.

[nmigen-issue-migration]

Comment by xobs
Wednesday Oct 02, 2019 at 01:08 GMT

---

@whitequark I've cloned the repository next to the existing migen repository, and changed my references to point there.

I was hoping nmigen would be a drop-in replacement for migen, however the addition of an installation step means that I will stick to migen for now.

[nmigen-issue-migration]

Comment by mithro
Wednesday Oct 02, 2019 at 01:10 GMT

---

@xobs - You install migen, why wouldn't you install nmigen?

[nmigen-issue-migration]

Comment by whitequark
Wednesday Oct 02, 2019 at 01:13 GMT

---

@xobs A user of nMigen is expected to be familiar with the absolute basics of Python package management. That's like saying "Yosys has dependencies, so I will stick to Vivado for now". I mean, sure, you can do that, but that's not a problem with Yosys or nMigen, it is a problem with your lack of desire to learn the fundamentals of the programming toolkit you are using.

[nmigen-issue-migration]

Comment by xobs
Wednesday Oct 02, 2019 at 01:24 GMT

---

@mithro I don't install migen, I check it out. I don't like using pip because it always breaks in spectacular ways. It's difficult to run a workshop offline, and isn't included in all of the official python installations. Running pip install has about a 50% success rate in my experience, so I very much like to avoid using it.

For example:

C:\Users\smcro>python -m pip install git+https://github.com/m-labs/nmigen.git
Collecting git+https://github.com/m-labs/nmigen.git
  Cloning https://github.com/m-labs/nmigen.git to c:\users\smcro\appdata\local\temp\pip-y_btvmjq-build
Unknown requires Python '~=3.6' but the running Python is 3.5.2
You are using pip version 9.0.1, however version 19.2.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

C:\Users\smcro>

With migen I just add it to my PYTHONPATH variable and go. No need to mess with installing anything. Plus it's much easier to make contributions, since it's just in a subdirectory.

[nmigen-issue-migration]

Comment by cr1901
Wednesday Oct 02, 2019 at 01:26 GMT

---

@xobs nMigen requires 3.6 and above, though I'm acutely aware that upgrading Python and pip3 on Windoze can be a painful process.

[nmigen-issue-migration]

Comment by whitequark
Wednesday Oct 02, 2019 at 01:29 GMT

---

@xobs In your example pip correctly tells you that nMigen requires Python 3.6, which is written in its README file:

nMigen is designed for Python 3.6 and newer. nMigen's Verilog backend requires Yosys 0.9 or a newer version.

[nmigen-issue-migration]

Comment by xobs
Wednesday Oct 02, 2019 at 01:29 GMT

---

I also have the official 3.7.3 installed:

C:\Users\smcro>python --version
Python 3.7.3

C:\Users\smcro>python -m ensurepip --default-pip
D:\Software\ecp5-toolchain-windows-v1.5\bin\python.exe: No module named ensurepip

C:\Users\smcro>python -m pip install git+https://github.com/m-labs/nmigen.git
D:\Software\ecp5-toolchain-windows-v1.5\bin\python.exe: No module named pip

C:\Users\smcro>

[nmigen-issue-migration]

Comment by whitequark
Wednesday Oct 02, 2019 at 01:31 GMT

---

OK, so what you're actually asking for is reliable installation instructions for Windows. I agree that this is something that is currently not provided but should be (especially given that you will also need Yosys to do anything).

[nmigen-issue-migration]

Comment by plaes
Wednesday Oct 02, 2019 at 14:03 GMT

---

I also have the official 3.7.3 installed:

C:\Users\smcro>python --version
Python 3.7.3

C:\Users\smcro>python -m ensurepip --default-pip
D:\Software\ecp5-toolchain-windows-v1.5\bin\python.exe: No module named ensurepip

C:\Users\smcro>python -m pip install git+https://github.com/m-labs/nmigen.git
D:\Software\ecp5-toolchain-windows-v1.5\bin\python.exe: No module named pip

C:\Users\smcro>

Since 3.3 Python has venv module which allows one to have different application-specific package installs.

[nmigen-issue-migration]

Comment by programmerjake
Wednesday Oct 02, 2019 at 15:59 GMT

---

maybe the windows installation issue should be in a separate issue

[nmigen-issue-migration]

Comment by whitequark
Thursday Oct 03, 2019 at 09:54 GMT

---

@xobs I've opened m-labs/nmigen#244 to track the Windows installation problems.

[nmigen-issue-migration]

Comment by mithro
Thursday Oct 03, 2019 at 22:46 GMT

---

Just FYI - I've worked with @xobs a bit to give him some more Python background knowledge and some probably terrible advice ;-)

[nmigen-issue-migration]

Comment by whitequark
Thursday Nov 28, 2019 at 22:11 GMT

---

@xobs FYI as of today, nMigen master is pure Python. You only need to install pyvcd and only if you use the simulator. (Actually, it could probably be made an optional dependency even for that case, though the utility of a simulator that can't write waveforms is marginal.)