flair==0.10 requires torch and will not accept pre-installed ROCM pytorch

[bennmann]

Still coming back to project from time to time in hopes of getting it to work with llama 70B (1t/sec 4bit quantized locally).

While trying to get a local model to work with this repo, I found that requirement "flair==0.10" would not accept AMD ROCM pytorch even when invoking the appropriate --index-url flag and ROCM nightly URL in pip.

If you ever release a follow-up to your work I encourage you to implement a local model version, including dependencies which are not NVIDIA only.

[yangkevin2]

Hey, yeah, the state of the art is indeed moving pretty fast these days. We're actually intending to open-source a version of this project that has the same high-level structure but supports newer LMs (e.g., LLama2), and is also rewritten to be easier to work with and modify. Flair will no longer be required in that version (and in the current version, you can probably get away with turning off the part that does use flair, with only a small loss). We're hoping to have the new version out in the next couple of weeks.

[bennmann]

Thank you Kevin!!

If you would like an AMD ROCM tester or any help getting AMD consumer cards to work with the new launch, I am happy to help even just with random errors as well. I have a couple hundred hours struggling through various AMD dependency purgatories at this point with multiple LLMs and have a local machine for inference with 16GB VRAM.

[yangkevin2]

Sure, will let you know when it's released!

[yangkevin2]

(Sorry, this is taking a bit longer than expected to get approved, but I'm hoping the new version should be out soon.)

[yangkevin2]

The new code is up at https://github.com/facebookresearch/doc-storygen-v2 !