A text generation plugin using local LLMs or other text generation methods.
Builds on top of transformers by Hugging Face.
- Generate text from a prompt using a local LLM
- Access all LLMs from the Hugging Face Model Hub
Since we depend on transformers, please make sure that you fulfill their requirements.
You also need Pytorch installed on your system, please refer to the pytorch installation instructions.
- Install Nendo
pip install nendo-plugin-textgen
If you have a CUDA GPU you can install the following library for an additional speedup:
pip install flash-attn --no-build-isolation
Then set ATTN_IMPLEMENTATION=flash_attention_2 in your environment variables.
Take a look at a basic usage example below. For more detailed information, please refer to the documentation.
>>> from nendo import Nendo
>>> nd = Nendo(plugins=["nendo_plugin_textgen"])
>>> nd.plugins.textgen(prompt=["Tell me about your favorite song."])[0]Visit our docs to learn all about how to contribute to Nendo: Contributing
Nendo: MIT License