Building applications with LLMs through composability, in Kotlin, Scala, ...

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

Develop DL models using Pytorch and Hugging Face

the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly

This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

Use custom tokenizers in spacy-transformers

Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels

Small library that provides functions to tokenize a string into an array of words with or without punctuation

Python script for manipulating the existing tokenizer.

A graphical user interface for the Elasticsearch Analyze API

Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.

Visualize some important concepts related to LLM architectures.

Multi-arch bindings for @huggingface/tokenizers.

🧰 The AutoTokenizer that TikToken always needed -- Load any tokenizer with TikToken now! ✨

[Unofficial] Simple .NET wrapper of HuggingFace Tokenizers library

Package to align tokens from different tokenizations.

Question and Answer web applicaiton using fine-tuned and pre-trained T5 models. Application runs on Streamlit.

Fine tuning pre-trained transformer models in TensorFlow and in PyTorch for question answering

Kingchop ⚔️ is a JavaScript English based library for tokenizing text (chopping text). It uses vast rules for tokenizing, and you can adjust them easily.