Tokenizer

This project does not have as purpose to be used to develop tokenizers. I just aim to develop this framework by myself to get a better understanding of how tokenizers work, by implementing in different ways, and comparing to performing ones, such as Tiktoken.
Explore the docs »

Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact

About The Project

Built With

	Backend	Frontend
Server
Libraries/Frameworks
Tokenizers

Getting Started

Installation

1. Clone the repo

   git clone git@github.com:art-test-stack/tokenizer.git

2. Run the Python server

• Create a virtual environment

  For example I use virtualenv:

virtualenv -p python 3.10 venv

• Install pip packages

pip install -r requirements.txt

• Run the server

uvicorn main:app --reload --host 0.0.0.0 --port 8000

3. Run the web app in another shell

• Install npm, for some doc look at npm getting started

• Run the app

npm run start

Roadmap

• [ ]

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE.txt for more information.

Contact

Arthur Testard - testardarthur@gmail.com

Project Link: https://github.com/art-test-stack/tokenizer

(back to top)