Byte-Pair Encoding tokenizer for training large language models on huge datasets. I don't know C yet, so most of the code comes from AI :D
I hope to learn C by rewriting it and making changes, fixes etc
-
Notifications
You must be signed in to change notification settings - Fork 0
High performance Byte-Pair Encoding tokenizer for large language models
License
jmaczan/bpe.c
About
High performance Byte-Pair Encoding tokenizer for large language models
Topics
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published