Skip to content

Latest commit

 

History

History
54 lines (40 loc) · 1.9 KB

README.md

File metadata and controls

54 lines (40 loc) · 1.9 KB
Raw

De-Tokenize

Package PyPI version Author License made-with-python Python 3.7 Downloads

What is it?

De-Tokenize is a Python package that offers efficient and precise reconstruction of tokens into their original sentence structure.

Contributor

Littin Rajan

Main Features

  • Effortless de-tokenization process
  • Lightweight package architecture
  • Eliminates external dependencies for hassle-free usage
  • Empowers users with robust and adaptable functionality

Where to get it?

The source code is currently hosted on GitHub at: https://github.com/littinrajan/detokenize

Binary installers for the latest released version are available at the Python Package Index (PyPI) and on Conda.

# conda
conda install detokenize
# or PyPI
pip install detokenize

Usage

from detokenize.detokenizer import detokenize

sample_tokens = ['These', 'are', 'some', 'tokens', '.']
sentence = detokenize(sample_tokens)

License

MIT

Contributing to De-Tokenize

Warmly welcome all contributions, including bug reports, bug fixes, documentation enhancements, improvements, new features, and innovative ideas.