tokenizer
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Here are 1,222 public repositories matching this topic...
Parser Building Toolkit for JavaScript
-
Updated
Dec 20, 2024 - TypeScript
Persian NLP Toolkit
-
Updated
Jul 16, 2024 - Python
Solves basic Russian NLP tasks, API for lower level Natasha projects
-
Updated
Oct 17, 2024 - Python
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
-
Updated
Apr 13, 2024 - Python
Self-contained Japanese Morphological Analyzer written in pure Go
-
Updated
Oct 24, 2024 - Go
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
-
Updated
May 16, 2023 - JavaScript
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
-
Updated
Dec 24, 2024 - Python
🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.
-
Updated
May 14, 2018 - Swift
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
-
Updated
Feb 27, 2024 - Python
专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference
-
Updated
Feb 3, 2021 - Java
Open Korean Text Processor - An Open-source Korean Text Processor
-
Updated
Mar 12, 2024 - Scala
The fast scanner generator for Java™ with full Unicode support
-
Updated
Sep 12, 2023 - Java
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
-
Updated
Jul 2, 2024 - Go
🌿 NodeJS PHP Parser - extract AST or tokens
-
Updated
Dec 5, 2024 - JavaScript
Python port of Moses tokenizer, truecaser and normalizer
-
Updated
May 26, 2024 - Python
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
-
Updated
Oct 29, 2023 - C
- Followers
- 10.6k followers
- Wikipedia
- Wikipedia