v3.0.0

What's Changed

Add Options.OmitWhitespace

Add optional extension methods using UAX29.Extensions

Tokenizer.GetWords → Split.Words

Full Changelog: v2.2.0...v3.0.0

v2.2.0

Added a new Position property, to know where in the original string/stream the current token is. It’s a count of bytes (UTF-8) or chars (UTF-16).

What's Changed

• Add Position property by @clipperhouse in #23

Full Changelog: v2.1.0...v2.2.0

v2.1.0

Added a new Ranges property on the tokenizer, an enumerator of the token boundaries.

Several optimizations and cleaner code.

What's Changed

• Implement Ranges by @clipperhouse in #19
• Refactor & simplify by @clipperhouse in #21

Full Changelog: v2.0.3...v2.1.0

v2.0.3

Added ToList and ToArray methods, refactors and renames.

If you are using v1.x of this package, v2 has been renamed:

dotnet add package uax29.net → dotnet add package UAX29

using uax29 → using UAX29

We now use methods:

Tokenizer.Create(input) → Tokenizer.GetWords(input)

Tokenizer.Create(input, TokenType.Graphemes) → Tokenizer.GetGraphemes(input)

v1.1.0

Add support for Stream (of UTF-8 bytes) and TextReader / StreamReader of char.

1.0.0

First implementation. Unicode tests are passing and perf looks good.

Install via Nuget dotnet add package uax29.net --version 1.0.0