Panlingo

Overview

Welcome to the Panlingo repository! 🚀

This project presents a comprehensive collection of language identification libraries for .NET. Its primary purpose is to bring popular language identification models to the .NET ecosystem, allowing developers to seamlessly integrate language detection functionality into their applications.

Libraries

Library	Nuget Release
Panlingo.LanguageIdentification.CLD2
Panlingo.LanguageIdentification.CLD3
Panlingo.LanguageIdentification.FastText
Panlingo.LanguageIdentification.Whatlang
Panlingo.LanguageIdentification.MediaPipe
Panlingo.LanguageIdentification.Lingua
Panlingo.LanguageCode

Contents

1. Models
   1. CLD2
   2. CLD3
   3. FastText
   4. Whatlang
   5. MediaPipe
   6. Lingua
2. Features
3. Platform Support
4. Key Concerns
5. TODO

Models

CLD2

• Wrapper docs: Documentation
• Original source code: CLD2 Repository

CLD3

• Wrapper docs: Documentation
• Original source code: CLD3 Repository

FastText

• Wrapper docs: Documentation
• Original source code: FastText Repository

Whatlang

• Wrapper docs: Documentation
• Original source code: Whatlang Repository

MediaPipe

• Wrapper docs: Documentation
• Original source code: MediaPipe Repository

Lingua

• Wrapper docs: Documentation
• Original source code: MediaPipe Repository

Key concerns

• Zero-dependency development.
• The original code of libraries (CLD2, CLD3, FastText, Whatlang) is used as submodules without additional modifications or improvements. Third-party code is not included into this repository.
• Preserve the original library behavior without breaking changes.

Features

Feature	CLD2	CLD3	FastText*	Whatlang	MediaPipe**	Lingua
Single language prediction	Yes	Yes	Yes	Yes	Yes	Yes
Multi language prediction	Yes	Yes	Yes	No	Yes	Yes
Supported languages	80	107	176 or 217	69	110	75
Unknown language detection	Yes	Yes	No	No	Yes	No
Algorithm	quadgrams	neural network	neural network	trigrams	neural network	trigrams
Script detection	No	No	Yes (only lid218e)	Yes	No	No

* When using these models: lid176, lid218e

** When using MediaPipe Language Detector

Platform support

Model	Linux	Windows	macOS	Blazor WASM
CLD2	✅	✅	✅	❌
CLD3	✅	✅	🚧	❌
FastText	✅	✅	✅	❌
Whatlang	✅	✅	✅	❌
MediaPipe	✅	❌	❌	❌
Lingua	✅	✅	✅*	❌

✅ — Full support | ❌ — No support | 🚧 — Under research

* ARM CPU only

TODO

• Research support for other platforms (Windows, macOS).
• Add more unit tests.
• Implement more native methods (FastText).
• Self-contained models (FastText + MediaPipe).
• Remove protobuf dependency (CLD3).

---

Feel free to open issues or contribute to the repository. Together, let's enhance the .NET language identification capabilities! 🌐

---

Happy hacking! 👩‍💻👨‍💻