Cross-platform support (#16)

* CLD3: try to build for Windows

* Update run-build.sh

* Update LanguageIdentification.CLD3.Native.csproj

* Add Windows in library logic

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update toolchain-mingw.cmake

* CLD3: native naming changes

* CLD3: fix EXPORT macros

* CLD3: try to fix linking for Windows

* Revert "CLD3: fix EXPORT macros"

This reverts commit dbfa20e.

* Update binding.cc

* Revert "Update binding.cc"

This reverts commit 069230f.

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Cross-platform tests: skip if not supported

* Update LinguaDetectorBuilder.cs

* Update github-ci.yml

* Cross-platform tests: skip if not supported

* Update LinguaDetectorBuilder.cs

* 0.0.0.21 -> 0.0.0.22

* GitHub CI: add tests for MacOS

* CLD2: add support for Windows

* Update MainTests.cs

* FastText: try to build for Windows

* Update github-ci.yml

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update CMakeLists.txt

* FT: try to use wine

* Update setup-build.sh

* CI: try to build images

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Different platforms for different models

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* FT: fix build for Windows

* CLD2, CLD3, FT: enforce using UTF8

* Update CMakeLists.txt

* CI: remove "build-docker-images" stage

* CI: remove custom code for FT

* Update binding.cc

* Update binding.cc

* Update binding.cc

* Update github-ci.yml

* Update binding.cc

* CLD3: change cld3_find_language return type

* CLD2: fix test

* FT: Try to build in Windows environment

* Update run-build.ps1

* Update run-build.ps1

* Update run-build.ps1

* Update run-build.ps1

* FT: WIndows build fixes + MacOS support for fun

* FT: try to fix build-native stage + artifact upload

* Update github-ci.yml

* Update github-ci.yml

* TF: Download artifact in "build-nugets" stage

* Update github-ci.yml

* Update github-ci.yml

* FT: download pretrained models for win & mac tests

* Update github-ci.yml

* Update github-ci.yml

* FT: try to fix build for MacOS

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* FT: fix MacOS build (-static-libgcc remove)

* FT: add MacOS support

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update CMakeLists.txt

* FT: /usr/lib/libc++.a

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update github-ci.yml

* Update CMakeLists.txt

* Update github-ci.yml

* Update github-ci.yml

* FT: brew install llvm

* FT: /opt/homebrew/opt/llvm/bin

* FT: /opt/homebrew/opt/llvm/lib

* Update github-ci.yml

* Update github-ci.yml

* Update github-ci.yml

* Update CMakeLists.txt

* FT: brew install llvm

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update github-ci.yml

* Revert "Update CMakeLists.txt"

This reverts commit 5226ff3.

* Update github-ci.yml

* Update github-ci.yml

* MacOS: amd vs x86

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update run-build.macos.sh

* MacOS: more flags

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update run-build.macos.sh

* Update CMakeLists.txt

* Update run-build.macos.sh

* Update run-build.macos.sh

* Update run-build.macos.sh

* Update run-build.macos.sh

* MacOS: build split for each arch

* MacOS: arch as argument

* Update LanguageIdentification.FastText.Native.csproj

* Update github-ci.yml

* Update CMakeLists.txt

* Update run-build.macos.sh

* Update github-ci.yml

* MasOS: pipeline fixes + IsSupported() unification

* MacOS: cross-arch naming of libfasttext.dylib

* MasOS: specific model path

* MacOS: another try to fix paths to models

* Update github-ci.yml

* Update FastTextDetector.cs

* Update FastTextDetector.cs

* 0.0.0.22 -> 0.0.1

* Running tests outside of Docker

* Update MainTests.cs

* CI fixes

* FT + MediaPipe: preload models from code

* Update Dockerfiles

* Test

* Update FastTextTests.cs

* Fixes

* Update github-ci.yml

* New test stage

* Tests: add CheckPackageVersion

* Fix CheckPackageVersion

* Fix version parsing

* Update github-ci.yml

* Update CMakeLists.txt

* CLD2 + CLD3 + FT: unification of CMakeLists.txt

* CLD2 + FT: no static linking on Linux

* CLD2 + CLD3

* Update README.md

* Try to use static linking for CLD2 + CLD3

* CLD2 + CLD3: Build fixes

* CLD2 + CLD3: fixes

* CLD2 + CLD3: yet another build fixes

* CLD2 + CLD3:  -Wno-narrowing everywhere

* CLD2 + CLD3: try to build for MacOS

* CLD2 + CLD3: fixe MacOS build

* CLD2 + CLD3: fix stupid mistakes

* Update CMakeLists.txt

* CLD2 + CLD3 + FT: try to use '-static'

* Using debian:bullseye-slim

* CLD2 + CLD3 + FT: no static libs for Linux

* Update setup-build.sh

* debian:bullseye-slim -> debian:bookworm-slim

* Update setup-build.sh

* Update setup-build.sh

* Update setup-build.sh

* CLD2 + CLD3: turn on MacOS support in packages

* Update setup-build.sh

* Update setup-build.sh

* Update setup-build.sh

* Update setup-build.sh

* Update setup-build.sh

* Update run-build.sh

* Update setup-build.sh

* Move zx to run-build

* Update Dockerfile

* Fixes

* Install NPM in run-build.sh

* Update Dockerfile

* Update run-build.sh

* Try to fix build for MacOS

* Update CLD2Detector.cs

* test

* Update CLD2Detector.cs

* CLD2 + CLD3: using DLLEXPORT

* CLD3 for Mac is on

* CLD2 for Mac is on

* FT: __attribute__((visibility("default")))

* CLD3: __attribute__((visibility("default")))

* CLD2: __attribute__((visibility("default")))

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update CLD2Detector.cs

* Update CLD3Detector.cs

* Update CLD2Detector.cs

* debian:bookworm-slim -> debian:bullseye-slim

* Fix

* Update github-ci.yml

* ubuntu:20.04 -> ubuntu:22.04

* Update github-ci.yml

* Bump versions and update readme

.github/workflows/github-ci.yml

@@ -8,20 +8,20 @@ on:
   workflow_dispatch:
 
 jobs:
-  build-nugets:
-    name: 🚀 Pack ${{matrix.projectName}}
-    runs-on: ubuntu-latest
+  build-native:
+    name: 💾 Build ${{matrix.task.title}} for ${{matrix.task.os}} (${{matrix.task.arch}})
+    runs-on: ${{matrix.task.os}}
     strategy:
       fail-fast: false
       matrix:
-        projectName: 
-          - LanguageIdentification.CLD2
-          - LanguageIdentification.CLD3
-          - LanguageIdentification.FastText
-          - LanguageIdentification.Whatlang
-          - LanguageIdentification.MediaPipe
-          - LanguageIdentification.Lingua
-          - LanguageCode
+        task: 
+          - { title: FastText, projectName: LanguageIdentification.FastText.Native, os: windows-latest, arch: x86_64, script: ./Scripts/run-build.ps1, artifact: fasttext.dll }
+          - { title: FastText, projectName: LanguageIdentification.FastText.Native, os: macos-13, arch: x86_64, script: ./Scripts/run-build.macos.sh, artifact: libfasttext.x86_64.dylib }
+          - { title: FastText, projectName: LanguageIdentification.FastText.Native, os: macos-14, arch: arm64, script: ./Scripts/run-build.macos.sh, artifact: libfasttext.arm64.dylib }
+          - { title: CLD2, projectName: LanguageIdentification.CLD2.Native, os: macos-13, arch: x86_64, script: ./Scripts/run-build.macos.sh, artifact: libcld2.x86_64.dylib }
+          - { title: CLD2, projectName: LanguageIdentification.CLD2.Native, os: macos-14, arch: arm64, script: ./Scripts/run-build.macos.sh, artifact: libcld2.arm64.dylib }
+          - { title: CLD3, projectName: LanguageIdentification.CLD3.Native, os: macos-13, arch: x86_64, script: ./Scripts/run-build.macos.sh, artifact: libcld3.x86_64.dylib }
+          - { title: CLD3, projectName: LanguageIdentification.CLD3.Native, os: macos-14, arch: arm64, script: ./Scripts/run-build.macos.sh, artifact: libcld3.arm64.dylib }
     permissions:
       contents: read
       packages: write
@@ -38,14 +38,79 @@ jobs:
       with:
         filters: |
           src:
-            - '${{env.ROOT}}/${{matrix.projectName}}/**'
+            - '${{env.ROOT}}/${{matrix.task.projectName}}/**'
+
+    - uses: dorny/paths-filter@v3
+      id: changes-native
+      with:
+        filters: |
+          src:
+            - '${{env.ROOT}}/${{matrix.task.projectName}}.Native/**'
+
+    - name: 🚀 Build ${{matrix.task.projectName}}
+      if: steps.changes.outputs.src == 'true'
+      working-directory: ${{env.ROOT}}/${{matrix.task.projectName}}
+      run: |
+        echo $(pwd)
+        git update-index --chmod=+x ${{matrix.task.script}}
+        chmod +x ${{matrix.task.script}}
+        ${{matrix.task.script}} ${{matrix.task.arch}}
+
+    - uses: actions/upload-artifact@v4
+      if: steps.changes.outputs.src == 'true'
+      with:
+        name: native-${{matrix.task.projectName}}-${{matrix.task.os}}
+        path: ${{env.ROOT}}/${{matrix.task.projectName}}/${{matrix.task.artifact}}
+        retention-days: 1
+        overwrite: 'true'
+        compression-level: 0
+        if-no-files-found: 'error'
+
+  build-package:
+    name: 🚀 Pack ${{matrix.task.title}}
+    runs-on: ${{matrix.task.os}}
+    needs: build-native
+    strategy:
+      fail-fast: false
+      matrix:
+        task: 
+          - { title: CLD2, projectName: LanguageIdentification.CLD2, os: ubuntu-latest }
+          - { title: CLD3, projectName: LanguageIdentification.CLD3, os: ubuntu-latest }
+          - { title: FastText, projectName: LanguageIdentification.FastText, os: ubuntu-latest }
+          - { title: Whatlang, projectName: LanguageIdentification.Whatlang, os: ubuntu-latest }
+          - { title: MediaPipe, projectName: LanguageIdentification.MediaPipe, os: ubuntu-latest }
+          - { title: Lingua, projectName: LanguageIdentification.Lingua, os: ubuntu-latest }
+          - { title: LanguageCode, projectName: LanguageCode, os: ubuntu-latest }
+    permissions:
+      contents: read
+      packages: write
+    env:
+      ROOT: ./src
+      NUGET_AUTH_TOKEN: ${{secrets.GITHUB_TOKEN}} 
+    steps:
+    - uses: actions/checkout@v4
+      with: 
+        submodules: 'true'
+
+    - uses: actions/download-artifact@v4
+      with:
+        pattern: native-${{matrix.task.projectName}}.Native-*
+        merge-multiple: true
+        path: ${{env.ROOT}}/${{matrix.task.projectName}}.Native/
+
+    - uses: dorny/paths-filter@v3
+      id: changes
+      with:
+        filters: |
+          src:
+            - '${{env.ROOT}}/${{matrix.task.projectName}}/**'
 
     - uses: dorny/paths-filter@v3
       id: changes-native
       with:
         filters: |
           src:
-            - '${{env.ROOT}}/${{matrix.projectName}}.Native/**'
+            - '${{env.ROOT}}/${{matrix.task.projectName}}.Native/**'
 
     - name: 📂 Setup .NET Core
       uses: actions/setup-dotnet@v4
@@ -57,38 +122,51 @@ jobs:
           8.0.x
         source-url: https://nuget.pkg.github.com/${{github.repository_owner}}/index.json
 
-    - name: 🚀 Pack ${{matrix.projectName}}.Native
+    - name: 🚀 Pack ${{matrix.task.projectName}}.Native
       if: steps.changes-native.outputs.src == 'true'
-      working-directory: ${{env.ROOT}}/${{matrix.projectName}}.Native
+      working-directory: ${{env.ROOT}}/${{matrix.task.projectName}}.Native
       run: dotnet pack -c Release -o out 
 
-    - name: 🚀 Pack ${{matrix.projectName}}
+    - name: 🚀 Pack ${{matrix.task.projectName}}
       if: steps.changes.outputs.src == 'true'
-      working-directory: ${{env.ROOT}}/${{matrix.projectName}}
+      working-directory: ${{env.ROOT}}/${{matrix.task.projectName}}
       run: dotnet pack -c Release -o out 
 
     - uses: actions/upload-artifact@v4
       if: steps.changes-native.outputs.src == 'true'
       with:
-        name: build-${{matrix.projectName}}.Native
-        path: ${{env.ROOT}}/${{matrix.projectName}}.Native/out
+        name: build-${{matrix.task.projectName}}.Native
+        path: ${{env.ROOT}}/${{matrix.task.projectName}}.Native/out
         retention-days: 1
         overwrite: 'true'
         compression-level: 0
 
     - uses: actions/upload-artifact@v4
       if: steps.changes.outputs.src == 'true'
       with:
-        name: build-${{matrix.projectName}}
-        path: ${{env.ROOT}}/${{matrix.projectName}}/out
+        name: build-${{matrix.task.projectName}}
+        path: ${{env.ROOT}}/${{matrix.task.projectName}}/out
         retention-days: 1
         overwrite: 'true'
         compression-level: 0
 
-  test-nugets:
-    name: 🧪 Test ${{matrix.projectName}}
-    runs-on: ubuntu-latest
-    needs: build-nugets
+  test-package:
+    name: 🧪 Test on ${{matrix.task.os}}
+    runs-on: ${{matrix.task.os}}
+    needs: build-package
+    strategy:
+      fail-fast: false
+      # https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#standard-github-hosted-runners-for--private-repositories
+      matrix:
+        task: 
+          - { os: ubuntu-24.04 }
+          - { os: ubuntu-22.04 }
+          # - { os: ubuntu-20.04 }
+          - { os: windows-2022 }
+          - { os: windows-2019 }
+          - { os: macos-13 }
+          - { os: macos-14 }
+          - { os: macos-15 }
     permissions:
       contents: read
       packages: write
@@ -100,36 +178,48 @@ jobs:
 
     - name: 📂 Files
       working-directory: ${{env.ROOT}}
-      run: mkdir -p local-nugets
+      run: mkdir -p local-packages
 
     - uses: actions/download-artifact@v4
       with:
         pattern: build-*
         merge-multiple: true
-        path: ${{env.ROOT}}/local-nugets
+        path: ${{env.ROOT}}/local-packages
 
     - name: 📂 Files
-      working-directory: ${{env.ROOT}}
+      working-directory: ${{env.ROOT}}/local-packages
       run: ls -R
 
-    - name: 📂 Build Docker Image
-      working-directory: ${{env.ROOT}}
+    - name: 📂 Use local NuGet (Linux)
+      if: ${{ startsWith(matrix.task.os, 'ubuntu') }}
       run: |
-        docker build -f ./test.Dockerfile -t langunage-identification-test-runner:latest .
-        docker build -f ./test-ci.Dockerfile -t langunage-identification-test-runner-ci:latest .
+        path=$(realpath "${{env.ROOT}}/local-packages")
+        dotnet nuget add source $path
 
-    - name: 🧪 Run Tests in Docker
-      working-directory: ${{env.ROOT}}
-      run: docker run --rm -v $(pwd):/src langunage-identification-test-runner-ci:latest
+    - name: 📂 Use local NuGet (Windows)
+      if: ${{ startsWith(matrix.task.os, 'windows') }}
+      run: |
+        $path = [System.IO.Path]::GetFullPath('${{env.ROOT}}/local-packages')
+        dotnet nuget add source $path
 
-    - name: 🧪 Run Test for LanguageCode
-      working-directory: ${{env.ROOT}}/LanguageCode.Tests
+    - name: 📂 Use local NuGet (OSX)
+      if: ${{ startsWith(matrix.task.os, 'macos') }}
+      run: |
+        path=$(realpath "${{env.ROOT}}/local-packages")
+        dotnet nuget add source $path
+
+    - name: 🧪 Run Tests for LanguageIdentification
+      working-directory: ${{env.ROOT}}/LanguageIdentification.Tests
       run: dotnet test -c CI
 
-  deploy-nugets:
+    - name: 🧪 Run Tests for LanguageCode
+      working-directory: ${{env.ROOT}}/LanguageCode.Tests
+      run: dotnet test
+
+  deploy-package:
     name: 🚚 Push ${{matrix.projectName}}
     runs-on: ubuntu-latest
-    needs: test-nugets
+    needs: test-package
     if: github.ref == 'refs/heads/master'
     strategy:
       fail-fast: false

README.md

@@ -89,14 +89,14 @@ functionality into their applications.
 
 ## Platform support
 
-| Model     |        Linux       |     Windows      |  macOS |  Blazor WASM   |
-| :-------- | :----------------: | :--------------: | :----: | :------------: |
-| CLD2      | :white_check_mark: | :construction:   | :x:    | :x:            |
-| CLD3      | :white_check_mark: | :construction:   | :x:    | :x:            |
-| FastText  | :white_check_mark: | :construction:   | :x:    | :x:            |
-| Whatlang  | :white_check_mark: | :construction:   | :x:    | :x:            |
-| MediaPipe | :white_check_mark: | :construction:   | :x:    | :x:            |
-| Lingua    | :white_check_mark: | :construction:   | :x:    | :x:            |
+| Model     |        Linux       |     Windows        |  macOS             |  Blazor WASM   |
+| :-------- | :----------------: | :----------------: | :----------------: | :------------: |
+| CLD2      | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x:            |
+| CLD3      | :white_check_mark: | :white_check_mark: | :construction:     | :x:            |
+| FastText  | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x:            |
+| Whatlang  | :white_check_mark: | :construction:     | :construction:     | :x:            |
+| MediaPipe | :white_check_mark: | :construction:     | :construction:     | :x:            |
+| Lingua    | :white_check_mark: | :construction:     | :construction:     | :x:            |
 
 :white_check_mark: — Full support |
 :x: — No support |

README_CLD2.md

@@ -5,8 +5,8 @@ Welcome to **Panlingo.LanguageIdentification.CLD2**, a .NET wrapper for the Chro
 ## Requirements
 
 - Runtime: **.NET >= 5.0**
-- OS: **Linux**
-- Arch: **AMD64**
+- OS: **Linux (Ubuntu, Debian)**, **Windows 10+** or **Windows Server 2019+**, **macOS**
+- Arch: **AMD64** (or **ARM** for macOS)
 
 ## Installation
 

README_CLD3.md

@@ -5,7 +5,7 @@ Welcome to **Panlingo.LanguageIdentification.CLD3**, a .NET wrapper for the Chro
 ## Requirements
 
 - Runtime: **.NET >= 5.0**
-- OS: **Linux**
+- OS: **Linux (Ubuntu, Debian)**, **Windows 10+** or **Windows Server 2019+**
 - Arch: **AMD64**
 
 ## Installation

README_FASTTEXT.md

@@ -5,8 +5,8 @@ Welcome to **Panlingo.LanguageIdentification.FastText**, a .NET wrapper for the
 ## Requirements
 
 - Runtime: **.NET >= 5.0**
-- OS: **Linux**
-- Arch: **AMD64**
+- OS: **Linux (Ubuntu, Debian)**, **Windows 10+** or **Windows Server 2019+**, **macOS**
+- Arch: **AMD64** (or **ARM** for macOS)
 
 ## Installation
 

src/LanguageCode.Tests/LanguageCode.Tests.csproj

@@ -7,6 +7,10 @@
 
     <IsPackable>false</IsPackable>
     <IsTestProject>true</IsTestProject>
+
+    <RootNamespace>Panlingo.LanguageCode.Tests</RootNamespace>
+
+    <Configurations>Debug;Release;CI</Configurations>
   </PropertyGroup>
 
   <ItemGroup>

src/LanguageCode.Tests/LanguageCodeTests.cs

@@ -1,7 +1,6 @@
-using Panlingo.LanguageCode;
-using Panlingo.LanguageCode.Models;
+using Panlingo.LanguageCode.Models;
 
-namespace LanguageCode.Tests
+namespace Panlingo.LanguageCode.Tests
 {
     public class LanguageCodeTests
     {

src/LanguageCode/LanguageCode.csproj

@@ -2,7 +2,7 @@
 
   <PropertyGroup>
     <TargetFrameworks>netstandard2.1;net5.0;net6.0;net7.0;net8.0</TargetFrameworks>
-    <Version>0.0.0.22</Version>
+    <Version>0.0.0.23</Version>
     <PackageId>Panlingo.LanguageCode</PackageId>
     <AssemblyName>Panlingo.LanguageCode</AssemblyName>
     <RootNamespace>Panlingo.LanguageCode</RootNamespace>

src/LanguageIdentification.CLD2.ConsoleTest/Program.cs

@@ -6,7 +6,7 @@ static void Main(string[] args)
         {
             using var cld2 = new CLD2Detector();
 
-            var text = "Hello, how are you? Привіт, як справи? Привет, как дела?";
+            var text = "Привіт, як справи?";
 
             var predictions = cld2.PredictLanguage(text);
 

src/LanguageIdentification.CLD2.ConsoleTest/Properties/launchSettings.json

@@ -2,6 +2,13 @@
   "profiles": {
     "Docker": {
       "commandName": "Docker"
+    },
+    "WSL": {
+      "commandName": "WSL2",
+      "distributionName": ""
+    },
+    "Project": {
+      "commandName": "Project"
     }
   }
 }

src/LanguageIdentification.CLD2.Native/.gitignore

@@ -1,2 +1,3 @@
 libcld2.so
+libcld2.dll
 build_temp/**

src/LanguageIdentification.CLD2.Native/CLD2NativeLibrary.cs

@@ -5,6 +5,6 @@ public static class CLD2NativeLibrary
         /// <summary>
         /// Name of native binary
         /// </summary>
-        public const string Name = "libcld2.so";
+        public const string Name = "libcld2";
     }
 }

src/LanguageIdentification.CLD2.Native/Dockerfile

@@ -1,5 +1,4 @@
-FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
-ARG BUILD_CONFIGURATION=Release
+FROM debian:bullseye-slim AS build
 
 WORKDIR /repo
 COPY ["src/LanguageIdentification.CLD2.Native/LanguageIdentification.CLD2.Native.csproj", "src/LanguageIdentification.CLD2.Native/"]