fix(developer): rewrite ldml visual keyboard compiler 🍒 🏠

[mcdurdin]

The visual keyboard compiler was never finished in 17.0. This rewrites it to:

1. Use the kmxplus data rather than reading from xml directly
2. Fill in visualkeyboard.header.kbdname
3. Support modifiers
4. Handle encoded characters like \u{1234}
5. Handle string variables like ${one}*

Additional unit tests have been added to verify the behavior of the visual keyboard compiler in more detail.

• String variable tests will be enabled in next commit (which is a cherry-pick of fix(developer): check vars string usage before definition #12404).

Other fixes:

1. The LDML XML reader was relying on its input being a Node.js Buffer even though it was declared Uint8Array, as it implicitly used Buffer.toString() to do text conversion. (Buffer subclasses from Uint8Array). This breaks when using Uint8Array directly and means we had an implicit dependency on Node.js. See also chore(common): xml2js -> fast-xml-parser #12331.
2. XML errors were not captured in the LDML XML reader. See also chore(common): xml2js -> fast-xml-parser #12331.
3. The unused and unfinished touch-layout-compiler.ts and keymanweb-compiler.ts have been removed along with corresponding unit tests and fixtures. These will be replaced by Core implementations; see epic: web-core 🎼 #12291.

Fixes: #12395
Cherry-pick-of: #12402

User Testing

Preparation:

1. Install Keyman Developer artifact from this PR.
2. Open the release/i/imperial_aramaic keyboards project in Keyman Developer, compile it, and locate the resulting build/imperial_aramaic.kmp.

TEST_LDML_VISUAL_KEYBOARD_WINDOWS: Install the imperial_aramaic.kmp file into Keyman for Windows 17.0.329. Verify that the On Screen Keyboard displays Aramaic characters.

TEST_LDML_VISUAL_KEYBOARD_MAC: Install the imperial_aramaic.kmp file into Keyman for macOS 17.0.329. Verify that the On Screen Keyboard displays Aramaic characters.

TEST_LDML_VISUAL_KEYBOARD_LINUX: Install the imperial_aramaic.kmp file into Keyman for Linux 17.0.329. Verify that the On Screen Keyboard displays Aramaic characters.

[keymanapp-test-bot]

User Test Results

Test specification and instructions

• ✅ TEST_LDML_VISUAL_KEYBOARD_WINDOWS (PASSED) (notes)
• ✅ TEST_LDML_VISUAL_KEYBOARD_MAC (PASSED) (notes)
• ✅ TEST_LDML_VISUAL_KEYBOARD_LINUX (PASSED) (notes)

Test Artifacts

• Android
  • Keyman for Android apk
  • FirstVoices for Android 16.0 apk
  • FirstVoices Keyboards for Android apk
  • KeyboardHarness apk
  • Keyman for Android 16.0 apk
  • KMSample1 apk
  • KMSample2 apk
• Developer
  • Keyman Developer
  • Compiler Regression Tests
  • kmcomp.zip
• iOS
  • Keyman for iOS (simulator image)
  • FirstVoices Keyboards for iOS (simulator image)
  • TestFlight internal PR build version - 17.0.330 (0.12406.12054)
• Keyboards
  • Test Keyboards
• Web
  • KeymanWeb Test Home
• Windows
  • Keyman for Windows
  • FirstVoices Keyboards for Windows
  • Text Editor (32 bit)
  • Text Editor (64 bit)

[srl295]

RSLGTM

[dinakaranr]

Test Results

I tested this issue with the attached "Keyman developer 17.0.330-test-12406" build on Windows 10.
Open the imperial_aramaic.kps project file.
Compile the build and generate the keyboard build.
Copy the imperial_aramaic.kmp file and share it to Windows, MacOS, and Linux.
Here I am sharing my observation.

• TEST_LDML_VISUAL_KEYBOARD_WINDOWS (Passed):

1. Installed the Keyman 17.0.329 keyboard build.
2. Installed the "imperial_aramaic.kmp" keyboard.
3. Open the Notepad application.
4. Open the OSK from the window tray icon.
5. Select the imperial_aramaic keyboard.
6. Enter some text in the notepad
7. Verified the OSK shows the imperial_aramaic letters.
   It works well. Thank you.

• TEST_LDML_VISUAL_KEYBOARD_MAC (Failed):

1. Installed the Keyman 17.0.329 keyboard build.
2. Installed the "imperial_aramaic.kmp" keyboard.
3. Open the "TextEdit" application.
4. Open the OSK from the toolbar.
5. Select the imperial_aramaic keyboard.
6. Enter some text in the "TextEdit"
7. Verified the OSK does not show the imperial_aramaic letters.
   It seems to be a problem. Thank you.
   Please refer to the screenshot below

• TEST_LDML_VISUAL_KEYBOARD_LINUX (Passed):

1. Installed the Keyman 17.0.329 keyboard build.
2. Installed the "imperial_aramaic.kmp" keyboard.
3. Open the "TextEditor" application.
4. Open the OSK from the toolbar.
5. Select the imperial_aramaic keyboard.
6. Enter some text in the "TextEditor".
7. Verified the OSK shows the imperial_aramaic letters.
   It works well. Thank you.

[mcdurdin]

@dinakaranr thank you for that test. I have addressed an issue in the compiler, so can you please retest on the Mac? Make sure you rebuild the keyboard before copying to the Mac for testing. Thanks!

@keymanapp-test-bot retest TEST_LDML_VISUAL_KEYBOARD_MAC

[dinakaranr]

Thank you for fixing this issue.

Test Results

I tested this issue with the attached "Keyman developer 17.0.330-test-12406" build on Windows 10.
Open the imperial_aramaic.kps project file.
Compile and regenerate the keyboard build.
Copy the imperial_aramaic.kmp file and share it to Windows, MacOS, and Linux.
Here I am sharing my observation.

• TEST_LDML_VISUAL_KEYBOARD_MAC (Passed):

1. Installed the Keyman 17.0.329 keyboard build.
2. Installed the "imperial_aramaic.kmp" keyboard.
3. Open the "TextEdit" application.
4. Open the OSK from the toolbar.
5. Select the imperial_aramaic keyboard.
6. Enter some text in the "TextEdit"
7. Verified the OSK appeared and the imperial_aramaic letters displayed
8. It works well. Thank you.
   Please refer to the screenshot below

[keyman-server]

Changes in this pull request will be available for download in Keyman version 17.0.330