Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial AudioVM implementation with pipewire and pulseaudio #576

Merged
merged 1 commit into from
May 28, 2024

Conversation

josa41
Copy link
Contributor

@josa41 josa41 commented Apr 24, 2024

Initial version of AudioVM with Pipewire backend and pulseaudio TCP remote communication layer for the guest VMs. Note that this is not really secure design (yet) basically all VMs can access the pulseaudio TCP service.

Description of changes

Separate AudioVM with HW passthrough audio device

  • Pipewire backend with pulseaudio tunnel for VM client

Checklist for things done

  • [X ] Summary of the proposed changes in the PR description
  • [X ] More detailed description in the commit message(s)
  • [X ] Commits are squashed into relevant entities - avoid a lot of minimal dev time commits in the PR
  • [X ] Contribution guidelines followed
  • Ghaf documentation updated with the commit - https://tiiuae.github.io/ghaf/
  • PR linked to architecture documentation and requirement(s) (ticket id)
  • [X ] Test procedure described (or includes tests). Select one or more:
    • [X ] Tested on Lenovo X1 x86_64
    • Tested on Jetson Orin NX or AGX aarch64
    • Tested on Polarfire riscv64
  • [X ] Author has run nix flake check --accept-flake-config and it passes
  • All automatic Github Action checks pass - see actions
  • Author has added reviewers and removed PR draft status

Testing

  • 3.5mm headset works
  • Integrated speaker works, automatically swapping to/from 3.5mm IO
  • Tested with Chromium, Element should also work
    Notes:
  • Integrated speaker volume is a bit low
  • Integrated mic does not work (possible kernel/driver issue)
  • USB/BT sound devices does not work (will be done separately)

@josa41 josa41 temporarily deployed to internal-build-workflow April 24, 2024 05:50 — with GitHub Actions Inactive
@josa41 josa41 marked this pull request as draft April 24, 2024 05:50
Copy link
Contributor

@leivos-unikie leivos-unikie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test results on Lenovo-X1-gen11

manual testing

  • Poweroff and reboot buttons don't work
  • All apps launch by clicking their icons
  • Audio output works both from integrated speakers and 3.5mm headphones, automatically swapping to/from 3.5mm IO
  • Audio output works both in chromium and element
  • Integrated microphone does not work
  • Youtube video real playback time does not differ from the video nominal lenght
  • Increasing volume works in audio-vm with “pactl set-sink-volume @DEFAULT_SINK@ 48000”

ci-test-automation

  • performance ok
  • bat tests passed

@josa41
Copy link
Contributor Author

josa41 commented Apr 24, 2024

Reboot & shutdown issue fixed. The issue was that Polkit package was removed from host along with pulseaudio. Apparently only pulseaudio had a dependency for Polkit so polkit was removed from build. Manually enabling polkit on host fixed the issue.
Will add separate tickets for adding USB sound device passthroughs and fixing integrated mic issue.
Also there is a separate ticket to add audio controller app in guivm which fixed the low volume issue.

@josa41 josa41 temporarily deployed to internal-build-workflow April 24, 2024 11:38 — with GitHub Actions Inactive
@josa41 josa41 marked this pull request as ready for review April 24, 2024 11:42
@leivos-unikie
Copy link
Contributor

Now reboot and poweroff buttons work.

@leivos-unikie leivos-unikie added the Tested on Lenovo X1 Carbon This PR has been tested on Lenovo X1 Carbon label Apr 24, 2024
@josa41 josa41 temporarily deployed to internal-build-workflow April 26, 2024 07:14 — with GitHub Actions Inactive
@leivos-unikie
Copy link
Contributor

Tested now also audio input with 3.5mm mobile phone headphones. Doesn't work. (Verified that the mic of the headphones work with my nixos work laptop.

@leivos-unikie
Copy link
Contributor

Tested 3.5mm mobile phone headphones mic again with instructions from Jon.

  • It works if plugged before chromium window is opened.
  • If headphones is plugged in when chromium window is already open the mic doesn't work. In this case closing all chromium windows and then opening a new chromium window sets mic working.

@leivos-unikie
Copy link
Contributor

Tested that 3.5mm headphones mic works also in Element app Jitsi Video Conference but only if plugged before pressing the button "Join Conference".

@leivos-unikie
Copy link
Contributor

Noticed something strange when pressing ghaf desktop poweroff button while 3.5mm headset connected and Jitsi Video Conference call active. Laptop power led did shut down but two leds stayed on:

  • speakers off / F1
  • mute / F4

Tried to reproduce the issue. Again the same setup. This time all leds of the laptop shutdown but I heard that fan is running. After few minutes fan stopped but the machine was still not shutdown. I could verify this by pressing the power button and nothing happening. Only after doing hard reset (pressing the power button 15 sec) I was able to boot the laptop.

Tried to reproduce with same setup except 3.5mm headset not plugged. This time laptop shut down normally.

@milva-unikie
Copy link

Noticed something strange when pressing ghaf desktop poweroff button while 3.5mm headset connected and Jitsi Video Conference call active. Laptop power led did shut down but two leds stayed on:

* speakers off / F1

* mute / F4

Tried to reproduce the issue. Again the same setup. This time all leds of the laptop shutdown but I heard that fan is running. After few minutes fan stopped but the machine was still not shutdown. I could verify this by pressing the power button and nothing happening. Only after doing hard reset (pressing the power button 15 sec) I was able to boot the laptop.

Tried to reproduce with same setup except 3.5mm headset not plugged. This time laptop shut down normally.

I noticed this same thing and just started trying to reproduce it. I tested 3.5 mm headset with Chromium and after that shut down the laptop with poweroff button. Mentioned leds stayed on.

@vilvo
Copy link
Contributor

vilvo commented Apr 29, 2024

Noticed something strange when pressing ghaf desktop poweroff button while 3.5mm headset connected and Jitsi Video Conference call active.
...
Tried to reproduce the issue. Again the same setup. This time all leds of the laptop shutdown but I heard that fan is running. After few minutes fan stopped but the machine was still not shutdown. I could verify this by pressing the power button and nothing happening.
...
Tried to reproduce with same setup except 3.5mm headset not plugged. This time laptop shut down normally.

I noticed this same thing and just started trying to reproduce it. I tested 3.5 mm headset with Chromium and after that shut down the laptop with poweroff button. Mentioned leds stayed on.

#565 could be adapted to log from relevant VMs to persistent storage to capture what's going on here. See Testing-section in the #565 PR description.

@leivos-unikie
Copy link
Contributor

Noticed something strange when pressing ghaf desktop poweroff button while 3.5mm headset connected and Jitsi Video Conference call active.
...
Tried to reproduce the issue. Again the same setup. This time all leds of the laptop shutdown but I heard that fan is running. After few minutes fan stopped but the machine was still not shutdown. I could verify this by pressing the power button and nothing happening.
...
Tried to reproduce with same setup except 3.5mm headset not plugged. This time laptop shut down normally.

I noticed this same thing and just started trying to reproduce it. I tested 3.5 mm headset with Chromium and after that shut down the laptop with poweroff button. Mentioned leds stayed on.

#565 could be adapted to log from relevant VMs to persistent storage to capture what's going on here. See Testing-section in the #565 PR description.

I tried rebasing on that log-vm and solving few merge conflicts. But the build fails.

@leivos-unikie
Copy link
Contributor

leivos-unikie commented Apr 30, 2024

Tried to see journalctl -b-1 on ghaf-host after shutdown with 3.5mm mic on in chromium-vm
No journal boot entry found from the specified boot offset (-1).

After normal shutdown journal -b-1 shows logs from the previous boot.

Then tried to leave this running on ghaf-host while doing the shutdown test:
journalctl -f > /home/ghaf/host-journalctl.log &

but there was nothing special, just repeating these:

Apr 30 13:00:55 ghaf-host vsockproxy[898]: Connection 6 RX 7132 bytes in 1.00 seconds, transfer rate 0.01 MB/s, rx total 185952
Apr 30 13:00:55 ghaf-host vsockproxy[898]: Connection 6 TX 29677456 bytes in 1.00 seconds, transfer rate 29.68 MB/s, tx total 353325956
Apr 30 13:00:56 ghaf-host vsockproxy[898]: Connection 5 RX 21391864 bytes in 1.00 seconds, transfer rate 21.38 MB/s, rx total 374717820
Apr 30 13:00:56 ghaf-host vsockproxy[898]: Connection 5 TX 4556 bytes in 1.00 seconds, transfer rate 0.00 MB/s, tx total 190508
Apr 30 13:00:56 ghaf-host vsockproxy[898]: Connection 6 RX 4556 bytes in 1.00 seconds, transfer rate 0.00 MB/s, rx total 190508
Apr 30 13:00:56 ghaf-host vsockproxy[898]: Connection 6 TX 21391864 bytes in 1.00 seconds, transfer rate 21.38 MB/s, tx total 374717820



@leivos-unikie
Copy link
Contributor

Tested 'shutdown with 3.5mm headset mic active' issue also with current ghaf mainline build (himalia 82). This issue is not in the mainline.

@josa41
Copy link
Contributor Author

josa41 commented May 2, 2024

The shutdown issue is a great finding. I can reproduce it also by starting the microphone/recording in chromium and then going to ghaf-host and using systemctl to stop audiovm. You can see that the recording in chromium stops (i use mic-test.com). Now audio-vm is turned off and shutting down the laptop seems to cause the issue, some times the indicator lights turn off. Checking that the fan at the bottom keeps spinning and also the laptop does not turn on with a short press of power button (needs the 15second shutdown first).
It seem the issue is between audioVM and chromiumVM connecting getting hanged by audio not responding.
It could also be audio hardware (or one of the audio related passthrough devices) getting stuck for audio VM shuttin down.
Anyway shutting down just audioVM may allow us easier way of getting some logs of what is going on.

@josa41
Copy link
Contributor Author

josa41 commented May 3, 2024

I was able to reproduce this with just audiovm so ssh to audiovm and record audio with pulseaudio tools
parecord -r output.wav

Now stop microvm@audio-vm.service from host and click the shutdown icon from guivm.
The audio indicator lights in most cases stay on and some cases the X1 bottom fan keeps spinning even when the lights may turn off.

@josa41
Copy link
Contributor Author

josa41 commented May 6, 2024

Figured out a way to reset the audio device on host after audio-vm has shut down and audio device is got in some buggy state. This can be demoed as below:

So after ssh to audio-vm and starting a recording

parecord -r output.wav

ssh to host and use systemctl to shutdown audio-vm to get to error state
After the audio-vm shutdown (when audiovm is recording) turning off the laptop will result in the bottom fan and parts of the laptop staying on..

This can be fixed from host by forcing a reset on the pci device (remove device and rescan bus):

sudo su
echo "1" /sys/bus/pci/devices/0000:00:00:1f.3/remove
sleep 1
echo "1" /sys/bus/pci/devices/0000:00:1f.0/rescan

Next step is figure out how to automate this..

@josa41
Copy link
Contributor Author

josa41 commented May 8, 2024

Tested sound and audio recording with chromium

  • Microphone works only with 3.5mm headset
  • Laptop shutdown issue fixed

@leivos-unikie
Copy link
Contributor

leivos-unikie commented May 10, 2024

Tested first the laptop shutdown issue with 3.5mm headset mic plugged

  • with chromium, ok
  • element app does not detect any microphone (Before the fix I was able to activate mic in Jitsi Video Conference in Element app.)

Copy link
Contributor

@leivos-unikie leivos-unikie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test results on Lenovo-X1

  • ci-test-automation bat tests passed / performance ok
  • All apps launch
  • No shutdown issues
  • Audio input/output works in chromium with 3.5.mm headset
  • Audio input/output does not work in element app, not with 3.5mm headset nor with integrated speakers/mic

@josa41
Copy link
Contributor Author

josa41 commented May 10, 2024

It seems element-vm network setup is done so that it does not resolve audio-vm.ghaf or other ghaf internal dns addresses. It does have access to ghaf addresses with direct ip address.
See dns settings in element-vm.ghaf:
systemd-resolve --status

Copy link
Contributor

@leivos-unikie leivos-unikie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebased josa41 audiovm branch on josa41 origin/element-dns-fix with
git rebase origin/element-dns-fix

to be able run tests for both PRs #576 and #600 merged together.

Test reults on Lenovo-X1:

  • Now 3.5mm headset audio input/output works in Element App
  • Intergrated speakers work.
  • Integrated mic does not work. Logi USB headset does not work. (this was expected, will be fixed separately)
  • All apps launch from icons
  • ci-test-automation tests pass and performance test are ok

modules/microvm/virtualization/microvm/audiovm.nix Outdated Show resolved Hide resolved
targets/lenovo-x1/appvms/chromium.nix Outdated Show resolved Hide resolved
targets/lenovo-x1/appvms/element.nix Outdated Show resolved Hide resolved
targets/lenovo-x1/appvms/element.nix Show resolved Hide resolved
targets/lenovo-x1/audiovmExtraModules.nix Outdated Show resolved Hide resolved
targets/lenovo-x1/audiovmExtraModules.nix Outdated Show resolved Hide resolved
targets/lenovo-x1/everything.nix Outdated Show resolved Hide resolved
targets/lenovo-x1/everything.nix Outdated Show resolved Hide resolved
Initial version of AudioVM with Pipewire backend and pulseaudio TCP
remote communication layer for the guest VMs. Note that this is not
really secure design (yet) basically all VMs can access the pulseaudio
TCP service.

Signed-off-by: Jon Sahlberg <jon.sahlberg@unikie.com>
@brianmcgillion brianmcgillion added Needs Testing CI Team to pre-verify and removed Tested on Lenovo X1 Carbon This PR has been tested on Lenovo X1 Carbon labels May 28, 2024
@leivos-unikie
Copy link
Contributor

I ran regression test with an image built before that last force-push.

  • ci-test-automation passed, performance ok
  • all apps launch

Tested that integrated speakers and microphone worked with chromium, as well as 3.5mm headset audio and mic works.

@brianmcgillion brianmcgillion merged commit 58f9b35 into tiiuae:main May 28, 2024
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs Testing CI Team to pre-verify
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants