Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add background blur feature for supported devices #2812

Draft
wants to merge 37 commits into
base: livekit
Choose a base branch
from

Conversation

toger5
Copy link
Contributor

@toger5 toger5 commented Nov 20, 2024

Fixes #121

This adds background blur via @livekit/track-processors for supported browsers.
The option can be found in the video tab of the settings.

The fallback will be a greyed out checkbox (on firefox for instance)
This way firefox users get informed why it does not work on their side if other users tell them where to set it up.

Sadly this is not exactly done the way we would want to set it up see: livekit/track-processors-js#50

  • bundle local assets not CDN
  • stringify related hang on startup

@toger5
Copy link
Contributor Author

toger5 commented Nov 20, 2024

how do we deal with lk modules that dont fullfill the ts lints?

@toger5
Copy link
Contributor Author

toger5 commented Nov 27, 2024

how do we deal with lk modules that dont fullfill the ts lints?

I found what we needed:
@types/dom-mediacapture-transform

Copy link
Member

@hughns hughns left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've suggested some cleanup. Furthermore I think you could deduplicate some of the code into something like a useBlur() or canBlur(). This might make adding tests easier too.

locales/en-GB/app.json Outdated Show resolved Hide resolved
src/livekit/useLiveKit.ts Outdated Show resolved Hide resolved
src/livekit/useLiveKit.ts Outdated Show resolved Hide resolved
src/livekit/useLiveKit.ts Outdated Show resolved Hide resolved
src/livekit/useLiveKit.ts Outdated Show resolved Hide resolved
src/settings/SettingsModal.tsx Outdated Show resolved Hide resolved
src/settings/SettingsModal.tsx Outdated Show resolved Hide resolved
src/settings/SettingsModal.tsx Outdated Show resolved Hide resolved
src/room/LobbyView.tsx Outdated Show resolved Hide resolved
locales/en-GB/app.json Outdated Show resolved Hide resolved
@toger5
Copy link
Contributor Author

toger5 commented Dec 2, 2024

Furthermore I think you could deduplicate some of the code into something like a useBlur() or canBlur(). This might make adding tests easier too.

I think @robintown has an idea how we might be able to reuse the track from the lobby to the in call view. Than this might come for free.

But in this pr we should at least get rid of the duplicated let blur = useMemo sections

locales/en-GB/app.json Outdated Show resolved Hide resolved
@hughns hughns changed the title Add background Blur Add background blur feature for support devices Dec 4, 2024
@hughns hughns mentioned this pull request Dec 4, 2024
@hughns
Copy link
Member

hughns commented Jan 6, 2025

Marking as draft whilst we fix these:

hook.js:608 error when trying to pipe IndexSizeError: Failed to construct 'ImageData': The source width is zero or not a number.
at Vz (index.mjs:291:1)
at MZe.drawVirtualBackground (index.mjs:230:14)
at MZe.transform (index.mjs:205:21)
at Object.transform (index.mjs:117:39)

It was also showing a green colour instead of blurring at one point. I'm not able to reliably reproduce that.

@hughns
Copy link
Member

hughns commented Jan 6, 2025

Also this:

image

@toger5
Copy link
Contributor Author

toger5 commented Jan 8, 2025

I also was not yet able to fix the following issues:

  • configure to not use blur -> press on a recent call from the list. -> turn on the blur it does not activate blur for me error when trying to pipe IndexSizeError: Failed to construct 'ImageData': The source width is zero or not a number.
  • unrelated but I now remember that I looked into this as well: start a call -> wait for the video in the lobby -> open settings observe that the video feed turns black for a couple of seconds.

See: https://matrix.to/#/!dXHEZfLxIQzAELxzQd:matrix.org/$i8KmqSqyJN4s65D8oXTjcSLTVgFo2uB30HWVlWAVk_Q?via=matrix.org&via=element.io&via=robin.town
and: https://matrix.to/#/!dXHEZfLxIQzAELxzQd:matrix.org/$ad9iclLVvo8doof3p8iFQBkL43koY-YxbV4PcCWUQfo?via=matrix.org&via=element.io&via=robin.town
for demo videos about those issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Background blurring
3 participants