Live2D with Lipsync (using audio file/link) #122

RaSan147 · 2023-12-14T00:31:54Z

Solving issues mentioned in #117

Changes: 1. updated model.motion(group, index, priority) to model.motion(group, index, priority, sound, volume, expression); 2. added model.stopSpeaking() 3. updated readme.MD with demos 4. Workflow will save files to dist (won't be gitignored) 5. Praying this new change doesn't break

voice volume expressions are now optional arg {name: value, ....}

guansss

Thanks again for the PR! I think we are getting close but some changes are still needed as described in the comments.

I noticed that some of the code is not properly linted. After making changes to the code, please run npm run lint:fix to automatically fix the linting errors, and address any remaining errors manually. (except for the triple slash reference errors, which I will fix later)

After you finish theses changes, I'll be adding some tests to make sure this feature works as expected.

src/cubism-common/MotionManager.ts

src/cubism-common/SoundManager.ts

guansss · 2023-12-14T11:45:20Z

src/cubism4/Cubism4InternalModel.ts

@@ -248,6 +257,11 @@ export class Cubism4InternalModel extends InternalModel {
        this.coreModel.addParameterValueById(this.idParamBodyAngleX, this.focusController.x * 10); // -10 ~ 10
    }

+
+    updateFacialEmotion(mouthForm: number) {


Could a name like setMouthForm be better? Because it's not changing the entire facial expression but only the mouth form. Also, update implies that this function will do some computations other than setting the value, so set will be more suitable here.

As a new API, this method should also be added to Cubism2InternalModel for consistency.

I'll test and run on the cubism 2 (well the issue is i tried and failed to set up the development env on my local system, but the github action worked fine even the codespace failed, i know my skill issue) So probably won't be able to run the npm lint (will try)

The development guide in DEVELOPMENT.md was a bit messy and I've rewritten it, now I guess there won't be problems if you follow the steps (if there is please let me know!)

It's not your issue but the codespaces being problematic with submodules, browser testing etc. So better run it locally.

Thanks a lot 😭

I decided to remove this method because setting this param is pretty straightforward and isn't really worth adding a method for it.

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

also remove cache buster and autoplay

into for_PR

guansss · 2024-04-15T07:18:27Z

Finally it's ready to merge! Before I merge it, are there any changes you would like to make or suggest?

RaSan147 · 2024-04-17T14:24:38Z

Sorry didn't notice, gimme a bit time, testing...

RaSan147 · 2024-04-17T14:27:21Z

btw can you please check the PR i've sent you on cubism folder repo...
that should fix the process not found (or you may tweak the results a bit)

RaSan147 · 2024-04-17T14:37:56Z

package.json

vite requires terser in this version (my fresh install was not working without it, so kindly add it the deps
i fixed it with npm install terser

RaSan147 · 2024-04-17T14:52:58Z

src/cubism-common/LipSync.ts

add another option, force or priority, if force or higher priority, will stop current audio, otherwise current one will play.

will add onFinish and onError callback as option

RaSan147 · 2024-04-17T15:05:05Z

Well, I'm gonna miss motion(...., {sound})
it was a great option (since its optional feature, removing it is kinda feeling like a bad idea)
that would help retain certain posture and motion while speaking. Also the expression and many more things are missing from the PR_version...
😥

liyao1520 · 2024-05-08T06:07:31Z

期待，更新

RaSan147 · 2024-05-08T07:33:01Z

Gotta re-test and look for compatible way to shift from patch to official version

liyao1520 · 2024-05-09T10:46:19Z

#150

live2d official website The demonstration video of the model can flexibly display mouth movements, and the lip-syncing looks quite natural. Demo Video

In this example model, not only can the mouth opening be set based on audio information, but vowel mouth shapes can also be set by adjusting 'ParamA', 'ParamE', 'ParamI', 'ParamO', 'ParamU'.

model.internalModel.coreModel.setParameterValueById('ParamMouthOpenY', mouthY)
model.internalModel.coreModel.setParameterValueById('ParamA', 0.3)

I feel there might be better methods to achieve lip-syncing. Can the model be set to correspond to the mouth shape based on the audio?

Also, Alibaba Cloud's TTS can output the time position of each Chinese character/English word in the audio. How can the model play the audio, and can it set the corresponding mouth shape based on the phonetic information?

Seeking guidance from the experts! 🙏

RaSan147 · 2024-05-25T02:47:03Z

Finally it's ready to merge! Before I merge it, are there any changes you would like to make or suggest?

Whenever you're ready. Thanks for all your hard works

mrskiro

+1

xiaoqiang1999 · 2024-07-19T01:57:02Z

@guansss 大佬快合并吧，期待发包❤️

t41372 · 2024-07-27T21:04:43Z

Eagerly awaiting merge!

tegnike · 2024-12-28T09:50:48Z

@guansss No merge yet??

cacard · 2024-12-30T09:57:07Z

这个是支持官方的MotionSync吗？

RaSan147 · 2024-12-30T21:21:19Z

这个是支持官方的MotionSync吗？

Nope, its not the official image to motion sync (like the official live2d module) we just used voice signal amplitude to get the lips approximate position. Also guansss sensei did some internal changes with this code (i am too noob to understand them all) but made it really good. But seems sensei is a bit busy.

stevelizcano · 2025-01-09T15:12:26Z

This is great work. Is there an example in this PR using the changes you have? Going to try and integrate it if it's possible.

My goal is to use real time lip syncing with the OpenAI realtime API. Would be pretty cool, but not sure if possible.

tegnike · 2025-01-09T15:20:01Z

@stevelizcano

Here is fork with Lipsync from @RaSan147 https://github.com/RaSan147/pixi-live2d-display

And I made a character chat app includeing Realtime API by using this fork.
Please try it https://github.com/tegnike/aituber-kit

liyao1520 · 2025-01-10T02:50:00Z

This is great work. Is there an example in this PR using the changes you have? Going to try and integrate it if it's possible.

My goal is to use real time lip syncing with the OpenAI realtime API. Would be pretty cool, but not sure if possible.

I have published an npm package to implement live2d motionsync.

GitHub: https://github.com/liyao1520/live2d-motionSync
npm: https://www.npmjs.com/package/live2d-motionsync
Demo: https://liyao1520.github.io/live2d-motionSync/

RaSan147 · 2025-01-10T10:30:12Z

This is great work. Is there an example in this PR using the changes you have? Going to try and integrate it if it's possible.
My goal is to use real time lip syncing with the OpenAI realtime API. Would be pretty cool, but not sure if possible.

I have published an npm package to implement live2d motionsync.

GitHub: https://github.com/liyao1520/live2d-motionSync npm: https://www.npmjs.com/package/live2d-motionsync Demo: https://liyao1520.github.io/live2d-motionSync/

This is really interesting, I'd love to give it a try.

RaSan147 · 2025-01-10T10:34:26Z

This is great work. Is there an example in this PR using the changes you have? Going to try and integrate it if it's possible.

My goal is to use real time lip syncing with the OpenAI realtime API. Would be pretty cool, but not sure if possible.

There are examples at the middle of readme, you can use the model and use realtime generated voice output (google voice or open ai or edge tts) to do lipsync. Thats the basic, you can even do expressions if mentioned to AI on what motion+expression to show and parse it.
https://github.com/RaSan147/VoiceAI-Asuna/blob/main/src/page/script_bot.js
this project of mine, use almost realtime output.

DominicStewart · 2025-01-10T10:43:19Z

@stevelizcano

Here is fork with Lipsync from @RaSan147 https://github.com/RaSan147/pixi-live2d-display

And I made a character chat app includeing Realtime API by using this fork.

Please try it https://github.com/tegnike/aituber-kit

Does this allow you to do lip sync using streamed audio rather than just audio files? The problem with this branch, as I began exploring, is that this basically requires an audio file. Meaning audio is generated from text to speech models, if you want to do something with AI, and you'd need to essentially put it in a file and give it to this library. Being able to stream audio data to the library and have the lips move is necessary for most use cases

RaSan147 · 2025-01-10T13:04:17Z

@stevelizcano
Here is fork with Lipsync from @RaSan147 https://github.com/RaSan147/pixi-live2d-display
And I made a character chat app includeing Realtime API by using this fork.
Please try it https://github.com/tegnike/aituber-kit

Does this allow you to do lip sync using streamed audio rather than just audio files? The problem with this branch, as I began exploring, is that this basically requires an audio file. Meaning audio is generated from text to speech models, if you want to do something with AI, and you'd need to essentially put it in a file and give it to this library. Being able to stream audio data to the library and have the lips move is necessary for most use cases

i'll look into it... do you have any code that can feed a function the audio stream?

RaSan147 added 22 commits October 6, 2023 18:16

Automated report

d378339

Update index.js links

3ff3a53

Update test.yml

8539e76

Update test.yml

cf0f4c4

Update test.yml

f53e25a

Fix Lip sync. Breaking change

7deba12

voice volume expressions are now optional arg {name: value, ....}

Automated report

06bfbd3

Update README.md and added video

1e89254

Merge branch 'master' into master

34831e8

Fix type error on ci test

2d0c512

clear dist, using workflow to get output files

2617eae

removed Version number

de764a1

Now supports other b64 audio

2877f47

Remove audio url validation

25ddef6

rename speakUp -> speak

b4a394a

rename resetMotions -> stopMotions

6f3a779

return false when audio play fails

e878d79

Added crossorigin for speak voice source

031ca86

Place missing CrossOrigin arg

ae1c923

Update MotionManager.ts

c0cb7f5

Update readme as per PR

c8eca3f

guansss requested changes Dec 14, 2023

View reviewed changes

RaSan147 and others added 7 commits December 14, 2023 19:05

Check any base64 data for audio

7f8ec7a

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

Remove autoplay attr from audio

06d7255

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

Remove cache blocker.

96a6f82

Co-authored-by: Guan <46285865+guansss@users.noreply.github.com>

remove wav only blob condition

9fc5a21

also remove cache buster and autoplay

Merge branch 'guansss:master' into for_PR

b005f73

RAN npm run lint:fix

ff1a588

Merge branch 'for_PR' of https://github.com/RaSan147/pixi-live2d-display

0495f5b

into for_PR

guansss added 6 commits April 8, 2024 18:48

test: add aborted lip sync test

9cb5c38

test: make lip sync test more stable

f53509b

test: make lip sync test more stable (for real)

d69a771

feat: merge lipSync.analyze() into lipSync.getValue()

94a7ea1

test: temporarily disable parallelism to fix random failures

87476a4

revert: remove updateFacialEmotion method

718215d

RaSan147 commented Apr 17, 2024

View reviewed changes

DominicStewart approved these changes Jun 7, 2024

View reviewed changes

mrskiro approved these changes Jun 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Live2D with Lipsync (using audio file/link) #122

Live2D with Lipsync (using audio file/link) #122

RaSan147 commented Dec 14, 2023

guansss left a comment

guansss Dec 14, 2023

RaSan147 Dec 14, 2023

guansss Dec 15, 2023

RaSan147 Dec 15, 2023

guansss Apr 15, 2024

guansss commented Apr 15, 2024

RaSan147 commented Apr 17, 2024

RaSan147 commented Apr 17, 2024

RaSan147 Apr 17, 2024

RaSan147 Apr 17, 2024

RaSan147 Apr 17, 2024

RaSan147 commented Apr 17, 2024

liyao1520 commented May 8, 2024

RaSan147 commented May 8, 2024

liyao1520 commented May 9, 2024

RaSan147 commented May 25, 2024

mrskiro left a comment

xiaoqiang1999 commented Jul 19, 2024

t41372 commented Jul 27, 2024

tegnike commented Dec 28, 2024

cacard commented Dec 30, 2024

RaSan147 commented Dec 30, 2024

stevelizcano commented Jan 9, 2025

tegnike commented Jan 9, 2025

liyao1520 commented Jan 10, 2025

RaSan147 commented Jan 10, 2025

RaSan147 commented Jan 10, 2025

DominicStewart commented Jan 10, 2025

RaSan147 commented Jan 10, 2025

Live2D with Lipsync (using audio file/link) #122

Are you sure you want to change the base?

Live2D with Lipsync (using audio file/link) #122

Conversation

RaSan147 commented Dec 14, 2023

guansss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guansss commented Apr 15, 2024

RaSan147 commented Apr 17, 2024

RaSan147 commented Apr 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RaSan147 commented Apr 17, 2024

liyao1520 commented May 8, 2024

RaSan147 commented May 8, 2024

liyao1520 commented May 9, 2024

RaSan147 commented May 25, 2024

mrskiro left a comment

Choose a reason for hiding this comment

xiaoqiang1999 commented Jul 19, 2024

t41372 commented Jul 27, 2024

tegnike commented Dec 28, 2024

cacard commented Dec 30, 2024

RaSan147 commented Dec 30, 2024

stevelizcano commented Jan 9, 2025

tegnike commented Jan 9, 2025

liyao1520 commented Jan 10, 2025

RaSan147 commented Jan 10, 2025

RaSan147 commented Jan 10, 2025

DominicStewart commented Jan 10, 2025

RaSan147 commented Jan 10, 2025