Feat/add record button and playback button in sampler widget

[Mubashirshariq]

fixes #3806

1)Added record button to record the audio in the browser itself
2)Added the playback button to play the recorded audio

[Mubashirshariq]

@walterbender @pikurasa i have added recording and playback buttons to the sampler widget. Could you please review and suggest any further changes needed?

[walterbender]

It sort of kind of works.
A couple of suggestions: (1) give some indication that recording is in progress; (2) give some indication that recording has stopped; (3) the wave form did not change for me (or maybe the amplitude was so small that I couldn't see it? the audio level seemed fine); (4) maybe the play button is disabled if there is nothing recorded yet?; (5) maybe as a separate MR, but it would be great to be able to crop the sound (start and end positions) in the widget.

[Mubashirshariq]

@walterbender sir I have made the changes as suggested by you

[walterbender]

I am not seeing any UX changes with your latest commit: I don't see the micorphone state change; the playback button status changing; or any signal on the waveform.

[Mubashirshariq]

@walterbender what I have done is that when the recording starts I am displaying a msg "recording started" and when I stop the recording the msg is displayed "recording complete" and I am disabling the playback button whenever there is no audio url
and regarding waveform I think the amplitude is too small to see but still I will look into it again

[walterbender]

I think it would be a better UX to change the button icon to show that recording has started/stopped...

[Mubashirshariq]

sure,will do that

[Mubashirshariq]

@walterbender could you please review it i have updated it.

[walterbender]

The UX is improved. But I don't seem to be able to use the sample in my code. The play button in the widget doesn't play my sample either. And the wave forms are still not working.

[Mubashirshariq]

Sure, will address the playback and waveform display issues

[Mubashirshariq]

@walterbender i have added the recorded audio sample to the list, and now the waveform is also getting displayed.could you please review it

[pikurasa]

It seems to now play the sample via the widget, but it doesn't yet seem to be something that can be used in a user's code.

@Mubashirshariq did you test it? Does it work within a user's code for you?

[pikurasa]

This is my test:

sampler-test.webm

In the video:

1. Pressing play for reference pitch, which did not seem to work the first time. Unsure why...
2. Recording (you cannot hear my voice, because it is only recording audio from the output of the computer)
3. Playback the recorded sound. This works! (Except the amplitude is not shown in visualization. This should be checked into.)
4. Pressing play for reference pitch, which worked the second time.
5. Re-recording the audio
6. Saving
7. Putting new sample into set instrument block
8. Putting notes into set instrument generated by sampler
9. Running the code

This represents a typical workflow

[Mubashirshariq]

@walterbender @pikurasa i was going through the code i noticed one thing, do we need to record the audio in .wav format ,because while uploading file we are checking that if the file format is .wav then only we are adding that sample .I think that might be the reason for waveform of the recorded file not being displayed

[pikurasa]

@walterbender @pikurasa i was going through the code i noticed one thing, do we need to record the audio in .wav format ,because while uploading file we are checking that if the file format is .wav then only we are adding that sample .I think that might be the reason for waveform of the recorded file not being displayed

It may be that it must be in .wav format, but I'm not 100% certain.

Here are the instructions for adding a sample: https://github.com/sugarlabs/musicblocks/tree/master/sounds/samples which does not mention mp4 format, if that is helpful.

@walterbender Can you confirm, please?

[walterbender]

I think that the file must be WAV in the current implementation.

[Mubashirshariq]

@walterbender @pikurasa sir could you please review it now i have used webm2wav converter to convert audio/webm to .wav ,now i am able to see the waveform and also now i can use it in my code

[walterbender]

Works for me. But I still don't see anything on the waveform/oscilloscope output.

[Mubashirshariq]

@walterbender i am able to see the waveform i am sharing the below video for reference ,maybe the amplitude of your waveform was too small

screen-capture.1.webm

[walterbender]

@pikurasa do you see activity on the waveform?

[pikurasa]

@pikurasa do you see activity on the waveform?

No, I don't.

I am currently testing on a3fe921

[pikurasa]

Also, watching that video, I'm confused. I would expect to see waveform activity soon after the record button was pushed. This is because it should be showing mic activity as the mic is recording.

[Mubashirshariq]

@pikurasa now how it is working is like first it records the audio then the audio sample is added to our list ,then whenever click play button it displays the waveform.Do you want it like during recording itself the waveform should be displayed

[pikurasa]

Do you want it like during recording itself the waveform should be displayed

If that's not too much trouble, yes, that would be best.

Then the person recording has visual feedback as to how well the mic is getting the input, as well as whether the levels are too low or too high.

[Mubashirshariq]

sure will do that

[github-actions]

This pull request has been open for more than 60 days without any activity. It will be closed in 3 days unless the stale label is removed or commented on.

[pikurasa]

@Mubashirshariq It would be nice to land this. Do you have any particular questions, or do you need feedback on anything?

[Mubashirshariq]

@pikurasa the issue I am facing is showing the waveform simultaneously while recording itself. I am not able to figure that out. However, once the recording is completed, pressing play button shows the waveform. If I try to use it in the code, it works well

[pikurasa]

@pikurasa the issue I am facing is showing the waveform simultaneously while recording itself. I am not able to figure that out. However, once the recording is completed, pressing play button shows the waveform. If I try to use it in the code, it works well

Perhaps we don't need to have that feature for now.

But we should probably put some indication that recording is happening.

[Mubashirshariq]

@pikurasa i am attaching few screenshots for your reference
1)Initially when the mic is off we have this symbol for mic

2)Once we hit the mic ,the symbol of mic is getting changed and also on the top msg is being displayed that recording started and playback button is disabled.

3)And when we hit the mic again the symbol changed back to default state we are showing the msg recording complete
and playback button is enabled.

4)Once i hit the play button i am able to see the waveform as follows

5)And once i click "save sample" then i am able to use it in my code as follows

[pikurasa]

I finally came around to testing this again, and I have some feedback:

1. The "reference tone" feature doesn't seem to work until after you've recorded some audio. This is a bit awkward as the user would want to hear the pitch first to mimic it.
2. There seems to be something that is causing all the "set instrument" blocks in a sequence to perform using the sample. You can see it in the recording linked below, but "set instrument = guitar" then "set instrment = sample" are both being performed as "set instrument = sample"; I'm including logs as well.
3. It may be worth exporting the new sample together with the set instrument block.

127.0.0.1-1722299531560.log
sampler-widget-testing.webm

[pikurasa]

@nathannnie this branch would be a good one for you to experiment with.

[pikurasa]

@Mubashirshariq Please address the few remaining improvements to the implementation of this.

[Mubashirshariq]

@pikurasa working on it!

[pikurasa]

@Mubashirshariq any updates on this?

Otherwise, we should probably just merge it, and add a feature request as an issue for live visual feedback of audio input.

[Mubashirshariq]

@pikurasa I was busy with college work that is why I didn't had time to do this,but I think we can merge this and create a separate Ticket for live waveform

[Mubashirshariq]

@walterbender I have resolved the merge conflict,I have created a separate ticket for live waveform we can merge this one and do the rest of thing in that issue.