Timestretching

[spamatica]

Hi Tim,

Tried your time stretching the other day and it works to some extent, yay :)

Not sure what to expect, if it actually can be used yet? I noticed it does not allow undo/redo at least.
As it is in the wave editor I guess it will be destructive edits, the actual wave file will be rewritten? Though I noticed afterwards that the wave file I was trying it on seemed unchanged...

Technically, how does it work?
And what do you see is currently missing?

[terminator356]

Nope. No destructive editing. The original wave is untouched and no copies are made.
It's all 'live' resampling and time stretching.
And if you 'bounce' to a final track, it should be correct, great care was taken there.
As I mentioned to oscillator, probably the more important thing is the automatic
non-destructive 'live' resampling. You can open any wave without having to make a resampled copy,
(you are given a choice). Also if you open your project at a different system samplerate
than when it was created, automatic live samplerate converters kick in. The project sounds normal
at any samplerate.

The lack of undo for actual stretch marker manipulations is an unfortunate oversight.
I can get around to adding that. But there's nothing to 'undo' for the actual wave
because nothing is destroyed. When you remove all stretch markers, the wave simply
goes back to normal !

What is missing at the moment is pitch shift markers. It has that ability, just need to figure out
how to expose it and provide user manipulation in the UI.
And of course the much bigger picture down the road is automatic tempo adjustment of wave parts,
and ultimately wave parts would become in units of ticks rather than frames, so that
when tempo is changed, the parts do not move, they stay in place just like our midi parts,
and the stretching kicks in and adjusts the wave accordingly.
This is ultimately how MusE should be, from the beginning, but when MusE was born we
didn't have these technologies to let us do that. So here we currently have wave parts that
move when tempo is changed. Not ideal.

[terminator356]

Further: I noticed that editing a MusE wave in an external editor seems to be broken -
even before my changes. I tested on an older release and it seems broken.
Does not seem to work at all. Need to test more...

So here we currently have wave parts that move when tempo is changed. Not ideal.

Recently I fired up old Cakewalk version 6 from around '98.
I was quite surprised that even back then, Cakewalk stretches the waves when tempo is changed !
But... It does it destructively - it operates on the entire wave as you wait for several seconds.
That's each and every time you change tempo.
Again, the technology did not exist to do 'live' stretching efficiently back then.
I suppose... We could do a similar thing in MusE, if the user so desires.
That is, even without this new 'live' stretching ability, we could do something similar
to old Cakewalk and stretch the waves destructively while the user waits.
That would get us to the point of 1998! Wave parts would not move with tempo changes...

[spamatica]

I suppose... We could do a similar thing in MusE, if the user so desires. That is, even without this new 'live' stretching ability, we could do something similar to old Cakewalk and stretch the waves destructively while the user waits. That would get us to the point of 1998! Wave parts would not move with tempo changes...

Lol, yeah, that's progress! We are practically already in the future! There's one reason I can think of, to commit the stretch to file, that would be cpu usage. But a better way might be to add the ability to freeze tracks. Anyway, thanks for the insight, I need to play more with it now, seems very usable! If that is how it works it should serve quite well without undo/redo. —

…

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/orgs/muse-sequencer/teams/musedevteam/discussions/24/comments/2?email_source=notifications&email_token=ABCFANZR2ZRTPVK46I2MLBTRBGXGZA5CNFSM4KPYKLHKYY3PNVWWK3TUL52HS4DFWNCGS43DOVZXG2LPNZIG643UKJSXA3DZVJRW63LNMVXHIX3JMTHAAAVLNM>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCFAN22TOBJIUKJJX27A73RBGXGZANCNFSM4KPYKLHA> .

[terminator356]

There's one reason I can think of, to commit the stretch to file, that would be cpu usage.

Yes, of course all this 'live' stuff takes CPU power.
But... all of this processing takes place in the disk thread !
By the time the realtime thread gets around to using the data, it has already been processed !
That's why it seems to work so well, with little impact on the realtime DSP usage.
(That reminds me, I must look at slightly bumping up the disk thread priority from
our current lowest possible setting to one that can throttle with heavy live processing.)

However, the provision is there for these converters to be used directly in the realtime thread if needed.
My thinking there, was planning for synchronization of audio with incoming external sync midi clock pulses.
It would need to respond instantly to changes in midi clock tempo, something that
cannot be done in the disk thread.
If we do this processing in the realtime thread, hey, that's when you'll see DSP usage go up...

[terminator356]

Not to ramble on, but I know you love audio, and this stuff is cool, so here are some crucial things:

1. I needed to inform the user if the project samplerate is different from the current system rate,
   and that live converters will be used if the rates are different.
   Prior to about 2012 our song file had no project samplerate xml entry. In order to accomplish this,
   I needed to 'guess' what the project's samplerate was. Thus was born a new set of "project discovery"
   functions which act exactly as our actual xml loading routines, but fake it and only look for
   information about the contained wave files and their samplerates. Then I inform the user
   "based on the wave files in the song it appears the project samplerate was ... yada yada...
   do you want to use that rate or select another rate or the current system rate... yada yada..."
   [Edit] You might ask "Why is project samplerate so important when wave files have an
   intrinsic samplerate value, and together with system samplerate that is all that is required
   to find a correction ratio?" True. But other things don't have such an intrinsic rate value -
   like our audio automation graphs. A project samplerate is important so that we can
   know what to do with such data.
2. You may well ask "Why not immediately write to file instead so that we can use the
   highest possible quality setting, which would otherwise slow down any such live converters?".
   But look closely. There are three quality settings for every converter type.
   Also, there are global settings in Settings, and there are local overrides for every
   wave event inside a wave part, accessible from their right-click context menus in the wave editor.
   The three quality settings groups are: GUI, Real time, and Offline:

• GUI is for anything drawing related. It is on the lowest quality by default.
• Real time is for listening - I'll refer to it as "auditioning" (thinking of changing the name from "Real time"),
  it's the actual sound you hear as you play through a song that you are arranging.
  It is on medium quality by default.
• Offline is for what I'll refer to as "rendering". When you finally 'bounce' to a final track.
  It is on the highest possible quality setting.
  (The GUI and Auditioning converters are actually two separate instances.)

Thus, with these quality settings capabilities, there is in effect no need to write to file
immediately because the GUI and Audition settings are (and can be) quite low to minimize
impact on performance. (But Rubberband is still notoriously a pig at low settings.)
Then, when you bounce to a track, the full quality of the Offline settings kicks in !

3. I am working to improve it. Currently there's no clone considerations - markers don't
   move in clones of a wave. Currently each Wave Event has a stretch graph, this is not good
   if a wave part contains multiple wave events, ie. from splitting a part, I need to move the
   graph maps up into the wave part so that it affects all the events as a whole.
   I have started to work on fixes but it became quite complex, I've got a branch where I
   introduce streaming classes so that the converters and normal plugins can be chained together.

4. It is possible to offload regular realtime audio (rack) plugin processing from the
   realtime thread to the disk thread. Just like these converters. IOW If you have a reverb
   on your wave track's rack, that reverb can be offloaded to the disk thread.
   It's very tricky but possible in theory. (One reason I'm trying to add streaming classes (above)).
   Recall that Florian did something similar, and in fact we have a branch he created called
   'audio streams'. I believe that was exactly what he was talking about. Do you remember?
   It would actually be quite easy if only considering Wave tracks and their effect racks.
   But it's tricky because I never use plugins in a Wave track! I feed all my wave tracks to Group tracks,
   and all my plugins are in the Group track !
   Thus I would need to make it really smart where it looks at all track types to see if it can offload
   any of the processing, not just wave tracks. It depends on what is routed and where, of course.

5. [Edit] Did you know that the metronome sounds were never resampled?
   For example if you ran MusE at 96K, the metronome sounded terriible - pips and squeaks.
   I fixed that. I run all the sounds through my converters at the highest quality Offline setting,
   once at startup. (Don't worry there's no live converters slapped on them at all times.)
   The metronome now sounds correct at any samplerate.