Slider Controlled/Uncontrolled Ideas

[OneHatRepo]

I have a music player. The Slider component normally animates showing progress through the song. But if the user wants to skip around, he grabs the slider and drags to a new location. Playback begins at that new point.

If the slider is controlled, the animation works fine.
If it is uncontrolled, the user can drag it and skip around in the track.

But how do I make it do both—be animated and draggable?
I have tried to make it controlled and then update it with onChange events, but the render doesn't happen fast enough. It lags woefully behind what the user is doing. I have tried switching it from controlled to uncontrolled, based on if the user is dragging, but the component won't go back to being controlled, and forever sits there. Any other ideas? Surely others have experienced this same problem.