Fix perf when streams don't change often (#17767)

Fix perf when streams don't change often (#17767) #825