Skip to content

Commit

Permalink
fix(MediaManager): use different HTTP streams for each variant
Browse files Browse the repository at this point in the history
- So that the variant's transform stream don't block each other by the stream backpressure mechanism, or does not block others when exits writing before reading the whole file.
  • Loading branch information
MrOrz committed Feb 28, 2024
1 parent 4b3b9a8 commit fd66c04
Showing 1 changed file with 5 additions and 2 deletions.
7 changes: 5 additions & 2 deletions src/MediaManager.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import fetch from 'node-fetch';
import { Bucket, Storage, File } from '@google-cloud/storage';
import { pipeline } from 'stream/promises';
import prepareStream from './lib/prepareStream';
Expand Down Expand Up @@ -160,9 +161,11 @@ class MediaManager {
// Must not await this promise because we still need to pipe body to another writable stream
// for hash calculation below
const uploadPromise = Promise.all(
variantSettings.map(({ transform, contentType }, i) =>
variantSettings.map(async ({ transform, contentType }, i) =>
pipeline(
body,
// Initiate separate HTTP stream here, as each variant may consume the stream differently,
// or even ends before reading the whole file.
(await fetch(url)).body,
transform,
tempVariantFiles[i].createWriteStream({ contentType, gzip: 'auto' })
)
Expand Down

0 comments on commit fd66c04

Please sign in to comment.