A demonstration of different RxJS methods for retrieving BLOBs (PNG files) via the Angular HttpClient API.
Sequential retrieval is demonstrated using the RxJS concatAll operator. Sequential means the retrieval of one BLOB does not commence until the retrieval of the previous BLOB has fully completed.
Concurrent retrieval is demonstrated using the RxJS mergeAll operator. Concurrent means the retrieval of many BLOBs can overlap and does not necessarily wait for the previous BLOB to complete. Concurrent retrieval can overload the server that is sending the BLOBs by asking for too many at once... The mergeAll operator accepts a parameter that sets a maximum limit on the number of concurrent retrievals active at any given time.
Computer networks are unreliable: Even with the 'reliable' TCP/IP protocol a request to a server can fail to complete for many reasons. This demonstration shows how the RxJS retry operator can be used in conjunction with the RxJS mergeAll operator to reattempt failed BLOB retrievals. For a more detailed discussion of retry tactics see Angular University's "RxJs Error Handling: Complete Practical Guide".
Important Caveat: This demonstration shows a difficult and overly complex method to load image PNG files into an Angular application. This is intentional for this demonstration so that the handling of BLOBs can be demonstrated.
Normally you would use <img> HTML tag as shown here in this stackoverflow question to display PNG image files.
My original use case for concurrent retrieval was to load GZip'ed BLOBs of latitude/longitude coordinates that described the polygon outlines of geographical features. YMMV!
The whole shebang depends on having Node.js installed... I'm running:
$ node -v
v12.18.4See the GitHub concurrent-http-server repository.
The concurrent-http-server project is a simple Express http server that returns a fixed set of 228 PNG images.
There are three primary REST endpoints and a PNG file BLOB retrieval endpoint.
- A REST endpoint that returns an array of strings enumerating the PNG filenames.
- A REST endpoint that allows the server to delay the return of the BLOBs in the PNG BLOB endpoint up to one second to augment the visual aspect of the demonstration as well as show the advantages of concurrent retrieval over sequential retrieval. It accepts a number from 0-1000 milliseconds of delay.
- A REST endpoint that that enables a random "503 Server Unavailable" failure in the PNG BLOB retrieval endpoint to allow experimenting with retries. It accepts a number from 0-100 representing the percentage of random failures. Set it to zero to disable random failures.
$ git clone https://github.com/krystalmonolith/concurrent-http-server.git
$ cd concurrent-http-server
$ npm install
$ npm startIf it is running correctly you should see:
$ npm start
concurrent-http-server listening at http://localhost:3000
Once the concurrent-http-server Express server is running, the concurrent-http Angular application can be installed and run as follows:
$ git clone https://github.com/krystalmonolith/concurrent-http.git
$ cd concurrent-http
$ npm install
$ npm startGenerating the following output:
$ npm start
> concurrent-http@1.2.0 start
> ng serve
✔ Browser application bundle generation complete.
Initial Chunk Files | Names | Size
vendor.js | vendor | 4.02 MB
polyfills.js | polyfills | 128.51 kB
styles.css | styles | 77.82 kB
main.js | main | 55.92 kB
runtime.js | runtime | 6.63 kB
| Initial Total | 4.28 MB
Build at: 2021-09-01T19:33:41.631Z - Hash: 0dac07b335f7f4582e3d - Time: 9779ms
** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **
✔ Compiled successfully.
As the above output says: Open the page at http://localhost:4200/
** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **
You should be presented a page that is initially blank with a title and three tabs: Click on any tab to initiate retries of the 228 PNG files from the server.
- When clicked the "RxJS concatAll()" tab demonstrates sequential retrieval of the 228 PNG files.
- When clicked the "RxJS mergeAll()" tab demonstrates concurrent retrieval of the 228 PNG files without any retries.
- When clicked the "RxJS mergeAll() with retry" tab demonstrates concurrent retrieval of the 228 PNG files with retries.
Above the images are two numeric spinners that control how the server responds:
-
The "Server Random File %" spinner controls the percentage of the PNG file requests that fail with a "503 Service Unavailable" error.
- Set it to zero to disable random failures.
- Set to a low number (1%-10%) to see some retrieval before a PNG file request failure occurs.
- Set it higher to really screw up the PNG file requests.
-
The "Server File Delay (msec)" spinner sets the delay the server introduces before responding to a PNG file request.
- Set it to zero to disable the server PNG file request delay.
- It defaults to 50 msec to best show the difference between sequential and concurrent retrievals.
- Crank it up to 100 msec to see how much better concurrent retrieval handles long server computational delays.
I started with an Angular CLI generated application with routing, then I added the Angular Material schematic.
Top level component AppComponent (app.component.ts & app.component.html ) contains the Angular Material <mat-tab-group> component and the router outlet. When a <mat-tab> is clicked the router outlet displays one of:
ConcatComponent( concat.component.ts & concat.component.html )MergeComponent( merge.component.ts & merge.component.html )MergeRetryComponent( merge-retry.component.ts & merge-retry.component.html )
ConcatComponent, MergeComponent,MergeRetryComponent are all composed of ImageGridComponent ( image-grid.component.ts & image-grid.component.html ) which uses an Angular Material <mat-grid-list> component to display the images.
ImageGridComponent is composed of a ParameterFormComponent ( parameter-form.component.ts & parameter-form.component.html ) and uses <mat-progress-bar> to display the retrieval progress.
ConcatComponent, MergeComponent,MergeRetryComponent all contain delegate functions to perform the actual retrieval of the images via functions ConcatComponent.loadImagesConcatAll, MergeComponent.loadImagesMergeAll, and MergeRetryComponent.loadImagesMergeAllWithRetry respectively. These delegate functions are invoked as part of ImageGridComponent.ngOnInit invoking ImageGridComponent.loadList.
ImageGridComponent.loadList shown below uses FileService.getFileList service function to enumerate the file names of the image files, then passes the list of files to the loadImageDelegate to perform the actual image retrievals via the delegate functions noted above.
loadList(): void {
// Clear out the old images and error information.
this.images = [];
this.imageCount = 0;
this.imagePercentage = 0;
this.imageFetchError = undefined;
this.imageFetchErrorFilename = undefined;
this.elapsedTimeMsec = 0;
// Fetch the list of files from FileService.
this.fileService.getFileList()
.subscribe(fileList => {
// Call the image loading delegate function to attempt to load all the images in the 'fileList'.
this.imageCount = fileList.length;
this.loadStartTime = Date.now();
this.loadImageDelegate!(fileList, this, this.pushImage).subscribe({
error: err => {
this.imageFetchError = `${err['status']} ${err['statusText']}`;
this.imageFetchErrorFilename = err['url'];
}
});
});
}
Image retrieval delegate function ConcatComponent.loadImagesConcatAll shown below uses the RxJS concatAll operator to perform sequential retrieval. It returns an 'outer' RxJS Observable that when subscribed uses the RXJS from function to create a stream of Observable... One Observable for each file name in the file list.
The RxJS Observable.pipe function then takes each file name Observable and:
- First uses the RxJS
mapoperator to transform the file nameObservableinto a PNG BLOBArrayBufferObservablevia theFileService.getFileservice function. - Second uses the RxJS
concatAlloperator to sequentially stream each PNG BLOBArrayBufferto thepipefunction's subscriber.
The output of the pipe operator is then subscribed to push the PNG BLOB into the <mat-grid-list>, and chain any errors out the 'outer' Observable error handler, or log a message to the Console if everything 'completes' successfully.
(The imagePusher callback is responsible for encoding the PNG BLOB to BASE64 before inserting it into the array backing the <mat-grid-list>. See ImageGridComponent.pushImage)
loadImagesConcatAll(fileList: Array<string>,
imageGridComponent: ImageGridComponent,
imagePusher: (imageGridComponent: ImageGridComponent, fileResponse: ArrayBuffer) => void): Observable<void> {
return new Observable(subscriber => {
from(fileList)
.pipe(
map((file: string) => this.fileService.getFile(file)),
concatAll() // SEQUENTIAL !!!
)
.subscribe(
(fileResponse: ArrayBuffer) => imagePusher(imageGridComponent, fileResponse),
(err: any) => subscriber.error(err),
() => console.log(`Image loading via concatAll() COMPLETE!`)
);
});
}
}
Image retrieval delegate function MergeComponent.loadImagesMergeAll shown below uses the RxJS mergeAll operator to perform concurrent retrieval.
loadImagesMergeAll(fileList: Array<string>,
imageGridComponent: ImageGridComponent,
imagePusher: (imageGridComponent: ImageGridComponent, fileResponse: ArrayBuffer) => void): Observable<void> {
return new Observable(subscriber => {
from(fileList)
.pipe(
map((file: string) => this.fileService.getFile(file)),
mergeAll(MergeComponent.CONCURRENT_GET_COUNT) // PARALLEL !!!
)
.subscribe(
(fileResponse: ArrayBuffer) => imagePusher(imageGridComponent, fileResponse),
(err: any) => subscriber.error(err),
() => console.log(`Image loading via mergeAll() COMPLETE!`)
);
});
}
MergeComponent.loadImagesMergeAll operates similarly to ConcatComponent.loadImagesConcatAll with two differences:
- It uses the RxJS
mergeAlloperator to concurrently stream each PNG BLOBArrayBufferto thepipefunction's subscriber. - The RxJS
mergeAlloperator accepts the constant numeric parameterMergeComponent.CONCURRENT_GET_COUNTthat limits the maximum number of concurrent retrievals to avoid overloading the server.
Image retrieval delegate function MergeRetryComponent.loadImagesMergeAllWithRetry shown below uses the RxJS mergeAll operator in conjunction with the RxJS retry operator to perform concurrent retrieval that can tolerate network and server failures.
loadImagesMergeAllWithRetry(fileList: Array<string>,
imageGridComponent: ImageGridComponent,
imagePusher: (imageGridComponent: ImageGridComponent, fileResponse: ArrayBuffer) => void): Observable<void> {
return new Observable(subscriber => {
from(fileList)
.pipe(
map((file: string) => this.fileService.getFile(file).pipe(retry(MergeRetryComponent.RETRY_COUNT))),
mergeAll(MergeRetryComponent.CONCURRENT_GET_COUNT) // PARALLEL !!!
)
.subscribe(
(fileResponse: ArrayBuffer) => imagePusher(imageGridComponent, fileResponse),
(err: any) => subscriber.error(err),
() => console.log(`Image loading via mergeAll() with retry() COMPLETE!`)
);
});
}
MergeRetryComponent.loadImagesMergeAllWithRetry operates similarly to MergeComponent.loadImagesMergeAll with two differences:
- The retrieval of the PNG BLOB is automatically retried via the
pipe(retry(MergeRetryComponent.RETRY_COUNT))appended to theFileService.getFilecall inside the RxJSmapoperator. - The RxJS
retryoperator accepts the constant numeric parameterMergeComponent.RETRY_COUNTthat limits the maximum number of retrievals per PNG BLOB to avoid retrying forever.
(This is an example of a inner pipe within a map operator within an outer pipe... Moving the retry pipe to after the outer pipe and before the subscribe call retries all the concurrent retrievals.)
This project was generated with Angular CLI version 12.2.1.
Hint: The Angular CLI is installed using the npm utility which is indirectly related to the Node.js project, i.e. when you install Node.js the npm utility is also installed.
Run ng serve for a dev server. Navigate to http://localhost:4200/. The app will automatically reload if you change any of the source files.
Run ng generate component component-name to generate a new component. You can also use ng generate directive|pipe|service|class|guard|interface|enum|module.
Run ng build to build the project. The build artifacts will be stored in the dist/ directory.
Run ng test to execute the unit tests via Karma.
Run ng e2e to execute the end-to-end tests via a platform of your choice. To use this command, you need to first add a package that implements end-to-end testing capabilities.
To get more help on the Angular CLI use ng help or go check out the Angular CLI Overview and Command Reference page.