Skip to content

Commit

Permalink
Update text
Browse files Browse the repository at this point in the history
  • Loading branch information
pereanub committed Oct 2, 2024
1 parent a32546d commit 38cfd47
Showing 1 changed file with 3 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,15 @@ First, the NPU plugin checks if the following conditions are met:
* The model does not contain states.

If the **conditions are met**, the NPU plugin attempts to compile and execute the original model with batch_size forced to 1. This approach is due to current compiler limitations and ongoing work to improve performance for batch_size greater than one.
If the compilation is successful, the plugin detects a difference in batch size between the original model layout
and the transformed/compiled layout. Then it executes the following steps:
If the compilation is successful, the plugin detects a difference in batch size between the original model layout (with batch size set to N)
and the transformed/compiled layout (with batch size set to 1). Then it executes the following steps:

1. Internally constructs multiple command lists, one for each input.
2. Executes each command list for the proper offsets of input/output buffers.
3. Notifies the user of the completion of the inference request after all command lists have been executed.

This concurrency-based batching mode is transparent to the application. A single inference request handles all inputs from the batch.
While performance may be lower compared to regular batching, this mode provides basic batching functionality for use with older drivers
While performance may be lower compared to regular batching (based on native compiler support), this mode provides basic batching functionality for use either with older drivers
or when the model cannot yet be compiled with a batch size larger than one.

If the **conditions are not met**, the NPU plugin tries to compile and execute the original model with the given
Expand Down

0 comments on commit 38cfd47

Please sign in to comment.