[Performance] Python inference runs faster than C++

[ashwin-999]

Describe the issue

Observe that ONNX model (FP32) executed in C++ runs slower than Python. it's much much worse with TensorRT execution provider.

I've tried exporting a F16 model with keep_io_types set to True. That isn't better either, infact worse.

But first want to talk about the F32 model in question.

To reproduce

Python script:

import onnxruntime
import numpy as np
import time

def test_run_onnx_model(model_path, input_shape, num_iterations=1):
    print(f"Available providers: {onnxruntime.get_available_providers()}")
    sess_options = onnxruntime.SessionOptions()
    # sess_options.log_severity_level = 0  # 0 = Verbose
    # sess_options.log_verbosity_level = 1  # Increase verbosity
    
    providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']
    
    session = onnxruntime.InferenceSession(model_path, sess_options, providers=providers)

    input_name = session.get_inputs()[0].name

    dummy_input = np.ones(input_shape, dtype=np.float32)

    session.run(None, {input_name: dummy_input})

    total_time = 0

    for _ in range(num_iterations):
        start_time = time.time()
        output = session.run(None, {input_name: dummy_input})
        end_time = time.time()
        total_time += (end_time - start_time)

    average_time = total_time / num_iterations
    print(f"Average inference time over {num_iterations} iterations: {average_time:.4f} seconds")

if __name__ == "__main__":
    model_path = "epoch103.nms.fp32.simplified.onnx"
    input_shape = (1, 6, 640, 640) 

    test_run_onnx_model(model_path, input_shape, 1000)

C++:

Model::Model(Ort::Env* env, const std::string& model_path){
    loadModel(env, model_path);
}

void Model::loadModel(Ort::Env* env, const std::string& model_path) {

    Ort::SessionOptions session_options;

    session_options.SetIntraOpNumThreads(1);
    session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL); 
    session_options.EnableMemPattern();
    session_options.EnableCpuMemArena();

    // CUDA Execution Provider
    OrtCUDAProviderOptions cuda_options{};
    cuda_options.device_id = 0; 
    cuda_options.arena_extend_strategy = 0;
    cuda_options.cudnn_conv_algo_search = OrtCudnnConvAlgoSearchExhaustive;
    cuda_options.do_copy_in_default_stream = 1; 
    session_options.AppendExecutionProvider_CUDA(cuda_options);

    // // TensorRT Execution Provider
    // OrtTensorRTProviderOptions trt_options{};
    // trt_options.device_id = 0;
    // trt_options.trt_max_workspace_size = 1 << 30;
    // trt_options.trt_max_partition_iterations = 1000;
    // trt_options.trt_min_subgraph_size = 1;
    // trt_options.trt_fp16_enable = true;
    // trt_options.trt_int8_enable = false;
    // trt_options.trt_engine_cache_enable = true;
    // trt_options.trt_engine_cache_path = "/dummy/path";
    // session_options.AppendExecutionProvider_TensorRT(trt_options);

    auto providers = Ort::GetAvailableProviders();
    std::cout << "Available providers: ";
    copy(providers.begin(), providers.end(), std::ostream_iterator<std::string>(std::cout, " "));
    std::cout << std::endl;
    
    session_ = std::make_unique<Ort::Session>(*env, model_path.c_str(), session_options);
    
    // Get input names and dimensions
    Ort::AllocatorWithDefaultOptions allocator;
    size_t num_input_nodes = session_->GetInputCount();
    input_node_names_.resize(num_input_nodes);
    input_node_dims_.resize(num_input_nodes);

    for (size_t i = 0; i < num_input_nodes; i++) {
        auto input_name = session_->GetInputNameAllocated(i, allocator);
		input_node_name_allocated_strings_.push_back(std::move(input_name));
        input_node_names_[i] = input_node_name_allocated_strings_.back().get();

        Ort::TypeInfo input_node_info = session_->GetInputTypeInfo( i );
        auto tensor_info = input_node_info.GetTensorTypeAndShapeInfo();
        input_node_dims_[i] = tensor_info.GetShape();

    }

    auto input_shape = input_node_dims_[0];
    for (auto dim : input_shape) {
        input_tensor_size *= dim;
    }

    
    // Get output names and dimensions
    size_t num_output_nodes = session_->GetOutputCount();
    output_node_names_.resize(num_output_nodes);
    output_node_dims_.resize(num_output_nodes);

    for (size_t i = 0; i < num_output_nodes; i++) {
        auto output_name = session_->GetOutputNameAllocated(i, allocator);
		output_node_name_allocated_strings_.push_back(std::move(output_name));
        output_node_names_[i] = output_node_name_allocated_strings_.back().get();
        
        Ort::TypeInfo output_node_info = session_->GetOutputTypeInfo( i );
        auto tensor_info = output_node_info.GetTensorTypeAndShapeInfo();
        output_node_dims_[i] = tensor_info.GetShape();
    }

    warmUpModel(10);

}

void Model::runInference(const std::vector<float>& input, std::vector<SomeObject>& detections) {

    Ort::Value input_tensor = Ort::Value::CreateTensor<float>(memory_info_, 
                                                        const_cast<float*>(input.data()), 
                                                        input.size(), 
                                                        input_node_dims_[0].data(), 
                                                        input_node_dims_[0].size());

    auto output_tensors = session_->Run(Ort::RunOptions{nullptr}, 
                                        input_node_names_.data(), &input_tensor, 1, 
                                        output_node_names_.data(), 1);
    
    const float* output_data = output_tensors[0].GetTensorData<float>();
    auto output_shape = output_tensors[0].GetTensorTypeAndShapeInfo().GetShape();
    
    num_detections_ = output_shape[0];

    detections.reserve(num_detections_);

   // code to populate detection hidden. (confirmed this piece isn't slow.. a for loop that populates detections) 

}

void Model::warmUpModel(int num_iterations) {
    
    std::vector<float> dummy_input(input_tensor_size, 0.0f);
    std::vector<SomeObject> dummy_detections;
    auto start = std::chrono::high_resolution_clock::now();
    for (int i = 0; i < num_iterations; ++i) {
        runInference(dummy_input, dummy_detections);
    }
    auto end = std::chrono::high_resolution_clock::now();
    auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();
    // print duration
}

Both Python and C++ run the same model, with inference being run 1000 times in C++ just as in Python.

• Python: 42.5ms/iter on average
• C++ (with OrtTensorRTProviderOptions commented out): 47.8ms/iter on average
• C++ (with OrtCUDAProviderOptions commented out): 86.0ms/iter on average

I suspect it's the large input copying between CPU <-> GPU that's slowing things down. I hate the fact that I am initializing Ort::Value input_tensor every single time. Can someone help me with binding I/O? I searched quite extensively online, and there's quite a few examples for Python and the ones I run into for C++ either don't compile or segfaults or gives me 0 detections. The output node is a dynamic node.. shape/size is not pre-determined.

Here are a few questions that I have in particular:

1. Would you say I/O is the primary suspect?
2. Beyond I/O what else could I be doing to improve the runtime?
3. Why is TensorRT being so slow?

for some context, all of the runtimes provided are on Jetson ORIN, being run from inside a docker container.

Urgency

No response

Platform

Linux

OS Version

22.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.20.0, commit id: e91ff94

ONNX Runtime API

Python

Architecture

ARM64

Execution Provider

Default CPU, CUDA, TensorRT

Execution Provider Library Version

CUDA Version: 12.2

Model File

No response

Is this a quantized model?

No