Cannot run ONNX on GPU: process didn't exit successfully exit code: 0xc0000005, STATUS_ACCESS_VIOLATION

[Redflashx12]

Hey,
I am having a few issues when trying to make ONNX run on the GPU and I don't understand why and I wanted to ask if you have any ideas which might help me.

I already found the similar issue #44 but I tried to implement both fixes, but neither worked for me.

I already had a few issues setting up the GPU the last few days/weeks on Windows 11 22H2, but now both CUDA and TensorRT is recognized. I am running CUDA 11.8 with cuDNN 8.9.0 and TensorRT 8.6.1.6 on 2x RTX 4090s for my company.

My cargo.toml entry of ORT looks like this:
ort = { version = "1.16.3", features = ["load-dynamic", "cuda", "tensorrt"] }

I am creating my Yolov8 model like this:

        let env = Environment::builder()
            .with_name(MODEL_NAME)
            .with_execution_providers([
                ExecutionProvider::CUDA(CUDAExecutionProviderOptions {
                    ..CUDAExecutionProviderOptions::default()
                }),
                ExecutionProvider::CPU(CPUExecutionProviderOptions::default()),
            ])
            .build()?
            .into_arc();
        let model = SessionBuilder::new(&env)?
            .with_optimization_level(GraphOptimizationLevel::Level1)?
            .with_model_from_file(path)
            .with_context(|| "Could not load ONNX model from file")?;

When running with the CUDA Execution provider, after I exit the following postprocessing of Yolo, I get this exit code:

process didn't exit successfully: `C:\Users\BatterySort01\RustroverProjects\machine_software\target\debug\deps\machine_software-8e1c30de558d0dba.exe ai::onnx_model::tests::test_positive_example_with_onnx_model --exact --nocapture` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)

    (more postprocessing steps)...
    
    predictions.sort_by(|box1, box2| box2.confidence.total_cmp(&box1.confidence));
    let mut result = Vec::new();
    while !predictions.is_empty() {
        result.push(predictions[0]);
        predictions = predictions
            .iter()
            .filter(|box1| {
                BoundingBox::iou(&predictions[0].bounding_box, &box1.bounding_box) < IOU_THRESHOLD
            })
            .copied()
            .collect()
    }
    result

Apparently there are issues when trying to return the result object because that's where my program crashes. There is no code stack afterwards.

However, when I'm running on the CPU, the are no errors/crashes. Any ideas maybe? My System Path variables look like this:

The user Path is this:

Thank you.

Kind regards
Anton

[decahedron1]

Are you using session.run in a different thread than the one the session was created in (or using await)?

I had also ran into STATUS_ACCESS_VIOLATION when using multithreading CUDA & DirectML, but CPU was fine. Seems like the same issue here. MS has said in the past that ONNX Runtime sessions can be shared between threads, but I guess either due to a bug or new change, that is no longer the case with some EPs.

[Redflashx12]

No, neither sadly. session.run is called within the same thread and I currently don't do any multihreading in this regard. I also just tried to set with_intra_threads() to 1 but that did not fix it either. Any other ideas? Or is this just a bug in the onnx runtime from MS?

I just have this code on the same thread which executes all after each other

impl ObjectDetector for ONNXModel {
    fn detect(&self, input: &MultichannelData) -> Vec<Prediction> {
        let (input, img_width, img_height) = preprocessing(input);
        let output = self.run_model(&input);
        postprocessing(output, img_width, img_height)
    }
}

And in run_model, ONNXModel is executing this:

    pub fn run_model(&self, input: &Array<f32, IxDyn>) -> Array<f32, IxDyn> {
        let input_as_values = &input.as_standard_layout();
        let model_inputs =
            vec![Value::from_array(self.model.allocator(), input_as_values).unwrap()];
        let outputs = self.model.run(model_inputs).unwrap();
        trace!("Outputs: {:?}", outputs);
        trace!("Outputs shape: {:?}", outputs.len());
        let output = outputs
            .get(0)
            .unwrap()
            .try_extract::<f32>()
            .unwrap()
            .view()
            .t()
            .into_owned();
        output
    }

[decahedron1]

Any other ideas? Or is this just a bug in the onnx runtime from MS?

Yeah it's looking like it. The only other idea I had was a CUDA version mismatch, but you're using 11.8 which shouldn't have an issue.

Maybe try upgrading to ort 2.0.0-alpha.2? (migration guide)