Et 4299 try onnx runtime

[krajewskiML]

Implements script responsible for running onnxruntime and tract comparison in the examples folder

[rustonaut]

This can be a dev-dependency examples, tests, benchmarks have access to dev dependencies.

[rustonaut]

depending internal details of OrtOwnedTensor<f32, _> and session.run(in_tensor)? the compiler could theoretically be able to deduce that output is not used session (besides costing time) has no side effects and then end up eliminating the while loop 🙃

practically this is impossible to happen as session will call C-FFI so the compiler doesn't know anymore that it doesn't have side effects etc.

still because of this you want to to pass unused output of benchmarks through std::hint::black_box (similar const input values, but we don't have them here)

libraries like criterion do that for you, but aren't supper trivial to use

[rustonaut]

Wrt. the issues with running it and so/dll loading:

The rust crate onnxruntime only provides rust bindings to the microsoft onnxruntime, it (for good reasons) does not bundle the runtime, nor does the runtime need to even be available to build it.

What that means is that the runtime must be preinstalled in the computer where you want to run it in a compatible version.

E.g. when build on my system libonnxruntime.so.1.7.0 i.e. ABI version 1.7.0 of the runtime.

[rustonaut]

Wrt. running it on ARM:

• it also needs to have the runtime pre-installed (luckily we don't try this on a phone)
• it might be easier to "just" build the example on the device
• alternatively you could consider cargo-cross or cargo-dinghy
  • e.g. if you ssh to the arm device this guid explains how to setup dinghy with ssh and then you can run cargo dinghy -d raspi run --example onnx_runtime and it will cross compile send it over ssh and feed back the output etc. (BUT the onnxruntime library from microsoft still needs to already be there compiled for ARM)

[rustonaut]

I think you can find a prebuild blob of the microsoft onnxruntime here: https://github.com/microsoft/onnxruntime/releases/tag/v1.15.1

only issue the rust bindings seem to be servely outdated, but it might still be ABI compatible idk. we would have to try out

[rustonaut]

was integrated in different PR AFIK