General issues with submodules

[MathijsdeBoer]

I've been playing around with Maturin and Pyo3 of a little while, and have gotten some neat things to work. Mainly, I want to use Pyo3 to move some of my math heavy Python code to Rust, so I can get speedups for processing. That is, a majorly Python project, with some aspects moved to Rust as opposed to a lot of the tutorials I can find assuming a majorly Rust project you can call from Python.

For example, my current project is based around doing a lot of ray/triangle intersections, which lends itself a bit more to a compiled language for speed. My first iteration, while functional, had a few misconfigurations that made working with it a bit of a hassle. Every time I wanted to use one of my structs, I'd have to type the entire slug to get there.

...
from evdkp import rs

def do_thing():
    ...
    mesh = rs.geometry.Mesh.from_file(str(mesh), num_samples=1_000_000)

While it would be much more ergonomic to be able to do

...
from evdkp.rs.geometry import Mesh
...

I've seen #759, which seems to be a pure Rust project, where the supermodule is the top-level package, but I'd like to use the Rust code as a submodule instead.

I've been making a few new projects with maturin, and got to being able to do

from linalg._rs import sum_as_string

def do_thing():
    print(sum_as_string(1, 2))

But when I then add a new Rust module as submodule:

linalg_test
| - python
    | - linalg
        | - py_module        <-- Straightforward python module that would be a sibling of `linalg._rs`
            | - __init__.py
        | - __init__.py          <-- Simple imports and a function to test
        | - _rs.pyi                <-- Works to type hint sum_as_string()
        | - py.typed
    | - tests
| - src
    | - submodule
        | - mod.rs
        | - some_struct.rs
    | - lib.rs
| - cargo.toml
| - pyproject.toml

And adding submodule to the top-level pymodule with

// submodule/some_struct.rs
#[pyclass]
pub struct SomeStruct {
    #[pyo3(get)]
    some_field: String,
}

#[pymethods]
impl SomeStruct {
    #[new]
    pub fn new(some_field: String) -> Self {
        Self { some_field }
    }
}

// submodule/mod.rs
#[pymodule]
pub fn submodule(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<SomeStruct>()?;
    Ok(())
}

// lib.rs
#[pymodule]
fn _rs(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
    m.add_wrapped(wrap_pymodule!(submodule::submodule))?;
    Ok(())
}

I can't do

from linalg.pymod import py_function
from linalg._rs import sum_as_string
from linalg._rs.submodule import SomeStruct

As it fails on from linalg._rs.submodule import SomeStruct with

ModuleNotFoundError: No module named 'linalg._rs.submodule'; 'linalg._rs' is not a package

But doing the thing I mentioned at the start:

from linalg.pymod import py_function
from linalg._rs import sum_as_string
from linalg import _rs

def do_thing(a, b):
    py_function()
    struct = _rs.submodule.SomeStruct("hello")
    print(f"{struct.some_field}")
    return sum_as_string(a, b)

does work. Somewhat. RustRover and PyCharm both yell at me for _rs.submodule not existing, while the code does run.

How could I make this work as expected? If at all. Googling the issue simply gives me a lot of techblogs that copy the basic template, and leave it at that, instead of diving in a little deeper.

---

Secondary question: How would I use type hints properly with submodules?

I know that the file setup, with py.typed and _rs.pyi in the root python package, described above works fine, as I got it from the official documentation, but I can't quite figure out how this would work if I get my submodules to work. The single _rs.pyi wouldn't also cover the _rs.submodule.SomeStruct type hinting, right?

[MathijsdeBoer]

I fixed the first issue using a later comment in #759, but it took me longer than I'd like to admit. Something like this really ought to be in the documentation without digging through issue comments

In short:

// ./src/lib.rs
use pyo3::prelude::*;

mod linalg;

#[pymodule]
fn _rs(_py: Python, m: &PyModule) -> PyResult<()> {
    linalg::init_linalg(_py, m)?;

    Ok(())
}

// ./src/linalg.mod.rs
mod vec3;

pub use vec3::Vec3;

use pyo3::prelude::*;

#[pymodule]
fn linalg(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<Vec3>()?;
    Ok(())
}

pub fn init_linalg(_py: Python, m: &PyModule) -> PyResult<()> {
    let linalg_child_module = PyModule::new(_py, "evdmeshlib._rs.linalg")?;
    linalg(_py, linalg_child_module)?;

    m.add("linalg", linalg_child_module)?;

    _py.import("sys")?
        .getattr("modules")?
        .set_item("evdmeshlib._rs.linalg", linalg_child_module)?;

    Ok(())
}

# ./python/evdmeshlib/__init__.py

from evdmeshlib._rs.linalg import Vec3


def test_vec3():
    v = Vec3(1, 2, 3)
    w = Vec3(4, 5, 6)

    print(v)
    print(w)
    print(v + w)


if __name__ == '__main__':
    test_vec3()

Unfortunately, the second issue, type hinting for submodules, is still unknown to me. RustRover is doing quite a bit of yelling at me for writing my Python files with this.

[davidhewitt]

Sorry for the slow reply on my side, am dealing with a sick family. Yes it really should be in documentation, the only reason I've never got round to that is because I keep hoping to work on #3294 and make this all nicer. That's not a good reason not to put up documentation for the current state of the world though. Sorry.

I'm not at a computer right now, but I think last time I tested with a jetbrains IDE, it liked type hints in the following structure:

./python/evdmeshlib/
     _rs/
        __init__.pyi  // put the root module type hints here
        linalg.pyi  // put the submodule type hints here
     __init__.py

But I may have remembered incorrectly. Do you want to try that and let me know how that looks?

[MathijsdeBoer]

No worries, rereading it I realize my wording unintentionally came across as a little crass, for which I apologize. I got a little frustrated in my own inability to figure this out. Hope your family will get better!

Re: Type hints:
I thought I had tried that, and I'm fairly sure I did. But that does seem to pick it up now. Strangely enough, putting the .pyi files next to their associated .rs files also seems to do something towards making type hints available, but it seems a little unreliable.

[nleroy917]

Something that ended up working for me was doing two things: 1) creating a python source directory with the same name as my package at the root, and then 2) for each submodule, creating a subfolder within that source folder with both an __init__.py and an __init__.pyi. file.

Inside __init__.py I simply write:

from module.submodule import *

Inside __init__.pyi I will create the type stubs:

def function(param1: str, param2: int) -> int:
    """
    Function description
    """

class SomeBigClass:
    def __init__(self, param1: str):
        """
        Describe how to instantiate a new instance
        """
    def class_fn(self, param1: int) -> int:
        """
        Doc string for class_fn
        """

So, in the end, my directory looks something like this:

.
├── module
│   ├── __init__.py
│   ├── __init__.pyi
│   ├── submodule1
│   │   ├── __init__.py
│   │   └── __init__.pyi
│   └── submodule2
│       ├── __init__.py
│       └── __init__.pyi
├── src
│   ├── lib.rs
│   ├── submodule1
│   │   └── mod.rs
│   └── submodule2
│       └── mod.rs
├── Cargo.toml
├── pyproject.toml
└── README.md

[austin362667]

Can "adding type hints for submodules" work when using declarative modules?