Rust function that accepts a Python Iterable?

[shner-elmo]

Hello, I want to make a Rust function that accepts a Python iterable (list, tuple, generator, etc.) of tuples: Iterable[tuple[str, str]].

I was using this code in v0.19 of PyO3, but now its not working anymore:

fn python_iterable_to_iterator(maybe_iterable: &PyAny) -> &PyIterator {
    maybe_iterable
        .call_method0("__iter__")
        .unwrap()
        .downcast::<PyIterator>()
        .unwrap()
}

fn add_keywords_from_iter(&mut self, words: &PyAny) {
    let iter = python_iterable_to_iterator(words)
        .iter()
        .unwrap()
        .map(|py_obj| py_obj.unwrap().extract::<(&str, &str)>().unwrap());

    self.inner.add_keywords_from_iter(iter); // this takes a Rust Iterator
}

Any suggestions are welcome

[shner-elmo]

I guess I could've just done this:

fn add_keywords_from_iter<'py>(&mut self, words: Bound<'py, PyIterator>) {
    Python::with_gil(|_| {
        let iter = words
            .iter()
            .unwrap()
            .map(|py_obj| py_obj.unwrap().extract::<(String, String)>().unwrap());
        self.inner.add_keywords_from_iter(iter);
    })
}

But I'm not sure if the use of the GIL token is correct

[jamestwebber]

I would be interested in learning how to do this too, it would be really nice for creating Rust functions that feel more natural in Python.

[shner-elmo]

This is how I did it:
https://github.com/shner-elmo/flashtext2/blob/0f4e1db7176874af40bf9395ed6e753f8a1fca4d/src/lib.rs#L71-L81
https://github.com/shner-elmo/flashtext2/blob/0f4e1db7176874af40bf9395ed6e753f8a1fca4d/flashtext2.pyi#L10

(my comment above is wrong, this accepts bothIterable[T] and Iterator[T]. When you call .iter() it will convert it to an iterator if its not already

[shner-elmo]

@jamestwebber check the message above

[jamestwebber]

Thanks for that example, I was able to use that to get started and implement what I wanted. 🎉

One edge that I'm still a little unhappy with is the typing (in Rust). I find myself calling extract a lot to get the values out of my Python iterator. I'm wondering if there's a better way to specify the function so that PyO3 knows the signature and I don't need as much boilerplate.

I have methods that look like:

fn add_edges(&mut self, edges: Bound<'_, PyAny>) -> PyResult<()> {
    for py_obj in edges.iter()? {
        let (x, y) = py_obj?.extract::<(usize, usize)>()?;
        self.add_edge(x, y);
    }
    Ok(())
}

It'd be slick to have something more like

fn add_edges(&mut self, edges: Bound<'_, [something more specific here?]>) -> PyResult<()> {
    for (x, y) in edges.iter() {
        self.add_edge(x, y);
    }
}

Maybe I'm overthinking this, the current version is already good. But I was curious if there's a better way.

[birkenfeld]

That would be hard since an iterator in Python can yield a different type on each iteration. There is no way to check that except during the loop, after it has yielded something.

[jamestwebber]

That makes sense! The version I have works fine. I just wanted to make sure I wasn't missing out on any PyO3 magic.