custom threadsafe sink hanging

[keithlostracco]

I've created a custom sink to log data from multiple threads to a Python output but I think I must have something wrong because it will hang under certain conditions. I thought I should be using spdlog::basic_logger_mt as it states in the wiki but it doesn't appear available in the namespace.

Here is my definition of the sink and a function that initializes logging:

template<typename Mutex>
class PythonSink : public spdlog::sinks::base_sink<Mutex>
{
protected:
    void sink_it_(const spdlog::details::log_msg& msg) override
    {
        spdlog::memory_buf_t formatted;
        spdlog::sinks::base_sink<Mutex>::formatter_->format(msg, formatted);
        auto str = fmt::to_string(formatted);

        nb::gil_scoped_acquire acquire;
        nb::print(str.c_str());
    }

    void flush_() override
    {
        nb::gil_scoped_acquire acquire;
        std::cout << std::flush;
    }
};

void init_logging()
{
    auto python_sink = std::make_shared<PythonSink<std::mutex>>(); 
    auto logger = std::make_shared<spdlog::logger>("python_logger", std::move(python_sink));
    logger->set_pattern("[%M:%S] [%t] [%l]: %v");
    spdlog::register_logger(logger);
}

nb is nanobind, a Python bindings library (similar to nanobind). nb::print() invokes the Python print function.

Am missing something here?

[gabime]

it doesn't appear available in the namespace.

What do you mean? something is wrong in your setup.

Will it also hang “nb::gil_scoped_acquire acquire;” is removed?

[keithlostracco]

There must be something not right with my setup, I used vcpkg in msvc and just noticed it was 1.12 and not 1.13 which I thought was odd. I'll clone the repository manually.

Will it also hang “nb::gil_scoped_acquire acquire;” is removed?

It will crash if I don't acquire the GIL. In a console I could just use std::cout and output directly from the cpp side but I need to output within a Jupyter notebook in which case I need to acquire the GIL.

[gabime]

but I need to output within a Jupyter notebook in which case I need to acquire the GIL

Why? is it multi threaded?

[keithlostracco]

Yes there are different implementations (this one is in vscode) but they each have their own event loop, where a Python process is either running a block or waiting for the user to run the next block. It's basically an interactive python session where you can write and execute and update cells of code.

In my case I need to support Jupyter for the library I'm developing, it's starting to get hard not being able log in Jupyter directly.

Thanks for super quick response btw.

[keithlostracco]

So not finding basic_logger_mt was just a dumb mistake. I didn't realize it was a file logger and wasn't including the right header...

Relating to the threading issue. It's feeling like a pretty tricky problem to navigate. I'm thinking it will be cleaner and less problematic if I just output to log files directly on the cpp side and avoid the Python Global Interpreter Lock and any other complications that could arise in any given Python environment.

Thanks for the quick feedback on this.