API for event loop integration

[garlick]

As noted in flux-framework/flux-core#1543, flux_sign_wrap() and flux_sign_unwrap() may, depending on selected mechanism, internally perform synchronous communication with outside processes / daemons. For example, when configured to use MUNGE, both wrap and unwrap exchange messages with the munge daemon. When configured to use curve, unwrap performs a fork/exec/wait on the IMP, and exchanges messages via stdin/stdout.

It is desirable to allow the security API to be integrated into event driven programs, such as flux-core comms modules, without introducing blocking behavior, as this limits scalability and reduces responsiveness.

See the adio example in the libev manual for one approach.

[garlick]

libzmq uses a different approach which hides the library's file descriptors and other event sources behind a "mailbox fd" that becomes readable when the host event loop should unblock and test for events to process. I believe this requires the library to spawn a separate event loop in another thread that can unblock on its own, handle events, and then signal the mailbox. (libzmq does spawn this "service thread")

libzmq chose to "edge trigger" this mailbox, which creates some headaches for event loop integration. In flux-core we followed the pattern outlined in the following blogs to make it work:

https://funcptr.net/2012/09/10/zeromq---edge-triggered-notification/
https://funcptr.net/2013/04/20/embedding-zeromq-in-the-libev-event-loop/

[garlick]

Couple of notes from discussion yesterday plus some thoughts:

• MUNGE currently only allows one transaction per connection, but this is not necessarily an impediment to adding an async interface should we decide to do that.
• I think we stated that that an IMP handshake was required for the curve "unwrap" (signature verification). Reviewing that design, this turns out to be untrue. Only access to the IMP CA public key is needed to verify a signature
• I did look a bit at GSSAPI in case we wanted to build a signing mechanism on top of that, and it does separate the creation/parsing of message from the I/O so that it looks like it would be no problem to use in an event-driven context

Here's the GSSAPI ref since it's potentially another example of a "sufficient" API for event loop integration:
https://docs.oracle.com/cd/E19683-01/816-1331/sampleprogs-fig-14/index.html

In the GSSAPI example, the functions are split into halves that produce and consume buffers, and wire communication is left to the user, e.g. (error code dropped):

s = connect_to_server(host, port); // user-provided synchronous connect

// snip - other init details (no I/O on s though, pretty sure)

gss_wrap(&min_stat, context, 1, GSS_C_QOP_DEFAULT, &in_buf, &state, &out_buf);

send_token(s, &out_buf); // user-provided synchronous send of full buffer
recv_token(s, &out_buf); // user-provided synchronous recv of buffer until EOF

gss_verify_mic(&min_stat, context, &in_buf, &out_buf, &qop_state);

I'm not sure how you would build an API like that for a function like flux_sign() which in some cases is an RPC, and in other cases is local. Something like that might be viable for MUNGE though...