Module guide

DRAFT

Overview

Flux services are implemented as comms modules. Each module runs in its own thread, currently within the flux-broker process, although it is planned to allow modules to run in standalone processes.

RFC 5/Flux Module Extension Protocol describes Flux extension modules, the static symbols they must define, how they are loaded and unloaded, and how they synchronize with the broker when they start and stop.

RFC 3/CMB1 - Flux Comms Message Broker Protocol describes how request messages are routed to services, and response messages are routed back to the requestor.

This document will serve as a brief practical guide to the module writer.

Service Name

The module declares its own name with the MOD_NAME() macro. Normally the module name is also the service name (e.g. the topic string prefix for the service provided by the module), however an additional name may be registered with the MOD_SERVICE() macro, which can be used when modules of different names provide the same interface.

Module Main

The module provides one entry point with the following prototype:

int mod_main (flux_t *h, int argc, char **argv)

The Flux handle h represents a shared memory connection to the broker. It is created before mod_main() is run, and destroyed After mod_main() exits.

Several request message handlers are registered on the handle to provide service for methods common to all modules. The methods are:

• shutdown - call flux_reactor_stop(3)
• stat.get - respond with JSON object containing message counters, etc..
• stats.clear - clear the stats counters
• debug - set/clear module debug flags (see below)
• ping - respond to ping request from flux-ping(1).
• rusage - respond with getrusage(RUSAGE_THREAD) results

These message handlers may be overridden by registering another message handler with the same topic string, if desired. In particular, it may be useful to return a module-specific JSON object in repsonse to stats.get or to trigger some module-specific teardown in response to shutdown. Just be sure to call flux_reactor_stop(3) when it is complete.

Normally mod_main() registers some message handlers of its own and enters the reactor loop with flux_reactor_run(3).

The module should not call exit(3), or any functions that call it, as this terminates the broker. Instead, if a fatal error occurs in the module, it should call flux_reactor_stop_error(3), which causes flux_reactor_run() to return an error.

If flux_reactor_run() returns an error, mod_main() should return with a negative return code and errno set.

Module Template

A typical mod_main() looks similar to this one:

static struct flux_msg_handler_spec htab[] = {
    { FLUX_MSGTYPE_REQUEST,  "userdb.lookup", lookup, 0, NULL },
    { FLUX_MSGTYPE_REQUEST,  "userdb.addrole", addrole, 0, NULL },
    { FLUX_MSGTYPE_REQUEST,  "userdb.delrole", delrole, 0, NULL },
    { FLUX_MSGTYPE_REQUEST,  "userdb.getnext", getnext, 0, NULL },
    { FLUX_MSGTYPE_REQUEST,  "userdb.disconnect", disconnect, 0, NULL },
    FLUX_MSGHANDLER_TABLE_END,
};

int mod_main (flux_t *h, int argc, char **argv)
{
    int rc = -1;
    userdb_ctx_t *ctx;

    if (my_initialization (flux_t *h) < 0) {
        flux_log_error (h, "failed to initialize module");
        goto done;
    }
    if (flux_msg_handler_addvec (h, htab, ctx) < 0) {
        flux_log_error (h, "flux_msghandler_add");
        goto done;
    }
    if (flux_reactor_run (flux_get_reactor (h), 0) < 0) {
        flux_log_error (h, "flux_reactor_run");
        goto done_unreg;
    }
    rc = 0;
done_unreg:
    flux_msg_handler_delvec (htab);
done:
    return rc;
}

MOD_NAME ("userdb");

Loading/Unloading

Modules may be loaded and unloaded dynamically with flux-module(1).

The instance's rc1 script (running on rank 0) takes care of loading modules at instance startup. The rc3 script unloads them.

Topics to be covered later

• listing module status (flux module list)
• module debug flags (flux module debug)
• the disconnect message
• reactive programming constraints
• modules as processes
• overriding the stats method
• security (handling request by guest users)
• argument parsing