Logback appender for ØMQ message transport layer

This project must be used with the Logback project in order to produce log messages over ZeroMQ messaging library.

This way, the application logs can be sent to remote systems / can be receive by remote systems in any language supported by ØMQ.

Installation

Requirements

Before installing and using this appender with Logback, you need to install ZeroMQ Java Binding library (2.1) on your system. This binding uses native library to work.

The ZeroMQ website is a great place to start installing the libraries, specially the Java binding page

If you want to develop or modify this plugin, I encourage you to read the ØMQ – The Guide which is very well documented.

You must download slf4j-api.jar, logback-core.jar and logback-classic.jar on their respective project pages SLF4J and Logback .

Installation

Put the following jar in the classpath of your Java application:

slf4j-logback-zeromq-0.0.1.jar

You must also put in the classpath the ØMQ Java binding compiled for your system:

jzmq-2.1.0.jar

Finally, put the SLF4J and Logback jars in the classpath:

slf4j-api-1.6.4.jar
logback-core-1.0.0.jar
logback-classic-1.0.0.jar

Configuration

The configuration of the ØMQ appender is done in the usual Logback configuration file

The appender must be declared in the logback*.xml file:

	<appender name="zeromq-bind" class="org.tlrx.logback.appender.ZMQSocketAppender">
		<!-- Define the ØMQ socket type -->
		<type>PUB</type>
		<!-- Only one of bind+connect must be declared -->
		<bind>tcp://*:9797</bind>
		<connect>tcp:localhost:9700</connect>
		<encoder>
			<!-- Message format -->
			<pattern>%-5level %logger{26} - %msg</pattern>
		</encoder>
	</appender>

Where:

Property	Description
<type>	Define the ØMQ socket type, value can be: PUB, SUB, REQ, XREQ. The default value is set to PUB
<bind>	The ØMQ socket will connect to the given address using bind method. Can be used only if socket is PUB or SUB type
<connect>	The ØMQ socket will connect to the given address using connect method
<pattern>	The pattern defines the format of the ØMQ message that will be sent by the appender. The format used the same PatternLayout of the default Logback ConsoleAppender
<highWaterMark>	A high water mark for the socket, after which messages will be discarded (if they’re not sent). This prevents infinite buffering (and ergo infinite RAM use) if the thing you are sending to is down. Defaults to 10000 messages

Examples of use

Publish log messages over TCP

Description

In this example, the log messages will be encapsualted in a ØMQ message and published by the appender on a given address.

Remote applications (subscribers) can connect to the ØMQ publisher socket and start receiving messages. The subscribers can also filter by log level the messages they want to receive.

Configuration

The configuration of the Logback appender defines a Publisher socket:

 <appender name="zeromq-bind" class="org.tlrx.logback.appender.ZMQSocketAppender">
        <type>PUB</type>
        <bind>tcp://*:9797</bind>
        <encoder>
            <pattern>%-5level %logger{26} - %msg</pattern>
        </encoder>
    </appender>

At runtime

Once started, the appender sends log messages like this:

DEBUG org.eclipse.jetty.util.log - doSelect true

Remote application can now connect to the publisher address to receive log messages. The ZMQSocketAppenderTest class gives an example of how to connect to the publisher and receive only ERROR messages.

If no subscribers is connected, the publisher will drop the message (this is a default ØMQ behavior).

Index log messages in Elasticsearch

Description

In this example, the ØMQ appender is configured to send log messages to the ØMQ transport plugin for Elasticsearch

This way, it is possible to index and search log messages with Elasticsearch features.

Configuration

The configuration of the Logback appender defines a XREQ socket:

<configuration>
...
    <appender name="zeromq-connect" class="org.tlrx.logback.appender.ZMQSocketAppender">
        <type>XREQ</type>
        <connect>tcp://localhost:9700</connect>
        <encoder>
            <pattern>POST|/logs/log/|{"origin":"my_app", "level":"%-5level", "message":"%msg", "thread":"%thread", "date":"%d{HH:mm:ss.SSS}", "logger":"%logger{26}", "exception":"%ex"}</pattern>
        </encoder>
    </appender>
...
</configuration>

The ØMQ transport plugin for Elasticsearch listen by default on the <tcp://*:9700> address.

In this configuration, the pattern for log messages reflects the format expected by the Elasticsearch plugin.

At runtime

Once started, the appender sends log messages to the Elasticsearch plugin, which index them in the logs index.