Making logs to be read with VisiData

[cool-RR]

Hey,

I'm just setting up logging for my application. I thought that instead of logging to a plain text file, it'll be cool to log into a kind of file that I will be able to open with VisiData as tabular data. I could write a CSV log formatter and use that, but I wonder whether any of you already had that use case and could offer a better suggestion.

[saulpw]

The best way is "structured logging", which is just outputting a series of JSON dictionaries, one per event. If you output one per line, that becomes JSONL. In Python I would do it like this:

def log(**kwargs):
    import datetime
    import json
    with open('logfile.jsonl', 'a') as fp:
        d = dict(timestamp=datetime.datetime.now().isoformat())
        d.update(kwargs)
        print(json.dumps(d)+"\n", file=fp)

log(event="start_oven", deg_F=350)

This is an easy function to use to log at any point, and you can build specialized functions on top of it if you want.

[cool-RR]

That's nice, but then if you'd look at the log of your program as tabular data, it'll be a sparse table since different log messages will use different fields. Also, because you're reinventing logging, your log won't include the logs from all the libraries you call.
I'm now experimenting with using this: 

class JsonlLogFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            'timestamp': self.formatTime(record, self.datefmt),
            'level': record.levelname,
            'logger': record.name,
            'path': record.pathname,
            'line': record.lineno,
            'message': record.getMessage(),
        }
        return json.dumps(log_entry)

logging.config.dictConfig({
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {'jsonl': {'()': JsonlLogFormatter,},},
    'handlers': {'file': {'class': 'logging.FileHandler', 'filename': self.log_path,
                          'mode': 'w', 'formatter': 'jsonl'}},
    'root': {'level': logging.DEBUG, 'handlers': ['file']},
})

It appears to work fine.

[saulpw]

Glad I could help.