DNS-Flags in Clickhouse?

[tomarcade]

Hi all

Logging to STDOUT captures all the appropriate DNS flags:

Nov 13 07:17:12 resolver go-dnscollector[2595738]: {"network":{"family":"IPv4","protocol":"UDP","query-ip":"192.168.1.100","query-port":"51165","response-ip":"192.168.1.1","response-port":"53","ip-defragmented":false,"tcp-reassembled":false},"dns":{"length":54,"id":60280,"opcode":0,"rcode":"NOERROR","qname":"microsoft.com","qclass":"IN","qdcount":1,"ancount":0,"nscount":0,"arcount":1,"qtype":"TXT","flags":{"qr":false,"tc":false,"aa":false,"ra":false,"ad":true,"rd":true,"cd":false},"resource-records":{"an":[],"ns":[],"ar":[]},"malformed-packet":false},"edns":{"udp-size":1232,"rcode":0,"version":0,"dnssec-ok":0,"options":[{"code":10,"name":"COOKIE","data":"-"}]},"dnstap":{"operation":"CLIENT_QUERY","identity":"resolver","version":"dnsdist 1.9.6","timestamp-
rfc3339ns":"2024-11-13T06:17:10.76926877Z","latency":0,"extra":"-","policy-rule":"-","policy-type":"-","policy-match":"QNAME","policy-action":"NXDOMAIN","policy-value":"-","peer-name":"@","query-zone":"-"}}

Nov 13 07:17:12 resolver go-dnscollector[2595738]: {"network":{"family":"IPv4","protocol":"UDP","query-ip":"192.168.1.100","query-port":"51165","response-ip":"192.168.1.1","response-port":"53","ip-defragmented":false,"tcp-reassembled":false},"dns":{"length":70,"id":60280,"opcode":0,"rcode":"NOERROR","qname":"microsoft.com","qclass":"IN","qdcount":1,"ancount":0,"nscount":0,"arcount":1,"qtype":"TXT","flags":{"qr":true,"tc":true,"aa":false,"ra":true,"ad":false,"rd":true,"cd":false},"resource-records":{"an":[],"ns":[],"ar":[]},"malformed-packet":false},"edns":{"udp-size":1232,"rcode":0,"version":0,"dnssec-ok":0,"options":[{"code":10,"name":"COOKIE","data":"-"}]},"dnstap":{"operation":"CLIENT_RESPONSE","identity":"resolver","version":"dnsdist 1.9.6","timestamp-rfc3339ns":"2024-11-13T06:17:10.769545507Z","latency":0,"extra":"-","policy-rule":"-","policy-type":"-","policy-match":"QNAME","policy-action":"NXDOMAIN","policy-value":"-","peer-name":"@","query-zone":"-"}}

Sending dnstap captured traffic to Clickhouse pushes only the following fields into the database (example for a query):
POST /?Query=INSERT%20INTO%20dns.records(identity,queryip,qname,operation,family,protocol,qtype,rcode,timensec,timestamp)

Is there a way, to send the whole output (incl. FLAG fields) to Clickhouse for further analysis? If yes, how should the table scheme in Clickhouse must be look like?

[dmachard]

Hi,

The current ClickHouse logger is basic and not efficient under high load. To improve flexibility and performance, the logger needs the following updates:

• Support for JSONEachRow insertion format
  

• Batch insert functionality
  

• A more structured data model
  

Could you please open an issue to track this enhancement?

Thanks,
Denis

[tomarcade]

Done: #877

[n5ke]

Hi, I would also love to see these improvements. In the meantime a decent workaround I used is to pipe the json into Vector (I used transport: unix and a unix socket for this), perform any transformations there and then sink it into Clickhouse or any other of the storage backends it supports (fluentd is also an option of course, I just found Vector simpler, lighter and faster)