Not populating all values for tcp_flags

[jvedman]

On multiple instances, I only see tcp_flags values of RA or FA. When the field has A, FPA, or PA (the only ones I can reliably find on search) tcp_flags doesn't populate at all in Elasticsearch. An example search result file (csv) is attached based on the following search:
tags:dcSense AND direction:in AND src_ip:(192.168.0.* OR 192.168.1.* OR 192.168.10.* OR 192.168.11.* OR 192.168.14.* OR 192.168.15.) AND !(src_ip: (fe80 OR 127.0.0.1)) AND !(dest_ip:( 192.168.0.* OR 192.168.1.* OR 192.168.0.255 OR 192.168.1.255 OR 192.168.10.255 OR 192.168.11.255 OR 192.168.14.255 OR 192.168.15.255 OR 255.255.255.255)) AND !(tcp_flags:(RA OR FA))
Outbound-Blocked-Traffic.txt

I am using your pfSense grok and conf.d files (though I've made some minor modifications along the way, including to GeoIP tag outgoing traffic). All of that is working fine, I just am not getting the tcp_flags field to populate on values other than RA or FA.

[patrickjennings]

The match on tcp_flags is on the word boundaries. So it should match all of RA, FA, FPA, etc. The fact that you are seeing RA and FA, to me, seems like it's not an issue on the ELK side but on the pfSense side.

Are you sending all syslog events in pfSense or just a subset? Are you seeing any _grokparsefailure tagged messages in your indices?

[jvedman]

So, I am sending all of my traffic to logs, and I am not seeing _grokparsefailure tags on any of the traffic in question. I see the data (in the attached file, they are all PA or FPA) in the data message where it is expected to be, it just doesn't get tagged. But I'm with you in wondering if this is something weird in how pfSense is translating the actual data into human-readable format (maybe there's an ascii issue?) since the grok is for a pretty straight-forward word regex. I'll pursue with them as well.
All Clients - Outbound Blocked Traffic (pfSense).txt

Thank you, Patrick. Appreciate your attention!

[jvedman]

Oh, and because I'm curious, I'm wondering if you're seeing the same thing. If you want to check your own logs, look specifically for blocked outbound TCP traffic on your LAN interface(s).

I'm trying to parse out the causes for outbound traffic being blocked on the firewalls I manage. There are lots of possible reasons and it would be a lot easier if I could weed out based on tcp_flags... but... ;-)

[patrickjennings]

The data I have access to right now is from over a year ago. But I do see some documents for FA, RA, FPA, and SA.

I would be curious to see whether you can capture some raw syslog traffic from pfSense? Either by setting up a secondary output in logstash or tcpdump could be an option.

[jvedman]

Yes, I'm sure I can do that. It might take me a bit, but I'll get it as quickly as I can manage.