GrayLog extractor for pfsense 22.01

Software & Web Applications
1

Hi All,
Im having problems getting certain items into fields when searching for messages.
My extractor that I am using is either Toms from his videos or some I found in the MarketPlace and when I try to validate the regular expression it fails.
For example, taken from the marketplace: ^filterlog:\s+.,(in|out),4,.,tcp,.*$
I get the error Does not match! Extractor would not run.

The Log Message Format in pfsense is set to syslog (RFC5424)

2

Here are the ones I am using for the firewall logs but I know some thingā€™s from the other log facilities doesnā€™t line up and just getā€™s stored unparsed.

{
  "extractors": [
    {
      "title": "pfSense filterlog: IPv4 TCP",
      "extractor_type": "regex",
      "converters": [
        {
          "type": "csv",
          "config": {
            "trim_leading_whitespace": false,
            "column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCPFlags,Sequence,ACK,Window,URG,Options",
            "strict_quotes": false
          }
        }
      ],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "FilterData",
      "extractor_config": {
        "regex_value": "^.*filterlog\\[.....\\]:(.*)$"
      },
      "condition_type": "regex",
      "condition_value": "^.*filterlog\\[.....\\]:(.*),(in|out),4,.*,tcp,.*$"
    },
    {
      "title": "pfSense filterlog: IPv4 UDP",
      "extractor_type": "regex",
      "converters": [
        {
          "type": "csv",
          "config": {
            "trim_leading_whitespace": false,
            "column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength",
            "strict_quotes": false
          }
        }
      ],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "FilterData",
      "extractor_config": {
        "regex_value": "^.*filterlog\\[.....\\]:(.*)$"
      },
      "condition_type": "regex",
      "condition_value": "^.*filterlog\\[.....\\]:(.*),(in|out),4,.*,udp,.*$"
    }
  ],
  "version": "4.2.7"
}
3

appreciate the quick feedback. still not parsing correctly for me . screenshot is what Iā€™m getting. protocol is not a field available. Are you running the same version of psense as I have noted? Im thinking maybe thereā€™s a way the logs or structured in the newer update

image
image2403Ɨ607 54.5 KB

4

I have my log type set to BSD 3164

5

Once I switch to BSD and away from syslog, messages stop showing up in graylog.

edit. to be clear, the Network IO metric s for the local inputs does show messages coming in but there are no more searchable messages.

6

i somewhat see the problem. ā€˜filterlogā€™ is not showing up in the example message. seeing how I cant force it in there Iā€™m going to probably assume its the way pfsense is now sending syslog data is different from prior versions.

image
image1832Ɨ913 101 KB

7

solved the problem.
The input must be put in as ā€œRaw/Plaintext UDPā€ and not Sysylog UDP.

Still donā€™t know if its the way GrayLog now parses or how pfsense creates the data. Either way that was the fix.

edit: Switching to Raw Plaintext for all my inputs has fixed the stream problems as well where even if I select the stream to use it would still show me all messages from all devices.

@LTS_Tom your video was/is a great starter video to this. I have around 50 or so devices pointing to it with 4 cores and 8GB of memory. Absolutly no issues. Future replacement for Splunk?!? Lets find out.