GrayLog extractor for pfsense 22.01

Hi All,
Im having problems getting certain items into fields when searching for messages.
My extractor that I am using is either Toms from his videos or some I found in the MarketPlace and when I try to validate the regular expression it fails.
For example, taken from the marketplace: ^filterlog:\s+.,(in|out),4,.,tcp,.*$
I get the error Does not match! Extractor would not run.

The Log Message Format in pfsense is set to syslog (RFC5424)

Here are the ones I am using for the firewall logs but I know some thing’s from the other log facilities doesn’t line up and just get’s stored unparsed.

{
  "extractors": [
    {
      "title": "pfSense filterlog: IPv4 TCP",
      "extractor_type": "regex",
      "converters": [
        {
          "type": "csv",
          "config": {
            "trim_leading_whitespace": false,
            "column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCPFlags,Sequence,ACK,Window,URG,Options",
            "strict_quotes": false
          }
        }
      ],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "FilterData",
      "extractor_config": {
        "regex_value": "^.*filterlog\\[.....\\]:(.*)$"
      },
      "condition_type": "regex",
      "condition_value": "^.*filterlog\\[.....\\]:(.*),(in|out),4,.*,tcp,.*$"
    },
    {
      "title": "pfSense filterlog: IPv4 UDP",
      "extractor_type": "regex",
      "converters": [
        {
          "type": "csv",
          "config": {
            "trim_leading_whitespace": false,
            "column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength",
            "strict_quotes": false
          }
        }
      ],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "FilterData",
      "extractor_config": {
        "regex_value": "^.*filterlog\\[.....\\]:(.*)$"
      },
      "condition_type": "regex",
      "condition_value": "^.*filterlog\\[.....\\]:(.*),(in|out),4,.*,udp,.*$"
    }
  ],
  "version": "4.2.7"
}

appreciate the quick feedback. still not parsing correctly for me . screenshot is what I’m getting. protocol is not a field available. Are you running the same version of psense as I have noted? Im thinking maybe there’s a way the logs or structured in the newer update

I have my log type set to BSD 3164

Once I switch to BSD and away from syslog, messages stop showing up in graylog.

edit. to be clear, the Network IO metric s for the local inputs does show messages coming in but there are no more searchable messages.

i somewhat see the problem. ‘filterlog’ is not showing up in the example message. seeing how I cant force it in there I’m going to probably assume its the way pfsense is now sending syslog data is different from prior versions.

solved the problem.
The input must be put in as “Raw/Plaintext UDP” and not Sysylog UDP.

Still don’t know if its the way GrayLog now parses or how pfsense creates the data. Either way that was the fix.

edit: Switching to Raw Plaintext for all my inputs has fixed the stream problems as well where even if I select the stream to use it would still show me all messages from all devices.

@LTS_Tom your video was/is a great starter video to this. I have around 50 or so devices pointing to it with 4 cores and 8GB of memory. Absolutly no issues. Future replacement for Splunk?!? Lets find out.