Anyone using Graylog and UNIFI ZBF?

youorme · May 9, 2025, 2:05am

After updating my UDM Pro to Zone Based firewalls, graylog only gets the triggers, I dont get the nice source/destination detailed firewall logs anymore.

Is anyone else seeing the same thing?

LTS_Tom · May 9, 2025, 4:38pm

Under the system settings then Activity Logging (Syslog) you need to have the “Firewall Default Policy” checked to get all the firewall data.

youorme · May 9, 2025, 8:16pm

Thanks Tom, I did verify that I have Default Firewall Policy set in the System → Traffic Logging → Activity Logging (Syslog).

Its weird for sure, If I try to hit my gateway over 443, I see the trigger in the UDMP gui that says blocked by firewall rule X. I see the same trigger in Graylog but I never get the detailed source X was blocked to destination Y over port 443.

Any suggestions? Thanks!

youorme · May 13, 2025, 12:20am

One last question, would you say your video from 2 years ago is still good with graylog 6.2? Thanks again

LTS_Tom · May 13, 2025, 10:27am

Yes, as I note in the video make sure you are pulling the latest compose file from my github GitHub - lawrencesystems/graylog: Graylog 6 setup which I keep up to date. Current version right now is 6.2.

youorme · May 13, 2025, 12:53pm

Thanks, im pretty sure my setup prior to graylog 6.2 is the root cause of the issue. I also updated graylog right after the ZBF update. I deleted all of my inputs, indicies, streams, pipelines and im starting from scratch again.

youorme · May 13, 2025, 2:06pm

well, i just figured it out. The triggers coming in were on my time zone while the network details were UTC timezone. I would get the triggers but the details were 5 hours ago. I only found this after starting from scratch. My user in graylog has my local timezone, i had to change the input to my local timezone too.

Not sure if there’s a better way but hopefully that helps someone.

LTS_Tom · May 13, 2025, 2:32pm

It should set it based on the time zone settings in my docker compose at lines 66 and 67

GRAYLOG_TIMEZONE: "America/Detroit"
TZ: "America/Detroit"

Having them in there twice is not a mistake, it fixes the issue you described.

youorme · May 15, 2025, 2:13am

Thanks, I originally changed it in my account but not in the server config. I saw in their youtube tutorial they do set the local tz in the config too. thx

youorme · May 16, 2025, 2:56am

Just a few more questions if you don’t mind:

Whats the benefit of storing the full message vs just using the message?
Do you prefer to parse the message into fields before doing any transformations or adding context to the data?

Thanks!

LTS_Tom · May 16, 2025, 10:57am

I store the full message in case something isn’t parsed properly I can look at it and don’t do an transformations or adding context.

youorme · May 16, 2025, 11:53am

ok thanks, so ill set my grok pattern on the message and not the full message. How are you selecting the firewall entry for parsing, are you using the description field? I was going to initially do a Regex for the full pattern, just looking for other suggestions. thanks again!

LTS_Tom · May 17, 2025, 10:36am

I need to build new extractors, I use ChatGPT to build them. If you do get new ones done feel free to post them here or a link to where you have them posted.

youorme · May 18, 2025, 12:30am

This is for UNIFI custom firewall rules and it works well. I cant say I came up with it all, I figured some out, took parts from multiple posts I found online and GROK AI for the rest.

%{DATA:device_name} \[(%{DATA:firewall_rule})(%{POSINT:firewall_number})\] DESCR="%{DATA:description}" IN=%{DATA:interface_in} OUT=%{DATA:interface_out} MAC=%{MAC:source_mac}:%{MAC:destination_mac}:%{GREEDYDATA:ethernet_header} SRC=%{IP:src_ip} DST=%{IP:dst_ip}.*?PROTO=%{WORD:protocol} SPT=%{NUMBER:src_port} DPT=%{NUMBER:dst_port} %{GREEDYDATA:additional_info}

LTS_Tom · May 21, 2025, 2:47pm

I made a new extractor that you can copy and paste into Graylog

github.com/lawrencesystems/graylog_extractors

UniF_version_9_AP_Events_and_Firewall

main

{
  "extractors": [
    {
      "title": "UniFi AP Events",
      "extractor_type": "grok",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "unifi_ap_event",
      "extractor_config": {
        "grok_pattern": "%{DATA:access_point} %{DATA:unifi_device_mac_address}(?:,%{DATA:ap_model})?: %{DATA:stahtd}: %{DATA:stahtd_process}\\[%{INT:stahtd_id}\\]: %{DATA:stahtd_event}: %{GREEDYDATA:json_data}"
      },
      "condition_type": "regex",
      "condition_value": "stahtd"
    },
    {
      "title": "UniFi Firewall Log",
      "extractor_type": "grok",
      "converters": [],

This file has been truncated. show original

michmoor · May 21, 2025, 2:55pm

nice, thanks Tom. You’re amazing.
I will say i have tried to have OpenAI create the extractors for me and it always fails. Whats your method? What do you ask it to do? I have a few more messages i need to split into fields. For example, i noticed if you ar eusing custom zones, the extractor doesnt work well with that. LAN_WAN is perfect tho.

LTS_Tom · May 21, 2025, 3:11pm

I started by giving it my old UniFI extractors which was not fully working with 9.1 so it had some context of my goals and built the rest with this prompt:

Please update the extractor with the improvements suggest and also here are some example firewall logs form UniFI 9.1 that also need to be parsed. Please parse the relevant fields and here is a guide to help understand the data “LTS-Studio” is the source system name. "[CUSTOM1_WAN-A-2147483647] " and “[CUSTOM1_LOCAL-A-2147483647]” are firewall rule names, “PROTO” refers to the protocol used “SRC” is the source IP “DST” is the destination IP. let me know If you need further information to parse this and create the updated extractor

And then I gave it some sample logs

LTS-Studio [CUSTOM1_WAN-A-2147483647] DESCR="[CUSTOM1_WAN]Allow All Traff" IN=br10 OUT=eth8 MAC=9c:05:d6:68:53:f0:3e:0b:2e:39:d1:82:08:00 SRC=192.168.100.64 DST=104.16.97.215 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=8940 DF PROTO=TCP SPT=41336 DPT=443 SEQ=1900301826 ACK=0 WINDOW=64240 SYN URGP=0 MARK=1a0000

LTS-Studio [CUSTOM1_LOCAL-A-2147483647] DESCR="[CUSTOM1_LOCAL]Allow All T" IN=br10 OUT= MAC=9c:05:d6:68:53:f0:3e:0b:2e:39:d1:82:08:00 SRC=192.168.100.64 DST=10.13.13.1 LEN=89 TOS=00 PREC=0x00 TTL=64 ID=26986 DF PROTO=UDP SPT=57677 DPT=53 LEN=69 MARK=0

LTS-Studio [CUSTOM1_LOCAL-A-2147483647] DESCR="[CUSTOM1_LOCAL]Allow All T" IN=br10 OUT= MAC=9c:05:d6:68:53:f0:3e:0b:2e:39:d1:82:08:00 SRC=192.168.100.64 DST=10.13.13.1 LEN=89 TOS=00 PREC=0x00 TTL=64 ID=48742 DF PROTO=UDP SPT=49954 DPT=53 LEN=69 MARK=0

LTS-Studio [LAN_WAN-A-2147483647] DESCR="[LAN_WAN]Allow All Traffic" IN=br0 OUT=eth8 MAC=9e:05:d6:68:53:e7:e0:4f:43:e6:b0:6f:08:00 SRC=172.16.16.9 DST=66.205.249.28 LEN=76 TOS=00 PREC=0x00 TTL=63 ID=61471 DF PROTO=UDP SPT=33022 DPT=123 LEN=56 MARK=1a0000

I think I need to do a video on how I prompt AI to get IT problems solved such as this.

michmoor · May 21, 2025, 3:25pm

I would and i can assure you many others here would like a Youtube short or maybe a quick video on using AI prompts and how to structure the sentences.

I like how you add “Please” to your query. Gotta be polite to the machines

youorme · May 21, 2025, 5:09pm

I don’t use extractors as they have been deprecated and Piplines are what was recommended from Graylog.

I’m still learning but its powerful and easy, i wrote grok patterns for about 6-7 different unifi log messages and I added the geo lookup. It works well, graylog is awesome.

LTS_Tom · May 21, 2025, 5:58pm

Per their documentation, they are not deprecated, but pipelines are preferred since they have more features.