GitHub Link for my pfsense 2.4 filterlog extractor
https://raw.githubusercontent.com/lawrencesystems/graylog_extractors/main/pfsense_24.json
Graylog Documentation
https://docs.graylog.org/en/4.0/index.html
1 Like
I get annoyed with the source being āfilterlog:ā with the : at the end. Simple fix with this extractor rule
{
"title": "source fix",
"extractor_type": "regex",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "source",
"target_field": "source",
"extractor_config": {
"regex_value": "^(.+):$"
},
"condition_type": "none",
"condition_value": ""
}
1 Like
have enabled remote logging options at pfsense 2.4.5p1
have setted up an input at gray log and showing that messages are coming
buw when i click show all messages nothing is there
something was wrong with the extractor
Check your timezones. the OVA version is set to UTC and iām in UTC+1 so after an hour the messages became visable
I did the Debian install instructions and set it to match my timezone.
@LTS_Tom - Thanks for explaining streams, never quite understood them. Going to update my Graylog config now.
Also, here are my firewall extractors, might want to check them out. Currently experimenting with Suricata extractors but my router has a hard time running Suricata.
Iām curious how the log format will change in pfSense 2.5.0, since theyāre moving away from clog and also offering the option to use RFC 5424 logs.
2 Likes
@lmm5247 - by default pfSense v2.5.0 will use BSD syslog which is the same as it was in prior versions. They added IETF Syslog or as they labeled it, Syslog (RFC5424) which is an improvement over the default/legacy and prior version (RFC3164) but wish they would have adapted syslog-ng for security (syslog-ng can be installed but would have been a step in the right direction pertaining to security).
The output if choosing RFC5424 is slightly different but I would recommend over the default version citing the precision timing which is useful for forensic analysis (timing), newer standard, not limited to messages <1024bytes and is independent of the transport.
Tom, Thanks for creating the Graylog video!
I had already had a go with Graylog but got stuck after creating inputs with everything going into the default/general index. You illuminated the concepts of indices, streams, and rules very nicely and now I have made LOTS of progress on Ubuntu.
Personally, I want to achieve the monitoring of failed logins to my remote desktop gateway and alert if a bad actor starts an attempt to brute force a remote desktop gateway password. Itās a common question on tender documents for my business.
One thing I have found useful, that I feel would be helpful for others, is the concept of āsidecarsā. In short - a sidecar allows for the pre-configuration collection agents on Windows or Linux hosts. When the agent is installed and the Graylog host IP and the ID for agent authentication set the agent starts to sent logs. I can set a template for a Windows Server, logging all or specific event log IDs, and attach that template to as many installed server agents as I like. It so simple, and quick to do.
It took me a long while to figure out how to collect Windows events, and early on found the Graylog documentation pages and their recommendation of using beats winlogbeat and filelogbeat packages. I now know there is also sysmon (from Windows system internals?) that can log every network connection between hosts - high volume of data, but excellent for a security breach postmortem.
While I got Elastic WinLogBeat to work (with lots of scratching around) I then found that there is superb Graylog sidecar installation package that also includes the WinLogBeat agent. It can be installed with the parameters needed to link it to Graylog. The sidecar agent then downloads the config from the Graylog server and keeps it updated - itās so simple.
I have a simple cmd script to silently install the sidecar agent with the necessary parameters for my Graylog instance. I now push the install script to as many hosts as I need in a short time. Once Graylog agent sidecars are configured (two clicks) the events start rolling in.
I think a follow-up video about collection agents that can be used, and using sidecars would help a lot of us out. An advanced follow-up video could be about having multiple Graylog nodes; I have multiple sites and donāt think it a good idea to send the logging over my WAN VPN. I feel I probably should have a node per site, and allow elastic search ti tie them together. If I figure out how to have multiple nodes I will post in the forums.
Best regards,
1 Like
Having some issues w/ graylog not recognizing gl2_source_input - I see the messages picked up by the input, created a stream w/ match rule on gl2_source_input, but no messages are show up in the stream 
Do the messages show up in the main indice?
Hey Tom,
I figured it out - the timezone for my system was set to UTC vs the firewall which is in EST.
Thanks man!
1 Like
Thanks Tom for sharing.
Your style and speed ensure one does not lose focus even for a second and every second is both informative and stimulating in the learning process.
Having watched your video a number of times, with lots of pauses, I managed to get the graylog server working using the ubuntu image on virtualbox (the easy part) and that linked up with the pfsense too.
I did have to try and tweak the regular expression to work on the log messages but all good to learn.
Your style and method of sharing ālearnedā knowledge is so helpful in getting the understanding in a rapid burst, much like in the matrix film series, and getting the āknow-howā to get going quickly.
Any suggestions, guidance on using RFC 5424 format would be welcome and I am looking forward to the next instalment too.
Thank you!
Ali
1 Like
I install graylog in docker. I can not access shell root account, admin password not wor.
@LTS_Tom - thanks for another great video. I really enjoy your content. Thanks!
For everyone - Iām new to Graylog so maybe I have missed it, but what is the reason for creating separate Inputs for your devices (pfsense, unifi and xcp-ng)?
Would it not be possible to created one Syslog UDP (514) input that would catch all the logs? In that way you wouldnāt have to manage and open new port on the firewall when adding a new source/input. Iām sure there is a good reason for creating separate input with different ports, but could someone please enlighten me - thanks.
Edit - I get that if you have extrators you need a specific input. But if you donāt use extractors what is the good reason for having multiple syslog inputs?
I like my data and retention policies (and everything around me) to be very organized with granular control. You can pile everything up in one place but you lose the organization and granular control.
When turning on the xcp-ng syslog from xen orchestra there is a ton of debug messages. Is there a simple way of tuning the log level from XO?
1 Like
Never tried, I prefer all the logs going over to Graylog