Hi Everyone,
I’m hoping this is a simple fix. Getting started with graylog followed Toms video. I am getting messages from my home Pfsense but I am unable to filter by Source IP. I believe this might have something to do with the extractor(s) but not 100% sure.
I attached pictures with similar log entries to the one used in the video and I am missing SourceIP field.
Hope this helps, here are my current extractors:
{
"extractors": [
{
"title": "pfSense filterlog: IPv4 TCP",
"extractor_type": "regex",
"converters": [
{
"type": "csv",
"config": {
"trim_leading_whitespace": false,
"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCPFlags,Sequence,ACK,Window,URG,Options",
"strict_quotes": false
}
}
],
"order": 1,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "FilterData",
"extractor_config": {
"regex_value": "^filterlog\\[.+\\]:(.+)"
},
"condition_type": "regex",
"condition_value": "^filterlog\\[.+\\]:(.+),(in|out),4,.*,tcp,.*$"
},
{
"title": "pfSense filterlog: IPv4 UDP",
"extractor_type": "regex",
"converters": [
{
"type": "csv",
"config": {
"trim_leading_whitespace": false,
"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength",
"strict_quotes": false
}
}
],
"order": 2,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "FilterData",
"extractor_config": {
"regex_value": "^filterlog\\[.+\\]:(.+)"
},
"condition_type": "regex",
"condition_value": "^filterlog\\[.+\\]:(.+),(in|out),4,.*,udp,.*$"
},
{
"title": "pfSense filterlog: IPv4 ICMP",
"extractor_type": "regex",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "FilterData",
"extractor_config": {
"regex_value": "^filterlog\\[.+\\]:(.+)"
},
"condition_type": "regex",
"condition_value": "^filterlog\\[.+\\]:(.+),(in|out),4,.*,icmp,.*$"
}
],
"version": "4.3.8"
}
Working from memory but I think these are the ones I found that I use at work.
Let’s you sort source/dest ip/port/etc.
I started up with Graylog a few weeks ago and it’s been really good. I’ve used it in tracking down some connection issues. In some cases pfblocker was incorrectly geo-locating IP addresses. In other cases the clients traffic wasn’t even making it to our firewall. I’m generating about 10GB of logs a day on my core firewall. Currently I’m planning on dedicating enough storage to maintain a months of logging locally, and will likely spend the coin to be able to use archiving to keep a years worth of data.