FreeNAS/TrueNAS issue I can not figure out

Hello, first post here, but been subbed to the YT channel for a few years. I think this is the correct section for this, mods please move if I am wrong.

So, I have been having issues with my FreeNAS/TrueNAS server. I get the following errors in the loag that come back after a few hours.

Oct 22 11:10:42 freenas 1 2020-10-22T11:10:42.551320-05:00 freenas.local collectd 1887 - - Traceback (most recent call last):
File “/usr/local/lib/collectd_pyplugins/disktemp.py”, line 63, in read
temperatures = c.call(‘disk.temperatures’, self.disks, self.powermode)
File “/usr/local/lib/collectd_pyplugins/disktemp.py”, line 63, in read
temperatures = c.call(‘disk.temperatures’, self.disks, self.powermode)
File “/usr/local/lib/python3.8/site-packages/middlewared/client/client.py”, line 421, in call
raise CallTimeout(“Call timeout”)
middlewared.client.client.CallTimeout: Call timeout

Once this error comes up, the system grinds to a near halt. So slow in fact that SSH commands take 10+ seconds to respond when sent. The web interface is also slow as can be. Google search has yet to yield a fix that works. Some say it is a firewall issue. I am using pfsense with pfblocker, but nothing is in the logs coming from this server or any of the jails on it. So, pretty sure it is not that. This issue has persisted through multiple updates, changing USB drives, and a new install of FreeNAS/TrueNAS as well. The only thing left I can think of is a strange drive failure of some sort. I only think this because the errors refer to disktemp. But the call timeouts look like network related issues. I am lost at this point. It is either somethings really screwy or stupidly obvious that I am overlooking.

Jails running are Plex and Syncthing via plugin and a custom jail with Unifi Controller.

Any help or at least some direction finding would be very apreciated at this point. I posted on the IX Systems forums a couple months back and never got much back from anyone.

File “/usr/local/lib/collectd_pyplugins/disktemp.py”, line 63, in read
temperatures = c.call(‘disk.temperatures’, self.disks, self.powermode)

I am not sure but it seems to be complaining about temps. Is the server/system hot? What type of setup do you have for monitoring. Do you run S.M.A.R.T checks?

Edit: Have you tried to look at the logs further, any more information?

Temps are all good. All the drives report in the 30-40c range. SMART checks are on. No notifications from it so far. I have noticed system services take up nearly 12GB of RAM when the errors have been going for a while. Usually sits around 6GB. This is based on the chart on the dash.

Your error is a red herring. All this is saying is that the check for temperatures timed out waiting for the middleware. The middleware is the software that actually does everything in FreeNAS, the web interface and other features ask it to do things and show the results. So really your system doesn’t grind to a halt when that error occurs, that’s just the first thing that gets logged when it does grind to a halt.

How to actually find what’s going wrong I don’t know. I think you’ll have better luck getting help from the FreeNAS forums, just because this is a very specific issue.

I can try over there again. They never responded when I posted a couple months back when this all started.

I was starting to think that the error was a symptom of another issue all together, but was not sure. And had no clue what to even start looking at. So, seeing at least one other person thinking that way is a good thing.

Well, left a message on the IX systems forums a couple days ago, again. So far the only response is another person getting similar errors and slowdown. Guess I will keep tabs on both forums and hope somone has an idea.

Little bump. Still not finding much of the IX forums.

So, other people having similar errors on IX systems forums lead me down a possible path. I started killing jails after a slowdown/reboot. So far after rebooting without syncthing running seems to have fixed things. Going to let it go another day or two and see what happens. Then I can start poking around that jail or if needed just nuke it and start over.