Truenas Fangtooth App Update Stuck

Software & Web Applications
1

Sorry if this is a double post. I thought I posted earlier today but I can’t find it now.

I recently upgraded to Fang Tooth and noticed that none of my apps are updating now. I see “update now” but the app.update task stalls and just keeps running in and endless loop.

I assume this is a catalog or other issue but is there a way to fix this without a complete reinstall of each app?

2

From the command line try to ping something such as google.com to make sure the system is online. If that works look in System-> Audit for errors or in /var/log/syslog

3
  1. I was able to ping out without a problem
  2. I didn’t see any errors in Audit and nothing related to the upgrade in syslog. Interestingly I tried to run the upgrade again and immediately looked in syslog and there was no evidence of it.
  3. I thought perhaps there was something with the middleware that was stuck and did a full reboot and now I cannot get back into the system (see console screen attached).

Not sure if these are related issues but clearly something isn’t happy with the upgrade. Any thoughts? Or do I need to open a ticket/report a bug over at IX?

image
image600×800 121 KB

4

That is hard to read, ssh into the TrueNAS and please post the text from the logs, not a photo.

5

Failing to start “systemd-journald” and getting DMA read errors is pretty serious - unless this is a bad bug in the kernel or core OS services, it looks like you have a hardware failure of some sort (maybe your boot disk) that coincided with the upgrade. Getting fuller text logs is required if you want to try to repair the install, but I would suggest instead reinstalling from scratch. If you have a config backup (you downloaded that when prompted before doing the upgrade, right?) and remove the data disks until the OS install is complete, everything should be back to how it was. You should be able to tell from doing a fresh install whether there are hardware issues or not.

6

Here was the key line

DMAR: [DMA Read NO_PASID] Request device [00:1e.0] fault addr 0x504b000 [fault reason 0x02] …
DRHD: handling fault status reg 202

What I neglected to mention is that this is on one of my G6 generation HP servers that I refuse to get rid of !

Turns out it was an IOMMU conflict (probably due to old outdated firmware). Once I set intel_iommu=off, problem went away.

** my original problem of apps failing to update was also fixed ! **

I wasn’t planning on PCI-pass through to VMs on this system anyway, but it is interesting to note that change in Fangtooth that wasn’t there in Electric Eel

7

And yes of course I backed up my config :slight_smile: